C++TESK Testing ToolKit

h1. C++TESK Getting Started

h2. Introduction

_Hardware verification_ is usually understood as the process of checking _behavior_ of hardware on conformity to its _specification_. Such a process can be done formally by means of, e.g., model checking, automatic theorem proving, etc. Also, verification can be done by means of _simulation of_ separated hardware modules with the help of _simulator_.

Accounting the complexity of hardware models under verification, the task of automation should have usually been solved before the actual verification. The more processes will be done automatically and the less manual labor will be needed, the more effective check will be made. Without touching upon the formal verification methods, in this course we will focus only on simulation-based verification. Moreover, we will further speak only about one of the existing verification tool, created in the Institute for system programming of RAS. The tool''s capabilities allow speaking about it as a powerful and quite modern solution. So, we will speak about using _C++TESK Testing ToolKit_ (or C++TESK for short).

C++TESK implements simulation based approach to verification. The main element of the tool is its _core library_, implemented in programming languages C and C++. All core components are arranged in one package and are available at http://forge.ispras.ru/projects/cpptesk-toolkit/files. The tool is designed for creating test systems using C++ for different models of synchronous hardware at different levels of abstraction. Test systems are created using any means, provided by C++, basing on the approach, macros and classes defined by C++TESK.

When creating test systems for simulation based verification, three main tasks are usually solved. The first one is _test sequence construction_, the second one is _checking of behavior correctness_, and the third one is test completeness estimation. C++TESK allows construction test sequences of two types: selection _random stimulus set_ from the previously described stimuli at each simulation cycle or _selective choice_ of stimuli based on techniques of _exploration_ of implicitly defined FSMs. Checking of behavior correctness is made at each simulation cycle by means of _executable reference model_, created by verification engineer at some level of abstraction. External model (e.g., system simulator) can also be used. _Test completeness_ is determined either by the number of testing cycles for randomly selected stimuli, or on the basis of the information about _completeness of FSM exploration_.

Common scheme of test system is represented in figure 1.

!figure1.png!

*Figure1. Generalized structure of test system*

Let us shortly describe elements represented at the common scheme.

# _Stimulus generator_ is a component making test sequence (stimuli sequence). It is adjusted by test scenarios;

# _Test oracle_ is a component receiving data flows from stimulus generator and target component, sending stimulus flow from generator to target component, estimating correctness of target system behavior;

# _Target system_ is a hardware model, developed at one of hardware description languages (here, in Verilog), receiving stimulus flow and responding to it by reactions which should be checked;

# _Coverage tracker_ is a component grabbing information about reference model functional coverage, which in general affects stimulus generator work (e.g., by information of reached coverage);

# _Test report generator_ is a component making reports (test traces) with information of traversed transitions, reached coverage, found errors, etc.

The following tasks will be overviewed below.

* Analysis of documentation for development of C++TESK test systems;

* Development of test oracles including reference models and their adapters;

* Definition of test coverage;

* Setting up stimulus generator;

* Verification itself.

h2. Documentation analysis for development of test systems

Under _verification_ we mean a process of checking observed behavior against specification. In the other words, verification is an establishing of the correspondence between target system behavior and its specification. Therefore, we should have not only the target system, but its specification too, being a document written at the beginning of target system development, slightly modified during development process, and containing information of target system functionality.

In reality specification is often very poor or even absent, and target system developers might have written only lists of _input_ and _output interface_ signals. In this case to conduct verification is difficult as to speak about bug in target system is possible only having information about correct behavior. Verification engineers have to interview target system developers, making list of requirements which are obligatory for target system. When list of requirements (specification) is obtained, verification can be started.

Practice shows that specification (especially, cycle accurate) is convenient to represent in the following ways.

1. by means of block diagrams (see figure 2).

!figure2.png!

*Figure 2. Example of block diagram for single operation*

Block diagram allows describing reference model behavior with any proximity to cycle accurateness. Figure 2 contains abstract representation of operation IO-WRITE (write via IO channel), starting from one-cycle request by the 20th interface. At the following cycle, device starts to receive data, taking 17, 19, 33 or 37 cycles depending on data length and presence of mask among sent data. This block diagram is insufficient for usage in reference model development and is supplied with _chart of cycle accurate operation description_, which should be described below. Operation finishes after receiving and saving of all data.

The following notations are used in block diagrams.

* Oval symbols mean points of operation start and stop; start point contains mnemonic designation of operation to be executed in this control flow path;

* Rectangle symbols mean blocks taking exactly one cycle for execution at current abstraction level. Process of reference model functioning is always described inside of such blocks. Block is allowed to be supplied with information about real execution time, other auxiliary information facilitating binding the scheme to reference model.

* Rhombus symbols mean branches in control flow. The symbol should contain condition of branching inside. There should be only one ingoing path and two outgoing paths;

* Forking of control flow is represented by bold point with two or more outgoing paths;

* Merging of several control flow paths is made also in bold point, meaning synchronization of control flow paths (waiting for the slowest one) to proceed to the outgoing path.

2. by means of _charts of cycle accurate operation description_ (see table 1).

*Table 1. Example of chart of cycle accurate operation description*

| Stimulus | Branch 0 | Microoperation 0 | Microoperation 1 | Branch 1 | Branch 2 | Microoperation 2 | Microoperation 3 |

| PRE:

  1) only one operation of the type may be executed simultaneously 2) val_wr_data_buff_nreg_to_IO(o)=1 | BRANCH:

  if val_mask=1 (in stimulus) microoperation 0 is started, else microoperation 1 is started. | PRE:

  - | PRE:

  - | BRANCH:

  if wr64=0 (in stim.), operation is done, else branch 2 is started. | BRANCH:

  if val_mask=1 (in stimulus),  microoperation 2 is started, else micro operation 3 is started. | PRE:

  - | PRE:

  - |

| Set stimulus parameters:

 val_wr_data_from_IO (i)=1 (strobe)

 wr_data_from_IO [15:0](i)=[0-7]

 Stimulus takes one cycle. || wr_data_from_IO [15:0] (i) contains mask. It is written into buffer(21) (with capacity of 4 32- or 64-bytes words).

 (repeat once). | wr_data_from_IO [15:0] (i) contains 2 data bytes. Data are written into buffer(21).

 (repeat 16 times). ||| wr_data_from_IO [15:0] (i) contains mask for the higher part of 64-byte transmission, being written in buffer(21).

 (repeat once). | wr_data_from_IO [15:0] (i) contains 2 bytes of the second part of 64-byte transmission. Data are written in buffer(21). (repeat 16 times). | 

||| POST:

 next cycle after val signal wr_data_buff_nreg_to_IO (o) (being written item in (21)) changes to the number of the following free item or to 8 if buffer (21) is full |    POST:

 next cycle after val signal wr_data_buff_nreg_to_IO (o) (being written item in (21)) changes to the number of the following free item or to 8 if buffer (21) is full ||| POST:

 - | POST:

 - |

| INPUT:

 val_wr_data_from_IO(i) - strobe

 wr_data_from_IO[15:0](i) includes

 - [15:3] - reserved;

 - [2] A5 – write to the high 32-bytes of 64-byte item flag;

 - [1] wr64 – 64-byte transmission flag;

 - [0] val_mask – mask flag. | INPUT:

 wr_data_from_IO [15:0] - mask | INPUT:

 wr_data_from_IO [15:0] - data ||| INPUT:

 wr_data_from_IO [15:0] - mask | INPUT:

 wr_data_from_IO [15:0] - data |