C++TESK Testing ToolKit

h1. C++TESK Getting Started

h2. Introduction

_Hardware verification_ is usually understood as the process of checking _behavior_ of hardware on conformity to its _specification_. Such a process can be done formally by means of, e.g., model checking, automatic theorem proving, etc. Also, verification can be done by means of _simulation of_ separated hardware modules with the help of _simulator_.

Accounting the complexity of hardware models under verification, the task of automation should have usually been solved before the actual verification. The more processes will be done automatically and the less manual labor will be needed, the more effective check will be made. Without touching upon the formal verification methods, in this course we will focus only on simulation-based verification. Moreover, we will further speak only about one of the existing verification tool, created in the Institute for system programming of RAS. The tool''s capabilities allow speaking about it as a powerful and quite modern solution. So, we will speak about using _C++TESK Testing ToolKit_ (or C++TESK for short).

C++TESK implements simulation based approach to verification. The main element of the tool is its _core library_, implemented in programming languages C and C++. All core components are arranged in one package and are available at http://forge.ispras.ru/projects/cpptesk-toolkit/files. The tool is designed for creating test systems using C++ for different models of synchronous hardware at different levels of abstraction. Test systems are created using any means, provided by C++, basing on the approach, macros and classes defined by C++TESK.

When creating test systems for simulation based verification, three main tasks are usually solved. The first one is _test sequence construction_, the second one is _checking of behavior correctness_, and the third one is test completeness estimation. C++TESK allows construction test sequences of two types: selection _random stimulus set_ from the previously described stimuli at each simulation cycle or _selective choice_ of stimuli based on techniques of _exploration_ of implicitly defined FSMs. Checking of behavior correctness is made at each simulation cycle by means of _executable reference model_, created by verification engineer at some level of abstraction. External model (e.g., system simulator) can also be used. _Test completeness_ is determined either by the number of testing cycles for randomly selected stimuli, or on the basis of the information about _completeness of FSM exploration_.

Common scheme of test system is represented in figure 1.

!figure1.png!

*Figure1. Generalized structure of test system*

Let us shortly describe elements represented at the common scheme.

# _Stimulus generator_ is a component making test sequence (stimuli sequence). It is adjusted by test scenarios;

# _Test oracle_ is a component receiving data flows from stimulus generator and target component, sending stimulus flow from generator to target component, estimating correctness of target system behavior;

# _Target system_ is a hardware model, developed at one of hardware description languages (here, in Verilog), receiving stimulus flow and responding to it by reactions which should be checked;

# _Coverage tracker_ is a component grabbing information about reference model functional coverage, which in general affects stimulus generator work (e.g., by information of reached coverage);

# _Test report generator_ is a component making reports (test traces) with information of traversed transitions, reached coverage, found errors, etc.

The following tasks will be overviewed below.

* Analysis of documentation for development of C++TESK test systems;

* Development of test oracles including reference models and their adapters;

* Definition of test coverage;

* Setting up stimulus generator;

* Verification itself.

h2. 1. Documentation analysis for development of test systems

Under _verification_ we mean a process of checking observed behavior against specification. In the other words, verification is an establishing of the correspondence between target system behavior and its specification. Therefore, we should have not only the target system, but its specification too, being a document written at the beginning of target system development, slightly modified during development process, and containing information of target system functionality.

In reality specification is often very poor or even absent, and target system developers might have written only lists of _input_ and _output interface_ signals. In this case to conduct verification is difficult as to speak about bug in target system is possible only having information about correct behavior. Verification engineers have to interview target system developers, making list of requirements which are obligatory for target system. When list of requirements (specification) is obtained, verification can be started.

Practice shows that specification (especially, cycle accurate) is convenient to represent in the following ways.

1. by means of block diagrams (see figure 2).

!figure2.png!

*Figure 2. Example of block diagram for single operation*

Block diagram allows describing reference model behavior with any proximity to cycle accurateness. Figure 2 contains abstract representation of operation IO-WRITE (write via IO channel), starting from one-cycle request by the 20th interface. At the following cycle, device starts to receive data, taking 17, 19, 33 or 37 cycles depending on data length and presence of mask among sent data. This block diagram is insufficient for usage in reference model development and is supplied with _chart of cycle accurate operation description_, which should be described below. Operation finishes after receiving and saving of all data.

The following notations are used in block diagrams.

* Oval symbols mean points of operation start and stop; start point contains mnemonic designation of operation to be executed in this control flow path;

* Rectangle symbols mean blocks taking exactly one cycle for execution at current abstraction level. Process of reference model functioning is always described inside of such blocks. Block is allowed to be supplied with information about real execution time, other auxiliary information facilitating binding the scheme to reference model.

* Rhombus symbols mean branches in control flow. The symbol should contain condition of branching inside. There should be only one ingoing path and two outgoing paths;

* Forking of control flow is represented by bold point with two or more outgoing paths;

* Merging of several control flow paths is made also in bold point, meaning synchronization of control flow paths (waiting for the slowest one) to proceed to the outgoing path.

2. by means of _charts of cycle accurate operation description_ (see table 1).

*Table 1. Example of chart of cycle accurate operation description*

| Stimulus | Branch 0 | Microoperation 0 | Microoperation 1 | Branch 1 | Branch 2 | Microoperation 2 | Microoperation 3 |

| PRE:

  1) only one operation of the type may be executed simultaneously 2) val_wr_data_buff_nreg_to_IO(o)=1 | BRANCH:

  if val_mask=1 (in stimulus) microoperation 0 is started, else microoperation 1 is started. | PRE:

  - | PRE:

  - | BRANCH:

  if wr64=0 (in stim.), operation is done, else branch 2 is started. | BRANCH:

  if val_mask=1 (in stimulus),  microoperation 2 is started, else micro operation 3 is started. | PRE:

  - | PRE:

  - |

| Set stimulus parameters:

 val_wr_data_from_IO (i)=1 (strobe)

 wr_data_from_IO [15:0](i)=[0-7]

 Stimulus takes one cycle. || wr_data_from_IO [15:0] (i) contains mask. It is written into buffer(21) (with capacity of 4 32- or 64-bytes words).

 (repeat once). | wr_data_from_IO [15:0] (i) contains 2 data bytes. Data are written into buffer(21).

 (repeat 16 times). ||| wr_data_from_IO [15:0] (i) contains mask for the higher part of 64-byte transmission, being written in buffer(21).

 (repeat once). | wr_data_from_IO [15:0] (i) contains 2 bytes of the second part of 64-byte transmission. Data are written in buffer(21). (repeat 16 times). | 

||| POST:

 next cycle after val signal wr_data_buff_nreg_to_IO (o) (being written item in (21)) changes to the number of the following free item or to 8 if buffer (21) is full |    POST:

 next cycle after val signal wr_data_buff_nreg_to_IO (o) (being written item in (21)) changes to the number of the following free item or to 8 if buffer (21) is full ||| POST:

 - | POST:

 - |

| INPUT:

 val_wr_data_from_IO(i) - strobe

 wr_data_from_IO[15:0](i) includes

 - [15:3] - reserved;

 - [2] A5 – write to the high 32-bytes of 64-byte item flag;

 - [1] wr64 – 64-byte transmission flag;

 - [0] val_mask – mask flag. | INPUT:

 wr_data_from_IO [15:0] - mask | INPUT:

 wr_data_from_IO [15:0] - data ||| INPUT:

 wr_data_from_IO [15:0] - mask | INPUT:

 wr_data_from_IO [15:0] - data |

Such a table represents target system behavior in cycle-accurate manner. Information in the table corresponds to block diagram and used for the following development of cycle-accurate reference model. The table is created for each operation, performed by target system. The column with stimuli comes first, where information about operation preconditions and set input signals should be written. There are four types of the other columns according to block diagrams: microoperation (one cycle of work at current level of abstraction), branching, forking, joining. The table supposes left-to-right execution of operations: the stimulus runs first, and then the following columns run one by one. Each microoperation contains information about its precondition, correspondent actions of reference model, postcondition to be checked after execution, used input and output signals. Each branch item has a condition and numbers of columns for jumps. Each fork and join items are supplied with numbers of input and output columns.

If cycle accurate reference model is not made in particular case (for example, it is taken from system simulator) or the model is abstract, specification in simple text form is enough. Therefore, if they exist in such a form, any additional representation of specification +is not+ necessary.

Let us proceed to analysis of requirements in context of usage С++TESK. To make С++TESK''s components _«reference model»_ and _«reference model adapter»_, input and output interfaces should be created. Each being verified unit has both input and output signals, which can be grouped in interfaces by their belonging to certain type of activity: writing or reading. Functionally complete sequence of interface call, leading the unit to some stationary state where the unit can stay infinitely, will be called an operation. Notice, that signals like CLK and RST do not usually belong to any interface. Each operation can use several input and output interfaces. Analyzing the documentation, one should let input interfaces contain *only input signals*, and output interfaces contain *only output signals*.

Let''s review an example of interface information extraction. Let''s take a short specification for microprocessor data hub unit Databox (DB), describing connection between DB and external unit Memory Access Unit (MAU).

_Arbiter (13), working with round priority, selects one request from 5 issues of short requests (they may come in parallel) to send into MAU. The pace of transmission is 1 request per 2 cycles in case of 32-byte or 1 request per 4 cycles in case of 64-byte request. The selected short request is stored into DB and is not sent to MAU.  MAU has the following interface (see the following table)._

*Table 2. Interface to MAU*

| Signal | Type | Semantics |

| val_data_reqack_to_mau_03 | O | Request validity flag. Request for lower part of 64-byte cache row |

| val_data_reqack_to_mau_47 | O | Request validity. Request for higher part of 64-byte cache row |

| cop_data_reqack_to_mau[2:0] | O | Operation code:

«011» - reset register LDB

«100» - send coherent answer

«101» - reset register STB

«110» - send data from register STB

«111» - send data and reset register STB |

| source_reg_data_reqack_to_mau[4:0] | O | Number of register LDB/STB |

According to this documentation fragment, abstract reference model requires input interfaces for all five resources of short requests. More information about interfaces is likely to be found later. Keep on reading: “request is sent to MAU”. MAU is an external to the being tested unit and test system has to control data sent there. To control them, first, they have to be taken; second, restrictions for data should be known. To have these tasks done, component “output interface” being correspondent to implementation output interface is created in reference model. This component is overloaded in reference model adapter (where it is bound to implementation signals) and start solving task of getting signals going from implementation and sending them to controlling component. Let this output interface be named as iface14. It will contain signals val_data_reqack_to_mau_03, val_data_reqack_to_mau_47, cop_data_reqack_to_mau, source_reg_data_reqack_to_mau. Notice, that subdivision of signals into interfaces is only logical in current version of C++TESK. Only selected signals are allowed in correspondent interface adapters but it is not checked. After finding of interfaces, the rest of documentation is ordered by means of block diagrams or table of cycle accurate operation description if cycle accurate model is needed. If cycle accurate model is not needed or it has been developed, one may proceed to the following chapter.

h2. Development of reference model

We proceed to development of test oracle being a control component. Having received stimuli from stimulus generator, it sends them to target system, receives its reactions and checks them (see figure 1). This checking requires reference values obtained from reference model. Test oracle can be said to be a «wrapper» of _reference model_. We will return to test oracle later, speaking about reference model now. Being used in C++TESK structure of reference model is represented in figure 3.

!figure3.png!

*Figure 3. Structure of С++TESK reference model*

Reference model contains the following main parts.

* _Input interface models_;

* _Functional model of target system_;

* _Output interface models_.

All data inside of model are carried by messages being instances of class _Message_. It is made for the purpose of unification of interfaces between test system components. Input and output interface models are instances of class _Interface_.

Functional model of system is a part of reference model class but can be written in a separated class. Functional model contains of _control logic_, _data processing_, and _interface commutation models_. First two parts can be implemented in separated classes, but the third part depends on interfaces in reference model class.

The following sub chapter describes message model development. Typically, at least two types of message are used – for input and output interfaces. After introduction of message model, we will return to reference model development.

h3. 2.1 Message model

All necessary macros and classes are located in file *<hw/message.hpp>*.

Message model is a class developed by means of macro MESSAGE (class_name) {}. Let us place empty constructor and destructor inside of macro''s code block. Then let us turn on copying constructor by adding macro SUPPORT_CLONE (class_name) after those two functions.

Each message contains one or several data fields. These fields do not necessary have the same names as names of DUV wires, but their names are to be convenient for test system developer. In the majority of cases, field names are equivalent to the wire names at certain abstract level, but do not go too far: one “global” field in message for data from all wires is not convenient at all. To include fields into message class, macro DECLARE_FIELD(name, length) is used. The length should not exceed 64 bits. This macro defines a variable in message class and creates get and set functions message_class_object.get_field_name and message_class_object.set_field_name for the variable. Manual initialization of fields is allowed in message constructor as they are available via their names. Although, values do not have to be assigned to the fields manually; this work can be done automatically by macro RANDOMIZE_MESSAGE(pointer_to_message_class_object). Notice, that all explicitly assigned values will be overwritten after macro being called. All fields should be registered in message class constructor by macro ADD_FIELD(message_class::field_name).

Header for message models (fifo_msg.h) can look as follows.

<pre><code class="cpp">

#pragma once

#include <hw/message.hpp>

namespace cpptesk {

namespace fifo {

MESSAGE(InputData) {

public:

  InputData();

  virtual ~InputData();

  SUPPORT_CLONE(InputData);

  DECLARE_FIELD(data, 8);

};

MESSAGE(OutputData) {

public:

  OutputData();

  OutputData(uint8_t datum);

  virtual ~OutputData();

  SUPPORT_CLONE(OutputData);

  DECLARE_FIELD(data, 8);

};

}}

</code></pre>

Notice, that directive #pragma once is wide distributed but not fully standard way to ask compiler include the header file into library only once (the other way is to use #ifndef, #define, and #endif).

File with message model implementation (fifo_msg.cpp) can look as follows.

<pre><code class="cpp">

#include <fifo_msg.h>

namespace cpptesk {

namespace fifo {

InputData::InputData(void) {

  ADD_FIELD(InputData::data);

  RANDOMIZE_MESSAGE(*this);

}

InputData::~InputData(void) {}

OutputData::OutputData(void) {

  ADD_FIELD(OutputData::data);

}

OutputData::OutputData(uint8_t output_data) {

  ADD_FIELD(OutputData::data);

  data = output_data;

}

OutputData::~OutputData(void) {}

}}

</code></pre>

h3. 2.2 Reference model

Necessary macros and classes are located in file <hw/model.hpp>.

Model class is created by macro MODEL (model_class_name) {}. Minimally necessary methods include constructor and virtual destructor. Selected in previous parts interfaces are defined in model class by macros DECLARE_INPUT(interface_name) for input interfaces of target system and DECLARE_OUTPUT(interface_name) for output interfaces of target system. Each output interface may be supplied with a method returning availability of the interface by means of standard for interface method interface_name.isReady(). Let such methods be declared as virtual bool isInterfaceNameReady() const. Notice that availability of interfaces depends on running operations using them: if some operation has been started on the interface, it will be busy till operation stops its execution.

Model also contains definition of operations being sequences of registered interface calls. Being, in fact, operations, model methods are declared by macro DECLARE_STIMULUS(method_name) and DECLARE_REACTION(method_name). They are not distinguished, but the first macro is typically used for definition of operations themselves, the second macro is used for operation parts requiring reading of data from output interfaces.

Header of reference model class (fifo_model.h) can look as follows.

<pre><code class="cpp">

#include <hw/model.hpp>

#include <fifo_msg.h>

namespace cpptesk {

namespace fifo {

MODEL(FIFO) {

public:

  FIFO();

  virtual ~FIFO();

  DECLARE_INPUT(iface1);

  DECLARE_OUTPUT(iface2);

  virtual bool isIface1Ready() const;

  DECLARE_STIMULUS(push_msg);         // operation “push data”

  DECLARE_STIMULUS(pop_msg);          // operation “pop data”

  DECLARE_REACTION(get_pop_msg);      // reaction “read output data”

  void push_item(int data);           // reference model function “push data”

  uint8_t pop_item(void);             // reference model function “pop data”

protected:

  std::vector<uint8_t> fifo;

};

}}

</code></pre>

Now let us speak a few words about implementation of reference model class. Constructor of the class must contain calls of interface constructor with the only parameter being text description of the correspondent interface. Sensible names are recommended as they will appear in diagnostics messages. Input and output interfaces are registered by means of ADD_INPUT (interface_name) and ADD_OUTPUT (interface_name) correspondingly.

Let us proceed to description of functions showing interface availability. They use standard for interface class method isReady and simply return its value: return interface_name.isReady ().

Let us now describe operations. Each operation starts from copying of input message into local variable by means of macro CAST_MESSAGE (input_message_type). Then message may be sent to the interface, set during operation start. If it is sent to an input interface, we are speaking about stimulus start, or about reaction start in another case. If we want to send message to an input interface with standard parameters (i.e., without changing of message, of interface name), this sending can be done by macro START_STIMULUS (start_mode). Start modes are the following.

*PARALLEL creates different process for sending and processing of sending results; the process is executed in parallel with operation commands written after START_STIMULUS;

*SEQUENTIAL stops execution of operation till stimulus execution is finished.

Being sent message will be serialized or turned into sequence of actions to DUV applied via its input wires. This work is done by serializers in reference model adapter (see next step).

Notice: if sending message or used interface are different from those being input parameters of function (what are “those being input parameters” will be clarified later), one should use macro  RECV_STIMULUS (start_mode, interface_name, message_object_name). In this case operation has to be started from START_PROCESS (), not from START_STIMULUS (start_mode), and to be finished by STOP_PROCESS () instead of STOP_STIMULUS () (see later).

Operation description may contain macro CYCLE (), telling to the test system that execution of this operation requires one cycle of delay. Operation is finished by macro STOP_STIMULUS ().

Parts of operations reading data from output interfaces are subject to separation into specific methods for definition as “reactions” (DEFINE_REACTION). In a reaction method reference model says to the test system that it has certain message to be obtained on a given output interface of DUV by means of macro SEND_REACTION (start_mode, interface_name, message_object_name).

Notice: SEND_REACTION works with only output interfaces!

Standard interface and message object may be obtained by means of macros GET_IFACE() and GET_MESSAGE() (under standard we means those, which are input parameters). Reaction methods (if they are developed) should be called from stimulus method. Best way to call reaction method is to call reaction_method_name (process, interface_name, message_object_name), where process is an implicitly defined operation execution context. To send the context is necessary for joining all operation parts together.

Development of DUV functional model (more precisely, control logic part) is not a specific task in test system development by means of C++TESK. This task is done by means of C++ standard library and that is why detailed comments about it will be omitted.

After all the steps, the following file with model implementation may be obtained (fifo_model.cpp).

<pre><code class="cpp">

#include <fifo_model.h>

namespace cpptesk {

namespace fifo {

FIFO::FIFO() {

  ADD_INPUT(iface1);

  ADD_OUTPUT(iface2);

}

FIFO::~FIFO() {}

bool FIFO::isIface1Ready() const { return iface1.isReady(); }

DEFINE_STIMULUS(FIFO::push_msg) {

  InputData data = CAST_MESSAGE(InputData);

  push_item(data.get_data());

  START_STIMULUS(PARALLEL);

  CYCLE();

  STOP_STIMULUS();

}

DEFINE_STIMULUS(FIFO::pop_msg) {

  InputData data = CAST_MESSAGE(InputData);

  START_STIMULUS(PARALLEL);

  OutputData outdata = OutputData(pop_item());

  get_pop_msg(process, iface2, outdata);

  STOP_STIMULUS();

}

DEFINE_REACTION(FIFO::get_pop_msg) {

  START_PROCESS();

  SEND_REACTION(SEQUENTIAL, GET_IFACE(), GET_MESSAGE());

  STOP_PROCESS();

}

void FIFO::push_item(uint8_t data) {

  assert(fifo.size() < 16);

  fifo.push_back(data);

}

uint8_t FIFO::pop_item(void) {

  assert(fifo.size() > 0);

  uint8_t data = fifo[0];

  fifo.erase(fifo.begin());

  return data;

}

}}

</code></pre>

h2. 3. Auxiliary tool VeriTool

Digital devices are developed in hardware description languages (HDLs) like Verilog and VHDL. In this paper we use only Verilog language (http://en.wikipedia.org/wiki/Verilog) as one of the most distributed. To connect design under verification (DUV) with test systems developed in different languages (not Verilog), Verilog standard describes special Verilog procedural interface (VPI).

To make files of VPI-environment (environment is understood to be a set of functions connecting Verilog-simulator running DUV and test system developed in C/C++ in standard to Verilog way), we will use licensed under GPL tool VeriTool (http://forge.ispras.ru/projects/veritool/files). VeriTool uses Verilog syntax analyzer built in open source Verilog simulator Icarus Verilog (http://sourceforge.net/projects/iverilog/). Installation of both tools can be done both manually and automatically by means of script from C++TESK package.

h3. 3.1 Command line options of tool VeriTool

Syntax of VeriTool command line looks as follows.

<pre>

$ veritool [options] input_files

</pre>

Supported command line options are represented below.