C++TESK Testing ToolKit

h1. C++TESK Test Engines

{{toc}}

h2. Introduction

This document describes _test engines_ included into library of C++TESK Testing ToolKit (hereafter С++TESK). _Test engine_ is a core of _stimulus generator_, which is a component of _test system_ being responsible for construction of stimulus sequence (_test actions_) for _target system_. Stimuli can be represented as parameterized target system operation calls. Sequence of stimuli is called _test sequence_. _Test scenario_, being input information for test engine, is a high-level specification of test defining available for test stimuli. Test sequence is constructed as a result of interpretation of given scenario by test engine. It should be noticed that all test engines have the same interface which let different test engines be used for execution of given test scenario.

Library of C++TESK includes two test engines: @fsm@ (_Finite State Machine_) and @rnd@ (_Random_), located in namespace @cpptesk::ts::engine@. The former (@fsm@) makes traversing of finite state machine _state graph_ described in implicit form in test scenario. The criterion of test completeness in this case is visiting of all states reachable from initial state (it means also traversing of all arcs outgoing from reachable states). Test engine @rnd@ constructs test sequence _randomly_ selecting random either stimulus or set of available stimuli at each work step. Test is finished when given work steps are passed. More detailed information about test engines can be found in correspondent chapters of this document.

Before reading the rest part of this document, it is recommended to read about development of test scenario by means of C++TESK. This information is contained, e.g., by document _«С++TESK Hardware Edition: Quick Reference»_ (chapter _«Development of test scenario»_).

h2. Test scenario structure

From test engine point of view, test scenario consists of two main parts: (1) _stimulus iterator_ and (2) _state calculation function_.

h3. Stimulus iterator

_Stimulus iterator_ is set by _scenario methods_ of scenario class. Each scenario method iterates parameters of some stimulus using typical loop-constructions being available in C++ (for, while etc.). Iterations are performed by means of _iteration variables_ which are the fields of _iteration context_ being a parameter of scenario method. Iteration variables can be accessed by means of macro @CPPTESK_ITERATION_VARIABLE(name)@. Macro @CPPTESK_ITERATION_BEGIN@ precedes iterations and macro @CPPTESK_ITERATION_END@ finishes them. Implementation of stimulus application to target system is located inside of set of loops in block @CPPTESK_ITERATION_ACTION{...}@. Stimulus application finishes by calling macro @CPPTESK_ITERATION_YIELD(verdict)@.

Typical scenario method has the following structure.

<pre><code class="cpp">

bool MyScenario::ScenarioMethod(AbcCtx &ctx) {

    int &a = CPPTESK_ITERATION_VARIABLE(a);

    ...

    int &z = CPPTESK_ITERATION_VARIABLE(z);

    CPPTESK_ITERATION_BEGIN

    for(a = 0; a < Na; a++)

    ...

    for(z = 0; z < Nz; z++) {

        CPPTESK_ITERATION_ACTION {

            ...

            CPPTESK_ITERATION_YIELD(...);

        }

    }

    CPPTESK_ITERATION_END

}

</code></pre>

Scenario method @ScenarioMethod@ for given reference model state (_specification state_) @ModelState@ produces stimulus set, where each stimulus is identified by integer numbers from range @[0, Nmax(ScenarioMethod, ModelState)-1]@, where @Nmax(ScenarioMethod, ModelState)@ is the number of executions of block @CPPTESK_ITERATION_ACTION@ for scenario methods of given reference model state.

Aggregated stimulus iterator is obtained by “aggregation” of iterators defined in scenario methods. Test scenario stimulus iterator describes a set of all possible stimuli and introduces current test scenario-wide system of stimulus identification. If reference model state ModelState is fixed, stimuli are identified by integer numbers from range @[0, Nmax(ModelState)-1]@, where @Nmax(ModelState)=i=0,n-1 {Nmax(ScenarioMethodi, ModelState)}@. Numeration of stimuli accounts order of scenario method registration.

h3. State calculation function

State calculation function is defined in state calculation method of scenario class. Type of return value can be scalar (int, float, etc.) or of string type @(std::string)@. Return value is interpreted as reference model state at some level of abstraction.

State calculation function is unnecessary test scenario component and used only by test engine @fsm@.

h2. Test engines

Test engines are controlled by command line parameters; some of them are common for all engines.

* _--cpu_ — running test host device identifier (this parameter is sensible only in distributed testing);

* _--length_ — length of test sequence (in current version of C++TESK Testing ToolKit test engine @fsm@ ignores parameter _--length_);

* _--print-progress_ — print of test progress.

h3. Common command line parameters

*Parameter cpu*

Parameter _--cpu integer_ sets identifier of host device (computer, microprocessor or core) which runs current test. This parameter is used only in case of distributed among several host devices testing. The main purpose of this parameter is varying of test executions at different host devices. E.g., this parameter is used for random seed setting and in choosing of the following stimulus (graph arc) by test engine fsm.

*Parameter length*

Parameter _--length integer_ restricts the length of test sequence generated by test engine. When the length is equal to the given number, test is finished.

*Option print-progress*

Option _--print-progress_ turns on print of test execution progress (state graph exploration).

*Default values*

Option _--print-progress_ is turned off by default, parameters _--cpu_ and _--length_ have the following values.

_--cpu 0 --length 1000_

h3. Test engine @fsm@

Test engine @fsm@ generates test sequence by finite state machine state graph exploration. The graph is represented implicitly in test scenario. The criterion of test completeness is visiting of all states being reachable from the initial state and traversing all arcs issuing from them.

*Brief algorithm description*

Test engine fsm works in the following way. At each step it has current state computed. There are two ways: (1) this state has non traversed arcs (it is more correct to speak about _stimuli_ not _arcs_ as in case of nondeterministic graph one stimulus can correspond to several arcs issuing from the same state) and (2) all arcs issuing from this state have been traversed. It the first case test engine selects one of the non traversed arcs, applies correspondent stimulus to the target system and test keeps on going. The second case has two alternatives: (2.1) there are other states (from having been visited) with non traversed arcs, and (2.2) all known arcs have been traversed. In the first case test engine finds the way to the state with non traversed arcs. Eventually situation (2.1) becomes situation (1). In the second case test is finished.

*Command line parameters*

Test engine fsm supports the following command line options.

* _--randomize_ — randomization of arc selection;

* _--nondeterministic_ — support of nondeterministic arcs;

* _--dump-fsm_ — saving of state graph in file after test finalizing:

** _--full-graph_ — saving of the whole graph;

** _--forward-tree_ — saving of spanning tree.

_Option randomize_

Option _--randomize_ turns on randomized selection of the following arc. It means that if test engine has a choice which arc should be selected, test engine choose arc randomly. If this option is not set, arcs are selected in the order prescribed by their order in test scenario.

_Option nondeterministic_

Option _--nondeterministic_ turns on support of _nondeterministic graphs_, i.e. graphs where several arcs labeled by the same stimulus can issue from the same state. The main difficulty in traversing of nondeterministic graphs is absence of possibility of determination which arc has been traversed by applying some or other stimulus. It makes finding of paths with non traversed arcs more difficult.

Setting of option _--nondeterministic_ allows usage of nondeterministic arc during path finding (increasing class of supported graphs). If this option is not set, paths are constructed via deterministic arcs.

_Option dump-fsm_

Option _--dump-fsm_ is used for saving of state graph in file after test finalizing. File has fixed name fsm.gv; in this file data are represented in Graphviz format, which is an open source graph visualization package (http://www.graphviz.org).

_Option full-graph_

Option _--full-graph_ (set together with --dump-fsm) saves state graph as a whole.

*Notice*: options --full-graph and --forward-tree are incompatible.

_Option forward-tree_

Option _--forward-tree_ (set together with _--dump-fsm_) saves only state graph spanning tree.

*Notice*: options _--forward-tree_ and _--full-graph_ are incompatible.

_Default values_

Options _--randomize_, _--nondeterministic_, and _--dump-fsm_ are not set by default.

If option _--dump-fsm_ is set, state graph is saved as a whole by default.

h3. Test engine @rnd@

Test engine @rnd@ constructs test sequence randomly selecting random stimulus or random stimulus set from the list of allowed in test scenario stimuli at each work step. Test is finished when given number of stimulus is called.

*Brief algorithm description*

There are two modes of test engine @rnd@ work. The first one is sequential (see chapter _«Option sequential»_) and parallel (see chapter _«Option parallel»_).

In sequential mode all the stimuli, set up by test action iterator, are equitable. At each work step, identifier of stimulus is randomly selected from range @[0, Nmax(ModelState)-1]@, and correspondent stimulus is applied.

In parallel mode stimulus with identifier 0 has a special meaning playing role of delay stimulus. All the other stimuli are not allowed to shift simulation time (they are “immediate” actions without waiting of reactions for them). In this mode at each work step random stimulus sequence (_multi stimulus_) is created. This sequence is finished by stimulus with identifier 0. In the other words, several stimuli are applied in parallel, and then some delay happens.

Size of multi stimulus depends on _load mode_. Test engine @rnd@ supports four load modes.

* _minimum load mode_ (see chapter _«Option min-load»_);

* _maximum load mode_ (see chapter _«Option max-load»_);

* _fixed load mode_ (see chapter _«Parameter fix-load»_);

* _variable load mode_ (see chapter _«Option var-load»_).

The criterion of test completeness is passing of certain number of test steps (see chapter _«Parameter length»_).

*Command line parameters*

Test engine rnd supports the following command line parameters.

* _--sequential_ — sequential mode;

* _--parallel_ — parallel mode:

* _--min-load_ — minimum load mode;

* _--max-load_ — maximum load mode;

* _--fix-load_ — fixed load mode;

* _--var-load_ — variable load mode.

_Option sequential_

Option _--sequential_ turns on sequential mode of test engine work. In this mode, test engine at each step selects random stimulus from the list of available in test scenario stimuli. Sequential mode is aimed for systems which does not support application of stimuli in parallel mode.

*Notice*: options _--sequential_ and _--parallel_ are mutually exclusive.

_Option parallel_

Option _--parallel_ turns on parallel mode of test engine work. In this mode, test engine at each step generates random stimulus sequence finished with stimulus with identifier 0 — delay stimulus (typically this is a scenario method without iterations called @nop()@ or @delay()@) . All the stimuli except stimulus 0 are supposed to run immediately. Despite their attempt to run some or other operation, their real start is performed only after calling stimulus 0 when time is shifted. Test step in parallel mode is schematically showed in figure 1.

TODO: no figure

*Figure 1. Test step in parallel mode*

*Notice*: options _--sequential_ and _--parallel_ are mutually exclusive.

_Option min-load_

Option _--min-load_ turns on minimal load mode allowing not more than one stimulus at each work step. This mode is typically used during debug of test system.

_Option max-load_

Option _--max-load_ turns on maximum load mode to apply all possible stimuli described in test scenario at each work step.

_Parameter fix-load_

Parameter _--fix-load load_percentage_ sets fixed load mode to apply certain part of stimuli at each work step. E.g., if _--fix-load 50_ is written, half of the whole stimulus set will be applied in parallel at each step.

_Option var-load_

Option _--var-load_ turns on variable load mode to let test system load be smoothly increased from minimum to maximum.

_Default values_

Option _--sequential_ is used by default.

If option _--parallel_ is used, variable load mode _--var-load_ is used by default.