Project

General

Profile

C++TESK Test Engines » History » Version 1

Mikhail Chupilko, 09/19/2013 05:24 PM

1 1 Mikhail Chupilko
h1. C++TESK Test Engines
2
3
h2. Introduction
4
5
This document describes _test engines_ included into library of C++TESK Testing ToolKit (hereafter С++TESK). _Test engine_ is a core of _stimulus generator_, which is a component of _test system_ being responsible for construction of stimulus sequence (_test actions_) for _target system_. Stimuli can be represented as parameterized target system operation calls. Sequence of stimuli is called _test sequence_. _Test scenario_, being input information for test engine, is a high-level specification of test defining available for test stimuli. Test sequence is constructed as a result of interpretation of given scenario by test engine. It should be noticed that all test engines have the same interface which let different test engines be used for execution of given test scenario.
6
7
Library of C++TESK includes two test engines: @fsm@ (_Finite State Machine_) and @rnd@ (_Random_), located in namespace @cpptesk::ts::engine@. The former (@fsm@) makes traversing of finite state machine _state graph_ described in implicit form in test scenario. The criterion of test completeness in this case is visiting of all states reachable from initial state (it means also traversing of all arcs outgoing from reachable states). Test engine @rnd@ constructs test sequence _randomly_ selecting random either stimulus or set of available stimuli at each work step. Test is finished when given work steps are passed. More detailed information about test engines can be found in correspondent chapters of this document.
8
9
Before reading the rest part of this document, it is recommended to read about development of test scenario by means of C++TESK. This information is contained, e.g., by document _«С++TESK Hardware Edition: Quick Reference»_ (chapter _«Development of test scenario»_).
10
11
h2. Test scenario structure
12
13
From test engine point of view, test scenario consists of two main parts: (1) _stimulus iterator_ and (2) _state calculation function_.
14
15
h3. Stimulus iterator
16
17
_Stimulus iterator_ is set by _scenario methods_ of scenario class. Each scenario method iterates parameters of some stimulus using typical loop-constructions being available in C++ (for, while etc.). Iterations are performed by means of _iteration variables_ which are the fields of _iteration context_ being a parameter of scenario method. Iteration variables can be accessed by means of macro @CPPTESK_ITERATION_VARIABLE(name)@. Macro @CPPTESK_ITERATION_BEGIN@ precedes iterations and macro @CPPTESK_ITERATION_END@ finishes them. Implementation of stimulus application to target system is located inside of set of loops in block @CPPTESK_ITERATION_ACTION{...}@. Stimulus application finishes by calling macro @CPPTESK_ITERATION_YIELD(verdict)@.
18
19
Typical scenario method has the following structure.
20
21
<pre><code class="cpp">
22
bool MyScenario::ScenarioMethod(AbcCtx &ctx) {
23
    int &a = CPPTESK_ITERATION_VARIABLE(a);
24
    ...
25
    int &z = CPPTESK_ITERATION_VARIABLE(z);
26
    
27
    CPPTESK_ITERATION_BEGIN
28
    for(a = 0; a < Na; a++)
29
    ...
30
    for(z = 0; z < Nz; z++) {
31
        CPPTESK_ITERATION_ACTION {
32
            ...
33
            CPPTESK_ITERATION_YIELD(...);
34
        }
35
    }
36
    CPPTESK_ITERATION_END
37
}
38
</code></pre>
39
40
Scenario method @ScenarioMethod@ for given reference model state (_specification state_) @ModelState@ produces stimulus set, where each stimulus is identified by integer numbers from range @[0, Nmax(ScenarioMethod, ModelState)-1]@, where @Nmax(ScenarioMethod, ModelState)@ is the number of executions of block @CPPTESK_ITERATION_ACTION@ for scenario methods of given reference model state.
41
42
Aggregated stimulus iterator is obtained by “aggregation” of iterators defined in scenario methods. Test scenario stimulus iterator describes a set of all possible stimuli and introduces current test scenario-wide system of stimulus identification. If reference model state ModelState is fixed, stimuli are identified by integer numbers from range @[0, Nmax(ModelState)-1]@, where @Nmax(ModelState)=i=0,n-1 {Nmax(ScenarioMethodi, ModelState)}@. Numeration of stimuli accounts order of scenario method registration.
43
44
h3. State calculation function
45
46
State calculation function is defined in state calculation method of scenario class. Type of return value can be scalar (int, float, etc.) or of string type @(std::string)@. Return value is interpreted as reference model state at some level of abstraction.
47
48
State calculation function is unnecessary test scenario component and used only by test engine @fsm@.
49
50
h2. Test engines
51
52
Test engines are controlled by command line parameters; some of them are common for all engines.
53
* _--cpu_ — running test host device identifier (this parameter is sensible only in distributed testing);
54
* _--length_ — length of test sequence (in current version of C++TESK Testing ToolKit test engine @fsm@ ignores parameter _--length_);
55
* _--print-progress_ — print of test progress.
56
57
h3. Common command line parameters
58
59
*Parameter cpu*
60
61
Parameter _--cpu integer_ sets identifier of host device (computer, microprocessor or core) which runs current test. This parameter is used only in case of distributed among several host devices testing. The main purpose of this parameter is varying of test executions at different host devices. E.g., this parameter is used for random seed setting and in choosing of the following stimulus (graph arc) by test engine fsm.
62
63
*Parameter length*
64
65
Parameter _--length integer_ restricts the length of test sequence generated by test engine. When the length is equal to the given number, test is finished.
66
67
*Option print-progress*
68
69
Option _--print-progress_ turns on print of test execution progress (state graph exploration).
70
71
*Default values*
72
73
Option _--print-progress_ is turned off by default, parameters _--cpu_ and _--length_ have the following values.
74
75
_--cpu 0 --length 1000_
76
77
h3. Test engine @fsm@
78
79
Test engine @fsm@ generates test sequence by finite state machine state graph exploration. The graph is represented implicitly in test scenario. The criterion of test completeness is visiting of all states being reachable from the initial state and traversing all arcs issuing from them.
80
81
*Brief algorithm description*
82
83
Test engine fsm works in the following way. At each step it has current state computed. There are two ways: (1) this state has non traversed arcs (it is more correct to speak about _stimuli_ not _arcs_ as in case of nondeterministic graph one stimulus can correspond to several arcs issuing from the same state) and (2) all arcs issuing from this state have been traversed. It the first case test engine selects one of the non traversed arcs, applies correspondent stimulus to the target system and test keeps on going. The second case has two alternatives: (2.1) there are other states (from having been visited) with non traversed arcs, and (2.2) all known arcs have been traversed. In the first case test engine finds the way to the state with non traversed arcs. Eventually situation (2.1) becomes situation (1). In the second case test is finished.
84
85
*Command line parameters*
86
87
Test engine fsm supports the following command line options.
88
* _--randomize_ — randomization of arc selection;
89
* _--nondeterministic_ — support of nondeterministic arcs;
90
* _--dump-fsm_ — saving of state graph in file after test finalizing:
91
** _--full-graph_ — saving of the whole graph;
92
** _--forward-tree_ — saving of spanning tree.
93
94
_Option randomize_
95
96
Option _--randomize_ turns on randomized selection of the following arc. It means that if test engine has a choice which arc should be selected, test engine choose arc randomly. If this option is not set, arcs are selected in the order prescribed by their order in test scenario.
97
98
_Option nondeterministic_
99
100
Option _--nondeterministic_ turns on support of _nondeterministic graphs_, i.e. graphs where several arcs labeled by the same stimulus can issue from the same state. The main difficulty in traversing of nondeterministic graphs is absence of possibility of determination which arc has been traversed by applying some or other stimulus. It makes finding of paths with non traversed arcs more difficult.
101
102
Setting of option _--nondeterministic_ allows usage of nondeterministic arc during path finding (increasing class of supported graphs). If this option is not set, paths are constructed via deterministic arcs.
103
104
_Option dump-fsm_
105
106
Option _--dump-fsm_ is used for saving of state graph in file after test finalizing. File has fixed name fsm.gv; in this file data are represented in Graphviz format, which is an open source graph visualization package (http://www.graphviz.org).
107
108
_Option full-graph_
109
110
Option _--full-graph_ (set together with --dump-fsm) saves state graph as a whole.
111
112
*Notice*: options --full-graph and --forward-tree are incompatible.
113
114
_Option forward-tree_
115
116
Option _--forward-tree_ (set together with _--dump-fsm_) saves only state graph spanning tree.
117
118
*Notice*: options _--forward-tree_ and _--full-graph_ are incompatible.
119
120
_Default values_
121
122
Options _--randomize_, _--nondeterministic_, and _--dump-fsm_ are not set by default.
123
If option _--dump-fsm_ is set, state graph is saved as a whole by default.
124
125
h3. Test engine @rnd@
126
127
Test engine @rnd@ constructs test sequence randomly selecting random stimulus or random stimulus set from the list of allowed in test scenario stimuli at each work step. Test is finished when given number of stimulus is called.
128
129
*Brief algorithm description*
130
131
There are two modes of test engine @rnd@ work. The first one is sequential (see chapter _«Option sequential»_) and parallel (see chapter _«Option parallel»_).
132
In sequential mode all the stimuli, set up by test action iterator, are equitable. At each work step, identifier of stimulus is randomly selected from range @[0, Nmax(ModelState)-1]@, and correspondent stimulus is applied.
133
134
In parallel mode stimulus with identifier 0 has a special meaning playing role of delay stimulus. All the other stimuli are not allowed to shift simulation time (they are “immediate” actions without waiting of reactions for them). In this mode at each work step random stimulus sequence (_multi stimulus_) is created. This sequence is finished by stimulus with identifier 0. In the other words, several stimuli are applied in parallel, and then some delay happens.
135
136
Size of multi stimulus depends on _load mode_. Test engine @rnd@ supports four load modes.
137
* _minimum load mode_ (see chapter _«Option min-load»_);
138
* _maximum load mode_ (see chapter _«Option max-load»_);
139
* _fixed load mode_ (see chapter _«Parameter fix-load»_);
140
* _variable load mode_ (see chapter _«Option var-load»_).
141
142
The criterion of test completeness is passing of certain number of test steps (see chapter _«Parameter length»_).
143
144
*Command line parameters*
145
146
Test engine rnd supports the following command line parameters.
147
* _--sequential_ — sequential mode;
148
* _--parallel_ — parallel mode:
149
* _--min-load_ — minimum load mode;
150
* _--max-load_ — maximum load mode;
151
* _--fix-load_ — fixed load mode;
152
* _--var-load_ — variable load mode.
153
154
_Option sequential_
155
156
Option _--sequential_ turns on sequential mode of test engine work. In this mode, test engine at each step selects random stimulus from the list of available in test scenario stimuli. Sequential mode is aimed for systems which does not support application of stimuli in parallel mode.
157
158
*Notice*: options _--sequential_ and _--parallel_ are mutually exclusive.
159
160
_Option parallel_
161
162
Option _--parallel_ turns on parallel mode of test engine work. In this mode, test engine at each step generates random stimulus sequence finished with stimulus with identifier 0 — delay stimulus (typically this is a scenario method without iterations called @nop()@ or @delay()@) . All the stimuli except stimulus 0 are supposed to run immediately. Despite their attempt to run some or other operation, their real start is performed only after calling stimulus 0 when time is shifted. Test step in parallel mode is schematically showed in figure 1.
163
164
TODO: no figure
165
*Figure 1. Test step in parallel mode*
166
167
*Notice*: options _--sequential_ and _--parallel_ are mutually exclusive.
168
169
_Option min-load_
170
171
Option _--min-load_ turns on minimal load mode allowing not more than one stimulus at each work step. This mode is typically used during debug of test system.
172
173
_Option max-load_
174
175
Option _--max-load_ turns on maximum load mode to apply all possible stimuli described in test scenario at each work step.
176
177
_Parameter fix-load_
178
179
Parameter _--fix-load load_percentage_ sets fixed load mode to apply certain part of stimuli at each work step. E.g., if _--fix-load 50_ is written, half of the whole stimulus set will be applied in parallel at each step.
180
181
_Option var-load_
182
183
Option _--var-load_ turns on variable load mode to let test system load be smoothly increased from minimum to maximum.
184
185
_Default values_
186
187
Option _--sequential_ is used by default.
188
189
If option _--parallel_ is used, variable load mode _--var-load_ is used by default.