Klever

Recent support for testing jobs made this issue priority even higher since tests are very small and their parallel execution can very considerably speed up job decision.

Branch "parallel-subjobs" implementing this feature was merged to master in 45c7ae5.

Starting from now users can specify the number of sub-jobs to be decided in parallel like the number of threads/processes for parallel build and verification task generation. By default this number is exactly 1 for all modes since it requires very thoughtful understanding of sub-jobs nature and corresponding resource requirements. For instance, lightweight tests (e.g. for rule specifications) can be decided in parallel (each test corresponds to one sub-job) since very little resources are required for them. In contrast verification of real Linux kernel modules, in particular during validation, can require very much resources, e.g. for generating environment models. Thus one should set this number in accordance with resources available/required.

One of consequence of this feature is that now all source files referred by error traces have absolute paths that can be noticed during error trace visualization. This looks to be the only appropriate solution when we deal with very distributed files where the most of them are shadows of original source files (they are copied into LKBCE working directory and then all compilers are invoked from that directory rather than from Linux kernel working source tree), some files are original files of rule specifications, some files are generated on the basis of rule specification files. Any way during visualization just some part of these paths is shown while the whole paths can be accessible by covering those ones.

The only thing I didn't implement is automatic resolution of source files overwritten during build process. At the moment there will be an assertion and everything will fail if there will be such the case. Let's see what we will get on real modules.

• Neither AVTG and its plugins except RSG nor VTG and its strategies can use working source tree any more. Instead they have to use CC working directories that are presented as values of attribute cwd of CC full descriptions.
• All processing (parsing, filtering, dumping and copying dependencies) of raw Linux kernel build commands are now performed directly when corresponding commands are executed. Before it was done in LKBCE and LKVOG. This was addressed in #6606 that I had to implement together with this feature since I needed to copy dependencies exactly when build commands were executed.
• All files referred by C source files passed to VTG and its strategies have absolute paths. Thus everybody can relax.
• One can introduce new locks if one will need to execute something exclusively. Unfortunately this can't be done via callbacks and thus it isn't recommended to do this except for very important cases (e.g. build processes of different sub-jobs are performed exclusively now).

Evgeny Novikov wrote:

Developers will see quite many changes. The most noticable ones are:

• Neither AVTG and its plugins except RSG nor VTG and its strategies can use working source tree any more. Instead they have to use CC working directories that are presented as values of attribute cwd of CC full descriptions.

I changed the way how RSG receives source tree roots in f3c0e15. Before that RSG hanged forever if there more than one verification object or/and rule specification in a given (sub-)job. This change also made accessible source tree root for AVTG itself unfortunately.

One more issue was with visualization of data reports uploaded during parallel decision of sub-jobs. This was due to sub-jobs required properly shared variable to keep results obtained. The issue was fixed in 968abd2.

Besides data representation was changed considerably to make things straightforwardly rather than to apply several artificial and non-logical transformations. That's why new Bridge will not visualize old data as well as old Bridge will not visualize new data. But since just data representation was changed old data can be actually converted to the new format.

Decision of validation sub-jobs in parallel revealed one more issue. It turned out that although all source files referred by CC build commands were copied to the LKBCE working directory references to them in CC build command descriptions weren't updated respectively. This caused unpleasant races that can in result in errors (missed or broken source files) in all components invoking preprocessor (SA, Weaver, ASE, ...).

The temporary workaround is to decide sub-jobs sequentially always.

The issue was likely fixed in af3b58e. Besides working source trees were replaced with shadow source trees since we copy all required source files into shadow source trees while working source trees can be overwritten.