Feature #9693
openFeature #9691: Integrate Juliet test suite as one more validation job
Develop a new algorithm for generating fragments for Juilet test cases
0%
Description
As far as I understand, it will be trivial. You will just need to combine together C source files compiled and linked together when building test cases independently on each other.
The another question is how to filter out C++ source files like we do for, say, Assembler files.
Updated by Ilja Zakharov over 5 years ago
- Status changed from New to Resolved
Implemented in juliet-suit.
To make this working use source code and Makefiles without any modifications. Then build with Clade the build base using target "individuals". You can disable collecting data with CIF or keep at least callgraph to speedup the process. Then copy the base and modify both meta.json files here setting two different options sets:
"Info.extra_CIF_opts":["-DINCLUDEMAIN", "-DOMITBAD"]
And in an another base:
"Info.extra_CIF_opts":["-DINCLUDEMAIN", "-DOMITGOOD"]
After that you can run the Klever framework providing it with these two build bases for good and bad test cases.
Note! That it will take really a lot of time, maybe a day or even more, since there are more that 50k tests in this job. Use for that a virtual machine, VerifierCloud cluster or anything else with maximum parallelism. After the run you will see a table with matching and wrong verdicts as that one which Klever provides for tests.
And an extra minor note. To run a single test or several tests use regular expressions or object file names like "testcases/CWE122_Heap_Based_Buffer_Overflow/s08/CWE122_Heap_Based_Buffer_Overflow__c_CWE805_char_snprintf_64.out" in the job json file. Also you can run "juliet:empty" and "juliet:memory safety" rules.
Updated by Evgeny Novikov almost 5 years ago
- Priority changed from Urgent to High
- Target version deleted (
3.0)
Now it does not seem that the Juilet test suite is good for benchmarking Klever. So, let's consider it later.
Updated by Evgeny Novikov almost 5 years ago
- Status changed from Resolved to Open
Anyway, the branch should be rebased according to the latest master.