Feature #6737
closedGet coverage from static verifiers if corresponding option is set
Added by Evgeny Novikov almost 9 years ago. Updated over 7 years ago.
0%
Description
Static verifiers (I know just about CPAchecker) can produce code coverage that can be useful for debugging purposes. Workers should get it and include into decision result files.
Note that coverage of original source files is assumed to be collected.
Files
| partially_coverage.zip (140 KB) partially_coverage.zip | Alexey Polushkin, 06/27/2017 02:38 PM | ||
| full_coverage.zip (2.24 MB) full_coverage.zip | Alexey Polushkin, 06/27/2017 02:38 PM | ||
| partially_coverage2.zip (148 KB) partially_coverage2.zip | Alexey Polushkin, 06/29/2017 06:17 PM | ||
| full_coverage2.zip (2.28 MB) full_coverage2.zip | Alexey Polushkin, 06/29/2017 06:17 PM | ||
| full_coverage3.zip (665 KB) full_coverage3.zip | Alexey Polushkin, 06/30/2017 03:48 PM | ||
| partially_coverage3.zip (33 KB) partially_coverage3.zip | Alexey Polushkin, 06/30/2017 03:48 PM | ||
| partially_coverag4.zip (32.9 KB) partially_coverag4.zip | Alexey Polushkin, 07/05/2017 05:13 PM | ||
| full_coverage4.zip (667 KB) full_coverage4.zip | Alexey Polushkin, 07/05/2017 05:13 PM | ||
| full_coverage5.zip (665 KB) full_coverage5.zip | Alexey Polushkin, 07/12/2017 03:22 PM | ||
| partially_coverage5.zip (33.2 KB) partially_coverage5.zip | Alexey Polushkin, 07/12/2017 03:22 PM | ||
| full_coverage6.zip (668 KB) full_coverage6.zip | Alexey Polushkin, 07/13/2017 05:25 PM | ||
| partially_coverage6.zip (33.4 KB) partially_coverage6.zip | Alexey Polushkin, 07/13/2017 05:26 PM |
Updated by Evgeny Novikov about 8 years ago
- Subject changed from Get coverage from static verifiers in development mode to Get coverage from static verifiers if corresponding option is set
- Assignee set to Alexey Polushkin
- Priority changed from Normal to Urgent
Updated by Alexey Polushkin over 7 years ago
- File full_coverage.zip full_coverage.zip added
- File partially_coverage.zip partially_coverage.zip added
I have attached archives, that correspond to coverage format.
Updated by Ilja Zakharov over 7 years ago
- Status changed from New to Feedback
- Assignee changed from Alexey Polushkin to Vladimir Gratinskiy
The partial archive does not contain header files that exist in directories with C files. This requirement has already been stated in the documentation but I fixed it a bit to be more explicit. You need to add them both to full ant partial representations of coverage archive. The rest is fine at the first glance.
Vladimir, do you need any additional examples or artifacts to start the development for #6739?
Alexey, add information about the branch with the implementation and update, please, the partial version of the archive.
Updated by Vladimir Gratinskiy over 7 years ago
Ilja Zakharov wrote:
Vladimir, do you need any additional examples or artifacts to start the development for #6739?
No, I can't - coverage.json in both examples doesn't have "value analysis" section.
Updated by Evgeny Novikov over 7 years ago
- Status changed from Feedback to Open
- Assignee changed from Vladimir Gratinskiy to Alexey Polushkin
Updated by Alexey Polushkin over 7 years ago
- File partially_coverage2.zip partially_coverage2.zip added
- File full_coverage2.zip full_coverage2.zip added
Updated examples of coverage. Value analysis data (only as example) is Lines of Code.
Also, I have pushed my work to 'coverage' branch
Updated by Ilja Zakharov over 7 years ago
This is not enough for developing and testing visualization. Expect more scenarios of data covered.
Updated by Alexey Polushkin over 7 years ago
- File partially_coverage3.zip added
- File full_coverage3.zip added
I have updated coverage examples. They contain more scenarios.
Updated by Alexey Polushkin over 7 years ago
- File deleted (
partially_coverage3.zip)
Updated by Alexey Polushkin over 7 years ago
- File full_coverage3.zip full_coverage3.zip added
- File partially_coverage3.zip partially_coverage3.zip added
Sorry, I've added not all files
Updated by Ilja Zakharov over 7 years ago
- Status changed from Open to Feedback
- Assignee changed from Alexey Polushkin to Vladimir Gratinskiy
Vladimir, is it detailed enough for you? The last added by Alexey example contains two variants of data in addition to coverage.
Updated by Vladimir Gratinskiy over 7 years ago
Ilja Zakharov wrote:
Vladimir, is it detailed enough for you? The last added by Alexey example contains two variants of data in addition to coverage.
Yes, it is enough and I already started.
Updated by Alexey Polushkin over 7 years ago
- File partially_coverag4.zip partially_coverag4.zip added
- File full_coverage4.zip full_coverage4.zip added
I've updated examples of coverage files.
1. Got rid of redundant nesting
2. Dirs in the root are renamed
Updated by Alexey Polushkin over 7 years ago
- File full_coverage5.zip full_coverage5.zip added
- File partially_coverage5.zip partially_coverage5.zip added
I've updated coverage files.
1. Returned statistics for function coverage
Updated by Vladimir Gratinskiy over 7 years ago
Alexey Polushkin wrote:
I've updated coverage files.
1. Returned statistics for function coverage
Root dirs in these examples are old.
Updated by Alexey Polushkin over 7 years ago
- File full_coverage6.zip full_coverage6.zip added
- File partially_coverage6.zip partially_coverage6.zip added
Vladimir Gratinskiy wrote:
Alexey Polushkin wrote:
I've updated coverage files.
1. Returned statistics for function coverageRoot dirs in these examples are old.
Sorry. I have updated examples with correct dirs.
Updated by Vladimir Gratinskiy over 7 years ago
First version of coverage visualization is ready (branch "bridge-coverage"). So everybody can test it (verification reports should have coverage for it).
Updated by Ilja Zakharov over 7 years ago
There is a question concerning sending reports with coverage. Seems that it is expected to send coverage data within a verification report, but it already contains files with a verification task. I propose to keep 'files' attribute as it was before and for coverage use a separate one 'coverage files'.
Updated by Vladimir Gratinskiy over 7 years ago
Now fields for files archive for all types of reports is "report files archive" (renamed from "file"). And for coverage archive - "coverage files archive". Coverage archive can only be uploaded for report with type "verification".
Updated by Evgeny Novikov over 7 years ago
- Status changed from Feedback to Closed
- Assignee changed from Vladimir Gratinskiy to Alexey Polushkin
- to migrate databases,
- to update CPAchecker,
- to specify one of full, partial, lightweight instead of none as value of option VTG strategy -> collect coverage within job.json.
- Coverage can be collected just for verification tasks and not for the whole verification (sub)jobs.
- Obviously not all verifiers (most likely just CPAchecker) as well as verifier configurations allow to get coverage.
- To calculate statistics on covered lines the total number of lines in files is used while most likely, say, empty lines, comments and perhaps some preprocessor directives should be ignored.
Likely there are other related limitations, bugs and features - please, report them separately.