Project

General

Profile

Actions

Feature #6737

closed

Get coverage from static verifiers if corresponding option is set

Added by Evgeny Novikov about 8 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Urgent
Category:
Scheduling
Target version:
-
Start date:
02/03/2016
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

Static verifiers (I know just about CPAchecker) can produce code coverage that can be useful for debugging purposes. Workers should get it and include into decision result files.

Note that coverage of original source files is assumed to be collected.


Files

partially_coverage.zip (140 KB) partially_coverage.zip Alexey Polushkin, 06/27/2017 02:38 PM
full_coverage.zip (2.24 MB) full_coverage.zip Alexey Polushkin, 06/27/2017 02:38 PM
partially_coverage2.zip (148 KB) partially_coverage2.zip Alexey Polushkin, 06/29/2017 06:17 PM
full_coverage2.zip (2.28 MB) full_coverage2.zip Alexey Polushkin, 06/29/2017 06:17 PM
full_coverage3.zip (665 KB) full_coverage3.zip Alexey Polushkin, 06/30/2017 03:48 PM
partially_coverage3.zip (33 KB) partially_coverage3.zip Alexey Polushkin, 06/30/2017 03:48 PM
partially_coverag4.zip (32.9 KB) partially_coverag4.zip Alexey Polushkin, 07/05/2017 05:13 PM
full_coverage4.zip (667 KB) full_coverage4.zip Alexey Polushkin, 07/05/2017 05:13 PM
full_coverage5.zip (665 KB) full_coverage5.zip Alexey Polushkin, 07/12/2017 03:22 PM
partially_coverage5.zip (33.2 KB) partially_coverage5.zip Alexey Polushkin, 07/12/2017 03:22 PM
full_coverage6.zip (668 KB) full_coverage6.zip Alexey Polushkin, 07/13/2017 05:25 PM
partially_coverage6.zip (33.4 KB) partially_coverage6.zip Alexey Polushkin, 07/13/2017 05:26 PM

Related issues 5 (0 open5 closed)

Related to Klever - Feature #8255: Move model files to correct dirRejected06/22/2017

Actions
Related to Klever - Feature #8352: Calculate and visualize total code coverageClosedEvgeny Novikov08/15/201710/09/2017

Actions
Related to Klever - Feature #8355: Calculate lines coverage without empty lines and some preprocessor directivesClosedAlexey Polushkin08/15/2017

Actions
Blocks Klever - Feature #6738: Upload coverage if soClosedAlexey Polushkin02/03/2016

Actions
Blocked by Klever - Feature #6664: Development modeClosedVladimir Gratinskiy02/01/2016

Actions
Actions #1

Updated by Evgeny Novikov about 8 years ago

  • Description updated (diff)
Actions #2

Updated by Evgeny Novikov over 7 years ago

  • Subject changed from Get coverage from static verifiers in development mode to Get coverage from static verifiers if corresponding option is set
  • Assignee set to Alexey Polushkin
  • Priority changed from Normal to Urgent

Updated by Alexey Polushkin almost 7 years ago

I have attached archives, that correspond to coverage format.

Actions #4

Updated by Ilja Zakharov over 6 years ago

  • Status changed from New to Feedback
  • Assignee changed from Alexey Polushkin to Vladimir Gratinskiy

The partial archive does not contain header files that exist in directories with C files. This requirement has already been stated in the documentation but I fixed it a bit to be more explicit. You need to add them both to full ant partial representations of coverage archive. The rest is fine at the first glance.

Vladimir, do you need any additional examples or artifacts to start the development for #6739?

Alexey, add information about the branch with the implementation and update, please, the partial version of the archive.

Actions #5

Updated by Vladimir Gratinskiy over 6 years ago

Ilja Zakharov wrote:

Vladimir, do you need any additional examples or artifacts to start the development for #6739?

No, I can't - coverage.json in both examples doesn't have "value analysis" section.

Actions #6

Updated by Evgeny Novikov over 6 years ago

  • Status changed from Feedback to Open
  • Assignee changed from Vladimir Gratinskiy to Alexey Polushkin

Vladimir will implement blocked issue #6739. There is #6738 in the middle.

Updated by Alexey Polushkin over 6 years ago

Updated examples of coverage. Value analysis data (only as example) is Lines of Code.

Also, I have pushed my work to 'coverage' branch

Actions #8

Updated by Ilja Zakharov over 6 years ago

This is not enough for developing and testing visualization. Expect more scenarios of data covered.

Actions #9

Updated by Alexey Polushkin over 6 years ago

  • File partially_coverage3.zip added
  • File full_coverage3.zip added

I have updated coverage examples. They contain more scenarios.

Actions #10

Updated by Alexey Polushkin over 6 years ago

  • File deleted (partially_coverage3.zip)
Actions #11

Updated by Alexey Polushkin over 6 years ago

  • File deleted (full_coverage3.zip)
Actions #13

Updated by Ilja Zakharov over 6 years ago

  • Status changed from Open to Feedback
  • Assignee changed from Alexey Polushkin to Vladimir Gratinskiy

Vladimir, is it detailed enough for you? The last added by Alexey example contains two variants of data in addition to coverage.

Actions #14

Updated by Vladimir Gratinskiy over 6 years ago

Ilja Zakharov wrote:

Vladimir, is it detailed enough for you? The last added by Alexey example contains two variants of data in addition to coverage.

Yes, it is enough and I already started.

Updated by Alexey Polushkin over 6 years ago

I've updated examples of coverage files.
1. Got rid of redundant nesting
2. Dirs in the root are renamed

Updated by Alexey Polushkin over 6 years ago

I've updated coverage files.
1. Returned statistics for function coverage

Actions #17

Updated by Vladimir Gratinskiy over 6 years ago

Alexey Polushkin wrote:

I've updated coverage files.
1. Returned statistics for function coverage

Root dirs in these examples are old.

Updated by Alexey Polushkin over 6 years ago

Vladimir Gratinskiy wrote:

Alexey Polushkin wrote:

I've updated coverage files.
1. Returned statistics for function coverage

Root dirs in these examples are old.

Sorry. I have updated examples with correct dirs.

Actions #19

Updated by Vladimir Gratinskiy over 6 years ago

First version of coverage visualization is ready (branch "bridge-coverage"). So everybody can test it (verification reports should have coverage for it).

Actions #20

Updated by Ilja Zakharov over 6 years ago

There is a question concerning sending reports with coverage. Seems that it is expected to send coverage data within a verification report, but it already contains files with a verification task. I propose to keep 'files' attribute as it was before and for coverage use a separate one 'coverage files'.

Actions #21

Updated by Vladimir Gratinskiy over 6 years ago

Now fields for files archive for all types of reports is "report files archive" (renamed from "file"). And for coverage archive - "coverage files archive". Coverage archive can only be uploaded for report with type "verification".

Actions #22

Updated by Evgeny Novikov over 6 years ago

  • Status changed from Feedback to Closed
  • Assignee changed from Vladimir Gratinskiy to Alexey Polushkin
The great news is that I merged to master everything required to get, to upload and to visualize coverage in fc107b76. So this the first time when we have a completely automatic way from getting coverage from a verifier to showing it to a user. To get coverage one needs:
  1. to migrate databases,
  2. to update CPAchecker,
  3. to specify one of full, partial, lightweight instead of none as value of option VTG strategy -> collect coverage within job.json.
There are several known issues I suppose to deal with separately:
  1. Coverage can be collected just for verification tasks and not for the whole verification (sub)jobs.
  2. Obviously not all verifiers (most likely just CPAchecker) as well as verifier configurations allow to get coverage.
  3. To calculate statistics on covered lines the total number of lines in files is used while most likely, say, empty lines, comments and perhaps some preprocessor directives should be ignored.

Likely there are other related limitations, bugs and features - please, report them separately.

Actions

Also available in: Atom PDF