Project

General

Profile

Feature #9412

Support more advanced calculation of total verdicts and similarity

Added by Ilja Zakharov 6 months ago. Updated 6 months ago.

Status:
New
Priority:
Urgent
Category:
Bridge
Target version:
Start date:
12/12/2018
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

The primary expert evaluation workflow should look as follows:
  1. Imagine there is no associated marks for a given report (Unsafe/Safe/Unknown).
  2. The expert needs to create a new mark.
  3. It becomes confirmed automatically for a given report and may be associated automatically with some other reports.
  4. Then the expert proceeds to a next report.
  5. If there is no associated marks, the expert should do the same (1-3).
  6. If there are some automatically associated marks and the expert finds one or more of them appropriate, the expert should confirm corresponding associations.
  7. Otherwise, the expert must create a new mark like in previous use cases (1-3, 5).

In any case there will be at least one confirmed association (in ideal, there should be the only confirmed association) for each report. And if there are confirmed associations, Bridge should not treat automatically associated marks when calculating total verdict and total similarity. If there is no confirmed associations, Bridge should behave as it does now.

PS: Let's exclude associations with 0% similarity from calculating total verdicts and similarity (inspired by #9413). This will improve a picture for automatically associated marks a bit since there will be less incompatible marks. But in general this will not help while the primary suggested feature will do. The pitfall may be confirmed associations with 0% similarity as it should be both used and not used in calculating total values according with various rules. I can hardly imagine cases when one will need to confirm associations with 0% similarity, but let's respect manual efforts (confirmation) first of all. So, let's calculate total values for confirmed associations with 0% similarity.

PPS: Let's exclude total similarity at all as like some other totals it is ambiguous. Some people likes one its semantics, other people prefers another one. But nobody will suffer much if there will be no total similarity at all since it is primarily useful just when considering particular error traces and their associated marks.

PPPS: I suggest to allow for mark authors to disable further mark changes. This will disable the only case (removing marks or changing them considerably) when somebody can affect evaluation of verification results made by mark authors. Of course, this does not include 3rd party marks which associations are confirmed.


Related issues

Related to Klever - Feature #8335: Similarity managementOpen08/11/2017

Actions
Related to Klever - Feature #8338: Add ability to specify similarity thresholdOpen08/11/2017

Actions
Has duplicate Klever - Feature #9413: Set total verdict and total similarity more precisely on base of automatic marks assotiationRejected12/12/2018

Actions

History

#1

Updated by Ilja Zakharov 6 months ago

  • Related to Feature #9413: Set total verdict and total similarity more precisely on base of automatic marks assotiation added
#2

Updated by Evgeny Novikov 6 months ago

Indeed, I dislike the suggested approach as it contradicts the primary workflow. Nobody prevents you to unconfirm new associatead marks - this way suits well the primary workflow. But I agree that often one is too lazy to do such the job.

I already suggested another way to reach the same result using other means. These means include protecting jobs from associations with all existing marks. So, you will need to create marks for all reports of protected jobs, and nobody will be able to affect your evaluation. One another mean to reach this is ability to freeze marks by their authors. Note that this use case forbids other issues, e.g. when you confirm some associations, but then one will change associated marks, your confirmations will be cancelled.

#3

Updated by Evgeny Novikov 6 months ago

We discussed this issue more. Now I agree that the improvement suggested by Ilja is good. So, the primary workflow will look as follows. Imagine there is no associated marks for a given error trace. You create a new mark. It becomes confirmed automatically. Then you go to a next error trace. If there is no associated marks, you should do the same. If there are some automatically associated marks and you find one of them appropriate, you should confirm a corresponding association. Otherwise, you will create a new mark like in previous use cases. In any case there will be at least one confirmed association (in ideal, there should be the only confirmed association). And if there are confirmed associations, Bridge should not treat automatically associated marks when calculating total verdict and total similarity. If there is no confirmed associations, Bridge should behave as it does now.

#4

Updated by Pavel Andrianov 6 months ago

I support the issue: incompatible marks are very distractive. One minor comment is that several confirmed associations may be present. This is not a general case, when we want to emphasize several marks as confirmed, but Bridge should not fail in this case.

#5

Updated by Evgeny Novikov 6 months ago

Pavel Andrianov wrote:

I support the issue: incompatible marks are very distractive. One minor comment is that several confirmed associations may be present. This is not a general case, when we want to emphasize several marks as confirmed, but Bridge should not fail in this case.

Bridge should not fail in any case. I also specified that there will be at least one confirmed association for each report. But there may be more ones. Perhaps, it is important to note, that in case when there are several confirmed associations Bridge should calculate total verdicts and similarity like when there are several automatic associations without confirmed ones.

#6

Updated by Evgeny Novikov 6 months ago

  • Description updated (diff)
  • Subject changed from Do not consider an error trace with manual confirmations as "incompatible marks" to Support more advanced calculation of total verdicts and similarity
#7

Updated by Evgeny Novikov 6 months ago

  • Related to deleted (Feature #9413: Set total verdict and total similarity more precisely on base of automatic marks assotiation)
#8

Updated by Evgeny Novikov 6 months ago

  • Has duplicate Feature #9413: Set total verdict and total similarity more precisely on base of automatic marks assotiation added
#9

Updated by Evgeny Novikov 6 months ago

#10

Updated by Evgeny Novikov 6 months ago

  • Related to Feature #8338: Add ability to specify similarity threshold added
#11

Updated by Evgeny Novikov 6 months ago

  • Description updated (diff)
#12

Updated by Evgeny Novikov 6 months ago

  • Description updated (diff)

Also available in: Atom PDF