Project

General

Profile

Feature #8338

Add ability to specify similarity threshold

Added by Evgeny Novikov almost 2 years ago. Updated 15 days ago.

Status:
Resolved
Priority:
Urgent
Category:
Bridge
Target version:
Start date:
08/11/2017
Due date:
07/02/2019
% Done:

100%

Estimated time:
Published in build:

Description

As was noted during discussions for #8335 one useful feature is an ability to specify a similarity threshold to associate marks with reports which are similar not for +0% (which is used now by default) but for the specified threshold. I suggest a new field should be added at the heavyweight creation/update mark page where an integer from 0 to 100 can be specified. If nothing will be specified than the default threshold should be used.


Related issues

Related to Klever - Feature #8335: Similarity managementResolved08/11/201707/11/2019

Actions
Related to Klever - Feature #9412: Support more advanced calculation of total verdicts and similarityResolved12/12/201807/10/2019

Actions
Blocks Klever - Feature #8339: Add user setting to select default error trace convertion function, comparison criterion and similarity thresholdNew08/11/2017

Actions

History

#1

Updated by Pavel Andrianov almost 2 years ago

Is this threshold associated with a mark? Imagine a situation: I am analyzing one launch and am creating marks with threshold 100%. Other person is analyzing other launch, but on the same machine and creating marks with threshold 50%. His marks are successfully applied to my job (launch) results, but I do not want such low threshold. Is there any protection?

#2

Updated by Evgeny Novikov almost 2 years ago

Pavel Andrianov wrote:

Is this threshold associated with a mark? Imagine a situation: I am analyzing one launch and am creating marks with threshold 100%. Other person is analyzing other launch, but on the same machine and creating marks with threshold 50%. His marks are successfully applied to my job (launch) results, but I do not want such low threshold. Is there any protection?

Good catch! This is a well known issue I thought about even when I wrote the specification. But it can be considered as a feature which can give quite much when playing carefully (e.g. one can find out duplicates of warnings or marks, suspicious reports or expert efforts and so on).

Indeed it isn't related with the similarity and its threshold - it is just related with the fact that there isn't any protection from associating third-party marks with your verification results even if they are private, i.e. other users can't see them and, say, create marks for them.

But as I already specified when discussing #8335 laziness can result in various difficulties anyway. From the other side I noticed that it can be not so bad if you will quickly "forget" about analyzed verification results as it usually happens.

Verification results protection should be implemented independently from this issue as well as #8335 since both of them doesn't help in this way in the general case (imagine that one can break your perfect statistics by creating malicious marks with similarity 100%). If you still think that it is useful for you, please, open another issue.

#3

Updated by Vadim Mutilin 7 months ago

Do we have a hope for the feature to be implemented in the nearest version?

#4

Updated by Evgeny Novikov 7 months ago

Do you need this in case when #9412 will be implemented?

#5

Updated by Evgeny Novikov 7 months ago

  • Status changed from New to Feedback
#6

Updated by Evgeny Novikov 7 months ago

  • Related to Feature #9412: Support more advanced calculation of total verdicts and similarity added
#7

Updated by Vadim Mutilin 7 months ago

Yes, we still need it even with #9412

#8

Updated by Evgeny Novikov 7 months ago

Pavel, what do you think? Will you agree that #9412 is a more proper way to evaluate verification results in general? This feature looks more like a cheat that can be supported and that will help in some cases to avoid manual work, but in many cases it will be wrong (both errors of first and second kind).

#9

Updated by Pavel Andrianov 7 months ago

I do not think it is a cheat. I would not like to resolve conflicts between 17% similar mark and 22% one, if, for example, they are incompatible. Also I do not see any wrong cases if we do not apply marks to a not similar traces. The only difference is less automatic applied marks.

#10

Updated by Evgeny Novikov 7 months ago

Pavel Andrianov wrote:

I do not think it is a cheat. I would not like to resolve conflicts between 17% similar mark and 22% one, if, for example, they are incompatible. Also I do not see any wrong cases if we do not apply marks to a not similar traces. The only difference is less automatic applied marks.

With #9412 you will need to either confirm one of existing associations or create a new mark for each report. And then you will have constant statistics unless somebody will delete marks having confirmed associations with reports of your job. I suggest to have an option to forbid this by mark authors.

Regarding your case, do you think that 17% is worse than 22%? Indeed, both associations may be irrelevant. Without investigation and manual evaluation no algorithm can provide you valid results.

#11

Updated by Pavel Andrianov 7 months ago

Regarding your case, do you think that 17% is worse than 22%?

Yes, both of them are irrelevant, so I do not like to open them, find differences or investigate them in any other way. I know, that similarity < 50% is useless. So, why we can not use this knowledge to prompt Klever. Let hide such marks and not distract the user.

Without investigation and manual evaluation no algorithm can provide you valid results.

In general case, yes. Anyway, there are other use cases, when a user needs to have only general overview. For example, several jobs with many unsafes and marks. Here manual confirmation will take a lot of time. With this feature a user may choose high similarity threshold to avoid automatic incompatible marks. The final results will be not so precise as with manual confirmation, but they will be much quicker.

#12

Updated by Evgeny Novikov 7 months ago

  • Target version set to 3.0
  • Priority changed from High to Urgent
  • Status changed from Feedback to Open

After all I agree that this is another use case, when users want to have another view of verification results (and associated marks) without much manual effort. Let's do it together with #9412.

#13

Updated by Vladimir Gratinskiy 15 days ago

  • % Done changed from 0 to 100
  • Status changed from Open to Resolved
  • Due date set to 07/02/2019

Implemented in bridge-3.0

Also available in: Atom PDF