https://forge.ispras.ru/https://forge.ispras.ru/favicon.ico?16490126692017-08-11T14:15:20ZOpen-Source ProjectsKlever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=310022017-08-11T14:15:20ZPavel Andrianovandrianov@ispras.ru
<ul></ul><p>Is this threshold associated with a mark? Imagine a situation: I am analyzing one launch and am creating marks with threshold 100%. Other person is analyzing other launch, but on the same machine and creating marks with threshold 50%. His marks are successfully applied to my job (launch) results, but I do not want such low threshold. Is there any protection?</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=310062017-08-13T12:42:18ZEvgeny Novikovnovikov@ispras.ru
<ul></ul><p>Pavel Andrianov wrote:</p>
<blockquote>
<p>Is this threshold associated with a mark? Imagine a situation: I am analyzing one launch and am creating marks with threshold 100%. Other person is analyzing other launch, but on the same machine and creating marks with threshold 50%. His marks are successfully applied to my job (launch) results, but I do not want such low threshold. Is there any protection?</p>
</blockquote>
<p>Good catch! This is a well known issue I thought about even when I wrote the specification. But it can be considered as a feature which can give quite much when playing carefully (e.g. one can find out duplicates of warnings or marks, suspicious reports or expert efforts and so on).</p>
<p>Indeed it isn't related with the similarity and its threshold - it is just related with the fact that there isn't any protection from associating third-party marks with your verification results even if they are private, i.e. other users can't see them and, say, create marks for them.</p>
<p>But as I already specified when discussing <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Similarity management (Closed)" href="https://forge.ispras.ru/issues/8335">#8335</a> laziness can result in various difficulties anyway. From the other side I noticed that it can be not so bad if you will quickly "forget" about analyzed verification results as it usually happens.</p>
<p>Verification results protection should be implemented independently from this issue as well as <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Similarity management (Closed)" href="https://forge.ispras.ru/issues/8335">#8335</a> since both of them doesn't help in this way in the general case (imagine that one can break your perfect statistics by creating malicious marks with similarity 100%). If you still think that it is useful for you, please, open another issue.</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354262018-12-13T11:57:13ZVadim Mutilinmutilin@ispras.ru
<ul></ul><p>Do we have a hope for the feature to be implemented in the nearest version?</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354492018-12-14T12:34:21ZEvgeny Novikovnovikov@ispras.ru
<ul></ul><p>Do you need this in case when <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Support more advanced calculation of total verdicts and similarity (Closed)" href="https://forge.ispras.ru/issues/9412">#9412</a> will be implemented?</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354502018-12-14T12:34:26ZEvgeny Novikovnovikov@ispras.ru
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354522018-12-14T12:34:45ZEvgeny Novikovnovikov@ispras.ru
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" href="/issues/9412">Feature #9412</a>: Support more advanced calculation of total verdicts and similarity</i> added</li></ul> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354542018-12-14T12:39:10ZVadim Mutilinmutilin@ispras.ru
<ul></ul><p>Yes, we still need it even with <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Support more advanced calculation of total verdicts and similarity (Closed)" href="https://forge.ispras.ru/issues/9412">#9412</a></p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354552018-12-14T13:08:42ZEvgeny Novikovnovikov@ispras.ru
<ul></ul><p>Pavel, what do you think? Will you agree that <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Support more advanced calculation of total verdicts and similarity (Closed)" href="https://forge.ispras.ru/issues/9412">#9412</a> is a more proper way to evaluate verification results in general? This feature looks more like a cheat that can be supported and that will help in some cases to avoid manual work, but in many cases it will be wrong (both errors of first and second kind).</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354642018-12-17T12:33:13ZPavel Andrianovandrianov@ispras.ru
<ul></ul><p>I do not think it is a cheat. I would not like to resolve conflicts between 17% similar mark and 22% one, if, for example, they are incompatible. Also I do not see any wrong cases if we do not apply marks to a not similar traces. The only difference is less automatic applied marks.</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354772018-12-17T19:34:04ZEvgeny Novikovnovikov@ispras.ru
<ul></ul><p>Pavel Andrianov wrote:</p>
<blockquote>
<p>I do not think it is a cheat. I would not like to resolve conflicts between 17% similar mark and 22% one, if, for example, they are incompatible. Also I do not see any wrong cases if we do not apply marks to a not similar traces. The only difference is less automatic applied marks.</p>
</blockquote>
<p>With <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Support more advanced calculation of total verdicts and similarity (Closed)" href="https://forge.ispras.ru/issues/9412">#9412</a> you will need to either confirm one of existing associations or create a new mark for each report. And then you will have constant statistics unless somebody will delete marks having confirmed associations with reports of your job. I suggest to have an option to forbid this by mark authors.</p>
<p>Regarding your case, do you think that 17% is worse than 22%? Indeed, both associations may be irrelevant. Without investigation and manual evaluation no algorithm can provide you valid results.</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354812018-12-19T08:16:52ZPavel Andrianovandrianov@ispras.ru
<ul></ul><blockquote>
<p>Regarding your case, do you think that 17% is worse than 22%?</p>
</blockquote>
<p>Yes, both of them are irrelevant, so I do not like to open them, find differences or investigate them in any other way. I know, that similarity < 50% is useless. So, why we can not use this knowledge to prompt Klever. Let hide such marks and not distract the user.</p>
<blockquote>
<p>Without investigation and manual evaluation no algorithm can provide you valid results.</p>
</blockquote>
<p>In general case, yes. Anyway, there are other use cases, when a user needs to have only general overview. For example, several jobs with many unsafes and marks. Here manual confirmation will take a lot of time. With this feature a user may choose high similarity threshold to avoid automatic incompatible marks. The final results will be not so precise as with manual confirmation, but they will be much quicker.</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=354822018-12-19T08:24:07ZEvgeny Novikovnovikov@ispras.ru
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Open</i></li><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li><li><strong>Target version</strong> set to <i>3.0</i></li></ul><p>After all I agree that this is another use case, when users want to have another view of verification results (and associated marks) without much manual effort. Let's do it together with <a class="issue tracker-4 status-5 priority-6 priority-high2 closed behind-schedule" title="Feature: Support more advanced calculation of total verdicts and similarity (Closed)" href="https://forge.ispras.ru/issues/9412">#9412</a>.</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=364232019-07-02T15:25:38ZVladimir Gratinskiygratinskiy@ispras.ru
<ul><li><strong>Due date</strong> set to <i>07/02/2019</i></li><li><strong>Status</strong> changed from <i>Open</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Implemented in bridge-3.0</p> Klever - Feature #8338: Add ability to specify similarity thresholdhttps://forge.ispras.ru/issues/8338?journal_id=377212020-01-16T14:13:36ZEvgeny Novikovnovikov@ispras.ru
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Closed</i></li></ul><p>In master.</p>