Project

General

Profile

Feature #8374

Comparison of verification job results does not work

Added by Evgeny Novikov almost 2 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Urgent
Category:
Bridge
Target version:
Start date:
08/22/2017
Due date:
10/12/2017
% Done:

100%

Estimated time:
Published in build:

Description

I got the following exception when I have tried to compare attached verification job results:

Comparison of reports' trees failed: Too many leaf reports for "40|51" 
Traceback (most recent call last):
  File "/var/www/klever-bridge/reports/views.py", line 690, in fill_compare_cache
    CompareTree(request.user, j1, j2)
  File "/var/www/klever-bridge/reports/comparison.py", line 136, in __init__
    self.tree1 = ReportTree(j1)
  File "/var/www/klever-bridge/reports/comparison.py", line 57, in __init__
    self.__get_tree()
  File "/var/www/klever-bridge/reports/comparison.py", line 81, in __get_tree
    self.__fill_leaves_vals(l_type)
  File "/var/www/klever-bridge/reports/comparison.py", line 114, in __fill_leaves_vals
    raise ValueError('Too many leaf reports for "%s"' % attrs_id)
ValueError: Too many leaf reports for "40|51" 

I suppose that this issue exists for a long period of time.


Files

Job-976d372039-0.zip (13.8 MB) Job-976d372039-0.zip Evgeny Novikov, 08/22/2017 01:50 PM
Job-b762ace2fb-0.zip (13.7 MB) Job-b762ace2fb-0.zip Evgeny Novikov, 08/22/2017 01:50 PM

History

#1

Updated by Vladimir Gratinskiy almost 2 years ago

  • Status changed from New to Feedback

These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.

#2

Updated by Vladimir Gratinskiy almost 2 years ago

I've changed error messages in branch "fix_8374".

#3

Updated by Evgeny Novikov almost 2 years ago

  • Status changed from Feedback to Open

Vladimir Gratinskiy wrote:

These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.

Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.

#4

Updated by Vladimir Gratinskiy almost 2 years ago

Evgeny Novikov wrote:

Vladimir Gratinskiy wrote:

These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.

Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.

bridge/vars.py:

JOBS_COMPARE_ATTRS = {
    JOB_CLASSES[0][0]: ['Verification object', 'Rule specification'],
    JOB_CLASSES[1][0]: ['Name', 'Verification object', 'Rule specification'],
}

For these jobs for the same pair rule-verification object there are reports that can't be compared, for example, 2 safes or 2 unsafes+safe. I can't calculate total verdict for such reports. I don't want to do comparison depending on specific attributes. So either create new job class or change these jobs class to validation.

#5

Updated by Evgeny Novikov almost 2 years ago

Vladimir Gratinskiy wrote:

Evgeny Novikov wrote:

Vladimir Gratinskiy wrote:

These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.

Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.

bridge/vars.py:
[...]

For these jobs for the same pair rule-verification object there are reports that can't be compared, for example, 2 safes or 2 unsafes+safe. I can't calculate total verdict for such reports. I don't want to do comparison depending on specific attributes. So either create new job class or change these jobs class to validation.

Now I understand. It looks like some new tests were added which made comparison impossible. Indeed I think that comparison is actually possible, you just need to add one more number outside of the main table that reflects the number of reports that can't be compared. By clicking that number a usual comparison of report trees with normal navigation can be shown. By showing these details one will be able to understand what is wrong. For instance, I will understand what tests are wrong or whether it is necessary to introduce new comparison attributes or even a new job class. This functionality will be helpful further as well.

#6

Updated by Evgeny Novikov almost 2 years ago

  • Tracker changed from Bug to Feature
  • Priority changed from Urgent to High

This isn't a bug, but we can improve comparison so that jobs like attached will be compared (indeed all jobs of the same class will be comparable).

#7

Updated by Evgeny Novikov almost 2 years ago

  • Priority changed from High to Urgent
  • Target version set to 1.0

Because of this feature is missed I can't compare more and more jobs. You can download two normal jobs for which comparison also does not work (1 and 2)

#8

Updated by Vladimir Gratinskiy almost 2 years ago

  • Due date set to 10/12/2017
  • Status changed from Open to Resolved
  • % Done changed from 0 to 100

The problem was solved in branch "feature_8374".

#9

Updated by Evgeny Novikov over 1 year ago

  • Status changed from Resolved to Closed

I merged the branch to master in 7f4fc2f9.

#10

Updated by Evgeny Novikov over 1 year ago

Don't forget to migrate your databases after update.

Also available in: Atom PDF