Feature #8374
closedComparison of verification job results does not work
100%
Description
I got the following exception when I have tried to compare attached verification job results:
Comparison of reports' trees failed: Too many leaf reports for "40|51"
Traceback (most recent call last):
File "/var/www/klever-bridge/reports/views.py", line 690, in fill_compare_cache
CompareTree(request.user, j1, j2)
File "/var/www/klever-bridge/reports/comparison.py", line 136, in __init__
self.tree1 = ReportTree(j1)
File "/var/www/klever-bridge/reports/comparison.py", line 57, in __init__
self.__get_tree()
File "/var/www/klever-bridge/reports/comparison.py", line 81, in __get_tree
self.__fill_leaves_vals(l_type)
File "/var/www/klever-bridge/reports/comparison.py", line 114, in __fill_leaves_vals
raise ValueError('Too many leaf reports for "%s"' % attrs_id)
ValueError: Too many leaf reports for "40|51"
I suppose that this issue exists for a long period of time.
Files
Updated by Vladimir Gratinskiy over 6 years ago
- Status changed from New to Feedback
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Updated by Vladimir Gratinskiy over 6 years ago
I've changed error messages in branch "fix_8374".
Updated by Evgeny Novikov over 6 years ago
- Status changed from Feedback to Open
Vladimir Gratinskiy wrote:
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.
Updated by Vladimir Gratinskiy over 6 years ago
Evgeny Novikov wrote:
Vladimir Gratinskiy wrote:
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.
bridge/vars.py:
JOBS_COMPARE_ATTRS = {
JOB_CLASSES[0][0]: ['Verification object', 'Rule specification'],
JOB_CLASSES[1][0]: ['Name', 'Verification object', 'Rule specification'],
}
For these jobs for the same pair rule-verification object there are reports that can't be compared, for example, 2 safes or 2 unsafes+safe. I can't calculate total verdict for such reports. I don't want to do comparison depending on specific attributes. So either create new job class or change these jobs class to validation.
Updated by Evgeny Novikov over 6 years ago
Vladimir Gratinskiy wrote:
Evgeny Novikov wrote:
Vladimir Gratinskiy wrote:
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.
bridge/vars.py:
[...]For these jobs for the same pair rule-verification object there are reports that can't be compared, for example, 2 safes or 2 unsafes+safe. I can't calculate total verdict for such reports. I don't want to do comparison depending on specific attributes. So either create new job class or change these jobs class to validation.
Now I understand. It looks like some new tests were added which made comparison impossible. Indeed I think that comparison is actually possible, you just need to add one more number outside of the main table that reflects the number of reports that can't be compared. By clicking that number a usual comparison of report trees with normal navigation can be shown. By showing these details one will be able to understand what is wrong. For instance, I will understand what tests are wrong or whether it is necessary to introduce new comparison attributes or even a new job class. This functionality will be helpful further as well.
Updated by Evgeny Novikov over 6 years ago
- Tracker changed from Bug to Feature
- Priority changed from Urgent to High
This isn't a bug, but we can improve comparison so that jobs like attached will be compared (indeed all jobs of the same class will be comparable).
Updated by Evgeny Novikov over 6 years ago
- Priority changed from High to Urgent
- Target version set to 1.0
Updated by Vladimir Gratinskiy over 6 years ago
- Due date set to 10/12/2017
- Status changed from Open to Resolved
- % Done changed from 0 to 100
The problem was solved in branch "feature_8374".
Updated by Evgeny Novikov over 6 years ago
- Status changed from Resolved to Closed
I merged the branch to master in 7f4fc2f9.
Updated by Evgeny Novikov over 6 years ago
Don't forget to migrate your databases after update.