Feature #8374
closed
Comparison of verification job results does not work
Added by Evgeny Novikov over 6 years ago.
Updated over 6 years ago.
Description
I got the following exception when I have tried to compare attached verification job results:
Comparison of reports' trees failed: Too many leaf reports for "40|51"
Traceback (most recent call last):
File "/var/www/klever-bridge/reports/views.py", line 690, in fill_compare_cache
CompareTree(request.user, j1, j2)
File "/var/www/klever-bridge/reports/comparison.py", line 136, in __init__
self.tree1 = ReportTree(j1)
File "/var/www/klever-bridge/reports/comparison.py", line 57, in __init__
self.__get_tree()
File "/var/www/klever-bridge/reports/comparison.py", line 81, in __get_tree
self.__fill_leaves_vals(l_type)
File "/var/www/klever-bridge/reports/comparison.py", line 114, in __fill_leaves_vals
raise ValueError('Too many leaf reports for "%s"' % attrs_id)
ValueError: Too many leaf reports for "40|51"
I suppose that this issue exists for a long period of time.
Files
- Status changed from New to Feedback
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
I've changed error messages in branch "fix_8374".
- Status changed from Feedback to Open
Vladimir Gratinskiy wrote:
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.
Evgeny Novikov wrote:
Vladimir Gratinskiy wrote:
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.
bridge/vars.py:
JOBS_COMPARE_ATTRS = {
JOB_CLASSES[0][0]: ['Verification object', 'Rule specification'],
JOB_CLASSES[1][0]: ['Name', 'Verification object', 'Rule specification'],
}
For these jobs for the same pair rule-verification object there are reports that can't be compared, for example, 2 safes or 2 unsafes+safe. I can't calculate total verdict for such reports. I don't want to do comparison depending on specific attributes. So either create new job class or change these jobs class to validation.
Vladimir Gratinskiy wrote:
Evgeny Novikov wrote:
Vladimir Gratinskiy wrote:
These jobs should belong to validation class as it has "Name" attribute. Or all verification jobs should have this attribute.
Another class intended for verification of Linux kernel modules leverages attribute Name for a long time without any problems, namely, it is used for tests with many sub-jobs and there is even a specific visualization of testing results. Perhaps I never compared verification results for different testing jobs although this is strange. Why does this attribute breaks comparison? Indeed I suggest to take it into account if it exists, when comparing verification results for this class as well.
bridge/vars.py:
[...]
For these jobs for the same pair rule-verification object there are reports that can't be compared, for example, 2 safes or 2 unsafes+safe. I can't calculate total verdict for such reports. I don't want to do comparison depending on specific attributes. So either create new job class or change these jobs class to validation.
Now I understand. It looks like some new tests were added which made comparison impossible. Indeed I think that comparison is actually possible, you just need to add one more number outside of the main table that reflects the number of reports that can't be compared. By clicking that number a usual comparison of report trees with normal navigation can be shown. By showing these details one will be able to understand what is wrong. For instance, I will understand what tests are wrong or whether it is necessary to introduce new comparison attributes or even a new job class. This functionality will be helpful further as well.
- Tracker changed from Bug to Feature
- Priority changed from Urgent to High
This isn't a bug, but we can improve comparison so that jobs like attached will be compared (indeed all jobs of the same class will be comparable).
- Priority changed from High to Urgent
- Target version set to 1.0
Because of this feature is missed I can't compare more and more jobs. You can download two normal jobs for which comparison also does not work (1 and 2)
- Due date set to 10/12/2017
- Status changed from Open to Resolved
- % Done changed from 0 to 100
The problem was solved in branch "feature_8374".
- Status changed from Resolved to Closed
I merged the branch to master in 7f4fc2f9.
Don't forget to migrate your databases after update.
Also available in: Atom
PDF