Actions
Feature #562
openStats server has to provide us with problems comparison
Start date:
11/16/2010
Due date:
% Done:
0%
Estimated time:
Published in build:
Description
I have thought about this at the beginning of stats comparison implementation but then it wasn't the task of high priority. From now it'll be very useful because of this can show the progress of our tool development more precisely. For example how time limits becomes memory limits and vise versa and so on.
The technical aspect is to extend comparison with conversions like P1->P2 where P1 is some first kind problem from a set of problems of all tools, P2 is some second kind problem of a given set. These conversions belongs to the general Unknown->Unknown conversion and show/hide must be available for them (by default conversions with different problems are shown, with the same problem are hide).
The issue comes from middle regression testing. After adding of problems patterns for build command extractor and some other changes the comparison with diff shows 66 real changes while the stats server shows just 3 changes. This is because 3 changes belong to the Unknown->Safe conversion while 63 changes belongs to the general Unknown->Unknown conversion that has subclasses:
The technical aspect is to extend comparison with conversions like P1->P2 where P1 is some first kind problem from a set of problems of all tools, P2 is some second kind problem of a given set. These conversions belongs to the general Unknown->Unknown conversion and show/hide must be available for them (by default conversions with different problems are shown, with the same problem are hide).
The issue comes from middle regression testing. After adding of problems patterns for build command extractor and some other changes the comparison with diff shows 66 real changes while the stats server shows just 3 changes. This is because 3 changes belong to the Unknown->Safe conversion while 63 changes belongs to the general Unknown->Unknown conversion that has subclasses:
- Time_limit->Out_of_memory
- ''->Compile error ('' means unmatched problem)
- ''->No Makefile or Kbuild file
- Exception,Parsing errors->Exception
- Exception,Parsing errors->Out_of_memory
Updated by Evgeny Novikov almost 14 years ago
- Priority changed from Normal to High
I think that this should be done earlier. Today night tests again contains conversion between unknown and unknown and we can't catch it effectively by means of stats server:
Out_of_memory -> Exception
Updated by Evgeny Novikov almost 14 years ago
- Status changed from New to Open
Mark 2 QA tasks that contains the same issue (even don't know how problems changed indeed).
Updated by Pavel Shved almost 14 years ago
- Priority changed from High to Normal
Well, for now that's not a big deal...
Updated by Evgeny Novikov over 13 years ago
This is especially useful in comparing different models like 32_1 and 32_2. I believe that we'll have a lot of different models soon so this should be done.
Also it will be very nice if stats server will print links to error traces directly in comparison tables.
Updated by Alexey Khoroshilov over 13 years ago
- Priority changed from Normal to High
Actions