Project

General

Profile

Actions

Feature #562

open

Stats server has to provide us with problems comparison

Added by Evgeny Novikov over 13 years ago. Updated over 12 years ago.

Status:
Open
Priority:
Normal
Category:
Statistics server
Start date:
11/16/2010
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

I have thought about this at the beginning of stats comparison implementation but then it wasn't the task of high priority. From now it'll be very useful because of this can show the progress of our tool development more precisely. For example how time limits becomes memory limits and vise versa and so on.
The technical aspect is to extend comparison with conversions like P1->P2 where P1 is some first kind problem from a set of problems of all tools, P2 is some second kind problem of a given set. These conversions belongs to the general Unknown->Unknown conversion and show/hide must be available for them (by default conversions with different problems are shown, with the same problem are hide).
The issue comes from middle regression testing. After adding of problems patterns for build command extractor and some other changes the comparison with diff shows 66 real changes while the stats server shows just 3 changes. This is because 3 changes belong to the Unknown->Safe conversion while 63 changes belongs to the general Unknown->Unknown conversion that has subclasses:
  • Time_limit->Out_of_memory
  • ''->Compile error ('' means unmatched problem)
  • ''->No Makefile or Kbuild file
  • Exception,Parsing errors->Exception
  • Exception,Parsing errors->Out_of_memory

Related issues 1 (1 open0 closed)

Related to Linux Driver Verification - Bug #548: Information about problematic launches can't make its way to results databaseNew11/15/2010

Actions
Actions #1

Updated by Evgeny Novikov over 13 years ago

  • Priority changed from Normal to High

I think that this should be done earlier. Today night tests again contains conversion between unknown and unknown and we can't catch it effectively by means of stats server:
Out_of_memory -> Exception

Actions #2

Updated by Evgeny Novikov over 13 years ago

  • Status changed from New to Open

Mark 2 QA tasks that contains the same issue (even don't know how problems changed indeed).

Actions #3

Updated by Pavel Shved over 13 years ago

  • Priority changed from High to Normal

Well, for now that's not a big deal...

Actions #4

Updated by Evgeny Novikov about 13 years ago

This is especially useful in comparing different models like 32_1 and 32_2. I believe that we'll have a lot of different models soon so this should be done.
Also it will be very nice if stats server will print links to error traces directly in comparison tables.

Actions #5

Updated by Alexey Khoroshilov about 13 years ago

  • Priority changed from Normal to High
Actions #6

Updated by Pavel Shved over 12 years ago

  • Priority changed from High to Normal
Actions

Also available in: Atom PDF