Bug #7231
closed
Uploading reports takes too much time
Added by Vitaly Mordan over 8 years ago.
Updated over 8 years ago.
Published in build:
1127662
Description
Uploading full reports for all Linux kernel modules (takes 600-700Mb) may take a few hours, while in LDV Tools such reports could be uploaded in a few minutes.
- Description updated (diff)
- Category set to Bridge
- Assignee changed from Evgeny Novikov to Vladimir Gratinskiy
You didn't mention that LDV Tools required extremely much time for visualization of any quite large data sets and were very-very limited in supported data representation.
Nevertheless I am sure that some optimizations for uploading large job archives can be done, so that uploading time will be decreased several times.
As a temporary workaround I recommend to use the production mode rather than the developer one for all large jobs. This will considerably reduce log file sizes, the total size of job archives and uploading time.
Suggestion: Since the majority of reports are safe verdicts (for example, for some task there are 77057 safes, 43 unsafes and 5572 unknowns), maybe we should not add verifier log to them, especially if we can access the log from the verifier component and most likely the user does not need them. Also in LDV Tools we added only N (which can be specified by the user) last lines of log to the report instead of the whole log.
For your first suggestion there are already opened issues: #6597 and #6598. Anyway these logs should have the way to get into the database - corresponding issues suggest to do this by default in the development mode.
Your second suggestion requires some analysis. I don't understand a good default value of N. Should it depend on verifier (its version/configuration)? It should be enough to catch verifier issues that can be somewhere in the middle of the log files. I guess that more reasonable is to cut off various debug and statistics information if the user asked so at the job start decision page (this can be done by default in the production mode). Anyway please open another issue for this, since this issue should be just for uploading optimizations in Bridge.
Feedback: Using production mode helps to significantly reduce size of an archive (in around 5 times), which also reduces time of its downloading.
- Status changed from New to Feedback
Maybe fixed in branch "bridge-optimizations". For small jobs (validation) the uploading time was decreased twice. I hope for big jobs it will be faster. I can't test it with huge jobs.
- Status changed from Feedback to Closed
- Published in build set to 1127662
The branch was merged to master in 1127662. Please, see #6710 for details.
Also available in: Atom
PDF