Bug #7231
closedUploading reports takes too much time
0%
Description
Uploading full reports for all Linux kernel modules (takes 600-700Mb) may take a few hours, while in LDV Tools such reports could be uploaded in a few minutes.
Updated by Evgeny Novikov over 8 years ago
- Category set to Bridge
- Assignee changed from Evgeny Novikov to Vladimir Gratinskiy
You didn't mention that LDV Tools required extremely much time for visualization of any quite large data sets and were very-very limited in supported data representation.
Nevertheless I am sure that some optimizations for uploading large job archives can be done, so that uploading time will be decreased several times.
As a temporary workaround I recommend to use the production mode rather than the developer one for all large jobs. This will considerably reduce log file sizes, the total size of job archives and uploading time.
Updated by Vitaly Mordan over 8 years ago
Suggestion: Since the majority of reports are safe verdicts (for example, for some task there are 77057 safes, 43 unsafes and 5572 unknowns), maybe we should not add verifier log to them, especially if we can access the log from the verifier component and most likely the user does not need them. Also in LDV Tools we added only N (which can be specified by the user) last lines of log to the report instead of the whole log.
Updated by Evgeny Novikov over 8 years ago
For your first suggestion there are already opened issues: #6597 and #6598. Anyway these logs should have the way to get into the database - corresponding issues suggest to do this by default in the development mode.
Your second suggestion requires some analysis. I don't understand a good default value of N. Should it depend on verifier (its version/configuration)? It should be enough to catch verifier issues that can be somewhere in the middle of the log files. I guess that more reasonable is to cut off various debug and statistics information if the user asked so at the job start decision page (this can be done by default in the production mode). Anyway please open another issue for this, since this issue should be just for uploading optimizations in Bridge.
Updated by Vitaly Mordan over 8 years ago
Feedback: Using production mode helps to significantly reduce size of an archive (in around 5 times), which also reduces time of its downloading.
Updated by Vladimir Gratinskiy over 8 years ago
- Status changed from New to Feedback
Maybe fixed in branch "bridge-optimizations". For small jobs (validation) the uploading time was decreased twice. I hope for big jobs it will be faster. I can't test it with huge jobs.
Updated by Evgeny Novikov over 8 years ago
- Status changed from Feedback to Closed
- Published in build set to 1127662