Feature #6710
closedThink on file compression
Added by Evgeny Novikov almost 9 years ago. Updated over 8 years ago.
100%
Description
Bridge keeps a lot of files (logs, specifications, etc.) that can be efficiently (de)compressed on the fly to save very much disk space.
Updated by Evgeny Novikov over 8 years ago
- Priority changed from Normal to High
Uncompressed logs can occupy too much space, e.g. for a couple of quite large jobs they need ~15 GB of disk space.
Updated by Vladimir Gratinskiy over 8 years ago
Implemented in branch "bridge-optimizations". All reports' files are compressed.
Updated by Vladimir Gratinskiy over 8 years ago
- Due date set to 06/16/2016
- Status changed from New to Resolved
- % Done changed from 0 to 100
Updated by Evgeny Novikov over 8 years ago
- Status changed from Resolved to Open
After corresponding migrations files referred by both existing and new error traces can't be found.
Updated by Vladimir Gratinskiy over 8 years ago
- Status changed from Open to Feedback
Evgeny Novikov wrote:
After corresponding migrations files referred by both existing and new error traces can't be found.
The problem was with '/' at the start of files' names again. It was fixed later in this branch. Now it has more migrations, but I didn't test them.
Updated by Evgeny Novikov over 8 years ago
- Status changed from Feedback to Open
I tested this and confirm that everything works fine now.
The only thing I would like to have is migrations that compress files also create a file with a list of these files to remove them manually fast later if everything will be okay.
BTW, is it correct: if I migrated to compressed files there is no more way to return back to uncompressed files?
Updated by Vladimir Gratinskiy over 8 years ago
Evgeny Novikov wrote:
I tested this and confirm that everything works fine now.
The only thing I would like to have is migrations that compress files also create a file with a list of these files to remove them manually fast later if everything will be okay.
BTW, is it correct: if I migrated to compressed files there is no more way to return back to uncompressed files?
Yes, it is correct.
Updated by Evgeny Novikov over 8 years ago
- Status changed from Open to Closed
- Published in build set to 1127662
The branch containing implementation of this feature and many other optimizations was finally merged to master in 1127662. I will put all comments here and just close other issues pointing to this one.
The easiest way to get these changes to local developer databases is to re-create them from scratch. Just very important data can be migrated with my and Vladimir help.
It is noticeable that we started to support Django migrations completely. So from this version everybody will be likely able to migrate all existing data without any troubles and nobody will need to re-create databases. In particular this assumes that nobody needs to manually create migrations by using manage.py makemigrations ... - migrations are placed into the repository and will be created together with corresponding changes if required. Users will just need to run manage.py migrate and perhaps to answer some questions. I updated documentation respectively.
Since this branch brought quite considerable changes in storing and retrieving quite large data it has sense to evaluate time consumption of various operations (downloading/uploading jobs/marks, sending reports, etc.) again and report new issues with new descriptions if so.