Feature #8086
closedRemove all files when it becomes clear that they won't be used anymore
Added by Evgeny Novikov about 7 years ago. Updated about 6 years ago.
0%
Description
This is already done partially but this isn't enough. For instance, report files are remained even after they are uploaded. After all this can require ~100 GB when checking one rule for all Linux kernel modules. Just Core and its components are subjected to this issue.
Updated by Evgeny Novikov about 7 years ago
- Assignee set to Ilja Zakharov
- Priority changed from High to Urgent
Let's do it together with #6608.
Updated by Evgeny Novikov over 6 years ago
- Target version deleted (
2.0)
This isn't actually important for Klever 1.0.
Updated by Evgeny Novikov over 6 years ago
- Assignee deleted (
Ilja Zakharov) - Priority changed from Urgent to High
It has sense to do this after Klever 1.0 will be released since Core will be considerably changed.
Updated by Ilja Zakharov over 6 years ago
- Category set to Infrastructure of Core
- Assignee set to Ilja Zakharov
- Priority changed from High to Urgent
- Target version set to 2.0
Updated by Ilja Zakharov over 6 years ago
- Assignee changed from Ilja Zakharov to Alexey Polushkin
Updated by Alexey Polushkin over 6 years ago
- Status changed from New to Resolved
Resolved in branch 'clean_dir2'
Updated by Ilja Zakharov over 6 years ago
- Status changed from Resolved to Open
Something is wrong here. Short jobs work nicely but verification of all modules hangs sometimes and sometimes leads to LKVOG crashes. A deeper investigation is needed as I am not sure about the reasons.
Updated by Evgeny Novikov over 6 years ago
- Category deleted (
Infrastructure of Core)
I believe this issue spreads over many different Core components, so, it isn't necessary to specify the particular category.
Updated by Ilja Zakharov over 6 years ago
- Related to Bug #8602: OSError: [Errno 39] Directory not empty added
Updated by Ilja Zakharov over 6 years ago
- Related to deleted (Bug #8602: OSError: [Errno 39] Directory not empty)
Updated by Ilja Zakharov over 6 years ago
- Blocks Bug #8602: OSError: [Errno 39] Directory not empty added
Updated by Evgeny Novikov over 6 years ago
- Blocks deleted (Bug #8602: OSError: [Errno 39] Directory not empty)
Updated by Evgeny Novikov over 6 years ago
- Related to Bug #8602: OSError: [Errno 39] Directory not empty added
Updated by Evgeny Novikov over 6 years ago
BTW, in #8602 I noticed that Core can't reliably collect total code coverage with all details, i.e. with setting "coverage": "full" in rule specs.json, because of VTG removes directories containing files needed by coverage reports.
Updated by Ilja Zakharov over 6 years ago
- Status changed from Open to Resolved
Implemented in clean_dir2.
In addition to files removal implemented by Alexey, I added fixes to avoid corrupting results due to incorrect work of VTG.
Updated by Evgeny Novikov about 6 years ago
- Target version changed from 2.0 to 1.0
I guess that everybody will be happy to have this feature implemented faster than Klever 1.0. Unfortunately, although there was quite much testing, I still expect some issues especially very tricky ones.
Updated by Ilja Zakharov about 6 years ago
- Blocks Feature #8084: Add marks for testing and validation sets added
Updated by Ilja Zakharov about 6 years ago
- Status changed from Resolved to Open
A bug found when full coverage is collected:
[JCR] Raise exception: Traceback (most recent call last): File "/home/debian/klever/core/core/components.py", line 425, in run self.main() File "/home/debian/klever/core/core/components.py", line 329, in callbacks_caller ret = attr(*args, **kwargs) File "/home/debian/klever/core/core/job.py", line 573, in collect_total_coverage os.path.join('total coverages', re.sub(r'/', '-', job_id))) File "/home/debian/klever/core/core/utils.py", line 469, in report elem.make_archive(directory=os.path.join(main_work_dir, 'reports'), prefix='{0}-'.format(cur_report_id)) File "/home/debian/klever/core/core/utils.py", line 411, in make_archive zfp.write(file, arcname=arcname) File "/usr/lib/python3.5/zipfile.py", line 1435, in write st = os.stat(filename) FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/klever/work/native-scheduler/scheduler/jobs/2f8576ac86eb8bfc10ab5ab168837cd8/klever-core-work-dir/job/lkbce/net/xfrm/xfrm_user.c' Traceback (most recent call last): File "/home/debian/klever/core/core/core.py", line 87, in main 'coverage_finished': multiprocessing.Manager().dict()}) File "/home/debian/klever/core/core/job.py", line 123, in start_jobs subcomponent.join() File "/home/debian/klever/core/core/components.py", line 329, in callbacks_caller ret = attr(*args, **kwargs) File "/home/debian/klever/core/core/components.py", line 543, in join raise ComponentError('Component "{0}" failed'.format(self.name)) core.components.ComponentError: Component "JCR" failed
Job archive (2 Gb because of coverage): https://getbox.ispras.ru/index.php/s/xpcUpFXaBrTQZx3
Updated by Ilja Zakharov about 6 years ago
Another job with the bug with the coverage collection option set to the default value: https://getbox.ispras.ru/index.php/s/G1ZfrpGiMv5HIIW
Updated by Ilja Zakharov about 6 years ago
- Status changed from Open to Resolved
Updated by Evgeny Novikov about 6 years ago
- Status changed from Resolved to Closed
At last we have this vital functionality in master since 6d71b32.