Project

General

Profile

Feature #8086

Remove all files when it becomes clear that they won't be used anymore

Added by Evgeny Novikov about 2 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Urgent
Category:
-
Target version:
Start date:
04/18/2017
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

This is already done partially but this isn't enough. For instance, report files are remained even after they are uploaded. After all this can require ~100 GB when checking one rule for all Linux kernel modules. Just Core and its components are subjected to this issue.


Related issues

Related to Klever - Feature #8272: Use good compression when creating zip-archivesClosed07/04/2017

Actions
Related to Klever - Bug #8602: OSError: [Errno 39] Directory not emptyClosed11/28/2017

Actions
Blocks Klever - Feature #8120: Execute intensive disk operations using RAMNew04/21/2017

Actions
Blocks Klever - Feature #8084: Add marks for testing and validation setsClosed04/18/2017

Actions

History

#1

Updated by Evgeny Novikov almost 2 years ago

  • Assignee set to Ilja Zakharov
  • Priority changed from High to Urgent

Let's do it together with #6608.

#2

Updated by Evgeny Novikov over 1 year ago

  • Target version set to 2.0
#3

Updated by Evgeny Novikov over 1 year ago

  • Target version deleted (2.0)

This isn't actually important for Klever 1.0.

#4

Updated by Evgeny Novikov over 1 year ago

  • Assignee deleted (Ilja Zakharov)
  • Priority changed from Urgent to High

It has sense to do this after Klever 1.0 will be released since Core will be considerably changed.

#5

Updated by Ilja Zakharov over 1 year ago

  • Category set to Infrastructure of Core
  • Assignee set to Ilja Zakharov
  • Priority changed from High to Urgent
  • Target version set to 2.0
#6

Updated by Ilja Zakharov over 1 year ago

  • Assignee changed from Ilja Zakharov to Alexey Polushkin
#7

Updated by Alexey Polushkin over 1 year ago

  • Status changed from New to Resolved

Resolved in branch 'clean_dir2'

#8

Updated by Ilja Zakharov over 1 year ago

  • Status changed from Resolved to Open

Something is wrong here. Short jobs work nicely but verification of all modules hangs sometimes and sometimes leads to LKVOG crashes. A deeper investigation is needed as I am not sure about the reasons.

#9

Updated by Evgeny Novikov over 1 year ago

  • Category deleted (Infrastructure of Core)

I believe this issue spreads over many different Core components, so, it isn't necessary to specify the particular category.

#10

Updated by Ilja Zakharov over 1 year ago

  • Related to Bug #8602: OSError: [Errno 39] Directory not empty added
#11

Updated by Ilja Zakharov over 1 year ago

  • Related to deleted (Bug #8602: OSError: [Errno 39] Directory not empty)
#12

Updated by Ilja Zakharov over 1 year ago

  • Blocks Bug #8602: OSError: [Errno 39] Directory not empty added
#13

Updated by Evgeny Novikov over 1 year ago

  • Blocks deleted (Bug #8602: OSError: [Errno 39] Directory not empty)
#14

Updated by Evgeny Novikov over 1 year ago

  • Related to Bug #8602: OSError: [Errno 39] Directory not empty added
#15

Updated by Evgeny Novikov over 1 year ago

BTW, in #8602 I noticed that Core can't reliably collect total code coverage with all details, i.e. with setting "coverage": "full" in rule specs.json, because of VTG removes directories containing files needed by coverage reports.

#16

Updated by Ilja Zakharov over 1 year ago

  • Status changed from Open to Resolved

Implemented in clean_dir2.

In addition to files removal implemented by Alexey, I added fixes to avoid corrupting results due to incorrect work of VTG.

#17

Updated by Evgeny Novikov about 1 year ago

  • Target version changed from 2.0 to 1.0

I guess that everybody will be happy to have this feature implemented faster than Klever 1.0. Unfortunately, although there was quite much testing, I still expect some issues especially very tricky ones.

#18

Updated by Ilja Zakharov about 1 year ago

  • Blocks Feature #8084: Add marks for testing and validation sets added
#19

Updated by Ilja Zakharov about 1 year ago

  • Status changed from Resolved to Open

A bug found when full coverage is collected:

[JCR] Raise exception: Traceback (most recent call last): File "/home/debian/klever/core/core/components.py", line 425, in run self.main() File "/home/debian/klever/core/core/components.py", line 329, in callbacks_caller ret = attr(*args, **kwargs) File "/home/debian/klever/core/core/job.py", line 573, in collect_total_coverage os.path.join('total coverages', re.sub(r'/', '-', job_id))) File "/home/debian/klever/core/core/utils.py", line 469, in report elem.make_archive(directory=os.path.join(main_work_dir, 'reports'), prefix='{0}-'.format(cur_report_id)) File "/home/debian/klever/core/core/utils.py", line 411, in make_archive zfp.write(file, arcname=arcname) File "/usr/lib/python3.5/zipfile.py", line 1435, in write st = os.stat(filename) FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/klever/work/native-scheduler/scheduler/jobs/2f8576ac86eb8bfc10ab5ab168837cd8/klever-core-work-dir/job/lkbce/net/xfrm/xfrm_user.c' Traceback (most recent call last): File "/home/debian/klever/core/core/core.py", line 87, in main 'coverage_finished': multiprocessing.Manager().dict()}) File "/home/debian/klever/core/core/job.py", line 123, in start_jobs subcomponent.join() File "/home/debian/klever/core/core/components.py", line 329, in callbacks_caller ret = attr(*args, **kwargs) File "/home/debian/klever/core/core/components.py", line 543, in join raise ComponentError('Component "{0}" failed'.format(self.name)) core.components.ComponentError: Component "JCR" failed

Job archive (2 Gb because of coverage): https://getbox.ispras.ru/index.php/s/xpcUpFXaBrTQZx3

#20

Updated by Ilja Zakharov about 1 year ago

Another job with the bug with the coverage collection option set to the default value: https://getbox.ispras.ru/index.php/s/G1ZfrpGiMv5HIIW

#21

Updated by Ilja Zakharov about 1 year ago

  • Status changed from Open to Resolved
#22

Updated by Evgeny Novikov about 1 year ago

  • Status changed from Resolved to Closed

At last we have this vital functionality in master since 6d71b32.

Also available in: Atom PDF