Bug #8602: OSError: [Errno 39] Directory not empty - Klever - Open-Source Projects

Actions

Copy link

#1

Updated by Ilja Zakharov over 6 years ago

Status changed from New to Resolved

Implemented in 8602-del-dirs.

Actions

Copy link

#2

Updated by Evgeny Novikov over 6 years ago

Status changed from Resolved to Closed

I hope that this issue has gone and merged the branch to master in f3b0b4de.

Actions

Copy link

#3

Updated by Evgeny Novikov over 6 years ago

Status changed from Closed to Open

Unfortunately the issue still exists:

[AAVTDG0] Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/vtg/__init__.py", line 540, in __generate_all_abstract_verification_task_descs
    shutil.rmtree(deldir)
  File "/usr/lib/python3.5/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'drivers/hwmon/pmbus/pmbus_core.ko/linux:arch:io'
[VTGWL1] Stop since some other component(s) likely failed
Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/vtg/__init__.py", line 338, in generate_verification_tasks
    self.launch_subcomponents(False, *subcomponents)
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 556, in launch_subcomponents
    launch_workers(self.logger, subcomponent_processes)
  File "/home/debian/klever/core/core/components.py", line 196, in launch_workers
    p.join(1.0 / len(workers))
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 521, in join
    raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "AAVTDG0" failed

Actions

Copy link

#4

Updated by Ilja Zakharov over 6 years ago

Another iteration of fixing: 8602-del-dirs.

Tests are still running and will finish tomorrow evening. However, I still was not able to reproduce the error occured after the last fix even solving jobs with solution time more than 20 hours.

Actions

Copy link

#5

Updated by Alexey Khoroshilov over 6 years ago

Ilja Zakharov wrote:

Another iteration of fixing: 8602-del-dirs.

Tests are still running and will finish tomorrow evening. However, I still was not able to reproduce the error occured after the last fix even solving jobs with solution time more than 20 hours.

After 10 hours I have got:
OSError: [Errno 39] Directory not empty: 'net/sctp/sctp_probe.ko/linux:drivers:clk2'
while similar tasks on another machine still works for 16 hours.
Klever 0.3.dev211+gf3b0b4de

Actions

Copy link

#6

Updated by Ilja Zakharov over 6 years ago

There are two reasons that lead to either corruption or failing runs. In particular the recent my change allows to avoid failures but do not prevent corruption status but with 100% tasks solved.

The first reason is incorrect work of general coverage collector:
[JCR] Raise exception:
Traceback (most recent call last):
File "/home/debian/klever/core/core/components.py", line 413, in run
self.main()
File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
ret = attr(*args, **kwargs)
File "/home/debian/klever/core/core/job.py", line 557, in collect_total_coverage
os.path.join('total coverages', re.sub(r'/', '-', job_id)))
File "/home/debian/klever/core/core/utils.py", line 463, in report
elem.make_archive(directory=os.path.join(main_work_dir, 'reports'), prefix='{0}-'.format(cur_report_id))
File "/home/debian/klever/core/core/utils.py", line 408, in make_archive
zfp.write(file, arcname=arcname)
File "/usr/lib/python3.5/zipfile.py", line 1435, in write
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/klever/work/native-scheduler/scheduler/jobs/65f7a8ee6dd5504699daf163e2a38fd8/klever-core-work-dir/job/vtg/drivers/tty/rocket.ko76d0b8a4c5c8/generic:memory/weaver/.tmp_timer.c.aux'
Traceback (most recent call last):
File "/home/debian/klever/core/core/core.py", line 81, in main
core.job.start_jobs(self, {'build': multiprocessing.Manager().Lock()}, {'report id': self.report_id})
File "/home/debian/klever/core/core/job.py", line 121, in start_jobs
subcomponent.join()
File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
ret = attr(*args, **kwargs)
File "/home/debian/klever/core/core/components.py", line 521, in join
raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "JCR" failed

The second is report loading failure:
[08.Dec.2017 22:51:43] The archive "108912-vjmbmeh6.zip" of report "/-/VTG/drivers/usb/serial/metro-usb.ko/linux:drivers:clk2/VTGW/Weaver" is not a ZIP file
Traceback (most recent call last):
File "/var/www/klever-bridge/reports/UploadReport.py", line 66, in init
self.__check_archives(self.data['id'])
File "/var/www/klever-bridge/reports/UploadReport.py", line 727, in __check_archives
raise CheckArchiveError('The archive "%s" of report "%s" is not a ZIP file' % (arch.name, report_id))
reports.UploadReport.CheckArchiveError: The archive "108912-vjmbmeh6.zip" of report "/-/VTG/drivers/usb/serial/metro-usb.ko/linux:drivers:clk2/VTGW/Weaver" is not a ZIP file

As a summary, it is better to fix both issues in #8086, since it also modifies the implementation of general coverage collection and improves deletion of files.

Actions

Copy link

#7