Project

General

Profile

Bug #8602

OSError: [Errno 39] Directory not empty

Added by Evgeny Novikov about 1 year ago. Updated 12 months ago.

Status:
Closed
Priority:
Immediate
Assignee:
Category:
Tasks generation
Target version:
Start date:
11/28/2017
Due date:
% Done:

0%

Estimated time:
Detected in build:
svn
Platform:
Published in build:

Description

New VTG removes plugin working directories when they finish in the production mode, namely, when option "keep intermediate files" is false. This helps to considerably reduce the required disk space, but sometimes fails with errors like:

[AAVTDG0] Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/vtg/__init__.py", line 525, in __generate_all_abstract_verification_task_descs
    shutil.rmtree(os.path.join(vobject, rule))
  File "/usr/lib/python3.5/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'drivers//char/tpm/tpm_tis_spi.ko/generic:memory'
[VTGWL1] Stop since some other component(s) likely failed
Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/vtg/__init__.py", line 336, in generate_verification_tasks
    self.launch_subcomponents(False, *subcomponents)
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 556, in launch_subcomponents
    launch_workers(self.logger, subcomponent_processes)
  File "/home/debian/klever/core/core/components.py", line 196, in launch_workers
    p.join(1.0 / len(workers))
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 521, in join
    raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "AAVTDG0" failed

It seems that all quite large verification jobs suffer from this issue, so it should be fixed ASAP.


Related issues

Related to Klever - Feature #8086: Remove all files when it becomes clear that they won't be used anymoreClosed2017-04-18

History

#1 Updated by Ilja Zakharov about 1 year ago

  • Status changed from New to Resolved

Implemented in 8602-del-dirs.

#2 Updated by Evgeny Novikov about 1 year ago

  • Status changed from Resolved to Closed

I hope that this issue has gone and merged the branch to master in f3b0b4de.

#3 Updated by Evgeny Novikov about 1 year ago

  • Status changed from Closed to Open

Unfortunately the issue still exists:

[AAVTDG0] Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/vtg/__init__.py", line 540, in __generate_all_abstract_verification_task_descs
    shutil.rmtree(deldir)
  File "/usr/lib/python3.5/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'drivers/hwmon/pmbus/pmbus_core.ko/linux:arch:io'
[VTGWL1] Stop since some other component(s) likely failed
Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/vtg/__init__.py", line 338, in generate_verification_tasks
    self.launch_subcomponents(False, *subcomponents)
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 556, in launch_subcomponents
    launch_workers(self.logger, subcomponent_processes)
  File "/home/debian/klever/core/core/components.py", line 196, in launch_workers
    p.join(1.0 / len(workers))
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 521, in join
    raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "AAVTDG0" failed

#4 Updated by Ilja Zakharov about 1 year ago

Another iteration of fixing: 8602-del-dirs.

Tests are still running and will finish tomorrow evening. However, I still was not able to reproduce the error occured after the last fix even solving jobs with solution time more than 20 hours.

#5 Updated by Alexey Khoroshilov about 1 year ago

Ilja Zakharov wrote:

Another iteration of fixing: 8602-del-dirs.

Tests are still running and will finish tomorrow evening. However, I still was not able to reproduce the error occured after the last fix even solving jobs with solution time more than 20 hours.

After 10 hours I have got:
OSError: [Errno 39] Directory not empty: 'net/sctp/sctp_probe.ko/linux:drivers:clk2'
while similar tasks on another machine still works for 16 hours.
Klever 0.3.dev211+gf3b0b4de

#6 Updated by Ilja Zakharov 12 months ago

There are two reasons that lead to either corruption or failing runs. In particular the recent my change allows to avoid failures but do not prevent corruption status but with 100% tasks solved.

The first reason is incorrect work of general coverage collector:
[JCR] Raise exception:
Traceback (most recent call last):
File "/home/debian/klever/core/core/components.py", line 413, in run
self.main()
File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
ret = attr(*args, **kwargs)
File "/home/debian/klever/core/core/job.py", line 557, in collect_total_coverage
os.path.join('total coverages', re.sub(r'/', '-', job_id)))
File "/home/debian/klever/core/core/utils.py", line 463, in report
elem.make_archive(directory=os.path.join(main_work_dir, 'reports'), prefix='{0}-'.format(cur_report_id))
File "/home/debian/klever/core/core/utils.py", line 408, in make_archive
zfp.write(file, arcname=arcname)
File "/usr/lib/python3.5/zipfile.py", line 1435, in write
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/klever/work/native-scheduler/scheduler/jobs/65f7a8ee6dd5504699daf163e2a38fd8/klever-core-work-dir/job/vtg/drivers/tty/rocket.ko76d0b8a4c5c8/generic:memory/weaver/.tmp_timer.c.aux'
Traceback (most recent call last):
File "/home/debian/klever/core/core/core.py", line 81, in main
core.job.start_jobs(self, {'build': multiprocessing.Manager().Lock()}, {'report id': self.report_id})
File "/home/debian/klever/core/core/job.py", line 121, in start_jobs
subcomponent.join()
File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
ret = attr(*args, **kwargs)
File "/home/debian/klever/core/core/components.py", line 521, in join
raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "JCR" failed

The second is report loading failure:
[08.Dec.2017 22:51:43] The archive "108912-vjmbmeh6.zip" of report "/-/VTG/drivers/usb/serial/metro-usb.ko/linux:drivers:clk2/VTGW/Weaver" is not a ZIP file
Traceback (most recent call last):
File "/var/www/klever-bridge/reports/UploadReport.py", line 66, in init
self.__check_archives(self.data['id'])
File "/var/www/klever-bridge/reports/UploadReport.py", line 727, in __check_archives
raise CheckArchiveError('The archive "%s" of report "%s" is not a ZIP file' % (arch.name, report_id))
reports.UploadReport.CheckArchiveError: The archive "108912-vjmbmeh6.zip" of report "/-/VTG/drivers/usb/serial/metro-usb.ko/linux:drivers:clk2/VTGW/Weaver" is not a ZIP file

As a summary, it is better to fix both issues in #8086, since it also modifies the implementation of general coverage collection and improves deletion of files.

#7 Updated by Ilja Zakharov 12 months ago

  • Related to Feature #8086: Remove all files when it becomes clear that they won't be used anymore added

#8 Updated by Ilja Zakharov 12 months ago

  • Related to deleted (Feature #8086: Remove all files when it becomes clear that they won't be used anymore)

#9 Updated by Ilja Zakharov 12 months ago

  • Blocked by Feature #8086: Remove all files when it becomes clear that they won't be used anymore added

#10 Updated by Ilja Zakharov 12 months ago

  • Status changed from Open to Resolved

There is a fix in 8602-del-dirs that should ensure jobs solution, however, a complete fix will be in #8086.

#11 Updated by Evgeny Novikov 12 months ago

  • Blocked by deleted (Feature #8086: Remove all files when it becomes clear that they won't be used anymore)

#12 Updated by Evgeny Novikov 12 months ago

  • Related to Feature #8086: Remove all files when it becomes clear that they won't be used anymore added

#13 Updated by Evgeny Novikov 12 months ago

  • Status changed from Resolved to Closed

I hope that the suggested fix will help, so I merged the branch to master in eb8dee3d.

#14 Updated by Evgeny Novikov 12 months ago

  • Status changed from Closed to Resolved

One more fix of a data race is available in branch rm-fix.

#15 Updated by Evgeny Novikov 12 months ago

  • Status changed from Resolved to Closed

I merged the branch to master in 561ccab.

Also available in: Atom PDF