Project

General

Profile

Bug #8602

OSError: [Errno 39] Directory not empty

Added by Evgeny Novikov 13 days ago. Updated about 4 hours ago.

Status:
Closed
Priority:
Immediate
Assignee:
Category:
Tasks generation
Target version:
Start date:
11/28/2017
Due date:
% Done:

0%

Estimated time:
Detected in build:
svn
Platform:
Published in build:

Description

New VTG removes plugin working directories when they finish in the production mode, namely, when option "keep intermediate files" is false. This helps to considerably reduce the required disk space, but sometimes fails with errors like:

[AAVTDG0] Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/vtg/__init__.py", line 525, in __generate_all_abstract_verification_task_descs
    shutil.rmtree(os.path.join(vobject, rule))
  File "/usr/lib/python3.5/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'drivers//char/tpm/tpm_tis_spi.ko/generic:memory'
[VTGWL1] Stop since some other component(s) likely failed
Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/vtg/__init__.py", line 336, in generate_verification_tasks
    self.launch_subcomponents(False, *subcomponents)
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 556, in launch_subcomponents
    launch_workers(self.logger, subcomponent_processes)
  File "/home/debian/klever/core/core/components.py", line 196, in launch_workers
    p.join(1.0 / len(workers))
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 521, in join
    raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "AAVTDG0" failed

It seems that all quite large verification jobs suffer from this issue, so it should be fixed ASAP.


Related issues

Related to Klever - Feature #8086: Remove all files when it becomes clear that they won't be used anymoreOpen2017-04-18

History

#1 Updated by Ilja Zakharov 11 days ago

  • Status changed from New to Resolved

Implemented in 8602-del-dirs.

#2 Updated by Evgeny Novikov 8 days ago

  • Status changed from Resolved to Closed

I hope that this issue has gone and merged the branch to master in commit:f3b0b4de.

#3 Updated by Evgeny Novikov 5 days ago

  • Status changed from Closed to Open

Unfortunately the issue still exists:

[AAVTDG0] Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/vtg/__init__.py", line 540, in __generate_all_abstract_verification_task_descs
    shutil.rmtree(deldir)
  File "/usr/lib/python3.5/shutil.py", line 484, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 482, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: 'drivers/hwmon/pmbus/pmbus_core.ko/linux:arch:io'
[VTGWL1] Stop since some other component(s) likely failed
Raise exception:
Traceback (most recent call last):
  File "/home/debian/klever/core/core/components.py", line 413, in run
    self.main()
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/vtg/__init__.py", line 338, in generate_verification_tasks
    self.launch_subcomponents(False, *subcomponents)
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 556, in launch_subcomponents
    launch_workers(self.logger, subcomponent_processes)
  File "/home/debian/klever/core/core/components.py", line 196, in launch_workers
    p.join(1.0 / len(workers))
  File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
    ret = attr(*args, **kwargs)
  File "/home/debian/klever/core/core/components.py", line 521, in join
    raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "AAVTDG0" failed

#4 Updated by Ilja Zakharov 4 days ago

Another iteration of fixing: 8602-del-dirs.

Tests are still running and will finish tomorrow evening. However, I still was not able to reproduce the error occured after the last fix even solving jobs with solution time more than 20 hours.

#5 Updated by Alexey Khoroshilov 4 days ago

Ilja Zakharov wrote:

Another iteration of fixing: 8602-del-dirs.

Tests are still running and will finish tomorrow evening. However, I still was not able to reproduce the error occured after the last fix even solving jobs with solution time more than 20 hours.

After 10 hours I have got:
OSError: [Errno 39] Directory not empty: 'net/sctp/sctp_probe.ko/linux:drivers:clk2'
while similar tasks on another machine still works for 16 hours.
Klever 0.3.dev211+gf3b0b4de

#6 Updated by Ilja Zakharov about 7 hours ago

There are two reasons that lead to either corruption or failing runs. In particular the recent my change allows to avoid failures but do not prevent corruption status but with 100% tasks solved.

The first reason is incorrect work of general coverage collector:
[JCR] Raise exception:
Traceback (most recent call last):
File "/home/debian/klever/core/core/components.py", line 413, in run
self.main()
File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
ret = attr(*args, **kwargs)
File "/home/debian/klever/core/core/job.py", line 557, in collect_total_coverage
os.path.join('total coverages', re.sub(r'/', '-', job_id)))
File "/home/debian/klever/core/core/utils.py", line 463, in report
elem.make_archive(directory=os.path.join(main_work_dir, 'reports'), prefix='{0}-'.format(cur_report_id))
File "/home/debian/klever/core/core/utils.py", line 408, in make_archive
zfp.write(file, arcname=arcname)
File "/usr/lib/python3.5/zipfile.py", line 1435, in write
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/klever/work/native-scheduler/scheduler/jobs/65f7a8ee6dd5504699daf163e2a38fd8/klever-core-work-dir/job/vtg/drivers/tty/rocket.ko76d0b8a4c5c8/generic:memory/weaver/.tmp_timer.c.aux'
Traceback (most recent call last):
File "/home/debian/klever/core/core/core.py", line 81, in main
core.job.start_jobs(self, {'build': multiprocessing.Manager().Lock()}, {'report id': self.report_id})
File "/home/debian/klever/core/core/job.py", line 121, in start_jobs
subcomponent.join()
File "/home/debian/klever/core/core/components.py", line 320, in callbacks_caller
ret = attr(*args, **kwargs)
File "/home/debian/klever/core/core/components.py", line 521, in join
raise ComponentError('Component "{0}" failed'.format(self.name))
core.components.ComponentError: Component "JCR" failed

The second is report loading failure:
[08.Dec.2017 22:51:43] The archive "108912-vjmbmeh6.zip" of report "/-/VTG/drivers/usb/serial/metro-usb.ko/linux:drivers:clk2/VTGW/Weaver" is not a ZIP file
Traceback (most recent call last):
File "/var/www/klever-bridge/reports/UploadReport.py", line 66, in init
self.__check_archives(self.data['id'])
File "/var/www/klever-bridge/reports/UploadReport.py", line 727, in __check_archives
raise CheckArchiveError('The archive "%s" of report "%s" is not a ZIP file' % (arch.name, report_id))
reports.UploadReport.CheckArchiveError: The archive "108912-vjmbmeh6.zip" of report "/-/VTG/drivers/usb/serial/metro-usb.ko/linux:drivers:clk2/VTGW/Weaver" is not a ZIP file

As a summary, it is better to fix both issues in #8086, since it also modifies the implementation of general coverage collection and improves deletion of files.

#7 Updated by Ilja Zakharov about 7 hours ago

  • Related to Feature #8086: Remove all files when it becomes clear that they won't be used anymore added

#8 Updated by Ilja Zakharov about 6 hours ago

  • Related to deleted (Feature #8086: Remove all files when it becomes clear that they won't be used anymore)

#9 Updated by Ilja Zakharov about 6 hours ago

  • Blocked by Feature #8086: Remove all files when it becomes clear that they won't be used anymore added

#10 Updated by Ilja Zakharov about 5 hours ago

  • Status changed from Open to Resolved

There is a fix in 8602-del-dirs that should ensure jobs solution, however, a complete fix will be in #8086.

#11 Updated by Evgeny Novikov about 4 hours ago

  • Blocked by deleted (Feature #8086: Remove all files when it becomes clear that they won't be used anymore)

#12 Updated by Evgeny Novikov about 4 hours ago

  • Related to Feature #8086: Remove all files when it becomes clear that they won't be used anymore added

#13 Updated by Evgeny Novikov about 4 hours ago

  • Status changed from Resolved to Closed

I hope that the suggested fix will help, so I merged the branch to master in commit:eb8dee3d.

Also available in: Atom PDF