Klever

I didn't find an administrator URL to unblock Bridge in case of failures. I suppose to make it accessible just by users with role Administrator.

Can't find commit where I already added such url. I've implemented it again (/tools/manual_unlock/), but now only for administrators.

Testing showed that there are some unexpected and unacceptable delays in requests processing.

I specified an incorrect priority. Of course this issue is the most important for Bridge now.

By some reason I don't observe very long delays when editing jobs anymore. The only case is when a job has quite many files - then about 5 seconds are required. That's why I at last merge the branch to master in 3f0ad507. Don't forget to migrate your database and be sure that parallel requests are actually blocked if this is necessary.

BTW, very seldom I saw quite strange errors when I conducted a lot of huge experiments. I guess that they were exactly due to data races and now they shouldn't happen anymore.

It turned out that there a couple of issues in the new implementation. So, when some request is terminated with an internal error and a lock file is remained (this also can happen, say, when a process is forcibly killed), then several requests can try to release this lock concurrently causing exceptions like:

[02.May.2017 10:41:23] Internal Server Error: /service/update_nodes/
Traceback (most recent call last):
  File "/home/novikov/work/klever/bridge/tools/profiling.py", line 48, in lock
    with open(self.lockfile, mode='x'):
FileExistsError: [Errno 17] File exists: 'media/.lock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/novikov/.pyenv/versions/virtual-env-3.6.1/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/novikov/.pyenv/versions/virtual-env-3.6.1/lib/python3.6/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/home/novikov/.pyenv/versions/virtual-env-3.6.1/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/novikov/.pyenv/versions/virtual-env-3.6.1/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/novikov/work/klever/bridge/tools/profiling.py", line 122, in wait
    locker.lock()
  File "/home/novikov/work/klever/bridge/tools/profiling.py", line 55, in lock
    os.remove(self.lockfile)
FileNotFoundError: [Errno 2] No such file or directory: 'media/.lock'

Another issue is that when some request really needs much time (more than hard coded 30 seconds) other requests will be executed in parallel that can cause races and most likely some issues with lock files creation/removing like above. I guess that you can easily check this.

As far as I understand both these issues are just for the development mode when we are trying to automatically recover from errors.

Fixed in "fix_8015".

I hope that this will help at least in case when one encounters really time consuming requests but one has to know such the limitations. I merged the branch to master in 3b60303.