Project

General

Profile

Actions

Feature #10351

open

Try more safe process creation method

Added by Evgeny Novikov almost 4 years ago. Updated almost 2 years ago.

Status:
New
Priority:
High
Assignee:
-
Category:
Infrastructure of Core
Target version:
-
Start date:
05/22/2020
Due date:
% Done:

0%

Estimated time:
Published in build:

Description

#9186 does not help to avoid hang ups. Indeed, hang ups can happen due to tricky deadlocks happen in child processes. If we will use more safe process creation method.

Actions #1

Updated by Ilya Shchepetkov almost 4 years ago

  • Priority changed from Urgent to High
  • Target version deleted (3.0)

By default on Linux Python multiprocessing module starts child processes using "fork" start method. There are a number of issues with this method, as outlined in this article. It is quite possible that "fork" is the reason of rare freezes that happen in Klever from time to time. The solution is to tell multiprocessing to use "spawn" start method instead. Example:

import multiprocessing as mp

def foo(q):
    q.put('hello')

if __name__ == '__main__':
    mp.set_start_method('spawn')
    q = mp.Queue()
    p = mp.Process(target=foo, args=(q,))
    p.start()
    print(q.get())
    p.join()

However, there are a few extra restrictions which code must satisfy to in order to be used with "spawn". These restrictions are outlined in the Python documentation. Currently converting native scheduler to use "spawn" is rather easy, but there are a lot of issues that arise during converting Klever.Core. Due to this the implementation is postponed until after the release of Klever 3.0.

Actions #2

Updated by Evgeny Novikov almost 2 years ago

  • Assignee deleted (Ilya Shchepetkov)
Actions

Also available in: Atom PDF