Feature #10351
openTry more safe process creation method
0%
Description
#9186 does not help to avoid hang ups. Indeed, hang ups can happen due to tricky deadlocks happen in child processes. If we will use more safe process creation method.
Updated by Ilya Shchepetkov over 4 years ago
- Priority changed from Urgent to High
- Target version deleted (
3.0)
By default on Linux Python multiprocessing module starts child processes using "fork" start method. There are a number of issues with this method, as outlined in this article. It is quite possible that "fork" is the reason of rare freezes that happen in Klever from time to time. The solution is to tell multiprocessing to use "spawn" start method instead. Example:
import multiprocessing as mp
def foo(q):
q.put('hello')
if __name__ == '__main__':
mp.set_start_method('spawn')
q = mp.Queue()
p = mp.Process(target=foo, args=(q,))
p.start()
print(q.get())
p.join()
However, there are a few extra restrictions which code must satisfy to in order to be used with "spawn". These restrictions are outlined in the Python documentation. Currently converting native scheduler to use "spawn" is rather easy, but there are a lot of issues that arise during converting Klever.Core. Due to this the implementation is postponed until after the release of Klever 3.0.