Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing set_start_method crash #873

Open
extremeCrazyCoder opened this issue Oct 28, 2024 · 0 comments
Open

multiprocessing set_start_method crash #873

extremeCrazyCoder opened this issue Oct 28, 2024 · 0 comments

Comments

@extremeCrazyCoder
Copy link

I don't know if this is supposed to be supported at all.

Describe the bug
The multiprocessing spawn method cannot be changed when using scalene.

To Reproduce
The following code will crash with the error:

Error in program being profiled:
 context has already been set
Traceback (most recent call last):
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/site-packages/scalene/scalene_profiler.py", line 1795, in profile_code
    exec(code, the_globals, the_locals)
  File "/home/USER/Dokumente/mptest/test.py", line 6, in <module>
    multiprocessing.set_start_method("spawn")
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/context.py", line 247, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set

import multiprocessing
import multiprocessing.process


multiprocessing.set_start_method("spawn")


def test():
    print("At", multiprocessing.process.current_process())


with multiprocessing.Pool(4) as p:
    for i in range(8):
        p.apply_async(test)

    p.close()
    p.join()

Expected behavior
No crash

Desktop (please complete the following information):

  • OS: Ubuntu 24.04.1 LTS
  • Browser doesn't matter
  • Version Scalene version 1.5.45 (2024.10.01) - also tried the current git version

Additional context
The error can be mitigated by using multiprocessing.set_start_method("spawn", force=True) but in that case it will crash when spawning the processes.

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/site-packages/scalene/scalene_profiler.py", line 1795, in profile_code
    exec(code, the_globals, the_locals)
  File "/home/USER/Dokumente/mptest/test.py", line 13, in <module>
    with multiprocessing.Pool(4) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/USER/anaconda3/envs/ox/lib/python3.12/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant