Issue34034
Created on 2018-07-03 18:06 by Santiago Hernandez, last changed 2022-04-11 14:59 by admin. This issue is now closed.
| Files | ||||
|---|---|---|---|---|
| File name | Uploaded | Description | Edit | |
| example.py | Santiago Hernandez, 2018-07-03 18:06 | A minimal example where the problem happens | ||
| Messages (4) | |||
|---|---|---|---|
| msg320995 - (view) | Author: Santiago Hernandez (Santiago Hernandez) | Date: 2018-07-03 18:06 | |
Hi, I am splitting some processing among different processes with the forkserver start method using multiprocessing. In python 3.4 to 3.6 this worked fine. In python 3.7.0 it's breaking with the following Traceback: ``` File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/process.py", line 112, in start self._popen = self._Popen(self) File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__ super().__init__(process_obj) File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__ self._launch(process_obj) File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: can't pickle weakref objects ``` I am attaching a minimum example where this happens. In the example this can be solved by making self._run_single a staticmethod. However, I would like to understand what is going on and what changed in python for this to throw an exception now, when before it did not. Thanks. Santiago |
|||
| msg321000 - (view) | Author: Antoine Pitrou (pitrou) * ![]() |
Date: 2018-07-03 19:38 | |
It could be many different things. The bottom line here, though, is that the Process class is not designed to be picklable (how would it work?), which is probably why you're seeing this. |
|||
| msg321190 - (view) | Author: Santiago Hernandez (Santiago Hernandez) | Date: 2018-07-06 19:49 | |
Thanks, I get it now, I hadn't seen that the pickling of the workers was Processes pickling. Yeah, I was not doing this anywhere else, it seems like slipped through once. But it called my attention that it used to work and now it doesn't. On Tue, Jul 3, 2018 at 4:38 PM, Antoine Pitrou <report@bugs.python.org> wrote: > > Antoine Pitrou <pitrou@free.fr> added the comment: > > It could be many different things. The bottom line here, though, is that > the Process class is not designed to be picklable (how would it work?), > which is probably why you're seeing this. > > ---------- > nosy: +pitrou > > _______________________________________ > Python tracker <report@bugs.python.org> > <https://bugs.python.org/issue34034> > _______________________________________ > |
|||
| msg321191 - (view) | Author: Antoine Pitrou (pitrou) * ![]() |
Date: 2018-07-06 20:05 | |
No problem! I'm going to close this issue as I don't think there's anything we can do here. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:59:02 | admin | set | github: 78215 |
| 2018-07-06 20:05:11 | pitrou | set | status: open -> closed resolution: not a bug messages: + msg321191 stage: resolved |
| 2018-07-06 19:49:46 | Santiago Hernandez | set | messages: + msg321190 |
| 2018-07-03 19:38:09 | pitrou | set | nosy:
+ pitrou messages: + msg321000 |
| 2018-07-03 18:06:26 | Santiago Hernandez | create | |
