Home Original page

[Python-Dev] Bug in generator if the generator in created in a C thread

Victor Stinner victor.stinner at gmail.com
Thu Mar 29 23:26:42 CEST 2012 
On 29/03/2012 21:35, Tim Lesher wrote:
>  From a theoretical standpoint, I can't quite decide what the real error is:
>
> 1) the fact that PyGILState_Release() destroys a temporary thread
> state that may still be referenced by some objects, or
> 2) the fact that some code is trying to keep frame objects after the
> creating thread state no longer exists.
>
> This week I've been leaning toward 2), but then I realized that
> keeping frames post-thread-death is not that uncommon (for example,
> debuggers and other diagnostic techniques like
> http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html).

The problem is not the frame, but the Python thread state referenced by 
the frame. It's a private attribute. My patch just updates this 
reference before running the generator (and it clears the reference when 
the generator excution is stopped or finished).

> Locally we added some (unfortunate) code to our 3.1.2 port to wrap
> PyGILState_Ensure(), which I thought had sidestepped the issue for us:
>
> void takeGIL()
> {
>      PyGILState_Ensure();
>      // This has the side effect of keeping such thread states alive until
>      // the interpreter is finalized; however, all thread state objects get
>      // unconditionally deleted during Py_Finalize, so they won't leak.
>      PyThreadState* pThreadState = PyGILState_GetThisThreadState();
>      if (pThreadState->gilstate_counter == 1)
>      {
>          ++pThreadState->gilstate_counter;
>      }
> }
>
> But clearly that can't be a correct answer (and it may not even be a
> functioning one, given that I'm seeing a similar issue again).

You may leak memory if your threads have a short lifetime and you create 
many threads. For example if one thread is only used to process one 
request and then is destroyed.

Victor
More information about the Python-Dev mailing list