Home Original page

HELP! Obscure Python Reentrancy Question

Courageous jkraska at san.rr.com
Sat Feb 23 01:40:10 EST 2002 
>How have you allocated interpreter state(s) versus your C++ scheduler
>threads (or fibers) and how are you maintaining the GIL?

I haven't considered how and when to grab the GIL, and what it
means from the perspective of two different cooperative threads,
each of which may (or may not!) be executing a Python code
fragment, at some partial state of completion. Let me help:

def f(task):
	do()
	some()
	task.Step(1)
	other()
	stuff()
	task.Succeed()
	
In the above example, do() and some() are ordinary internal or
external python procedure calls. task.Step(1) kicks this codelette
back to the scheduler, which is written in C++. The scheduler is
dealing with other tasks, some in Python, some not, which are
basically doing the same thing. Eventually, the scheduler comes
back to the task, and then other() and stuff() are executed.
task.Succeed() then kicks this task back to the scheduler and
it is deallocated. It is an uncategorical error to end a task
without a success call of some kind.

>When the other thread in 5 resumes and "contains" more Python, does
>that mean that from the perspective of the Python script you are
>"returning" from the call in 4, or does that mean there's some
>independent Python code you are just starting to execute?

Well, each Python code fragment, much of the time, never really
"finishes". Thing of them as complex coroutines. So the scheduler
may be invoking an independent piece of Python code, or it may
awaken older code. Technically, it may reawaken the code which
just went to sleep.

>It seems to me that depending on how you want to link your Python code
>execution with your scheduler thread states, you may want to have an
>interpreter (or at least thread state) per thread,...

I have tens of thousands of threads. They access eachother's data
in some cases. They send messages to eachother, indirectly, through
the simulation queue, sometimes.

Basically what's going on here is that the code is bunched up into
lock steps which only occur at defined points in _simulation time_.
Someone using the API writes "task.Step(20)" and then knows that
the code which follows this won't happen until 20 simulation seconds
have jumped ahead from simulated current time. This is managed by
kicking the current task back to the scheduler and putting an entry
on a priority queue.

Theoretically, at some time in the future, I might want to have
a worker pool of threads, to support running this on SMP machines.
It's okay if Python can't take a lot of advantage of that, but
I do need to make it behave well and not block.

I didn't think that Python was structured to allow multiple interpreters
in a single process?

If it is, how are module imports resolved across interpreters? I need
module globals to behave globally.

C//
More information about the Python-list mailing list