Yes, threads are a somewhat unexplored territory in Python, and
combining threads with embedding in an application will certainly
uncover problems...
> However, I am embedding Python in an application, and there is no
> interpreter running.
> My program is a GUI, and calls Python via the call_object API. The Python
> event handler code creates a new thread, and it starts OK. Then the event
> handler code exits, but with the interpreter lock still acquired. The
> thread remains stalled on the lock. When another event handler is called,
> the thread continues, etc.
>
> I resolved this problem by doing the reverse of BGN_SAVE/END_SAVE. Before
> calling the interpreter, I acquire the lock (thread_restore()), and after
> call_object returns, I release it (thread_save())
Your basic analysis of the problem is correct. *Anything* you do with
Python objects has to be done with the interpreter lock held. As you
found out, the way to acquire the lock is to call restore_thread()
(funny name, OK). This shouldn't be called if you are already holding
the lock. To release the lock, call save_thread().
> However, this does not resolve all the problems. There are other ways the
> interpreter is called that I can not intercept. print_error() is a problem,
> but the biggest one hard to work around seems to be deleting objects. In my
> specific example, when the user closes a window, a Python object is
> DEC_REF'd. This may force destruction of the object, which may/will cause
> the interpreter to be called, exiting with the lock in place.
If you DELREF (or INCREF!) objects or call print_error, you must also
have the lock. What exactly do you mean by "exiting" here? Is there
some path out of the interpreter that does not return from the DELREF
call? If so I'd like to know about it.
> Working around these is getting ugly, and some have no clean solution (eg,
> the print_error should a thread exit with an exception).
I'm not sure whether you are referring to problems in the existing
Python code (where it should release the lock but doesn't) or in your
own code. As far as I know, I call print_error() only with the
interpreter lock held. You should do the same. The call to exit() in
print_error() may be a problem -- perhaps this should be handled in a
different way anyway.
> It would seem the solution is to acquire/release the lock on
> interpreter entry/exit, as well as the current scheme of releasing
> the lock while executing.
How do you define "interpreter exit"? In many cases the interpreter
calls itself. Maybe the solution is to have a few special calls that
acquire and release the interpreter lock?
--Guido van Rossum, CWI, Amsterdam <mailto:Guido.van.Rossum@cwi.nl>
<http://www.cwi.nl/cwi/people/Guido.van.Rossum.html>