coroutines and continuations ( in Python? - long discussion )

Steven D. Majewski (sdm7g@elvis.med.virginia.edu)
Fri, 29 Apr 1994 13:58:25 -0400

A while ago there was a (crossposted) thread about about pipes:
most of the discussion was about Perl, but it caused me to repost my
redirect.py modules ( to show how *easy* the problem was in Python! ;-),
and also to think more about the utility of having a more general
2-way interface.

The tostring/tolines functions in redirect are able to package
the printed output from a function and return it as a python
object. This gives some of the capability of unix pipes without
having to actually pipe thru the OS - the non-trivial semantic
difference being that redirect does not produce any concurrency -
the inner function must run to completion before the redirecting
wrapper can return the collected output to the calling function.

Actual concurrency of independent threads is not required, though:
coroutine would be sufficient since the read/write is a
blocking operation that could be an implicit transfer of control.

I know Guido is not as enamored of coroutines as old Icon-ers like
Tim and I. He has (more than once, I think) stated the view that
anything you could do with coroutines, you can do with classes.
This view is certainly true from a theoretical POV - complemantarity
of the object=method+data and closure=function+environment and all
that, and coroutines are just a subset of continuations, etc., etc.
But the difference is that saving of state into an objects instance
variables has to be explicitly programmed, where coroutines and
continuation are implicit. ( And being able to turn a pair of
read/writes in a pair of functions into a coroutine return/continue,
mediated transparently by a file-like python object would seem to
be a potential big win for code-reuse. )

It is possible to program a pair of classes that cooperate with each
other, but there is no way to make a class that can link two arbitrary
functions by their read/write methods. The flow of control is not
symetrical: one of the functions must RETURN before the other can
get continue.

The fact that Python frames are (heap allocated) objects and not
just pushed on the stack, means that they can persist whether that
function is active or not. In fact, an exception passes you a link
to a whole bunch of frameobjects in it's traceback object.

A frame object appears contain all of the necessary state to serve
as a continuation except that the interpreter's stactpointer
and blockpointer are not preserved, and when a stack frame is
unwound by an exception, the stack is emptied by poping and
DECREF-ing the contents.

I looks to me that it would no be difficult to give Python
continuations: if a stackpointer and blockpointer are added
to the frameobject, all that is missing is an operation to
package a frameobject-continuation, and a 'resume' operation
( a change to ceval.c ) that takes a frame/continuation object
instead of creating a new frameobject.

I am far from a complete understanding of ceval and what happens
to frame and blocks in Python, so I may be missing some obvious
problem ( And all the details of *exactly* how to do this aren't
completely clear to me yet, either ).

Comments ?

What I initially wanted was some sort of cooperative coroutine
control between two functions mediated by a file-like object
that used it's read/write methods to alternate control on the
two functions. Except for inet servers, most of the cases where
I might want threads seem to actually fit a blocking coroutine
model better than parallel threads, and in the cases where it
doesn't, I'm quite happy to use unix processes for threading.
I would like to have threads on dos/mac, but I don't know if
any of the portable (on unix) threads packages work on those
platforms, or how much work it would be to get really portable
threads written in C to work in Python. But since the easiest
method of adding coroutine control appears to be the more
general case of adding continuations to Python, I would think
that this also gives a possible method of adding preemptive
threads to the Python interpreter in a portable manner by
coding it into the virtual machine rather than the real machine.

Comments ?

[ Encouraging or _discouraging_ comments invited! If I'm blind
to some fatal flaw or don't understand what's going on in
ceval, I'ld be happy to hear about it before I actually
waste time in trying to implement it! ]

-- Steve Majewski (804-982-0831) <sdm7g@Virginia.EDU> --
-- UVA Department of Molecular Physiology and Biological Physics --
-- Box 449 Health Science Center Charlottesville,VA 22908 --
[ "Cognitive Science is where Philosophy goes when it dies ...
if it hasn't been good!" - Jerry Fodor ]