Re: SUSPEND (continuations)

Tim Peters (tim@ksr.com)
Tue, 10 May 94 00:13:31 -0400

> [steve m]
> ...
> I did not completely agree with Tim and Guido's analysis of the
> problems involved. This is not the refutation of their view - only
> the tool I need to discover if they are correct or not. i.e. all this
> implements is "resumable functions" - I have yet to demonstrate that
> a more complete threading and/or coroutine control can be built in
> a similar manner. ( SO it's only proof of half the concept, so far! ;-)

We're not really disagreeing -- you just haven't figured out what I said
yet <grin>. From the start, I said I thought resfuncs (as originally
sketched) were implementable. In every msg _except_ the start, I said I
thought anything fancier than that would be a nightmare.

Back when we were talking about how C frames interacted with this, you
wrote:

> But 'SUSPEND' opcode would do a return from ceval just like 'RETURN'
> opcode, except it would return a ( retval, cont-obj ) tuple instead of
> just retval.
> ...
> Ceval would return and the C-stack would unwind, leaving behind a
> "restartable" python frame.

For a simple resfunc, that's true, because its ENTIRE continuation state
is captured in a single frame, and it returns directly to its caller.
But suppose you're _more_ than one level deep when you SUSPEND:

def W():
suspend whatever

def V():
whatever = W()

def U():
V()

U()

SUSPEND only returns to its caller, so only the C frames built up in
calling from V to W get unwound. If "suspend" is only implementing a
resfunc, no problem! But if "suspend" is trying to implement a more
general continuation, the C frames for the call from U to V, and from the
top level to U, are not unwound by the suspend. What happens to them?

In one seductively misleading sense that's often not a problem, because
the back-pointers in the frames tell you where to return -- you don't
need the info in the C stack for that, so long as you're staying entirely
within Python. Still, those C frames _are_ stacked up, and you've got no
clear way to get rid of them (Guido's solution-sketch was to do the calls
inline, hence not build the C frames to begin with).

The implication is that you can get a general suspend to work (so long as
the continuation remains entirely in Python), but the C stack will
continue to grow and grow. If you _don't_ stay entirely within Python,
it's much worse: say a user application embeds Python, and one of their C
routines calls a Python function. In this case, the C stack is an
_essential_ part of the continuation while you're in the Python code.
Right? Else you can't get back to the C caller.

The other half, which we haven't talked about, is what happens when you
do a resumption -- i.e., when you _execute_ a continuation:

def W():
global cont # some continuation object
cont() # give control to it

def V():
W()

def U():
V()

U()

Here again you've got no way to nuke the C frames that were built up when
top-level called U, which called V, which called W. So the C stack keeps
growing on the execute side too. For resfuncs this isn't a problem,
because a resfunc will, by definition, return to its caller (unlike a
general continuation object, which "in theory" replaces the whole current
call chain with whatever it was at the time the continuation was
captured).

So my prediction remains <wink> that you'll get a working resfunc
implementation out of this, and maybe _think_ you have a working more-
general implementation until you try it on a fat program & discover the C
stack is taking over your disks ... OTOH, tame resfuncs were all I was
after from the start <grin>.

Keep it up! You understand the details of ceval better than I do now,
and I'd love to be proved wrong on this one.

first-time-for-everything<snort>-ly y'rs - tim

Tim Peters tim@ksr.com
not speaking for Kendall Square Research Corp