Paul F. Dubois
Computer Science Project Leader, X-Division
Lawrence Livermore National Laboratory1
dubois1@llnl.gov
Python extensions are usually created by writing the glue that connects Python to the desired new functionality in the C language. While simple extensions do not require much effort, to do the job correctly with full error checking is tedious and prone to errors in reference counting and to memory leaks, especially when errors occur. The resulting program is difficult to read and maintain. By designing suitable C++ classes to wrap the Python C API, we are able to produce extensions that are correct and which clean up after themselves correctly when errors occur. This facility also integrates the C++ and Python exception facilities.
This paper briefly describes our package for this purpose, named CXX. The emphasis is on our design choices and the way these contribute to the construction of accurate Python extensions. We also briefly relate the way CXX's facilities for sequence classes allow use of C++'s Standard Template Library (STL) algorithms on C++ sequences.
Anyone who has extended Python using C knows that it is not hard to get something simple working fast. There is even a small Tk-based GUI that will write the framework of a C extension which you can just fill in. However, once you attempt to write an extension of any significant size or complexity, you quickly find it is difficult to maintain correctness, and that the main size and complexity of your program quickly becomes dominated by the extensive coding required for detecting errors and maintaining correct reference counts.
Attempts have been made to enhance the writing of Python extensions within the C language setting. These include work by Don Beaudry2, Jim Fulton3, and David Beasley4. Our work is in a different direction.
We are using Python as part of a steering system for a large program written in C++. There is a strong fit between the characteristics of C++ and the needs of the Python extension writer. We felt that a suitable set of C++ classes could be used to alleviate these difficulties. In particular, C++'s facilities for defining behaviors during object creation, copying, assignment, and destruction allow us to get reference counting right. We describe here the package we have implemented, called "CXX".
Sometimes the most difficult part of writing a Python extension in C is dealing with errors. If you are part way through a complicated method and discover that you are unable to proceed and are going to need to trigger a Python exception, you need to release any references owned by local objects. If you do not, a memory leak will occur. Depending on the flow of logic within a routine, it may be very tedious to keep track of which owned local references have been created so far.
In C++, the destructors of any local objects that have been fully constructed will be called when an exception is thrown, and so on up the call chain until either the job terminates or the exception is "caught". This will release precisely the right references, without any effort on our part. By using a simple coding style, the extension author can convert the C++ exceptions thrown in Python extensions into Python exceptions, thus giving the Python user a chance to regain control.
A secondary goal was to bring the power of the STL to Python container classes. By defining a suitable facility, we are able to directly apply STL algorithms (such as sort) to Python sequences, and to use iterators over Python sequences. Python sequences can thus be treated in a similar programming style to that used for any other container class.
CXX is written
in standard C++. The source code is available for free redistribution
as a beta release as part of the LLNL Python extensions distribution,
which is available at
ftp://ftp-icf.llnl.gov/pub/python/LLNLDistribution.tgz. Please
see the legal notices enclosed in that package.
C++ compilers are rapidly catching up to the ISO standard. Suitably strong compilers for compiling CXX are available on the platforms of interest to us. Nonetheless, some compilers are not yet able to compile CXX due to its use of modern template techniques and namespaces. A small configuration file provides an attempt at a work-around for the namespace problem. In our own testing, C++ compilers from Microsoft (Visual C++ 5.0 SP3 or later) and from Kuck and Associates have been used successfully. We have heard of other successes but do not have a complete list.
The CXX class hierarchy is rooted in class Object, in namespace Py. An instance of Object holds a pointer to a Python object and owns a reference to that object. When this instance is destroyed, the destructor decrements the Python object's reference count. An Object can be created that holds a pointer to any kind of Python object. The Object class contains methods that correspond to each of the general Python operations. Binary operators such as plus are defined so that we have access to those behaviors for Objects. Stream output is defined to use the object's string representation str().
Sometimes we want to work with the properties of Python objects that are specific to their actual type. We define descendants of Object for each of the Python types; these more specific classes test the Python object to which they are going to point to make sure it is a proper member of the desired type. When this test fails, an exception is thrown. Thus, creating an instance of one of these more specific types not only gives us access to the API appropriate to such an object, it also acts as a runtime type-check.
Here is a simple example. The following program fragment is part of the test routine for the Dict class. Dict, List, String, and Int are CXX classes which correspond to Python's dictionary, list, string, and integer objects.
Dict a, b; List v; String s ("two"); a ["one"] = Int(1); a [s] = Int(2); a ["three"] = Int(3); v = a.values (); sort (v.begin (), v.end ()); b = a; b.clear();
Here we see that a Python dictionary object a is created, and three key/value pairs are added to it. Then the list of values in extracted into a Python list object, and the list object is sorted using the STL algorithm sort. Finally, a second reference b to the dictionary is created, and the dictionary-specific method clear is called to empty it.
Note that at any point if an error had occurred in one of the underlying calls to the Python C API, the result would have been the propagation of an exception. In the act of leaving the routine that contains the above fragment, the objects we had created would be correctly destroyed.
Note that in the previous example we were able to use the STL algorithm sort on an instance of class List. This is possible because each of the classes in CXX that corresponds to a sequence type in Python has been declared to inherit from a special templated class SeqBase<T>. Here the template parameter T is intended to be some descendent of Object which defines the most specific type expected for the element of the sequence. Of course, for the standard Python sequences, this type is Object itself: that is, the most we know about an element of a Python list is that it is an Object. Since this is such a common case, the name Sequence is available as a short-hand for SeqBase<Object>.
SeqBase<T> defines a number of facilities that assist the C++ programmer in dealing with sequences. These are:
List mylist; mylist = ... some list ... for (List::const_iterator j = mylist.begin(); j != mylist.end(); ++j) { cout << *j; }
If you are not familiar with STL iterators, this doubtless looks very strange to you. Think of an iterator as a kind of pointer that when incremented knows how to advance itself to the next item in a container. Traditional pointers can only do this with contiguous data structures. Iterators are thus a generalization of C pointers.
Each of the CXX classes has a variety of constructors available, appropriate to the particular type. For example, class Float instances can be constructed from C doubles:
double d; Float x (d); // make a Float whose value is d
Each of the classes has a constructor which accepts an existing PyObject*. The class instance increments the reference count on the object, thus giving itself an owned reference to the object. When the instance is destroyed, its destructor decrements the reference count.
When a Python object is created by a call to the Python C API, the result is often an owned reference. In that case we want the resulting CXX class instance to take over this ownership. (Since we wrap a great deal of the Python C API in CXX classes, this is more of an issue within CXX's implementation than it is for the end user.) To this end CXX defines a helper class FromAPI whose net effect is to decrement the reference count on a PyObject*. For example, PyDouble_FromDouble is a routine in the Python C API that returns an owned reference to a Python float object. Thus, were we to do:
double d; Float x (PyDouble_FromDouble (d)); //Incorrect
the reference count would be incorrect. (Of course, we would not normally do this since the constructor Float x (d) would do the job much more simply.) Instead, we should write:
Float x (FromAPI (PyDouble_FromDouble (d)));
It is part of the philosophy of CXX that no illegal objects are ever permitted to exist. Each and every CXX constructor must end with its pointer pointing to a legitimate Python object acceptable to that class.
An alternative design is possible, in which the pointer can be null as well. We came to a decision after trying it both ways. The current design has the virtue of eliminating any concern about whether an object represents a legitimate object; in effect, we are eliminating the possibility of a dangling pointer. No consumer of CXX ever need test an object to see if it is a legitimate representative of its class, because if it was not its constructor would have thrown an exception.
The downside of this decision is a burden on those extending CXX to add new types. You must be able to (at least temporarily) construct a legal member of the parent class as part of your own constructor process. If you are inheriting directly from Object, there is no problem doing that, since you can use Object's default constructor, which yields a reference to Python's None object.
This is not the only place in CXX where a minor inefficiency was accepted in order to ensure correctness. It is difficult to measure the performance effects of such choices without completely recoding the package a different way. We have not yet had the opportunity to recode an existing C extension to compare performance but have no concerns in this regard; generally, minor inefficiencies in the compiled part of an interpreter extension are insignificant for total performance.
The object hierarchy in CXX is as follows. Inheritance is shown by indentation. Besides the Object family, there is a family of exception classes and some classes to help in the creation of Python modules and extension objects. All names in CXX are contained within the namespace Py.
TABLE 1. CXX Class Hierarchy
Object Type Module Integer Float Long Complex Char (Strings of length 1) SeqBase<T> Sequence (same as SeqBase<Object>) String Tuple List Array (NumPy array) MapBase<T> Mapping (same as MapBase<Object>) Dict Exception StandardError IndexError RuntimeError ... (more classes corresponding to the Python exception hierarchy). MethodTable ExtensionModule PythonType PythonExtension<T> ExtensionType<T>
In addition there are a number of functions defined at the global (namespace Py) level. These include the usual binary arithmetic operators and stream output operators.
The documentation shows the tables of methods for each class. Class Object defines a large set of methods that is thereby made available on all of its children. Notable among these are:
To make a Python extension module is now straight-forward. Let us begin by examining the form to use for a module method.
The generic form of the extension module is the same as when using C. First you write a function whose signature is
PyObject* mymethod (PyObject* self, PyObject* args)
In this form, we know that the argument self is unused, and that the argument args is actually always a tuple. We will therefore always have the same structure to our method:
PyObject* mymethod (PyObject* self, PyObject* args) { Tuple the_arguments (args); try { .... do stuff return the_answer; } except (Exception&) { return Null(); } }
The try/except clause converts any Python API errors or CXX-detected errors into an exception which is caught in this except clause and converted into a Python exception. (You can also catch the exception instance and clear the exception, as explained in the documentation). Keyword arguments can be handled by using the Dict dictionary object in an analogous manner to the use of the Tuple object above.
In writing the "do stuff" part of the method, we are now greatly assisted by CXX:
This will throw an exception if the first argument is not a string, or if there is no first argument. (Class Tuple also has methods which can check for a certain number or a range of numbers of arguments).
Here for example is a method written using CXX that sums the set of float arguments given to it.
using namespace Py; static PyObject * ex_sum (PyObject* self, PyObject* args) { Tuple a(args); try { Float f, g; int i; f = 0.0; for (i = 0; i < a.length(); i++) { g = a[i]; f = f + g; } return new_reference_to (f); } catch (const Exception&) { return Null (); } }
The function new_reference_to (Object ob) returns an owned reference to the object ob. In this way the Python float object we have created to hold the answer survives the destruction of the variable f that occurs when we return from ex_sum. If you want a method that doesn't return anything you return Nothing() and to signal a Python exception you return Null().
Note in this example how the assignment g = a[i] not only extracted the ith argument but ensured that it was a float object. It would have also worked perfectly well not to do this step but directly add f + a[i]. This might be desired, in fact, if you did not want to insist that the arguments be floats.
As usual, we now need an Python "init" function for our extension module. In this routine, which must have C linkage so that Python can find it, we create the extension module and add the method(s) desired to it, as well as any objects we wish to seed into its dictionary (here, as an example, we add the constant pi).
extern "C" void initexample ();
void initexample () { static ExtensionModule example ("example"); example.add("sum", ex_sum, "sum(arglist) = sum of arguments"); Dict d = example.initialize(); d ["pi"] = Float(3.14159); }
The declaration of example as static is to ensure that the object will survive the call to the routine initexample. We could have also declared, outside of initexample,
static ExtensionModule* example;
and then created the module with in initexample with:
example = new ExtensionModule("example")
Note the simplicity compared to writing the same thing in C, with its mysterious "static forward", Python type tables, etc.
CXX also contains a facility for construction of new Python types. This facility is not yet completely satisfactory, but we believe it is a step forward. The key point is that to begin a new Python type we inherit from class PythonExtension, which in turn inherits from PyObject. Thus, at one blow we have made our type a descendant of PyObject, created a type object for it, and created a function that will check whether an object is of this new type. Then we initialize the object in a similar manner to the extension module, adding behaviors and their descriptions. We also write the methods in a similar way.
PythonExtension is a templated class, and the template argument we give it is, shockingly, the class we are defining. This is an example of what Scott Meyers has called the "Curiously Recursive Template Pattern"5,6. PythonExtension sets up a Python type object unique to this type, creates a static function
static bool check (PyObject *)
that tests membership, and sets a deletion behavior that ensures the calling of the class' destructor in the case of its Python reference count going to zero.
Here, for example, is the start of a class "r" defining new objects of type "r" similar to the Python range object:
class r: public PythonExtension<r> { public: long start; long stop; long step; r (long start_, long stop_, long step_ = 1L) { start = start_; stop = stop_; step = step_; }
~r() { std::cout << "r destroyed " << this << std::endl; } ...
In a method similar to the way we implemented the module method above, we define the Python behaviors of the new object, such as r_repr, r_getattr, r_length, and methods we choose such as amethod, value, etc. Then in some module's initialization procedure is a call to init_rtype:
void init_rtype () { r::behaviors().name("r"); r::behaviors().doc("r objects: start, stop, step"); r::behaviors().repr(r_repr); r::behaviors().getattr(r_getattr); r::behaviors().sequence_length(r_length); r::behaviors().sequence_item(r_item); r::behaviors().sequence_slice(r_slice); r::behaviors().sequence_concat(r_concat); r::methods().add("amethod", r_amethod); r::methods().add("assign", r_assign); r::methods().add("value", r_value); }
Extension objects defined in this way can be used in the same way you usually work with PyObject pointers. However, they have an additional desirable property that ordinary Python extension objects do not: they do not have to live on the heap. Of course, as in C++ one must be careful not to retain a reference to a local object after the routine returns. But, with care, one can have the efficiency of objects using stack rather than heap memory. With correct use both new/deleted objects and stack objects will coexist nicely.
A wrapper for this object can be created by using class Class ExtensionObject<r>. This gives us a wrapper class that is a child of Object. The necessary acceptance test used in this wrapper is r::check, defined for us when we inherited from PythonExtension<r>. Such a wrapper is useful as a wrapper of "r" objects. Unfortunately, it does not contain any methods specific to class r, so to make a really useful extension wrapper for class r we would need to write one that inherits from ExtensionObject<r> and adds wrappers for the methods of interest.
We see that the strengths of C++ are a good match for the weaknesses in the Python language extension process. Use of C++ in this way should lead to much smaller, cleaner, and easier to write extensions, with confidence in their correctness. Reference-counting errors, among the most difficult to prevent and to diagnose when using C, are avoided automatically, even in the difficult case of when an error occurs.
Additional benefits to this approach are the merging of the Python and C++ exception facilities and the ability to write extension modules and objects much more naturally and safely. The facility is completely extensible via inheritance to the users' own classes.
In the future, we would like to eliminate the need for the try/catch blocks in each method and hoist that task up above the call to each method. Geoffrey Furnish has done some work in this area but has not had time to complete it. Recent additions to Python may also allow some improvement in the extension object area.
Research remains on further integrating Python and C++ but we believe CXX represents a significant step forward.