Productivity

Over the course of this project, we have been able to write on average over 175 lines of debugged, documented, tested code per developer per day. Over a period of 660 FTE days, we produced a total of approximately 121K lines, of which 77K were written in Python. Even without considering that a line of Python is typically equivalent to 10 or more lines of C, we were extremely pleased with this result.

The entire product, including third party open source modules, actually contains on the order of 1.2 million lines of code, of which 274K lines are Python.

So why was using Python so productive, even when only 63% of the code we wrote was in Python? There is no single answer to this question, but several:

1) Simple syntax -- Although use of indentation to indicate program structure sometimes turns off first-time Python users, the reduced typing burden that comes with avoiding {}'s and similar syntactic sugar does matter over the course of writing and rewriting hundreds of thousands of lines of code.

2) Dynamic high-level data typing -- Lack of strong data typing, another commonly cited "weakness" in Python, is in practice a significant advantage for most kinds of software development. The bottom line is that you don't need strong data typing in the context of a language like Python. Common coding errors are caught anyway by the type system: You still can't add a number to a string, reference past the end of an array, or call an non-existent class method. Dynamic high-level data typing cuts out great volumes of support code and makes it possible to write flexible and introspective code (more on this below).

3) Powerful, easy-to-use data structures -- Python's built-in list and dictionary data structures can be used in combination to build just about any fast runtime data structure in a snap. This further reduces the amount of support code you need to write.

4) Extensive standard library -- Python comes with a vast standard library supporting everything from string and regular expression processing to XML parsing and generation, Web services tools, and internet protocol support. Many common programming tasks have already been built into the standard library, making it possible to do more with less code. Third-party modules are also available for database access, CORBA, COM, statistics, math, image processing, and much more.

5) Introspection -- Python's flexible data typing system extends also to code documentation, classes, methods, and even the way in which methods are called. Python makes introspection extremely accessible to the programmer and, remarkably, introspective code remains readable and maintainable. This can be very useful in redirecting I/O to classes of your own design, writing a tracer that determines code coverage, packing up data to store on disk or send over the network, developing glue code, writing table-driven algorithms, extracting documentation from code at runtime, applying design-by-contract development methods, and in building various types of meta-classes. For almost every programming task, Python makes it not only possible but quite easy to build meta-code where one might otherwise end up building gobs of manually crafted code.

6) Faster development and deeper prototyping -- Python increases speed of development to the point where prototyping can be integrated into and interleaved with the primary development process. When it takes only half a day to try out a new approach to a problem, rather than the week it might take in C or C++, programmers are more often empowered to rework existing imperfect code, and to try out new ideas. This results in the more rapid incorporation of experience into an application's design, and leads to higher code quality.

Cross-platform Deployment

Wing IDE runs on a variety of Posix operating systems and Windows. Throughout our development process, we've been very happy with the way that Python performed across platforms. The same Python source or compiled Python byte code files can be shipped to clients regardless of target platform, making support quite easy.

Speed, Scalability, and Stability

When we started to write in Python, our previous experience in compiled languages led us to believe that we would be spending a fair amount of time either optimizing code or converting it into C or C++ once we had prototyped it. As it turned out, most of the time Python produced a snappy end product that didn't require any extra work.

This happened partially because most Python code is really just a thin interpreted layer over functionality that is written in C or C++. In our case, this included not just Python's fast built-in data structures and standard libraries but also the bulk of the GTK GUI development layer and the Scintilla source editor.

In the course of development and in responding to thousands of support tickets over a three year period, we have never run into any significant problems with Python itself, either in scalability or stability. Wing IDE can handle software projects with thousands of Python files, and in many cases can run for weeks without problems. To our knowledge, we have yet to see Wing IDE crash because of a flaw in the Python interpreter or its standard libraries.

Mixed-language Development

Python is almost always fast enough but we did run into a few cases where the interpreter did introduce too much overhead. The Wing IDE debugger and the source code analysis engine both contain modules that engage in extremely CPU intensive processing. These modules needed to be written in C in order to squeeze out as much speed as possible. Fortunately, Python is designed to make it quite easy to call back and forth between Python and C or C++.

In most cases, we wrote and debugged code first in Python, and then converted by hand into C. This approach worked well for us. Working initially in Python was much more efficient and the conversion process relatively painless.

Analysis of our records shows that 360 days were spent on 77K lines of Python code and 300 days (almost as much) on 44K of C, C++, or other code. From our experience with code conversions, we believe it is roughly correct for most types of performance-critical code to equate one line of Python with ten lines of C or C++ code. This means that about 5-10% of our application functionality is in C or C++ and the rest is in Python. Even considering that the C/C++ code is somewhat more complex than most of the Python code, these results confirm without any doubt that working in Python is far more productive than working in C or C++.

In hindsight, we believe that we could have converted smaller units of code into C, by writing more general data-driven processing engines, and by more carefully selecting code to convert instead of converting whole modules at a time. Our primary goal for Python in the future is to be able to use it more often, even in performance-critical sections of code. This effort should benefit from projects like pyrex, which allows the use of Python-like code in the development of compiled extension modules, and psyco, which is a just-in-time compiler for Python.