Way too old! But even 1.1.1 has memory leaks. There must be ones in
1.2 (currently in beta) but I've fixed several ones that were in
1.1.1.
> -- What techniques do you use to isolate unexpected heap growth? The
> growth may or may not be a memory leak, but I'd like to know how to
> determine that without reading the source for every module I use.
This question is about as general as "how do you debug a program".
You don't need to read every module you use but you must understand
the general data flow within your program. If you have a program that
can run for a long time, do you expect (based upon the algorithm you
use, etc.) that it would accumulate heap space or not?
> -- Is it reasonable to expect that memory footprint can be controlled
> in a non-trivial, long-running application written in python? I know
> this is a rather vague question. Perhaps a better one to ask would
> be has anyone successfully applied python in this way?
The people at Infoseek have some interesting stories. Again, if you
have a long-running process that doesn't accumulate log information or
anything like that, and you experience (by observing "ps" output) that
it grows without bounds, you've either hit a memory leak in Python or
one in your application.
> -- Are there known leaks or conditions that produce leaks
> (notwithstanding the regex one that passed by the other day)?
Yes. I don't have a list -- it's buried somewhere in my ChangeLog
files.
> -- Would a later release provide substantive improvement in the
> memory footprint area?
Yes. As I mentioned above 1.2 does plug a number of serious holes.
> -- Is anyone aware of memory footprint related problems with dbm (I use it
> rather heavily)?
Hmm... Dbm is magic for me. If you suspect anything here, perhaps
you can construct a small test program that repeatedly opens a dbm
database, accesses it a few times (or a lot), and closes it -- and see
if your process size changes.
> -- has anyone created iterator types for sucking the contents out of a
> mapping type (especially something like dbm)? The 'keys()' method
> doesn't seem to scale real well with dbm hash files that are large.
> Am I thinking about this wrongly?
This is indeed a problem with dbm. I suppose it would be an easy
modification to provide access to the firstkey and nextkey operators
that the dbm library provides.
> I appreciate any pointers or help.
You're welcome!
--Guido van Rossum, CWI, Amsterdam <mailto:Guido.van.Rossum@cwi.nl>
<http://www.cwi.nl/cwi/people/Guido.van.Rossum.html>