Re: Problems with big (?) dictionary

Guido.van.Rossum@cwi.nl
Wed, 09 Mar 1994 10:08:03 +0100

> I'm trying to load a module which contains a dictionary with
> 3600 keys. Under Python 1.0.0 I get an error:
>
> shift: no mem in addchild
> Traceback (innermost last):
> File "<stdin>", line 1
> MemoryError
>
> If I reduce the dictionary to around 1800 entries it loads ok.
>
> I also tried loading my dictionary under Python 0.9.8. The 1800
> entry version was MINUTES faster loading, but I get the same error
> with the 3600 entries.

You don't say which platform you are using but I have a hunch it's
MS-DOS, the only memory-starved platform on which Python run (and on
which it has only recently moved from 0.9.8 to 1.0.x).

The reason is that the parse is a little stupid and requires a lot of
memory.

I bet you are generating this directory in another program, e.g. using
repr(). It is probably *much* faster to invent your own encoding of a
dictionary and write that to a file. For example, write the items of
the dictionary to a file one per line, where each line contains a
(key, value) pair. To be specific:

def dumpdict(dict, fp):
for key in dict.keys():
value = dict[key]
fp.write('%s, %s\n' % (`key`, `value`))

def readdict(fp):
dict = {}
while 1:
line = fp.readline()
if not line: break
key, value = eval(line)
dict[key] = value

Let me know if this makes sense for you...

--Guido van Rossum, CWI, Amsterdam <Guido.van.Rossum@cwi.nl>
URL: <http://www.cwi.nl/cwi/people/Guido.van.Rossum.html>