To get data into Folio Views it must be converted to a tagged text format called Folio Flat File (FFF). FFF can be created directly from word processing documents but it can also be generated from databases and structured text formats such as XML as discussed in this paper.

All the software aspects of the electronic publishing process—from data capture quality assurance through to the final generation of a >2 GB Folio Views Infobase—are written in Python.

XML simplifies SGML in many ways. The most important from a software development perspective are that XML has a much more fixed lexical format, simplified grammar features and much simpler parsing algorithms.

XML is a meta-language—a language for creating languages. An XML derived language is a grammar consisting of named information elements organized into a hierarchical and/or recursive structure. The order and occurrences of information elements can be controlled by a grammar specification known as a DTD (Document Type Definition).

SGML and XML have conceptual similarities to YACC, BNF and Regular Expressions. SGML and XML applications (tag languages) can be thought of as particular grammars expressed in YACC/BNF or as a regular expression.

XML documents consist of a root element that can contain other elements and/or primitive data content creating arbitrarily deep structures. The presence of elements in data is signaled by the use of tags that indicate where the elements start and end. The lexcial format of the tags remains the same, regardless of the tag names used.

Consider the task of modeling a simple quiz show dialog. A simple quiz show has a host and a single contestant. The quiz consists of one or more items. Each item has a question and an answer.

The first line:

serves to indicate that this is an XML document.

The second line:

Elements can have associated attributes—small nuggets of extra information that further qualify the element. In this example the quiz element type has an associated host attribute.

Parsing XML

An XML parser is a utility program that works through an XML document tokenizing it into a series of markup tokens and primitive data tokens. A class of XML parser known as validating XML parser will also check to see if the XML document meets the structural constraints spelled out in the associated DTD. Depending on the parser, it might also generate some form of post-parse output.

Output of an XML Parser

Note that each major token in the XML file is allocated a line in the output. A line starting with "(" signals a start-tag, "A" signals an attribute, ")" signals and end-tag. "-" signals primitive data content and so on.

This output format contains much—but not all—of the markup from the original file. Note that the comment "<!-- A simple quiz-->" has disappeared.

Relationship of XML to HTML

XML is not an enhanced HTML language as it is sometimes described. HTML is a fixed set of tags created in conformance with the SGML standard. HTML is thus a tag-language. XML on the other hand, is a simplified subset of SGML. It is not a tag-language. It is a meta-language like SGML before it. Tag languages based on XML cover everything from electronic commerce to Web Browser bookmark exchange. HTML—currently based on SGML—is also migrating to become a tag language derived from XML.

XML Programming Standards

The XML software development community has spawned a number of XML programming standards that are in various stages of completion. The two most important for Python programmers are SAX and DOM:

SAX: Simple API for XML.

Oversimplifying, SAX is a naming convention/interface specification for event driven XML programs that originated with Java but has been successfully implemented in other languages including Python.

The idea is that by developing your applications in accordance with the SAX API, you have the freedom to swap XML parsers in and out without breaking any application code. Being able to swap parsers in an out is useful because XML parsers are optimized for different things—speed, memory footprint, levels of XML compliance and so on.

DOM (Document Object Model):

XML Processing Tools

Although XML is relatively new, numerous XML tools already exist—especially for software developers. Java probably has the most comprehensive support for XML but CPython is running a close second. Moreover, thanks to JPython, the Python programmer inherits many of the Java based tools. Perl, Tcl, C, C++, Scheme and Ruby also have XML tools in various stages of development.

XML Recap

Here is a recap of the acronyms and concepts presented in this section.

Problem: Rapidly changing publishing technology

It seems that with every passing month there is a new publishing delivery format—requiring yet another re-working of document content to utilize its features. In the last few years we have battled with: Windows 3.1 Help, Windows 95 Help, Microsoft Multimedia Viewer, RTF 2, RTF 6, RTF 7, PDF, Postscript, Maker Interchange Format, TeX, Troff, Lotus Notes 3 and 4, Folio Views 3 and 4, HTML 2,3 and 4. Dynamic HTML, HTMLHelp, NetHelp etc. etc.

As new formats arrive, older formats become obselete. Any data caught in these formats can be marooned and become known as a "legacy data format" requiring costly conversion to the next legacy data format...

Problem: High Publishing Volumes

The floppy disk has given way to the 600MB CD which will soon give way to the 16GB DVD etc. The sheer volume of textual information that can be packed onto these digital delivery media is new territory for the publishing sector. Managing, searching, developing user interfaces for such vast volumes of information is hard.

Problem: "A la Carte" rather than "Table D'Hote"

Users are increasingly demanding that published products be tailored to them as individuals with their own likes and dislikes. Different users want different views of the same information, different document orders, formatting, search functionality etc. etc.

Problem: New Publishing Paradigms

Hypertext is becoming an increasingly important part of electronic publishing. Creating and managing large collections of hypertext links—and tailoring them on a per user basis—is a real problem.

Loose leaf publishing on paper is giving way to incremental updates of large document collections over the Web. Achieving this seamlessly and automatically is non trivial.

XML has a number of important facets that help address the electronic publishing problems outlined in the last section.

Open Systems

XML is an open, vendor neutral, formally defined standard.

Pure Content

XML encourages the removal of presentation information from document data. Presentation information is layered onto XML data at point of presentation using style sheets.

Contrast this with WYSIWYG where presentation is wired into content and notoriously difficult to extract:

Note the intertwining of content and formatting. Although some of this intertwining can be removed by using paragraph and character styles, this only provides two levels of hierarchy with which to describe your content. Document structures are considerably deeper than two levels of hierarchy.

As a simple example, consider the quiz example presented earlier. It is easy to see that a question element would have the question paragraph style and an answer element would have an answer paragraph style. However, one level of grouping above this is the item element. There is no way to achieve this grouping as there is no concept of a paragraph group.

Future Proof

XML is independent of the application that created it. Thus if the vendor/developer of my editor or database or browser disappears, access to my data does not disappear with them. A number of SGML companies have come and gone over the years but the data their tools created never became "legacy data". Likewise, XML data will never become legacy data. It will always be possible to programmatically extract the content and structure of an XML document. Always.

Content Reuse

XML documents are essentially hierarchical databases. Information in them can be programmatically located, harvested and re-used over and over again. Contrast this with the typical single-use lifestyle of a WYSIWYG document.

With XML it becomes feasible to target multiple output formats from a single XML source document. Moreover, it becomes feasible to do it in a completely automated fashion.

Expertise and Tools

There is a wide variety of XML tools, techniques and expertise available. There is great support for it in Python and other languages such as Java and C. There is a growing list of end-user tools such as editors, databases, browsers etc.

Support standards

There is a growing set of XML support standards: XSL and CSS2 for creating style sheets, XLink for powerful hypertext linking etc.

Industry adoption

XML is popping up all over the place. Everywhere from E-commerce to Corba/COM bridges to bookmark exchange languages.

LumberJack is the name of a Python package for SGML/XML processing developed by the authors over the years for use in electronic publishing production systems.

LumberJack does not directly include any SGML or XML parsers. It communicates with parsers via an extended ESIS style notation expressed as simple Python tuples.

LumberJack supports both event-driven (SAX style) and tree processing (DOM style) programming. Furthermore, it is possible to switch from SAX style to DOM style processing on-the-fly.

BEEs (Bundled ESIS Events)

Simple BEE Programming with LumberJack

Event driven programming is a natural and common technique for XML/SGML programming. The idea is that as nodes are encountered in a depth first, left to right walk of the hierarchy described by the document, events are dispatched to handling methods with certain names. If a FOO element opens then a method FOO_HANDLER() is called. For data, a HANDLE_DATA() method is called.

Note the unpleasant state variables in the event driven example in the previous section. A Boolean is used to retain state information about whether or not the parse is within a title element or not. Also, a string variable is required to accumulate data as multiple data events might be dispatched for any one title element. When the number of such variables is low, they are manageable but it rapidly gets messy when the number of variables increases.

The need for such state-space variables is largely removed if random access to the tree structure is available. This is the most powerful (and also the most resource intensive) form of XML processing with LumberJack. The entire hierarchy represented by the XML document is turned into a random access data structure known as an LJTree. LJTree objects have a rich set of navigation and cut/paste functionality and make extensive use of Python lists in creating lists of interesting nodes from LJTrees.

Switching between SAX style and DOM style

The process culminated in the design of three DTDs for the project:-

A file naming convention was used to associate each debate file with its corresponding volume and date. For example this filename:

is interpreted as the debate record for the 16^th of November 1978 occurring in debate volume 309.

The Index DTD: This is the DTD used to capture the index pages at the back of each volume.

The data capture to XML was outsourced to a specialist data entry vendor. Outsourcing such a large and complex markup task presents many problems not least of which is ensuring that the markup used is the markup you would have used yourself.

There are many ways to mark up a document so that it is valid per the constraints expressed in the DTD but still contains structural errors.

DTDs allow for many, many combinations of elements to occur in XML documents. It is important that conversion software deal with all the required possibilities. We created a test suite of sample XML files that contained the hierarchical patterns most likely to occur in real XML files and used these whilst developing software for use later on in the process.

It is also important to get a feel for how big a resultant electronic database is going to be without having to wait perhaps 6 months for the data to be captured. Simply copying a handful of XML files over and over again does not work as compression rates and inverted index sizes will not be representative of real data.

To get around this, we created a small set of XML files to act as seed data for an obfuscating Python program. This program generated valid XML files but with different data content. It worked by modifying, deleting or inserting single characters from words. In doing so, it was necessary to avoid generating dangerous characters such as "<", ">" and "]". These could have caused the resultant XML files to be invalid.

Information Harvesting

During prototyping of the Folio Views deliverable, many questions arose that required harvesting reports from the XML repository. E.g. listing variations on speaker names, average length of speech and so on. Python/LumberJack was used as a very effective ad hoc query language for this.

Some of the report generation required dealing with multiple tree structures simultaneously. For example, traversing the project file, selecting files in particular volumes then performing simple regular expression matching on these files. The combination of LumberJack for tree navigation and Python's re module made this form of information harvesting much easier.

Folio Views Generation

Folio Views is a powerful platform but Folio Flat file is an unpleasant file format to generate directly from XML. Moreover, to get the most out of Folio’s advanced search features such as data types and fields you really need an "engine" to do a lot of housekeeping for you. We developed such an engine in Python specifically for this project.

To generate a Folio Infobase of exactly the same document collection:

Distributed Build

The file format in the last section also provided us with a convenient way of distributing the build across multiple machines.

To do a distributed build, we point a collection of PCs at the source directory for the document repository.

On my machine I type:

On my colleague Neville's machine I type:

Both of us share the drive Y: on which the XML repository sits. The make Python script when invoked with the -n switch only processes volumes with a machine id equal to the command line parameter.

Hypertext Management

For each volume of debate there is an index with tens of thousands of hypertext links. We needed a way to discover broken links prior to building to Folio Views.

We used a Python/LumberJack program to process the files for each volume and generate a report of any hypertext links that were broken. It turned out to be very useful to generate this as an XML file with hypertext links back to the source of the broken link. We could generate these link error files overnight and view them very easily in MultiDoc Pro.

Python and XML are to my mind, a marriage made in heaven. No software aspect of this project took more than one man week to prototype and a team of three programmers—sometimes working continents apart—could pick up code, understand it and be moving forward making changes to it very quickly.

Sure it is not fast. By writing it in, say C++ we could probably get the build time for each days debate to a matter of seconds. However it would have taken us many man weeks to write the code in C++.

The timing of this project criss-crossed the release of Python 1.5, the standardization of XML and the development of SAX and DOM.

If I had to do it all over again I would probably make the following changes:-

1) I would investigate the direct use of SAX and DOM rather than LumberJack. If SAX and DOM gain the sort of popular support that is hoped for them, I will find it easier to build development teams in the future when Python and XML become mainstream computing technologies.

2) I would investigate using an XML aware Web Browser and associated style sheet (CSS2 or XSL) to replace MultiDoc Pro as the viewing tool. At the time of writing, both Internet Explorer and Netscape are showing signs of direct support for XML rendering.

Finally, if XML continues its onward march, direct support for it in tools such as Folio Views, Internet Browsers and so on is just around the corner. When this happens the Irish Parliamentary Debate Record will be well positioned to take full advantage.

As for the production system, inevitably, it will change significantly over time to deal with new publishing requirements. We will end up writing lots of new stuff and abandoning lots of old stuff. We fully expect to have to rewrite it completely in the future at least once. In years gone by, when the electronic production systems were all in C, C++ or Perl, I would find such a scenario worrying.

This production system is in Python so I worry a lot less...

[8] A Python wrapper around James Clark's C based XML parser by Jack Jansen

[11] By Stefane Fermigier, fermigie@math.jussieu.fr