Application Server Components

The Applications Server components are the core of the application development and supervisory control platform for the Devil Framework product line. They provide a unified environment for data collection, data historization, communication, process automation and applications deployment and integration.

Application Server component architecture Zoom in

A component is designed as a plugins-based system. Plugins can be added, removed and updated at run-time, manually or through scheduled policies. Different policies are supported so a component can change functionalities and configurations at scheduled times. Every functionality of a component is implemented via plugins.

Components are designed to work as a tree-like networked structure, with one master node (MCP) where all configuration and plugins installation and updates are performed. All other nodes (Collectors and Agents) automatically get proper configuration and software updates. New nodes can be added at any time to scale data processing, reach new geographic zones or to connect to new devices.

Tree-like distributed system topology Zoom in

Components are designed to work on insecure, unreliable and not-always-connected communication channels. All communication is encrypted (using the pycrypto module) and authenticated through an internal PKI infrastructure (all nodes have their PKI certificates). A store and forward mechanism is used either for the configuration and code propagation systems either for events transmission. Connections can be initiated from any component: parents can call children, and children can call their parent (connections can activated at scheduled times, when specified events occur or when special conditions are detected).

The Application Framework creates a distributed environment where all nodes, from a programming and operative point of view, are seen as an unique system exposing a single API. Operations can be performed on any child node from any parent node, transparently.

IceBridge Graphical Console

The IceBridge console is a multi-platform application written in Python (using the QT/PyQT toolkit/wrapper for the user interface) that provides all tools needed for development, configuration, management and interactive use of the Application Server system.

Like the Application Server's components, the console is a plugins-based system. Plugins are dynamically downloaded from the server when needed or when new versions are available. No user or administrator intervention is needed. Plugins can add, change, augment or hide any console functionality, providing a complete per-user customizable experience.

A complete configuration and visual development environment is integrated into the console (including a version control system for Views and an interactive remote Python shell). Visualization and control forms (called Views) can be added, modified and tested in real-time. New view components (Widgets) can dynamically be made available by plugins.

Real-time system configuration and interface design with "live" widgets Zoom in

Process specific widgets, views, windows and applications can be easily developed and deployed. The system is ready for internationalization; multiple languages are supported and every user can use its preferred language. Moreover, a translation tool (inspired by the QT Linguist Tool) and a WikiWiki-like documentation editing and viewing tool are integrated into the system.

Control view with a custom 3D visualization widget running on Linux and KDE Zoom in

Productivity

Python is our "secret weapon". It gives us order-of-magnitude increases in productivity thanks to:

• Simple syntax and clear coding style: it effortlessly forced us to use a consistent and expressive coding style. Now it's easy to look at a piece of code and understand at a glance how it works (nearly impossible with the first Perl version).
• Rapid development cycle: Python removed the tedious and long code/build/run/crash development cycle.
• It allowed us to development one of the hottest features of our system: when we upgrade or install some new code into the system, it dynamically updates itself, either the client and the server. Imagine this: with a simple double-click you install a plugin on the central server, update it (propagating the changes to all the nodes of the system that are using it), run it, download its client stuff into the console and make it active (dynamically updating the GUI). Everything without restarting nor the server nor the client.
• Extensive and complete standard library and great external modules.
• And when there's no standard library or external module, it's really easy to build wrappers around a third-party library (especially with the ctypes module)
• Easy and fast code refactoring: we have some old code that gets updated all the time.

Cross-platform Development

Porting our application to the different supported platforms was really simple thanks to the great portability of both Python and PyQT and most of the external modules we used in the project. The only real problems we had were caused from platform specific modules (mostly on Window) and from the development of the installation systems (Windows again).

GUI Development

The choice of the PyQT/QT toolkit was fundamental for the development of the client side application. We extensively used other toolkits (Tk and wxPython) before approaching PyQT. No toolkit can compare with its easy of use, power, stability and portability.

As an example, the Mac OS X port of the IceBridge console was possible in less than a week, with most of the time spent in learning how to correctly build the single packages from source and configure the packaging system.

The IceBridge console running under Mac OS X Zoom in

Speed, Scalability and Stability

One of our concerns was the speed of our system. Python proved to be "fast enough" for our uses as the most processor-intensive stuff is already written in C (GUI toolkit, GUID generators, numeric manipulations, etc.). We only had to write, in Python, our custom RPC protocol, because XML-RPC (and it's Python implementation) proved to be too slow and processor-intensive.

Scalability was an issue that we solved at design time, but Python let us create programming constructs like the transparent distributed RPC system and extensible API system.

In regard to stability, we think that Python's great exception handling mechanism gave us an enormous advantage. Our coding errors almost never stop the application and are just about always recoverable.

Real-life Applications

Some sample scenarios where we used the Devil Framework are:

• Management and control of all the technology systems for large-scale retailers from the local stores and the head offices (air-conditioning systems, power consumption control and reduction, data analysis, etc).
• Interface and integration of production systems and accounting systems for manufacturing groups with different production plants and business units located all over Europe (real-time price simulation, live production data analysis and visualization, automated anomaly detection and response, data correlation, etc.).
• Management and control of remote environmental monitoring units (GPS connections, automatic update and reconfiguration of the units, data normalization and analysis, etc.)