Application Development Using IronPython

Application Development Using IronPython

Who is this guy?

I'm Menno Smits <http://freshfoo.com/>

I've been developing professionally with Python for about 10 years.

Most of my work has been on the Linux platform but I currently work at Resolver Systems on a product called Resolver One. It's a unique Python development environment with a familar spreadsheet user interface.

I maintain an open source Pythonic IMAP client library called imapclient.

What is this talk is about?

Please ask questions if I'm not being clear!

The .NET Framework

Before we talk about IronPython we need to talk about .NET. You've probably heard of .NET. It's Microsoft's big programming platform. It's actually a well thought out framework (if sometimes a little over-engineered) .

.NET is not tied to a particular programming language, although the defacto language is C#. There are many .NET languages: C#, IronRuby, IronScheme, F#, Boo, of course IronPython and many more.

All .NET programs, regardless of what language they are written in, run in a virtual machine called the Command Language Runtime (CLR). This virtual machine provides a number of useful facilities including automatic garbage collection and a powerful JIT compiler. The CLR executes bytecode called Common Intermediate Language (CIL) in an environment called the Common Language Infrastructure (CLI). Because of the CLR/CIL/CLI, modules written in different .NET languages can interoperate seamlessly.

An independent open-source implementation of .NET exists, called Mono. It is quite good but is not complete and tends to lag behind the Microsoft implementation.

So what is IronPython?

IronPython is a complete re-implementation of Python in C#. It is actually a compiler which converts Python code into .NET bytecode (CIL) in memory. The bytecode is passed to the CLR for execution.

IronPython started as an independent open source project started by Jim Hugunin but is now run as an open source project from within Microsoft. The open source license used is BSD-style. This means there are no restrictions for commercial use and no copyleft requirements.

IronPython runs on Mono so it is possibly to use it on non-Windows platforms.

Why Use IronPython? (1)

If you don't need .NET you don't want IronPython. IronPython is at its best for .NET programmers.

For .NET programmers, Python is nicer than C#. It is also a ready made scripting language for embedding in applications.

The .NET runtime (the CLR) has a highly optimised JIT compiler, and has seen a lot of work to ensure that multi-threaded applications can take advantage of multi-core processors. This is something that CPython can't do (for the foreseeable future) because of the Global Interpreter Lock.

IronPython integrates seamlessly with the .NET framework so native .NET types can be passed between IronPython and other code without conversion.

Third party components includes a huge range of sophisticated GUI components. Due to the Windows culture you usually have to pay for them! However we use a couple of big components in Resolver, and there just aren't equivalents available for CPython.

Why Use IronPython? (2)

IronPython can be embedded inside .NET programs written in other languages. This makes it an excellent scripting or extension language.

IronPython is much easier to extend with C# than CPython is with C. This makes it an attractive proposition for projects that need to add lots of performance critical extension code.

Now that IronPython falls under the Microsoft umbrella of products it may be easier to "sell" into certain corporate environments where well-known vendors are favoured.

Why Not Use IronPython?

IronPython runs the PyStone benchmark about thirty percent faster than CPython. However, according to the Computer Language Shootout (a better benchmark - but there is no such thing as a perfect benchmark) IronPython is generally a bit slower than CPython.

We have however seen interesting, and unexplained, speedups from time to time. IronPython does take advantage of the JIT compiler which is the likely cause of this.

Compiling Python code to modules is an intensive task, which can make startup times long for large programs. You can now pre-compile to assemblies to speed this up.

Mono is good, but still incomplete. .NET is a predominantly Windows platform.

Due to the way IronPython implements classes you can't create IronPython classes which can be consumed from another .NET language.

IronPython has no C-API. There are alternative implementations (and wrappers) of many C extensions from the standard library, but not for all of them.

This means it is currently not possible to import pre-compiled CPython extensions in IronPython. The IronClad project (started by Resolver Systems) aims to address this issue.

Installing IronPython

IronPython currently relies on the .NET 2.0 framework. If you have been keeping up with Windows updates it is likely that this is already installed on your system. If not, download it from Microsoft and install it.

IronPython is distributed as a zip archive. You download it and extract it to where ever is convenient. You may want to add the installation directory to the system-wide PATH environment variable.

If you want to use the standard Python library with IronPython (and you probably will), you'll need to install Python 2.4 and point the IRONPYTHONPATH environment variable to the root library directory (eg. c:Python24Lib). You'll then be able to import the standard library modules from within IronPython.

Interactive Shell (1)

ipy -D -X:TabCompletion -X:ColorfulConsole

One of the things we all love about Python is the interactive shell. IronPython has it's own shell which is quite similar to the CPython shell. It is started by running ipy.exe. By passing some extra command line options you get colours and tab completion.

Interactive Shell (2)

IronPython console screenshot

Using .NET Assemblies (1)

Using .NET Assemblies (2)

import clr
clr.AddReference('System')

from System.Threading import Thread

t = Thread()

Creating a Simple UI Dynamically

A Quick Demo

import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import (Application,
                                  Button, Form)

form = Form()

def onClick(sender, eventArgs):
    form.Text = 'Thanks for clicking!'

button = Button(Text="Please click")
button.Click += onClick

form.Controls.Add(button)

Application.Run(form)

This short program shows off some of the key concepts when programming with IronPython.

I'll now show you how this works live from the IronPython shell.

Note that although we can create UI elements from the shell it's usually better to use a visual designer tool instead. It quickly becomes tedious to lay out controls using code. We'll look at this later.

Let's Run It

Quick example created from the shell

Creating an Application

To illustrate some key concepts relating to IronPython development I'm going to show the code for a small application. This program recursively scans a directory tree and collects information about any MP3 audio files it finds.

MP3 files usually contain an ID3 tag which gives information about the song. In order to examine these tags I've used the ID3Sharp module written by Chris Woodbury.

Project Layout

mp3stats.py - the main GUI and app code
tags.py - simple class to collect ID3 tag stats
AddReferences.py - small module to load
                   external assemblies
CSharp/
    CSharp.csproj - Visual C# project file
    MainFormBase.cs - C# Base class
    ...
dll/
    CSharp.dll - .NET assembly containing the GUI.
                 Generated from C# sources above.
    ID3Sharp.dll - .NET assembly with ID3 tag routines

I've chosen this directory layout for the project. This layout is fairly arbitrary. The IronPython sources are all in the main directory. The CSharp sources and resources for the GUI are in the CSharp directory. The .NET assemblies that are used by the code are in the DLL directory.

Creating the GUI

It's usually easiest to use a visual designer tool to create the GUI. At Resolver Systems we use Visual Studio's form designer to create the form. This is then compiled to a .NET assembly which is imported by IronPython code.

Only the basic UI work is done in Visual Studio. UI behaviour is implemented in IronPython. We'll see how this is done in just a moment.

Creating the GUI

images/visualcsharp.gif

Setting GUI Properties

namespace CSharp
{
    partial class MainFormBase
    {
        ...
        public System.Windows.Forms.Button browseButton;
        public System.Windows.Forms.TextBox pathTextBox;
        public System.Windows.Forms.Button goButton;
        public System.Windows.Forms.Label artistsLabel;
        public System.Windows.Forms.Label totalLabel;
        public System.Windows.Forms.Label albumsLabel;
        public System.Windows.Forms.Label genresLabel;
        public System.Windows.Forms.Label missingTagsLabel;
        private System.Windows.Forms.Panel panel1;
        private System.Windows.Forms.Label label1;
        ...
    }
}

Here's a fragment of the code generated by Visual C# from the form designed using the Form designer. Notice how the attributes that need to be accessible from IronPython have been given friendly names and marked as public. If the attributes aren't public or protected then we obviously can't use them from IronPython.

AddReferences.py

import sys
import os

baseDir = os.path.split(os.path.abspath(__file__))[0]
dllDir = os.path.join(baseDir, 'dll')
sys.path.insert(0, dllDir)

import clr
clr.AddReference('System')
clr.AddReference('System.Windows.Forms')
clr.AddReferenceToFile('CSharp')
clr.AddReferenceToFile('ID3Sharp')

This small module adds references to the external assemblies that we use. I've put it in a separate module to avoid clutter in the main application code.

Although it looks a little scary it's actually fairly simple. It just inserts the dll directory into sys.path so that the .NET assemblies in that directory can be imported. Then, references to the standard .NET APIs and our own DLLs are added so that they can be imported from IronPython code.

Extracting ID3 Tags

import os
from ID3Sharp import ID3v2Tag, ID3v1Tag

class TagStats(object):
    FIELDS = ('Genre', 'Artist', 'Album')
    ...
    def collect(self, basePath):
        for mp3Path in self.walkAndMatch(basePath, self.isMp3):
            tag = ID3v2Tag.ReadTag(mp3Path)
            if tag is None:
                tag = ID3v1Tag.ReadTag(mp3Path)
            if tag is not None:
                for field in self.FIELDS:
                    fieldSet = getattr(self, field.lower()+'s')
                    fieldSet.add(getattr(tag, field))
                self.total += 1
            else:
                self.tagless += 1
    ...

Here's a shortened version of tags.py. It contains the TagStats class which recursively walks through a directory tree and collects statistics about any MP3 files that are found.

You can see how classes from the ID3Sharp module are simply imported and uses like any other Python class.

The code looks for ID3 version 2 tag first, falls back to version 1 if that fails and if no tag is present just records that no tag was found.

Extending the Form

import AddReferences
from System.Windows.Forms import (Application, DialogResult,
                                  FolderBrowserDialog)
from CSharp import MainFormBase

class MainForm(MainFormBase):

    def __init__(self):
        MainFormBase.__init__(self)

        self._folderDialog = FolderBrowserDialog()
        self._folderDialog.ShowNewFolderButton = False

        self.pathTextBox.Text = r'c:\...
        self.browseButton.Click += self.onBrowse
        self.goButton.Click += self.onGo
...

This is the first half of mp3stats.py.

The AddReferences module we looked at earlier is imported so that we can import the .NET assemblies required.

The GUI form class, MainFormBase, is imported from CSharp.dll and subclassed. The members that were created in Visual C# by adding GUI controls are immediately available as attributes of self. This includes the buttons and other controls that were marked as public.

Event handlers are added to the Browse and Go buttons. We'll look at these in just a moment.

So that the user has GUI for selecting a directory to start scanning at, a FolderBrowserDialog instance is created for later use. This class is from the standard .NET API in System.Windows.Forms.

The Browse Button

def onBrowse(self, _, __):
    self._folderDialog.SelectedPath = self.pathTextBox.Text
    if self._folderDialog.ShowDialog() == DialogResult.OK:
        self.pathTextBox.Text = self._folderDialog.SelectedPath

Here's the handler for the Browse button which we connected up in the form __init__(). This method is called when the Browse button is clicked.

It copies the path from the path text entry text box to the FolderBrowserDialog instance and causes the dialog to be displayed. This blocks the application until the dialog returns. If the user clicked OK (indicated by the return value of ShowDialog()) then we copy the path the user selected back to the path entry text box.

Pretty simple stuff.

The Go Button

def onGo(self, _, __):
    stats = TagStats()
    stats.collect(self.pathTextBox.Text)

    self.totalLabel.Text = str(stats.total)
    self.genresLabel.Text = str(len(stats.genres))
    self.artistsLabel.Text = str(len(stats.artists))
    self.albumsLabel.Text = str(len(stats.albums))
    self.missingTagsLabel.Text = str(stats.tagless)

This is the handler for the Go button that was connected up earlier. It's run when the Go button is clicked.

This creates an instance of the TagStats class we looked at earlier, tells it collect the stats and then updates the various labels on form to reflect what was found.

This basic implementation blocks while the statistics are gathered. If we had more time we could implement a more advanced implementation that would show progress. This could be done by making collect() a generator function which yields every directory.

Running the Application

Tell Windows what form to run and start the message loop so that events start being processed.

Application.Run(MainForm())

The last line in mp3stats.py calls Application.Run() with an instance of the form class. We saw this line in the earlier example. This initialises the application and starts the message loop. Once this is called the application acts on the events that we have set up handlers for.

How it Looks (1)

mp3Stats with folder selection dialog open.

How it Looks (2)

Showing some stats...

mp3Stats showing some results

A Real World IronPython App

To give you an idea of what can be achieved with IronPython I'll now show you a large real-world application written in IronPython. The application is Resolver One which is developed at my employer, Resolver Systems.

Resolver One is a spreadsheet application mixed with a Python development environment. The spreadsheet itself is represented as a Python program. It embeds an instances of IronPython to execute spreadsheet calculations.

It is probably the largest IronPython program around with 35690 lines of application code (8th May 2008).

Resolver One is strictly developed using eXtreme Programming technique such Test Driven Development, Pair Programming and rapid iterations. We have a 3:1 ratio of test:application code.

It is easy to import external Python modules from within your spreadsheet. This opens up all sorts of interesting ways to extend your spreadsheets. For example, import live data from the web and send results to a database.

Spreadsheet as Python Program

images/resolver-1.gif

In Resolver One, the spreadsheet is a Python program. As changes are made to the grid you can see the Python code update.

The formula language used is immediately familar to spreadsheet users and many of the common spreadsheet functions are available (eg. SUM)

The code pane at the bottom allows for entry of arbitrary (real) Python code. Some parts of the code are generated by Resolver One based on entries to the grid. These sections can't be modified directly by the user. Other sections of the code allow for free entry of Python code. This allows new spreadsheet functions to be defined and global operations to be performed on the spreadsheet from Python.

R1: Python Expressions & Objects

images/resolver-2.gif

Here's another example. A new function Square has been defined in the code pane using standard Python code.

Resolver One allows for any Python object to be inserted into a grid cell. This of course includes functions. Here I've stored the Square function into the cell D1 (using the formula =Square).

As well as a standard spreadsheet formula syntax, the Resolver One formula language also allows for Pythonic expressions such as list comprehension. In fact, the formula language is a superset of the Python syntax. It extends Python expressions, allowing for spreadsheet niceties such as references to cell ranges (eg. A1:C1)

Now let's look at the cell F1. You can see its expression in the formula bar. The cell contains a list comprehension which applies the cell D1 to the cell range A1:C1. Notice how this expression calls a cell; this causes object stored in the cell to be called which in this case is the Square function. This cool feature allows functional programming using the spreadsheet grid.

Getting Resolver One

That's it

The example code along with this presentation will be available at http://freshfoo.com/

Any questions?