Software | Secret Software | Writing

Managing the house with Perl

These days, a lot of the code that I write for myself, out of work time, comes as a result in changes in my life situation. When I went to Japan for a month, I wrote some code which helped me maintain a diary and newsletter. Recently, I've moved house, and now have the joy of housemates again.

On top of everything else, this means all sorts of daily tasks require additional administration - bills need to be divided up, the house network needs better organisation, there needs to be a shopping list for communally-bought items, and so on. Being a lazy hacker, I shun additional administration and code around it. And since there are quite a few overly-geeky houseshares around who might benefit from automating their admin, I took the time to write HouseShare.pm.

What It Does

HouseShare is a web-based administration system for a shared house. When it's completely finished, it will look after the network, the phone bill, the shopping list, and passes messages and information between housemates. At the moment, it does a reasonable chunk of those things.

When you first connect to HouseShare with your web browser, you'll see a menu a little like this:

    [ Image one ]

Here you see the front page for the Trinity House (that's my geekhouse) installation of HouseShare. At the bottom of the page are the latest blog entries - notes that all housemates should see.

Let's add a new computer to the house network, by following the link on the computer icon. This presents us with a list of the currently configured computers, and prompts us for information about a new one:

    [ Image two ]

You'll notice that the system also suggests the next available IP address for us. Hosts on the network can be renamed, reconfigured, or deleted; changes to the network will be reflected in the DNS server, which is controlled by the whole HouseShare application.

HouseShare is a modular system, and additional components can easily be added and updated. The phone bill and communal shopping modules haven't yet been written, although they have been designed and I'll talk about their opereation later on, but they will slot in with one additional database table and additional Perl module each.

How It Does It

The heart of the HouseShare system is a combination of two of the Perl modules I talk about most in these columns: Class::DBI, and the Template Toolkit. As Kake Pugh points out in http://www.perl.com/pub/a/2003/07/15/nocode.html, these two modules are almost made for each other, allowing you to go straight from a database to HTML with very little Perl in the middle.

Most of the magic which runs HouseShare is done in the appropriately named HouseShare::Magic class. This is a subclass of Class::DBI, which all the HouseShare classes inherit from. Its job is to provide all the bridging code necessary to get from the database to HTML output.

One of the most important methods it provides, for instance, is list. The various listing pages for computers, users and so on is all provided by this one list method in HouseShare::Magic. Even more interestingly, most of the pages are produced by exactly the same Template Toolkit template. This raises an obvious question: how does the template know whether or not it's dealing with a user or a computer, or something else?

Class::DBI helps with some of this, providing methods like columns which returns a list of a database table's columns. If we tell the template the names of each table's columns, we can write code like this to turn a list of objects into a table:

    [% FOR item = objects;
        "<tr>";
        FOR col = classmetadata.columns;
            NEXT IF col == "id";
            "<td>";item.$col;"</td>";
        END; 
        button(item, "edit");
        button(item, "delete");
        "</tr>";
    END %]

Another extremely useful piece of code is UNIVERSAL::moniker, which adds two methods to every class: moniker and plural_moniker. These methods transform a class name like HouseShare::Computer into computer and computers respectively.

Now code like

    <h2> Listing of all [% classmetadata.plural %]</h2>

will say "Listing of all computers" and "listing of all users". If we have a class like HouseShare::PhoneNumber which represents numbers that users have registered as having called recently, we can override the moniker and plural_moniker methods appropriately:

    package HouseShare::PhoneNumber;
    sub plural_moniker { "recently called phone numbers" }
    sub moniker { "phone number" }

and the same template will still make sense.

HouseShare::Magic contains a do-everything templating method, process, which finds the templates, sets up the Template object, and creates a default set of arguments for it to use. The more interesting of these are classmetadata. We've already seen the use of columns and plural; here's the classmetadata argument in full:

    $args->{classmetadata} = {
        name => $class,
        columns => [ $class->columns ],
        colnames => { $class->column_names },
        moniker => $class->moniker,
        plural  => $class->plural_moniker,
        description => $class->description
    };

Two methods in that metadata section may not be immediately recognisable: description and column_names are provided by HouseShare::Magic itself, and supposed to be overridden in child classes. Column names maps a database table's columns to names that are sensible for display; the default implementation just uppercases the first character:

    sub column_names { 
        my $class = shift; 
        map { $_ => ucfirst $_ } $class->columns 
    }

However, for some classes you'll want to specify more human-readable column names. For instance, in the computer table, the column for the IP address is ip. With the default version of column_names, this will come out as "Ip", which is horrific. Instead, we provide a better mapping:

    sub column_names {
        ip => "IP Address",
        hostname => "Hostname",
        owner => "Owner",
        comment => "Comment"
    }

Now our table can have a nice friendly header:

    <TR>
    [% FOR col = classmetadata.columns.list;
        NEXT IF col == "id";
        "<TH>"; classmetadata.colnames.$col; "</TH>";
    END %]

Similarly, description provides a human-readable description of what the class represents.

This more or less covers what process does, and everything else that spits out HTML is implemented in terms of that. For instance, the list method which produces the lists of things just looks like this:

    sub list {
        my $class = shift;
        $class->process("list", { objects => [$class->retrieve_all] });
    }

This looks for a template called "list", and passes as additional arguments an array called objects, which are all the table's rows.

As we've seen with our list template, we then go through all the columns of this class's database table, and asks each object for its details. This works perfectly for things like comments and IP addresses, but when I asked a computer for its owner, I was surprised to see that my computers had an owner of 1, rather than simon.

This is because, in the database schema, the owner is stored as a foreign key into the users table. We've set up a Class::DBI has-a relationship to say that each computer has an owner, and therefore, quite correctly, calling owner on the HouseShare::Computer object returns a HouseShare::User.

Unfortunately, this object stringifies to the ID, which is not what we want. (At least it doesn't stringify to HouseShare::User=HASH(0xgarbage), which would be even less useful.) We want to display the actual username.

There are two ways we could do this. I did it first a good way, and this helped me to see a better way. The good way is to allow each class to override the default template. We do this in the magic template processing method by providing a series of template search paths:

    my ($class, $name, $args) = @_;
    my $template = Template->new({ INCLUDE_PATH => [
        File::Spec->catdir($HouseShare::templatehome, $class->moniker),
        File::Spec->catdir($HouseShare::templatehome, "custom"),
        File::Spec->catdir($HouseShare::templatehome, "factory")
    ]});

This means if we call HouseShare::Computer->list, Template Toolkit will first look for templates in the /opt/houseshare/templates/computer directory, then in /opt/houseshare/templates/custom, (where installation-specific customisations can be made) and finally in /opt/houseshare/templates/factory, where the factory settings are found. This allowed me to put code into templates/computer/list to fiddle with the owner column:

    IF col == "owner"; item.owner.username; ELSE; item.$col; END;

Now we can have our HouseShare::Computer class-specific templates in one location, out of the way. That was the good way.

The better way is to realise that Class::DBI is only trying to be helpful when it stringifies a HouseShare::User object to the ID, and it could easily be persuaded to stringify it to something more useful instead. So, putting the following code in HouseShare::User:

    __PACKAGE__->columns(Stringify => qw[ username ]);

solves the problem without having to mess with specific templates.

Editing records

So much for displaying things. What about editing them? Well, there's the wonderful Class::DBI::FromCGI method, which turns a set of posted CGI form parameters into a Class::DBI object in your specified class, handling untaining via CGI::Untaint. That solves half the CGI problem, allowing you to create and update objects just be receiving form field values - the other half of the problem involves creating the CGI form in the first place. As it turns out, there's a nice, generic way we can do this too.

In the process of writing my houseshare application, I found myself writing the Class::DBI::AsForm module. This provides a to_cgi method, returning a hash mapping columns to HTML form elements.

If we feed this hash to our template too, we can create a generic form for adding entries to a database table like so:

    <h3>Add a new [%classmetadata.moniker%]</h3>
    <FORM METHOD="post">
        <INPUT TYPE="hidden" NAME="action" VALUE="create">
        <INPUT TYPE="hidden" NAME="class" VALUE="[%classmetadata.name%]">
         [% FOR col = classmetadata.columns;
            NEXT IF col == "id";
            "<b>";classmetadata.colnames.$col;"</b> : "; 
            classmetadata.cgi.$col;
            "<BR>";
            END;
        %]
        <INPUT TYPE="submit" NAME="create" VALUE="create">
    </FORM>

Editing objects is very similar: just replace the relevant row in the list table with a set of calls to to_field($col) on the object. This produces a HTML snippet for the column in question, optionally taking notice of has-a relationships. For instance, when we edit a computer, at some point our template will do the equivalent of

    object.to_field("owner")

The owner of a computer is a HouseShare::User, and to_field knows this, so it produces a drop-down box of the users, with the current owner selected:

    <select name="owner">
        <option value=1 selected> simon </option>
        <option value=2> heth </option>
        ...
    </select>

Hence the add box we used to add a new computer to the network was generated completely generically, using a generic template and no special code in the computer class.

We've mentioned briefly the FromCGI module which is used to process these forms when the data is returned. Here's the code which does this, again in the generic Magic class:

    sub do_edit {
        my $class = shift;
        my $r = Apache->request;
        my $obj = $class->retrieve(shift);
        my $h = CGI::Untaint->new(%{Apache::Request->new($r)->parms});
        $obj->update_from_cgi($h);
        $class->list;
    }

I've removed some of the error-checking code for the purposes of clarity: we'll be passed in an object ID by the front-end handler, and then CGI::Untaint reads and verifies the CGI form parameters. Sending this CGI::Untaint object to the update_from_cgi method, as provided by Class::DBI::FromCGI does the rest, and we direct the user back to the list page.

Identifying users

What other information do we feed to our magical process method? You'll notice that at the top right-hand corner of our page, there was a little box with our name, demonstrating that the system had recognised the current user. This is done by passing in a HouseShare::User instance in to the template arguments:

    $args->{me} = HouseShare::User->me;

The me method tries to work out which of the housemates the remote user viewing the page actually is. How can we do that? Well, given that we know about all of their computers and we can determine which IP address their browser connected from, we can tell who owns the computer making the request. This is obviously a weak form of authentication, but in a houseshare situation where everyone has physical access to each other's kneecaps - sorry, I mean, computers - there's not much point in having any stronger authentication.

On the other hand, it is important to ensure that this request is actually coming from inside the house's network. The last thing you want is some random stranger out there on the Internet messing with your milk budget. To demonstrate the HouseShare system to the world at large, I added a demo mode which means that people can access and view the web site, but not change anything.

To work out who the user is, we start by knowing their IP address, information we get from the environment:

    sub my_ip {
        return $ENV{'REMOTE_ADDR'} || 
        inet_ntoa(scalar gethostbyname(hostname() || "localhost"));
    }

The first line checks the REMOTE_ADDR as set by the web server; the second line assures that this function will still work properly when HouseShare routines are called from the command line. As well as helping with debugging, we'll see later that this overcomes an interesting problem...

Now we can ask the main HouseShare module for the house's network, and construct a NetAddr::IP representing the network range:

    sub me {
        my $class = shift;

        my $net = NetAddr::IP->new(HouseShare->config->{network});

Now, if our IP address is not in this range, we switch to demo mode and don't return a user:

    if (!$net->contains(NetAddr::IP->new($class->my_ip))) {
        $HouseShare::demo = 1; 
        return;
    }

And now we can see if we have a computer in the house registered to this IP:

    if (my @computer = HouseShare::Computer->search({ip => $class->my_ip})) {
        return $computer[0]->owner;
    }

The owner method will, quite properly, return a HouseShare::User so our work is done.

Now comes the interesting problem. Suppose you've just installed HouseShare, and you go to the web site and want to start adding computers and users. Unfortunately, the computer doesn't currently know who you are - and it can't look you up by computer, because you don't have any computers registered either! To bootstrap the system, the installation program prompts for the first user's information and creates a HouseShare::User object for them. From then on, any access from unregistered IP address inside the network is assumed to be from this first "administrator" user:

    my @users = $class->retrieve_all;
    return $users[0];

And that, basically, is the HouseShare::User class.

The Front-End

Finally, in our tour of the HouseShare classes, let's look at the front-end mod_perl handler which ties the whole system together.

Here's the entire handler:

    sub handler {
        my $r = shift;
        my @pi = split /\//, $r->uri();
        shift @pi while @pi and !$pi[0];
        @pi = qw(user process frontpage) unless @pi; 
        my $class = "HouseShare::".ucfirst(shift(@pi));
        my $method = shift @pi || "present";
        return DECLINED if !$class->require || !$class->can($method);
        $r->send_http_header("text/html");
        $class->$method(@pi);
        return OK; 
    }    

I will admit that this code is a little insecure, although it's not easy to see how to exploit it - you'd have to find a dangerous class method in a HouseShare class. This is not the way I would recommend you code, but it was a neat hack. The idea is that the URL http://houseshare.my.house/computer/edit/5 gets turned into HouseShare::Computer->edit(5);

If there isn't a method, we call the generic method present; this means that things like the HouseShare::Blog class (based on Bryar, the subject of my previous articles) can be accessed from http://houseshare.my.house/blog.

And if there isn't even a class, such as when we hit the front page, we're effectively sent to HouseShare::User->present("frontpage") which displays the frontpage template. (The choice of the user class to do this is somewhat arbitrary.)

One thing you might notice about that handler routine is that it's simple and compact, a theme which runs through the whole system - in fact, the system currently weighs in at only 250 lines of actual Perl code, and 300 lines in the templates. All the heavy lifting is either done with existing modules or abstracting tasks away to a generic layer, such as the Magic class.

Actually Doing Stuff

So far, we've discussed a lot of infrastructure - a framework for doing neat things with databases and the web. However, it's now very trivial to build on top of this framework to add "real work" functionality to HouseShare.

For instance, one piece of real work we can do with HouseShare::Computer is to update the DNS tables when a Computer object is added or modified. Thankfully, with Class::DBI's trigger support, this is a very simple matter. Assuming we have a subroutine called build_zonefile which does the equivalent of

    print $_->hostname, " IN A ", $_->ip, "\n"
        for HouseShare::Computer->retrieve_all;

(Except naturally with a little more smarts...) we can trivially arrange for this to be called when anything changes:

    __PACKAGE__->add_trigger(after_update => \&build_zonefile);
    __PACKAGE__->add_trigger(after_create => \&build_zonefile);

Now new hosts will automatically be added to the DNS server, and updates will automatically be reflected - as usual, with only a tiny bit of code.

What It Will Do Soon

HouseShare currently does around 50% of what I would like to see it do. It was very simple to plug in HouseShare::Blog as a 20-line subclass of Bryar to add a blog to the site, and an Apache::MP3 instance to share music around the house; I also plan to add a CGI::Wiki wiki to share information about who to call when the power fails, and so on. (Oh, hang on a second...)

The next big step will be integrating Tony Bowden's Data::BT::PhoneBill to download and parse phone bill data. The methodology here is similar to what we've seen so far: a Phonenumbers class and database table will register known numbers and associate them with people who are likely to have called them, and then a method on that class will grab the data, share out the cost and process a template displaying the results.

The final piece (for now) will be something to record purchases of house essentials and share the cost between the housemates; again this will be a simple class with a database table, and a class method to do the work. This simple approach can be extended to all kinds of functionality to the system - in fact, the framework we've drawn up can be used in a huge number of applications, and we use an (admittedly more complex) variant of the idea at work as a basis for all kinds of e-commerce sites.

HouseShare is available from my CVS repository at http://cvs.simon-cozens.org/viewcvs.cgi/?cvsroot=HouseShare; it's currently a little underdocumented but email me if you want to install it and hack on it and I'll help you through it.

I sometimes think that HouseShare is a little bit overkill for the job it does; it's currently slightly more useful than a whiteboard in the hall. But it's been a lot of fun, which is the main thing, and in the course of writing it, I've learnt a lot about Class::DBI, Template Toolkit, abstracting functionality away, and making generic templates do a wide range of tasks. I hope through reading this, you have, too.