Planet CompSoc

Hello internet! I am at FOSDEM 2010 in Brussels. I tried the fosdem-maemo schedule application for my Nokia N900, and decided to write an alternative app which is easier to use with my fingers, and looks more like a Maemo application.

The result is foschart. It's just something I knocked together in a few hours yesterday, but it's pretty usable already. It supports showing talks grouped by track, by room, and just in chronological order, and a list of favourites. It's all happily kinetic-scrollable, etc., and is very snappy once it's started.

There's no proper release or package yet; if you want to package it up properly, please do! But for now, apt-get install python-hildon, then copy foschart.py and schedule.xml to /opt/foschart, and foschart.desktop to /usr/share/applications/hildon. Then it should show up in your application list, and away you go. As ever, patches welcome. Enjoy!

Update:

The illustrious Jonny Lamb has made a package!

2010 will be a bad year for IPv4 – this is exactly why I designated native IPv6 support as fundamental, not merely a “nice-to-have”, when setting up Splice last July. Hats of to Bytemark for supplying IPv6 with their hosting. I’m sure the fact that it’s excluded from their SLA is something that won’t be the case in eighteen months’ time, and meanwhile, I and the rest of the crew are using it quite happily.

Footnote: those of you reaching for the comments button to sarcastically remark that this website appears to not be available over IPv6 will be pleased to hear that I intend to fix this in the immediate future.

Why I've IPv6 enabled my server and how I did itWe hear the horror stories every now and then. We're running out of IPv4 space and soon the world as we know it is going to end. One day we'll run out of addresses and computers will have to live behind the evil jail of NAT. There is another way, and that is of course IPv6. ISPs are mostly burying their heads in the sand, and won't implement it until there are more IPv6 enabled websites. Webmasters are waiting on IPv6 addresses to be handed dished out by hosting companies who are too lazy to do anything until ISPs dish out addresses to surfers. A vicious cycle indeed. Luckily, we host with Bytemark, who dish out native IPv6 addresses so we don't have to rely on hackish Sixxs tunnels. I have a /56 on my VPS. The first thing to do was to to bring up 2 addresses in /etc/network/interfaces. I tried putting a separate ipv6 stanza (the way you're supposed to do it), so I used post-up commands by placing this under iface ethe0 inet static

  #Bring up additional ipv6 addresses on same if
  up ip -6 addr add 2001:41c8:10a:200::1/56 dev eth0
  up ip -6 ro add default via fe80::1 dev eth0
  up ip -6 addr add 2001:41c8:10a:200::2/56 dev eth0

I saved myself a reboot by also running the up commands on the command line. Just as I did for v4, I planned to use the first address for everything, except lighttpd, because Apache is already using port 80. I added an extra Listen line to /etc/apache2/ports.conf

Listen [2001:41c8:10a:200::1]:80

All of my vhosts listen on *:80, so that's good. Lighttpd's server.bind syntax only supports one bind address and port, but you can bind to multiple addresses like so:

server.use-ipv6 = "enable"
server.bind = "::ffff:212.110.165.233"
$SERVER["socket"] == "[2001:41c8:10a:200::2]:80" { }

Next I turned on ejabberd's IPv6 support by adding the inet6 keyword to thgis stanza in /etc/ejabberd/ejabberd.cfg:

{listen,
[
{5222, ejabberd_c2s, [inet6, {access, c2s}, {shaper, c2s_shaper}]},
...
]}

Next up wasmy IRC bouncer, ZNC. I'd told it to explicitly bind to a certain IP address so I could have a vanity address. That needed disabling so I can connect to IPv6-only IRC servers (which to be honest isn't going to happen anytime soon.) Postfix has IPv6 support since 2.2, and i have 2.5.5 so that should just work, as it currently binds to all addresses. For good measure, i added inet_protocols=all to /etc/postfix/main.cf For Dovecot, I added listen = [::] to /etc/dovecot/dovecot.conf. Note that listen = * refers to all IPv4 only. Bytemark's hosted TinyDNS servers support IPv6 records (prefix 6 for automatic rDNS, prefix 3 otherwise), but I stupidly totally forgot about this and used this generator to cook up some AAAA records to match my A records. A little testing with the SixXS IPv6-IPv4 and IPv4-IPv6 Website Gateway, which is basically an IPv6 -> IPv4 web proxy that refuses to retrieve anything hosted on IPv4, and I confirmed everything was good to. Stop Press! Aren't we forgetting something? In keeping with the tradition set by kame, and followed by Google and many others, I needed a bouncing logo thats only shown to surfers that connect via IPv4. Lucky I had an animated gif that I'd made earlier. In Django, you can do something like

":" in request.META["REMOTE_ADDR"]

to work out if your surfer is an IPv6 surfer. IPv4 users can sneak a peak at using the SIXXS gateway See it's that easy. If your host gives you IPv6 space, then you have no excuse not to be leading the way to the move to adopting IPv6.

My e-mail still isn’t as shiny as I’d like. In particular, my use of Exim Filters to sort incoming mail into folders lacks the ability to mark messages as read (although it’s still miles ahead of the dreaded Procmail). This would be handy for high-traffic mailing lists which I don’t have time to read on a daily basis, but which I find it hard to ignore the “unread” icon next to the folders for.

One day, I should probably move to using the Dovecot LDA and its sieve implementation, which supports the “imap4flags” extension, thus allowing marking messages as read, making them turn purple in Thunderbird, and all sorts of other cool stuff. Sadly, life (or this afternoon) is too short.

In the meantime, I’ve solved the problem in the usual way I deal with life’s imperfections: gratuitious Python – http://pastebin.org/77199 run from a crojob every five minutes.

(Disclaimer: letting scratty little bits of Python anywhere near something as important as your e-mail is probably a Very Bad Idea.)

The camera/photo viewer on the N900 has a pretty nice tag cloud widget, which lets you quickly label your photos before you upload them to Flickr. (The novelty hasn't yet worn off!) But an autocompletion accident left me with a tag in the widget that I'd really prefer not to be there when I'm showing off my nice new phone to people.

I spent a happy¹ few hours trying to figure out where it gets the set of tags from. The viewer asks Tracker for the most commonly-used tags, but this tag wasn't used on any of my photos, so wasn't coming from there. In fact, it didn't appear in any of Tracker's database files! After a bit of investigation, I discovered that the photo viewer keeps its own independent set of recently-used tags, not in Tracker, but in GConf, at /apps/osso/image-viewer/recent_tags. Lest you should find yourself in my position, a quick

gconftool --set --type list --list-type=string /apps/osso/image-viewer/recent_tags '[]'

will expunged your undesired utterances from the cloud. Bug report time. Next stop: finding a tool that lets the user remove typos from the autocompletion database …

1. Grr.

My home machine, on an ADSL line, runs an OpenVPN server, so I can connect remotely and access resources (because otherwise I'd need to use IPv6, or something).  Now, the OpenVPN server pushes various routes to remote clients, because my home network is a little complex.  Today, I finally got my netbook (runing Ubuntu 9.10 Karmic (netbook remix)) connecting properly to the VPN (turns out things are happier if you set comp-lzo consistently), only to discover that network-manager was routing _everything_ over it - not ideal, given the VPN server is on the thin end of the ADSL connection...

Anyway, turns out you can fix this by modifying the VPN connection in Network Manager, going to IPv4 Settings, changing the method to "Automatic (VPN) addresses only", then going to "Routes..." and ticking the box marked "Use this connection only for resources on its network".

And now I have speedy internets again (and a working sixxs tunnel)

Earlier this year, I hacked together a stateless RSS reader called Bogroll.

It’s been doing sterling service for me at news.dnorth.net ever since. Today, I’ve sorted out a 0.2 release with the following improvements:

• Now caches etags/Last-Modified headers to avoid fetching a feed if it hasn’t changed since last time (thank you, Mark Pilgrim, for chapter 14 of Dive Into Python 3, which reminded me to be a good citizen in this regard). I was pleased to discover that the Universal Feed Parser it’s built on top of already supports gzip and deflate compression to save bandwidth.
• Now supports just one category per feed, because having articles appear in several categories just seems wrong to me
• Each category now really does contain the most recent X articles from the relevant feeds, because I’ve fixed the severely broken sort-by-date logic

A fair bit of refactoring has gone on under the hood, and the code now looks a bit more like an app and less like a ten-minute bodge. The next round will involve getting some proper unit tests in place, and possibly AJAX magic to load the articles lazily on the page.

You can download the 0.2 zip, or get the latest version from subversion if you like to live dangerously. The cool kids all seem to be using Git or Mercurial these days, but I haven’t found the need (or overcome the inertia) yet.

Enjoy. Feedback welcome to the usual address.

So the turkey is (at least partly) eaten, the mince pies are disappearing fast, the wrapping paper has been picked up off the floor and the presents played with. 2009 is done.

It’s hard to say what I’ll remember 2009 most for, because it’s been such a packed year for me. Perhaps I’ll remeber it as the last of my three happy years at Magdalen, the year I finished my degree, the year I set up a hosting co-operative with six friends, the year I moved away from home, the year I got a job or the year I took over as joint church treasurer.

It wasn’t a perfect year, of course – my personal TODO list is still 39 lines long – but never mind, I’m fairly sure there’s another year just around the corner. One item a week. What could possibly go wrong?

Happy New Year to all 3.5 of my readers (and all those following along via Facebook; do come and read the website this is all automatically imported from), and I look forward to seeing as many of you as possible in 2010.

During the planning of our Christmas card at work this year, a mad idea came up. Do we ignore mad ideas? No, we tackle them head-on.

The idea was to build an internet-controlled snow machine - you'd hit the button on the website, and watch a member of the Torchbox team get pelted with snow. When we first came up with the idea, we dismissed it as being "too complex", but after a while, we came around.

Cue three days of frantic development and phoning round to get the parts. I'll be posting a full build article, with all our source code, once we're done.

The brief summary is that we have a Django app which handles rate-limiting of snow, and tracking who has clicked the button, which then communicates with our snow machine using Artnet and DMX. Ustream is used to stream the video back to the internet.

Still, I imagine you want to see it in action, so head over to snow.torchbox.com, and have a go. It's only online 10 - 5 UK time, and only until Tuesday (we can't fill our office up with paper snow forever), but it's still very good fun, even if you don't know any of us!

Let’s be honest about this. Looking back, I should have known better. Nevertheless, as I describe the problem that ate far too many hours of my weekend, judge for yourself whether I was entirely to blame for How It Went…

The problem

A friend of mine has a laptop. It’s about five years old, and it runs Windows XP. This means, inevitably, that it’s a mess. My personal metric of measuring how rodgered a machine is by the number of icons in its system tray gave it a ten, and that’s pretty nasty. Nevertheless, until last Friday, there was nothing wrong with it that an uninstall fest followed by a defrag wouldn’t have fixed.

Enter the Internet Man

Last Friday, a chap called round to set up some broadband for my friend, who’d previously been on dial-up. I’m not going to name the ISP concerned, since it gives me more freedom to say nasty things about them, but suffice to say that they’re big enough that they really should have done better.

Their engineer fixed the simple(ish) problem with wiring which was stopping the broadband from working, and what he should have done at that point was connect the laptop to the router (less than six inches away on the same desk) using the supplied ethernet cable, and left. What he actually did was shove the supplied CD in the drive, which helpfully installed a few hundred megabytes of crapware onto the machine, then hooked it up to the broadband via wireless. Then he left.

Despite the above totally unnecessary shoving of stuff onto it, the laptop struggled manfully on (system tray count now up to 12) and seemed superficially fine.

Enter Dragon Naturally Speaking

My friend makes extensive use of Dragon NaturallySpeaking, a voice-recognition product which seems to knock the socks off everything else on the market when it comes to actually coping with different accents. And it was here that the problem first manifested itself: trying to use Dragon to voice-control Internet Explorer caused it to crash with an error message along the lines of the one described in this Microsoft knowledgebase article.

Enter the sucker, stage left

At this point, I fetched up, and agreed to see if I could fix the problem. Although the above KB article looked ideal, being the first hit on Google when I exercised the too-useful-to-be-documented Windows feature of “Ctrl-C copies the text of the active dialog box to the clipboard”, the hotfix it supplies claimed to be already present in service pack 3 of Windows XP. Just about the only virtue of this laptop was that it was fully up to date on patches and service packs, so what now?

Don’t press that button

Being rather short on ideas at this point, I decided to fire up the nearest thing broken Win32 boxes have to a magic bullet, namely System Restore. The machine refused to roll back to any of the restore points at first, but restarting into safe mode fixed that, and it was soon rolled back to the Friday, at a time before the problem occured.

Unfortunately, Dragon now seemed completely broken, giving the error message described in this support article. And no, of course there were no backups, I hadn’t taken one before I started, and my friend is no different to most of the non-geeky people I know in not backing up, except for dragging his holiday snaps onto CD once every six months.

To their credit (and they’re about the only players in this story to be awarded any), Nuance’s suggestion in the article of how to manually restore the user files for Dragon did work, after I rolled back the fatal system restore [or rather, didn't, because it didn't seem to have made a pre-restore restore point. Fortunately picking one from the Saturday seemed to work].

So, after two hours feeling my friend’s anxiety at the thought of having to retrain the speech recog from scratch – not a pleasant accompinement to the sick, swoopy feeling we get when we know we’ve just permanently erased some irreplaceable data – we were back at square one with the original problem. One last shot in  the dark, disabling the ISP’s nasty extensions to IE, seemed to fix the issue.

So who do we blame here? Laptop vendors, for selling machines so laden with rubbish before they even leave the factory that DLL hell seems assured the minute they meet with real life usage? Or, we could blame the idiots who seem to write the nasty unsigned drivers for most hardware on the market. We can definitely blame the ISP’s engineer for installing the crapware, but perhaps he’d been trained to, and anyway, why do ISPs think we need a CD full of crap to supplement the TCP/IP standard that’s been around for several decades? Is it really asking too much of Johnny User to plug in a cable or enter some Wifi passwords in to the applet that’s sodding well supplied with Windows, thus making the poorly writtten replacements from laptop manufacturers and ISPs alike completely superfluous? We could also blame Microsoft for making system restore not clever enough to cope with software like Dragon. Or possibly blame Nuance for not fixing or documenting what has apparently been a known incompatability for several versions of Dragon*.

Certainly, we can blame me. I clearly need to have “I will not agree to even slightly ‘fix’ someone else’s computer without taking a full disk-image of it first” tattooed across my forehead. I also clearly need to reimmerse myself in the happy world of properly written software which I’m lucky enough to earn a living in and try to forget the horrors of the last 48 hours.

* The most useful reference I could find on Google was this page. The Nuance KB doesn’t mention system restore. Then again, perhaps their customers simply don’t know this is the cause of the issue, or don’t get round to reporting it. It’s not like I have.

A couple of days ago, I released version 0.2.1 of Bustle, someone's favourite D-Bus profiler. As the version number suggests, there aren't really any big new features; most of the changes just make it a bit nicer to use, like showing you all the bus names a service owns, ellipsizing strings, a slightly less spartan UI, etc. Having finally gotten around to cutting a release, I've started wondering what to work on next. There are various small things I have in mind, such as searching, filtering, integrating the various statistic tools (bustle-time and friends) into the UI, and so on, but it'd be nice to have a larger goal to work towards.

One recurring feature request is the ability to see messages' arguments. This isn't currently possible because the simple plain-text logs produced by the monitor (which is a variation on the theme of dbus-monitor --profile) only includes the message header. I've thought for a while that the right thing to do would be to log the raw dbus messages, together with a timestamp, but wasn't sure what the files would look like. (Maybe shove the timestamps into the message headers?) Rob had a nice idea: why not log to pcap files? This avoids inventing a new format—the UI would just use libpcap and feed each message through the dbus parser—and would also let you look at the logs in WireShark, if you're into that kind of thing. I'm hoping to find some time to give this a shot soon. (Maybe on a cold Christmas evening, in front of a fire?)

In the meantime, have a peek at what your D-Bus-using applications are up to, and let me know what's missing!

Or "why your mail from hotmail may not reach me"Lately I've been having a major problem with spam sent from hotmail addresses using hotmail's own SMTP servers. This has evaded sanesecurity's signatures and SpamAssassin, even though I've fed about 100 of these messages to sa-learn and also given some to steveb at sanesecurity. Some of these new spams I've been getting even have a negative SpamAssassin score! They are mostly random waffle (not 411, viagra or sex), with a hyperlink embedded. My email address is never in the To: or Cc: fields.. I have tried contacting Hotmail in the past about issues such as this, and its like talking to a brick wall. Previously, the most obvious spam was rejected by our SMTP servers at SMTP-time, and that which was likely to be spam, but not certain, was delivered to a Junk folder. Now, I am going to configure my server to accept then silently drop all email which is routed via a hotmail.com SMTP server where my name is not in the To: or Cc: fields. (Technical issues with the way the SMTP servers are setup prevent me from rejecting it at SMTP-time at present.) This means that if you use Windows Live mail to Bcc me an email then I will no longer receive it. You will not receive an automated reply, in the interests of reducing backscatter.

I’ve long been of the opinion that Firefox and Thunderbird are a bit like democracy: aclaimed far and wide as major achievements and bastions of a civilised society, but actually, honestly, a bit crap in many ways. Sadly, we’re stuck with all three until someone manages to come up with some compelling alternatives.

If you’re reaching for your e-mail client at this point to tell me I’m being unduly harsh, look me in the eye and tell me you think the way Firefox cheerfully caches DNS lookups and ignores such things as TTLs is a good idea. Or, even harder, give me one good reason why Thunderbird doesn’t check all IMAP folders for new messages by default. You can’t; in both cases it’s a disgrace.

All this being the case, I wasn’t what you’d call hopeful when, last Sunday, I had to write some e-mails on a train. And to do so, I needed to refer to some other e-mails in my Inbox. Since said train lacked anything as useful as a wireless internet service*, I’d need some sort of offline IMAP facility**.

As I bashed Thunderbird offline IMAP into Google, I was expecting a half-baked plugin at best, and “can’t be done” at worst. What I was actually very pleasantly surprised to find is that this functionality is built into Thunderbird.

What astonished me even more is that it actually worked. Faultlessly. So perhaps democracy can be salvaged after all.

* And, let’s be honest, because I still haven’t got organised and bought a phone with internet capabilities.

** Nobody uses POP3 in the twenty-first century, right?

In my neverending quest to save the time of those using RDBMSen, South 0.6.2 is released.

It mostly contains bugfixes, but there's now support for extending introspection to third-party apps without having to edit their source.

I'll probably start working full steam on South 0.7 soon - there's a refactor of the migration engine (mostly done), of startmigration (not started), and a few changes to command names to make them nicer (the old ones will still work), and to the behaviour of default values.

I'm hoping 0.7 will be the last release before 1.0, at which point I will claim I've been working in an octal version numbering sequence all along.

An oft-requested feature in Empathy and Telepathy is support for OTR (Off The Record) encryption of messages, interoperating with the OTR plugin for Pidgin and other popular IM clients. We've been resisting implementing it so far, mainly because we think there are better ways to do end-to-end encryption of messages and audio and video calls over XMPP, which we hope to implement in the not too distant future.

However, a nice aspect of OTR as compared to other encryption solutions is that it allows you to plausibly deny having taken part in a conversation. We believe this to be an example of a wider trend towards deniability on the internet, a position which is backed up by the growing popularity of Tor, and by several modern browsers allowing you to cover your browsing tracks out of the box.

As a result, we've been working hard to help secure your privacy while you're using Empathy. We've had to do this quietly for various legal reasons, but we're proud to announce Empathy's new Private Mode. When enabled, your contact list will be anonymized, as will your entry on your contacts'. Thus, you can conduct conversations with anyone without fear of repercussions from their discovering your identity, or of anyone else knowing the conversation took place:

It's not obvious how to bring these privacy benefits to Jingle video calls. We came up with a technique we refer to as Kitten Secrecy (patents pending in all relevant countries), and managed to integrate it with Empathy with the help of our friends at Collabora Multimedia, who constructed a fantastic GStreamer element using only two leaky queues!

We think the results speak for themselves. The source is not quite ready for release yet, but (lawyers permitting) we hope it'll be public by the end of the month. Hope you can wait until then!

— the Telepathy and Empathy teams

---

If you're about to leave an angry comment:

At the Boston Gnome summit, Robert McQueen, Sjoerd Simons and I sketched out a plan for the API for end-to-end encryption of communications (implemented using XTLS, OTR or anything else) and how we'd implement this API for OTR. Work's just started on a challenge-response authentication API, which is a prerequisite. Stay tuned; or, jump onto the Telepathy list or #telepathy on Freenode if you're interested in helping out!

Dear lazyweb, does anyone else find that Spotify mysteriously hangs at the “logging in” stage when run under Wine? This suddenly happend on both my Linux boxes after it had been working fine for weeks.

Various posts on the Ubuntu forums suggested firewall issues, but nothing had changed, and Spotify continue to work on Windows on both machines.

In the end, I applied the nuclear fix of

$ mv ~/.wine ~/.wine-with-broken-spotify

and reinstalling Spotify from scratch, which worked. Might not be so handy for anyone with other apps under Wine, though.

Jason's hands were tired after typing two days of excellent notes on sessions at the Gnome Summit, so I took over writing up the Telepathy BOF (which was largely about Telepathy integration in Gnome Games).

Gnome Games's tubes code is broken because of Empathy moving to Mission Control 5, which broke the contact chooser they were using (which used Mission Control 4). A Canonical person (your scribe did not catch who it was) has written a contact selector in C which just uses telepathy-glib, which Gnome Games will use and then start working again. This widget could form the basis of the long-anticipated telepathy-gtk.

Rob pointed out that it's currently a bit of pain to request a channel for yourself: you can't just call one D-Bus method and get a channel back, you also have to implement a Client.Handler object on which MC will call the HandleChannels() method. Sjoerd noted that Empathy has helper code for doing this in simple cases, which could be moved to tp-glib (it's under the LGPL).

Jason wants a way to share high scores with your contacts. (Digression about a gnome-games high score server on gnome.org ensued, the notes of which your scribe lost in a kernel panic. One main point is that global high scores end up just featuring incredibly good scores and people setting their name to obscenities.) Jason wonders if g-g could publish your high scores to your contacts in the background?

• Rob and Will suggested that in principle this could use PEP on XMPP, which allows you to publish a blob of information to your XMPP server and be notified when your contacts' blobs change. (This is how Geolocation works in Empathy, for instance.) Unfortunately Google's server doesn't support PEP, so most people wouldn't be able to use this; plus, it would need an extension in Gabble.
• Another way to do it would be to initiate file transfers behind the scenes.
• Rob suggested using tubes. Gnome Games would come with a tiny daemon that handles tubes for the "org.gnome.games.highscores" service, serving up your high scores to anyone who asks. Then, when you open a high scores table, it would use ContactCapabilities to discover which of your online contacts support that service, and then ask them for their latest high scores (with a cache and a rate limit, obviously). It could show a throbber while it's doing this. Jason thinks this sounds like a good idea. Prerequisites: you need the latest MC and Gabble 0.9 in order for ContactCapabilities to work, but that's it.

J5 wondered if any g-g people are documenting how they're using tubes, because he was always confused by them, and he reckons this is a very important thing for app developers. Rob suggested pushing this into tp-book, and Sjoerd noted that Danielle has a helper which lets you say "give me a stream to this person" and get a GIOChannel back branch with methods to convert between telepathy address GValues and GSocketAddresses, which could conceivably be extended to set up a socket automatically. J5 thinks that if patterns for using tubes were really well documented, people would jump on the chance to use them.

Jason mentioned that people were discussing having a help option which jumps you to #gnome-games-$lang. J5 said that Ximian tried that, but found that people would just end up in empty chatrooms or paste goatse at each other. Integration with DevHelp would be nice to let people post examples etc.

J5 suggested that another good way to improve documentation is to make writing it a requirement for SoC. Rob noted that Telepathy hackers know that you need to use, eg., a Handler and a Tube, but it's hard for people really immersed in the stack to remember which prerequisites people need to learn in order to understand that stuff. Sandy said they'd been discussing documenting requirements and standards for SoC students, and thinks it's a great idea to ask students to blog stream-of-consciousness "this is what i did" updates. People have to make sure that they do this as they go along, because you forget the learning process after a few weeks.

Advantage of peer-to-peer high scores: you don't get the problem of one incredibly good person dominating. ajaxxx suggested that you could make high scores decay over time, or once you reach a particular level, to address this problem.

Tetrinet is latency-sensitive: will that be a problem with Telepathy, particularly using MUC tubes? Rob said just try it and if it's too slow it'll get faster as the implementation of Tubes improves. Sjoerd noted that for tetrinet you probably want to just export the Tetrinet server over a multi-user stream tube, rather than using d-tubes.

Telepathy should use UPnP to make FT and tubes fast in more cases. This is on the TODO list.

I'm just getting ready to fly away to Boston for the Gnome Summit. I'm looking forward to meeting people and seeing MIT, as well as getting the chance to spend more than a few hours in Boston (unlike every other time I've been there).

Inevitably, I haven't been organised enough to propose a Telepathy- or Empathy-related session, but Rob McQueen, Sjoerd Simons, Andres Salomon, Dafydd Harries, Shaun McCance (when he's not busy running a pair of interesting-sounding documentation sessions!) and myself will be around if people are interested; maybe something will coalesce. If window manager theming is more your kind of bag, Thomas Thurman's running a session on CSS in Metacity/Mutter. It'd be great to talk about integrating IM with the Gnome Shell; Moblin's people panel and many parts of Maemo make interesting use of Telepathy, and it'd be nice to have something similar on the desktop.

Speaking of Maemo, going to Boston means I'm not at the Maemo Summit in Amsterdam, which is a real shame: I'd love to meet more of the Maemo community, hear what people have up their sleeves for the N900, and discuss how Telepathy could help. Happily, Marco Barisione's giving a talk about how Telepathy's used on Maemo, and how you can use it too; relatedly, Travis Reitter and Mathias Hasselmann will speak about the address book, one of the heaviest users of Telepathy. Also, Marc Ordinas i Llopis is hosting a BoF on extending the (frankly stunning) Hildon desktop, and Ian Monroe is giving a talk with Sergiy Dubovik about preparing Qt4 applications for Fremantle and Harmattan. I hear Philippe Kalaf is also floating around somewhere. ☺

See you in (the wrong) Cambridge!

In my opinion, there's just not enough Subversion servers on the internet.

That's an opinion you'll probably never hear me express - I much rather everyone move over to a DVCS (*ahem* mercurial) - but Subversion has, some would say quite rightly, earned its place as the dominant VCS for nearly every part of the IT community (apart from big, lumbering, financial companies).

Recently, I've been slowly switching away from subversion for my personal projects into Mercurial. This, in itself, a painless process, but with my public projects - particularly South - this has the unintended effect of slightly alienating some of my users. Most are alright with adapting - after all, it's quickly becoming the norm to use a DVCS - but there's still a few people left out.

One of the particular issues I have is with svn:externals. A lot of apps - including some we have at work - rely on svn:externals to pull in external dependencies into a libs folder along with the project itself. Externals is one of the few features of subversion that I thought was pretty much perfect, and it was sad to see my move to Mercurial break it.

This week, at DjangoCon, Chris Wanstrath did a nice talk on DVCSen. One of the questions led him onto hg-git - the awesome git backend plugin for Mercurial, that the GitHub guys wrote - and how they first investigated the idea of an svn gateway to expose their repositories transparently to subversion users. From what I gathered, subversion's wire format proved too tricky to deal with, and so they turned elsewhere.

However, the idea of a subversion gateway intrigued me greatly. What better way to transparently serve South so people can still pull it using svn:externals while still developing in Mercurial? With that in mind, I started looking into Subversion's wire protocols and their delta format (svndiff) sometime yesterday afternoon.

After some digging, reading an obscure academic paper and liberal application of Wireshark, I grew confident enough that I could at least implement something. A day later, and I'd like to present the very first version of what I'm calling Heechee (if you get the pun, ten nerd points).

Heechee is a transparent mercurial-as-subversion gateway. It serves a Mercurial repository as a Subversion WebDAV-based repository. It's still in the early stages, but at the moment it will serve its own mercurial repository to subversion in such a way that you can check out the repository, and update to various revisions within it.

You can check it out at BitBucket. It's pretty alpha code, and make sure you have the dependencies mentioned in the README, but it works, which greatly surprises me. I plan to much improve the code to support more 'advanced' features, like being able to do more than checkout and update, as well as exposing tags and branches correctly. There's even the chance I'll stick Git support in, when I've had a play with Dulwich.

Still, hopefully I'll soon have it so it can serve South's code via svn, and thus restore harmony to my land of VCS users.

So, please, take the code and have a play with it. It's not formally licensed and released yet, but it will probably be Apache licensed. Feedback and suggestions for features, as well as how to linearise trees into svn histories, much welcomed!

Have verbal nagios alerts with festivalHere's a fragile script I cooked up to do it Verbal Nagios

South 0.6.1 is now available for public consumption; it fixes quite a few bugs.

If you want to see what's been fixed, have a gander at the Release Notes - there's a few significant bugs that have been fixed, including problems with SQLite, GeoDjango, and M2M fields.

Sorry for the short but sweet post - I'm in the process of writing my talk for DjangoCon! I'm giving a talk about South, with both a good grounding for those who haven't grokked migrations yet, as well as some tips on more advanced usage. Come watch it on Tuesday morning!

A script to automatically generate the shell prompt colour from the username and hostnameI like to have a different coloured shell prompt for every PC, so I don't end up executing commands on the wrong machine. Previously, I have edited .bashrc manually on every new machine, but it would be nice if this was done automatically when I check out my dot files from the repository onto the machine. If you make the following changes to your .bashrc, the colour and font-weight of the shell prompt will be chosen from a hash of your current username and the machines' hostname. My terminals have a black background. If yours have white, you might want to remove the white/yellow options and add black and brown instead, so the text shows up. Find if [ "$color_prompt" = yes ]; then change the next line to PS1='${debian_chroot:+($debian_chroot)}\[\033[`color_from_hostname`m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' Then, at the end of the file add: function color_from_hostname { hash=`(echo $USER; hostname) | md5sum | awk '{print $1}'` case ${hash#${hash%?}} in "0") c="1;34" ;; # Light blue "1") c="0;34" ;; # Dark blue "2") c="1;32" ;; # Light green "3") c="0;32" ;; # Dark green "4") c="1;36" ;; # Light cyan "5") c="0;36" ;; # Dark cyan "6") c="1;31" ;; # Light red "7") c="0;31" ;; # Dark red "8") c="1;35" ;; # Light purple "9") c="0;35" ;; # Dark purple "a") c="1;33" ;; # Light yellow "b") c="0;33" ;; # Dark yellow "c") c="1;37" ;; # Light white "d") c="0;37" ;; # Dark white # Run out of colours. Lets have some repeats "e") c="0;34" ;; "f") c="1;32" ;; esac echo $c }

Finally, Debian now does this when you /etc/init.d/apache2 reload, rather than silently failing and killing your websites.For a long time, I've been meaning to make a patch to the force-reload part of /etc/init.d/apache2, so that it performs an apache2 -t, to test if the new configuration files have valid syntax, and aborts the reload if there is an error, rather than silently killing your websites dead in their tracks. However, it seems that Debian or upstream have beaten me to it, as my init file (2.2.9-10+lenny2) seems to have this feature. However, I don't seem to test the syntax of a conf file without reloading Apache. I get this error when running apache2 -t: apache2: bad user name ${APACHE_RUN_USER} The following command does the right thing though: /usr/sbin/apache2ctl configtest

Some hints about using Backupninja to backup Debian servers to an Ubuntu boxAndrew and I currently use Backupninja for backing up our servers and laptops to our Ubuntu PVR, because this is the only machine at home thats guaranteed to be on 24-7. Backupninja was recommended to us by Michael and David. It runs on the servers that you want to back up, and uses a conf.d directory to store tasks, such including pgsql dumps which save to /var/backups, and then there is a final task which executes rdiff-backup to push over the files. As far as I can tell, rdiff-backup uses a combination of diffs, symlinks and rsync to have a snapshot history yet only use the bandwidth and disk space that a normal rsync would use. Backupninja insists that the version of rdiff-backup is the same at both ends, and although you can override it with the ignore-version conf file entry, I'd rather not as chaos and unrecoverable backups would probably follow. Since we had multiple Debian servers to be backed up, and only one one Ubuntu recipient, we chose to leave the Ubuntu boxes with the stock lenny version of rdiff-backup and change the Ubuntu version to match. (Also, rdiff-backup is not in lenny-backports) At the time of writing, Debian Lenny had v1.2.5 and Ubuntu Jaunty v1.2.7. We installed rdiff-backup_1.2.5-1build1_i386.deb on the Jaunty box but when the cron ran, got this lovely error: Fatal: rdiff-backup does not have the same version at the source and at the destination. After was not actually caused by the rdiff-backups having different versions, but by Ubuntu Bug 345086. Until v1.2.7, Ubuntu packaged rdiff-backup wrong. It installs to /usr/local/bin, not to /usr/bin as on Debian. As a result the Debian servers couldn't find the binary when connecting over ssh. I could have rebuilt the v1.2.5 deb from source so that it installs to /usr/bin, but in the end I settled on ln -s /usr/local/bin/rdiff-backup /usr/bin/rdiff-backup I've also filed a bug against Backupninja, with a patch so that it checks ssh's exit value and gives a more appropriate error if it can't find rdiff-backup in the right place on the remote server.

I've now made the move from svn to mercurial, so my repositories can now be found hereMercurial is the natural successor to svn - its commands are a superset so you don't have to relearn anything to get the same functioanlity. Also, it is distributed so you can check in changes offline to your local hg (Hg is the periodic table symbol for mercury) and then check them in remotely at a later time. I converted my svn projects to mercurial ones using this guide (I recommend you do it on your repo server or else it makes loads of connections and takes ages) and also installed hgweb using this guide

Need a Windows Vista DVD for its recovery mode, but your system manufacturor didn't give you one?Then nip over to Neosmart. They have torrents of the Windows Vista DVDs stripped of all the files that the installer uses, so the discs are only usable as recovery discs. Very useful if you mess up your Windows bootloader...

It is with great pleasure (and a measurable sense of relief) that I announce the release of South 0.6, a new release bringing quite a few new features, although most of them aren't immediately obvious.

Perhaps the biggest change is the move from parsing source files to get the field definitions (ugh, I know) to introspecting the fields to get their definitions. As well as being more stable and making my skin crawl less, this also means that dynamically-created models and fields are now fully supported.

South still falls back on the parser module for custom fields if it has to - since it doesn't know how to introspect them - but we encourage you to implement a South-readable interface for custom fields; see our wiki page explaining how to do that.

The other noticeable change is that the migrate command is much zippier, thanks mostly to the hard work and lazy-evaluation-patches of Juan and Simon from Akoha. Note that the SQL for migrations is no longer printed out as they're running any more - it was getting a bit verbose - but you can re-enable it with --verbosity=2. You can also completely shut South up using --verbosity=0, if you desire.

For easier conversion, there's a new one-shot convert_to_south command - take care to read the wiki page on that, however, as the caveat is that you must run a migrate --fake on any other computers running the same codebase. We're going to make that a bit more intuitive for the next release.

The release notes contain the rest of the changes - some big, some small - and remember, we've switched to Mercurial for revision control, so if you're still running off Subversion please change your checkouts.

After a long few months, during which I've increased my recommendations to run off trunk, South 0.6 is almost here.

This release contains a whole bunch of new features - more of which will be written up on the Release Notes page soon - but in the meantime, I've decided to adopt a more usual release process.

This means two things: First, I'm releasing a release candidate of South 0.6 for general testing and final critical-bug reporting; please see the Download page and follow the instructions to check out the 'stableish' tag, or if you already have a checkout please 'hg pull' and 'hg up' it.

The Git mirror is also, for once, up to date, and there's also a tar.gz available for those who don't fancy either DVCS.

The second change is that I'll actually maintain a stable branch from now on as well, much like Django. This means there will be a 0.6 branch for any fixes that are easy or trivial to backport, and in the unlikely case someone should manage to conceive of a security vulnerability in a migrations library fixes for that will go in too. This should hopefully mean you and your production servers can track a real release number and not miss out on bugfixes.

Note that a few of the planned features haven't made it into this release in time, but I felt I had to get something out there. My much-vaunted fixture migration didn't make it in, but introspection, the major feature, did.

Still, please, test the release! If things look good, I'll officially tag and release 0.6 early next week.

Just installed Ubuntu Jaunty fresh onto a new ext4 partition on my Advent 4211 netbook. Am delighted to report that WiFi, suspend and hibernate all work flawlessly out of the box, so I can finally stop using Windows on this machine.

It’s amazing what sort of shrinkage – both in size and price – you can miss if you’re not buying in a particular market. Yesterday, out for lunch with some friends, one of them showed off his latest toy, a 2GB USB drive. Which, as you can see even in these amateurish photos, is similar in size to an SD card:

It’s called the Tuff ‘n’ Tiny, apparently, and if you live in the UK, Ryman are selling the 2GB version online for £5.99 and two for a tenner on the high street.

I can also confirm that mine has partitioned quite happily into two FAT32 partitions, and, with prodding, boots Linux off one of them [the second. I had to use usb-creator, followed by install-mbr to get the bootloader to work]. I don’t think I’ll be following John’s suggestion of RAIDing two together just yet, but who knows…

People often waffle about whether Linux is ready for the desktop, so I thought I'd examine it the other way around. Be warned there is some humour intended here.I recently started working for a company which has a lot of Windows on its network, and although I could install any OS on my workstation, I decided to install Windows because Windows it allows Outlook and nicer integrating with the corporate Exchange server and other things such as easier remote management of Windows servers. Also, I thought that it would be less troublesome to use the same platform as my colleagues. Before I begin, you should note that I have used Windows only rarely in the past 3 years and am very used to the Linux way of doing things, but I don't think that this makes the examination unfair as most articles examining the suitability of Linux for the desktop are written by people who are used to Windows and assume that everyone else is. If they can be biased, then so can I :) No central repository: This makes installing software a pain as you have to visit each vendor's website or find their CD to install software. Dependencies aren't automatically installed, such as .net framework. Valve have proved with Steam that paid-for repositories can work for games, so why isn't there one for other software No middle click paste: Extremely annoying if you expect it to be there No SSH client out of the box: Linux distributions usually include some sort of program for accessing Windows servers over RDP, but Windows users are expected to find third-party software Random errors on clean installs: On a fresh install, Microsoft Virtual Server 2005 installed then moaned with a seemingly random error code. VirtualBox's Bridged Networking failed. Useless scripting languages out of the box: Windows machines don't include Perl or Python out of the box, only Batch scripting and VBScript (eeugh) More prone to viruses and crapware: Windows machines always need third-party resource-hogging virus scanners and firewalls Licensing overhead: Windows costs money and bugs you for a serial key on installation, which is an added hassle So, Windows fans, maybe you should consider looking in the mirror before laying into Linux. Windows felt much clunkier and awkward to use than Ubuntu to me, and as such I am declaring that from my point of view, Windows is not yet sufficiently polished to be deemed suitable for use on the Desktop

Just over 18 months ago, I signed up for a 256slice from Slicehost to host this website, my email, etc. Later this week, I shall be shutting the machine down and cancelling my account with them.

For the record, this has nothing to do with the level of service I’ve received from them – I’ve always found their support team quick to respond and helpful, and their articles site and wiki are both very handy.

However, a combination of fluctuations of the pound against the dollar, a surge in demand for RAM by my applications and sites, and my getting increasingly fed up with transatlantic ping times of 130ms meant the machine was becoming unfit for purpose and overloaded.

Thus, I have now found a reasonably cheap way to bring “my stuff” home to hosting in the UK. About which, I will be writing more shortly!

So goodbye, Slicehost, and keep up the good work. I’d certainly recommend you to anyone who lives in America and needs a VM.

And Ubuntu gets seven out of ten

Last week, it was clear my parents’ ageing XP box needed some attention – in fact, let’s be honest, it had been crying out for a zap-and-reload since last year.

This time, though, I was determined. With my sister safely moved off onto her new laptop, there should be no reason not to migrate the box to Ubuntu. The only things in use on it were Firefox, Thunderbird and Openoffice…so how hard could it be?

Five days on, I’m in a position to report that the answer is “harder than it should have been” – here’s what happened:

Problem 1: Audio CD autoplay

Dad wanted the machine to start playing his CDs automatically when he put them in, much like Windows does. Seemed perfectly reasonable to me, as I blithely assured him it’d be a matter of a few mouse clicks.

It wasn’t.

Much Googling and swearing later, this hack (fifth comment down) seems to do the necessary, though I had to bump the sleep interval to six seconds.

Problem 2: No matter how close KPatience comes, it’s not “real FreeCell”

…largely because, despite generating the same deals from the same numbers as the Windows version, it counts every loss of the same game number as a loss in the stats, whereas the Microsoft one counts them as one loss.

Fortunately, freecell.exe runs fine under wine, so copy it across and save yourself the bother.

In conclusion

Other than the above minor niggles, all seems well, and hopefully the machine should now do a few more years’ trouble-free duty.

So, as the title of this post says, Windows XP has now been deposed as my OS of choice for “real people”. Whoever would’a thought it?

Update, 18 July 2009

I forgot to say that probably the most impressive feature of the switch was that Ubuntu flawlessly detected and printed a test page on our trusty old Epson Stylus C62. That’s the sort of slickness we like to see.

Hello, internet! I'm at the Gran Canaria Desktop Summit. I'm a big fan of this "conference beside a beach" concept: interesting people, talks and projects, in the sun beside the Atlantic. There are a *lot* of Collabora people here; the head-count on the roof of the hotel last night exceeded 20, with still more people arriving today (and others stuck in Madrid). As ever, it's good to catch up with colleagues and others!

Bringing GUADEC and aKademy together seems to have been a partial success. It’s interesting to learn about projects from the other side of the fence, and I've had some productive conversations about the state of using Telepathy in KDE. That said, it’s a bit sad that so much of the cross-desktop schedule seemed to be a talk about a GNOME implementation of a concept, followed by a KDE implementation of the same idea. Maybe putting the duplicated effort in people's faces could help improve matters, but that seems unlikely.

Earlier today I gave a talk about profiling and optimizing D-Bus APIs (grab the slides!), including Bustle, a D-Bus visualization and profiling tool I've written to help visualize bus activity and find issues. I think it went quite well, and it lead to some interesting questions and discussions afterwards. In the talk, I mentioned a few examples of sub-ideal API design in other frameworks, but largely focused on the problems we've found in the Telepathy API, and how we’ve gone about fixing them. A couple of people mentioned that this might have reinforced the impression that Telepathy is really hard to use, which was not what I intended at all! On the contrary, the improvements I talked about have made Telepathy a lot more developer-friendly than it was when I first got involved: both the D-Bus API (if you ignore the deprecated bits) and the higher-level client-side convenience libraries (telepathy-glib and telepathy-qt4) have become significantly easier to use. It's telling that a lot of the changes that have made developers' lives easier have also made the D-Bus API more efficient, sane, and correct.

Of course, there's still room for improvement. As I mentioned, contact lists are quite awkward to work with, which is a shame because most Telepathy UIs would like to use contact lists. :-) libempathy-gtk has a couple of contact list widgets you could reuse, some of which I'd like to see in a smaller, API-stable libtelepathy-gtk; I hear that there's talk to extract the Python contact list widget that's being written for GNOME Games to a library that other Pythonic Telepathy applications (such as pyhallebard for which I can't find a website right now — thanks Anonymous!) could use.

In other news, Tracker 0.7 sounds (among other things) like a potential solution to the silly unfortunate situation where every music player uses its own library database, and thus your DAAP server either needs a separate database or to be built into your music player. Mojito looks pretty interesting; I'd love to see a UI for it on my desktop. It annoys me that I have to check so many disjoint sources at the moment. It might be nice if it could show me blog post titles from my LiveJournal or Facebook contacts; I suppose I should start hacking. :)

It's about time I posted an update on what's going on with South development, so here it is.

Firstly, there hasn't been much work done in the last few weeks, due to exams and there being a Ball to help run. Now that's over, development will crack down. 0.6 is going to focus on the following new features:

• Less parsing, more introspection. The default way of getting fields' definitions will be to introspect them; the implementation of this is pretty much complete already, and can be found in the 'noparsing' Mercurial branch. Custom fields (which cannot be introspected withour prior knowledge) have two options; provide a south_field_triple method to tell us their definition (preferred way), or as a fallback the parser will be used. This feature solves many issues, including dynamic models, and also doesn't lose the ability to work with most custom fields right from the start.
• Reverse dependencies. Just brought up by one of the Ella developers, this is a good idea that should have been in South a while back. You'll be able to specify 'this other migration depends on us' as well as 'we depend on that other migration' - this helps pluggable apps around a common core.
• Fixture migration. Since I know this is going to bite me in the ass at some point, I'll be rolling in the fixture-migration someone mentioned at EuroDjangoCon (uses the test runner to make a temporary database, loads an old fixture set, an old schema, migrates, dumps the new fixture). I haven't fully tested this yet, so it might not make it for reasons of impossibility, but I'm reasonably confident.
• Signals. There will be post-migrate and pre-migrate signals added to help developers add in custom code that needs to be run after migrations or before migrations (the latter helps people trying to carry out a deployable app rename, for example)

As well as those new features, there's also around 25 bugs that need fixing, and will be tackled soon, hopefully. I aim to get a beta release of these out in the next month, especially in the light of South's allegedly high penetration rate.

The last piece of news is that I'm moving my main development efforts to Mercurial (and specifically BitBucket), and only pushing mostly-stable trunk phases to Subversion. There's a github mirror of whatever hg-git deems to be main development, but that's for convenience only, and because I can do it in one command.

Fixing an ext3 partition that has corrupt superblocks and won't mount because the kernel thinks its ext4.

Yesterday, my USB ext3 500GB disk complained that it could not write any files because it was full. I deleted a few files, then had a look at df -h, which claimed i was using 16TB of a 16TB drive. I unmounted the drive, and then replugged it in. This time it did not auto-mount.

It turns out that some of the superblocks had become corrupted. fsck wouldn't run because it couldn't read the superblock. No worries, I thought. I'll just use dumpe2fs to tell me where the backup superblocks are and pass one of these as an argument with -b to e2fsck to force it to use one that was still intact. Only dumpe2fs failed with an error that I forgot to make a note of. Also, gparted claimed that the partition was ext4.

In the end I ran mkfs.ext3 -n /dev/sdb1. -n only displays what would be done if the disk was formatted, but does not actually format. One of the things it does display though, is the deterministic locations of where it would create the superblocks for a partition this size. Armed with this information, I could then try them one in turn with the -b switch on e2fsck until I stumbled across one that was intact. fsck found an awful lot of errors (probably because the disk was heavily in use when the problem occured), but I seem to have the majority of my data intact.

The moral of the story is to always have backups! You may not be as lucky as me.

I guess I might as well park these here as anywhere else:

Some hilarity courtesy of bash.org: #693046, #9214, #105199, #626249, #610952

Funny source code comments – “throw up”, snigger. Nothing like Java-based humour.

Batteries Feel Included: 307 – it’s a bit strange, but it made me nearly fall off my chair laughing

A genius article from 1995 explaining the trouble with the internet.

This post brought to you by the department of “I’ve finished finals and have absolutely naff all to do for the next week”.

I had a lot of interesting chats with people about South last week at EuroDjangoCon, and several eyebrows have been raised at me both parsing models and then storing their definitions as dicts.

While this is a design that is partially born of backwards compatibility, it was mostly a conscious design choice I had to make, and so I'd like to now describe why I made it.

Firstly, for those who aren't aware, South's magic autodetection and auto-writing-migration code is based off of a file called 'modelsparser.py', which does what it says on the tin; it opens models.py, parses it, and extracts definitions directly.

Now, my first response to this is also 'ew', but there's a reason for it: custom fields. If it wasn't for those, introspection of the existing fields would work just fine, given a long list of special cases (I in fact have a file called modelsinspector.py sitting around that does just that - if you're interested, it's in the 'noparsing' branch on BitBucket. It doesn't do much, but it proves the concept).

However, with custom fields in the game, you have two options: hope they follow Django conventions and try to introspect common options, or put the onus on custom field authors to add a hook that makes it spit out a definition. While I would obviously choose the second one (possibly added to some of the first), it simply doesn't provide the drop-in compatability I want South to provide.

However, as Alex Gaynor and several others have surmised, I could just pickle everything and stick that at the bottom. No introspection needed, no weird namespace errors because the models dictionary is simply a collection of eval-able strings.

However, I don't like pickle (the Python kind; the delicious food kind, on the other hand, is more than welcome in my household). The security issues don't matter here - after all, we're passing raw strings to eval instead - but it has two major features I don't like. Firstly, it's not at all human-readable - and while I don't imagine people editing their migrations by hand, I like to be able to see what is going on, and I like the fact that South still doesn't depend on you using startmigration - a design choice I'm very proud of.

Secondly, I've been burned by pickles in the past. If you remove a module that a pickled thing uses, pickle will crash and burn, and if you've not done them right, you won't know quite where the error happened. (For the purposes of argument, of course, I'm assuming I'd do it right, so this isn't really a point. Still...)

There's yet another option (see, my job isn't easy) - custom-pickling, which Alex suggested last Wednesday; it consists of roughly freezing the __dict__ of the field. This has some advantages - it's more human-readable - but you get the errors similar to my current approach (where datetime.datetime wasn't unfreezing if people didn't have datetime imported).

My main gripe, with both this and pickle, is that the internals of fields aren't guaranteed to be consistent between releases of Django, just the external API, and so South uses a method that lets it use the external API - in my mind, this is less hassle, since that's the thing that's really not going to change very much (and even if it does, you can easily see what South is trying to call, and run that in a shell yourself to see where the problem lies).

So, that's why models are frozen as a massive lump of dictionaries and triples. As with many things, it probably wouldn't be necessary if models had been done slightly differently (so there was a method that always spat out a nice, portable way to reincarnate a field), but with no reason for that previously, I would have been thoroughly impressed by the core developers' foresight if it existed.

As for parsing versus introspection... I'm still divided. Parsing always has, and still feels, wrong, but introspection might just be less reliable. If you're interested, go peek at the branch I mentioned above, and see what you make of it.

And now, I will stop blathering and return to my small hole at the base of the mighty Django Mountain, where I am attempting to fit square pegs into round holes. Good day.

Quick tip for anyone like me who’s foolish enough to cram Exim4, Spamassasin, Apache with Wordpress running on it, and a pile of other stuff on to a server with 256MB of RAM: Install one of the Wordpress caching plugins. Otherwise, your server is liable to start swapping frantically if you get a dozen or so hits on one of the Wordpress-powered sites.

Indeed, it was possible to nuke jasper just by mashing the F5 key whilst sitting on the front page of this site.

You can now sign in to leave comments here (yes, dear readers, both of you!) – using an OpenID if you have one, and in related news, I’ve got m’self a shiny new OpenID to go and do likewise on other people’s sites.

For those interested, the slides from my migrations talk at EuroDjangoCon are up. I believe the videos will be around soon.

Why settle for a doorbell when you could make a SIP-enabled intercom?

Revision for finals is boring me, so I thought I'd take a break and my mind started wondering... wouldn't it be cool to give door-to-door salesmen the runaround, analogous to that Asterisk script that sends known cold callers into menu hell. Of course this would be possible if your door entry system incorporated an intercom that was a SIP device.

By butchering a cheap £3 Argos phone which you wouldn't miss if it got vandalised/rained on and connecting it to an indoor ATA, the intercom would then have its own identity in the Asterisk PBX, and visitors could press some big green button which dialled '9' or something in the intercoms context. We could then ask visitors questions which they can answer using the touch-tone keypad, e.g. specifying who they want, if they're salesman etc. One could even change the extensions.conf on Halloween.

If you're not in (ie your RFID rabbit is not on the house mir:ror reader), the intercom could take a message which could be saved on the server, or even better, it could call your mobile phone and patch the extensions together if you really wanted. Obviously you'd combine this with my Asterical script so you don't get calls in the middle of something important.

If you had lots of time, and either owned your house or had an extremely nice landlord, you could make Asterisk call a script that controlled an arduino, which could physically open the door. This would almost certainly invalidate your insurance policies, but it would make it possible to let in callers from the comfort of your own chair, and even remotely by dialling a PIN number on the phone.

Of course this is all crazy and I don't really plan on doing it, but my blog needed a new post and this was all I could come up with.

As I write, the sun is up, the bells are ringing, and Oxford is dragging itself out of bed for another day. It’s slightly sobering to think that I’ve just listened to the college choir for my last May morning as an undergraduate…hopefully not my last time from inside college, though – time to start making friends with lower year groups.

I’m going to miss Magdalen. But meanwhile, eight weeks still to enjoy!

a.k.a. “How to query for all objects not in a given subclass”

I ran across this problem today – do please leave a comment if you know of a better way to solve it than what follows.

You have two Django models, one of which inherits from the other, e.g.:

class Order(models.Model):
    # some fields

class DiningOrder(Order):
    # some more fields

So, how do I write a query (using the Django ORM) which returns all the Orders which are not DiningOrders?

Apparently, this does the trick:

>>> Order.objects.all().count()
762L
>>> DiningOrder.objects.all().count()
93L
>>> Order.objects.exclude(id__in=[d.id for d in DiningOrder.objects.all()]).count()
669L

I'm pleased to announce the release of South 0.5; this has been a long and exciting release, with many new features...

• ORM Freezing, a new feature inspired by Migratory, which lets you freeze the historical state of your current models into a migration so you can then use them via an ORM as if your models.py file existed at that point in time. There's a good description of how exactly this is used in part 3 of the new tutorial.
• Automatic change detection; using the frozen model states, South can now diff your current models.py file against how it looked last time you made a migration, and write a migration to make up the difference. This essentially allows django-evolution style hinting, and means you'll only have to manually write more advanced migrations. It's discussed more in part two of the new tutorial.
• A much more robust models.py parser, using the Python parser module. It's a dirty secret that South automatically makes migrations by directly examining your models' source code; however, the new parsing engine hits 99% of all Django use cases, and has provisions for custom hooks so more wacky fields (such as django-denorm's decorated-function-based-fields) can tell South their equivalent definitions.
• A move into a south/ subdirectory, to fit with Python packaging guidelines, and make South more friendly with environments such as buildout and pip.

I've been very excited about the new features for a while, as those who have talked to me will tell you, and thanks to those people who used the trunk releases and submitted a great set of bug reports, it's finally ready for general consumption!

Additionally, for those going to EuroDjangoCon, don't miss my talk on migrations next Monday; as well as South and its new features, I'll be covering and comparing django-evolution and a few other migration solutions.

Happy migrating!

My main problem with RSS readers is that if I go away for a few days and come back, I don’t really have time to catch up with all the posts which came up whilst I was away. What I needed, therefore, was a stateless reader which would just show the newest 10 entries or so.

Add in category filtering and a web interface – such as it is – and you get Bogroll, which I’ve just released under the GPL. You can grab the 0.1 release, check out the code from subversion, and have a nosey at my installation.

Incidentally, if anyone can think of a better name for it, do get in touch.

Hands up who sees what the problem is with:Which obviously means "hey, you've done something stupid, and someone may be maliciously attempting to log in, when they shouldn't be able to, so for your sake I'm going to completely ignore the file".

The real question, of course, is why was the file in my configtool cache given the wrong permissions to start with?  I blame one of the various 2.6.26+xen-related crashes.

Returning things to normal (by nuking /var/lib/configtool/* and re-syncing) seems to have fixed the problem, allowing me to log in as root once again.

Stopping overnight in Reading last weekend, with a complete lack of reading material, I picked up an actual dead-tree copy of PC Pro magazine for the first time in years. I’ve read it on and off since discovering three years’ worth of back issues on a shelf at the back of my school library – the output of Jon Honeyball, in particular, is usually good for a laugh – followed, more often than not, by some serious thinking about the issues raised.

The column that most caught my eye this time, though, was one which explained the Rebel Simcard, which appears to offer a way of running an iPhone 3G on the network of your choice, without having to risk bricking the phone or voiding the warranty.

That’s good, but unfortulately, it’s still not good enough for me to send the £400 or so I have earmarked for a smartphone in Apple’s direction. It still doesn’t support tethering, the inability to run a SIP client in the background would be a huge annoyance, and developing for it would require the purchase of a £68 SDK (OK) and a £1000+ Mac to run it on (not OK, and neither is having to distribute code through the app store).

All of the above are things which the phone is technically capable of doing, but has been prevented from for various reasons. Meanwhile, my personal preference for keyboards – even small ones – over touchscreens which I struggle to operate with my medium-sized fingers, is another negative of the iPhone.

At the moment, I’m hanging on to the money and pinning my hopes on the Palm Pre, which should arrive in the UK later this year. If it’s as good as the hype says it is, supports tethering, and can be purchased on its own without an expensive contract I don’t need or want, it should fit the bill very nicely.

Memo to self for the future:On the other hand, I had 1.82TB in my crazy array for a while...

Growing xfs filesystems is easy - just remember to have the filesystem mounted:
xfs_grow /dev/vg0/data

When working on Telepathy, I've often wanted to be see which D-Bus methods are being called on whom, when signals are emitted, and so on. Timing information is also handy: I'd like to figure out why cold-starting Empathy takes 12 seconds, and it'd be much easier if I could look at a diagram rather than staring at the unreadable output of dbus-monitor.

Previously, Alban wrote a tool that used a patched version of mscgen, and produced appropriate input with a dbus-monitor-like Python script. I wanted some more D-Bus-specific diagrams, and ended up reimplementing both the monitoring component (by forking dbus-monitor, as its --profile output did not contain quite enough information) and the diagram-drawing component (using Cairo). I'm happy to present an initial release of Bustle:

There's a Telepathy-specific hack in the tool to shorten object paths, but it shouldn't make the tool any less useful for looking at other D-Bus traffic.

I haven't made binary packages yet, I'm afraid, so you'll need to grab the source tarball and build it if you want to try it out. In Debian-land, the dependencies are libdbus-1-dev libglib2.0-dev libghc6-mtl-dev libghc6-cairo-dev libghc6-gtk-dev libghc6-parsec-dev; see README in the source tree for how to build and use it.

The astute among you may have noticed from the dependencies that the diagram-building component is implemented in Haskell, using the excellent bindings to Gtk+ and Cairo. I got a prototype going within a few hours, and the strong correctness guarantees that the type system provides meant that I could refactor it mercilessly with confidence. I'm sure that I would have spent many frustrating hours chasing type bugs had I written it in Python, which is a more conventional high-level language for prototyping and writing tools like this. Next time you're frustrated by such bugs, you should give Haskell a try. :-)

Edit: Bustle now lives at willthompson.co.uk/bustle.

Let’s be honest, shall we: if you’re reading this, you’re probably the sort of person who’s been where I was last week: on the recieving end of a phone call from a friend/relative/friend of a friend/friend’s relative, enquiring if you could help them with a “computer problem”…

In this case, it all seemed pretty simple: the chap on the other end of the phone was rightly concerned that his second-hand Windows XP box had come without any antivirus software installed.

“No problem”, I blithely replied, “just install the free edition of AVG“.

And therein began the difficulty: no matter how carefully I spelled it out, the chap insisted that visiting the URL for AVG’s site resulted in a 404 error. Other sites were visible, but not that one. Emailing the link produced the same result, and so, thoroughly puzzled, I agreed to a site visit the next day.

Sure enough, from both IE and Firefox, the machine would not display pages from avg.com. Further experimentation revealed that most of the other antivirus vendors’ sites were similarly blocked, along with microsoft.com.

This rang faint bells with me – viruses have been blocking such sites for years – and full of confidence that I was going to be able to pull off a quick fix and look like a genius, I pulled up the C:\Windows\System32\Drivers\etc\hosts file, expecting to find a bunch of domains being redirected to 127.0.0.1. What I actually found was nothing which shouldn’t have been there.

So then I broke out the usual tools: Spybot, Rootkit Revealer, a manual installation of AVG via disk from another machine, HijackThis … several hours of the machine wrongly being pronounced clean later, I remembered reading that Microsoft’s OncCare sometimes managed to disinfect machines other products didn’t.

I downloaded the free trial (via another machine as the Microsoft site was being blocked) and fired it up. Its first quick scan found nothing, but it also recommended installing Internet Explorer 7. I think it must have been the running of the Microsoft malicious software removal tool during the IE install which finally disinfected the machine to the point where a full OneCare scan found and removed the last of what it proclaimed to be the Conficker worm.

More Windows-savvy readers might well be saying at this point that I should have recognised the Conficker symptoms earlier. The truth is, though, that after nearly three years using Linux on my machines, it’s starting to grate to have to find my way around Windows and fix knackered installs thereof, and I’ve been cheerfully avoiding articles about the latest threats in the Windows world, smug in the knowledge that it’s not my problem any more. .. until the phone rings.

It turns out that the aspect of the infection which had me confused the most – the blocking of the AV sites – is achieved by hooking into the relevant windows API calls and tampering with their behaviour. The question has to be asked, though – why the hell is it possible for code running as a limited user – or, come to that, any user – to intercept API calls like this?

Asside from the still-woeful state of its security, Windows really does feel prehistoric to me now – no central mechanism for installing and updating software, leading to every app having its own updater (launched at startup for maximum slowdown of the machine, of course), no decent command-line, and all the nonsense of antivirus, antispyware and anti-everything-else.

It must be ten years since I started getting the phone calls from people with broken machines, and yet it’s still happening. I saw the future of home computing last week, and it was a broken Windows box. Time to think the unthinkable and start installing Ubuntu as the standard remedy?

What do you do with images that render in Firefox fine, but appear as a red cross in Internet Explorer 6 & 7?

I was recently making West Lindsey Churches Festival 2009 website. Systematic Print, the company who designed and printed the brochure for the event, had supplied me with a DVD-R of all of the images used in the brochure, as large TIFs. Since there were 82 churches, obviously I wanted to batch convert these to 200pixel-wide JPEGs. My first port of call was ImageMagick's command-line convert utility. This appeared to work, except that the resulting JPEGs all seemed to be 500kB bigger than expected. (Even a 1x1 image was over 500kB). This was fixed using mogrify with the -strip parameter instead of convert.

All was going well until the site was tested in Internet Explorer 6 and 7, both of which didn't like these strange JPEGs, and kindly rendered red crosses instead. This is strange, as every other program (Firefox, GIMP, Windows Image & Fax Viewer etc) could render them fine.

My next port of call was GIMP. If an image converted by GIMP could be rendered by IE, then I could use GIMP's command-line batch mode to process all 82 images. In fact, there is already a set of Perl scripts called batchgimp would do the trick. Unfortunately, IE refused to render the images created by GIMP too.

In the end I had to reboot to Windows and download Irfanview, a free image viewing and editing program, which unfortunately doesn't "just run" under Wine. It has a batch mode that can be accessed by pressing the B key on the keyboard. I then ran ren *.jpg *.tif.jpg (Warning: globbing works differently in Windows shells. This can only done with a for loop in bash) on the resulting images to give them the names that convert had made, and used WinSCP to copy them direct to the webserver, overwriting the old images.

The moral of the story: Always check your websites in multiple browsers. Strange things can happen even if you dont't have any CSS or fancy stuff. And if you have strange images, give Irfanview a try. It worked for me.

Have you ever felt the burning need to ride aimlessly aimfully around the London Underground? Well, then I have good news for you.

My new in-development project is Mornington Square (holding page only so far!) - it's a adaptation of the classic game of Mornington Crescent, a game universally enjoyed by young and old.

The key part of the adaptation is that, unlike other online versions, this one is played by riding around the real underground. A GPS phone (and a location broker like Fire Eagle) updates the server with where you are (this does mean climbing out of stations each time you make a move).

The rough gameplay outline goes as follows (if you want just the technical nuggets, skip a bit):

First, contestants start either in the same location, or in predetermined equidistant-in-time-from-the-destination locations. They're all heading towards the same place - say, Mornington Crescent.

Before each move, the server analyses each player's location and assigns them a pseudorandom set of rules they must follow for the next move. Rules limit the lines you can travel on - so, for example, if you're told that the Occidental Grouping Restriction applies, you can't use Aldgate or Aldgate East. (The rules all have explanations for new players, and a map of possible moves, including restrictions imposed by rules, is shown)

The choice isn't random; the server will try several tens of sets of rules and select the one it think looks most appropriate (which is usually the one blocking any short routes - it knows.)

Then, each player looks at what moves they're allowed (you're always only allowed to move on one line at a time, so you have to choose an appropriate next stop), and gives the server a time estimate, and then they start moving.

Players must get to their next stop in the time they allocated; if they are late, they suffer a 5-minute time penalty. If they are early, they must wait until the end of the time they originally allocated - this encourages good time estimation!

The whole process repeats until someone makes it to the destination and their time runs out (or their penalty period does) - then they're the winner.

You're required to submit your location at the end of every move, to ensure you're in the right place. One possible measure for stopping location cheating is to ask the person to stand next to a local landmark you know the position of, but this shouldn't be a problem early on.

The server also has an overview page showing all the players, their current moves, and generally useful information.

So, it sounds good, but it actually works, as well! So far the rule-calculation code and map drawing code is all done, so it's just the location and timing left before there's a fully-functional beta, which I shall hopefully be demoing at BarCampLondon6 (possibly by actually playing it that evening).

The site is built using Django, of course, along with my usual use of Cairo for drawing the custom Underground maps for each player's move (they're geographical, and not the proper Underground style yet, however).

Rules are defined by exclusion functions - a rule can exclude any route, set of station names, or stations matching a regular expression. An A* algorithm does the route planning, scoring each ruleset based on the shortest route to each destination, allowing more time for interchanges between lines (I have basically invented a small travel planner).

I am perhaps too excited at the prospect of an excuse to go around riding the underground, and emerging from the surface, only to find that the server has decided that on the next move, wild lifts are high, and so I can't go through Covent Garden, but it'll all come out in testing (which I shall be doing in secret before the next DJUGL).

Yesterday night mostly featured Psapp at Cargo. I was expecting to have a great time, and they didn't disappoint: it was easily the best live show by a *tronica band I've seen. (Though, that's not really a fair comparison: all of their music is readily re-arrangable for a live band, particularly their more recent tracks.) Energetic, accomplished, quirky, etc. etc. Kazoos? With squeaky lobsters duct-taped to the side? Oh yes. Plus, closing on Everybody Wants To Be A Cat was a fun touch.

Meanwhile, back in a different genre, Pure Reason Revolution have just released their new album, Amor Vincit Omnia. Apparently, at some point in the three years since The Dark Third they decided that their largely guitar-based nü prog sound could be improved by, well, replacing it entirely with synth bass and some more synths and heavily processed downtuned guitars. To my surprise, it works pretty well: it's not what I was expecting, and it feels a bit overproduced, but they haven't got rid of the vocal harmonies, and the songs are still interesting. That said, for a band who go in for Latin titles and general pretentiousness, they should really have learned to pronounce all three words of Deus Ex Machina. It should be interesting to see if and how they adapt older songs to their new style live.

Which brings me to a question: dear readers, is anyone interested in seeing them play at Dingwalls on Tuesday? I have come to have a spare ticket, and it would be nice for it to be used. If so, let me know via whatever means you feel is the most appropriate.

Work on South has been going full steam ahead lately, with some good results.

Two major features that I've been wanting to implement for ages are now in SVN trunk, and due for imminent release with South 0.5. They are:

• ORM Freezing: Much like the strangely similar feature of Migratory (which is what inspired this), South now has the ability to 'freeze' models to make them accessible inside the migration, so you can write data migrations more easily. Things like fake models needed for the other ends of ForeignKeys (when creating tables and the like) are also now implemented using the same system, to make things cleaner.

• Automatic change detection: Before this was impossible, but now, if you freeze all of an app's models into a migration, South can compare that with the current state and automatically write a migration for the changes that have occurred. There's a setting to automatically freeze the app into every migration to make this more usable, too.

I'm very excited about these new features; they should make the next release of South one not to be missed. There's also other changes under the hood, as well, including a new, much more robust models.py parser. Documentation for these features is up already, and the tutorial will be extended (given a second part) soon, in order to cover them - they're features we want everyone using, since they'll save you time, and, hopefully stress.

If you feel like testing the new features before the next release, just check out trunk and have a play. Your feedback is also more than welcome on our new South mailing list, which you should also have a look at.

As many of you may know, My laptop's motherboard broke and I had to get it repaired. Here's my review of Fujitsu Siemens' in-warranty repair service.

The motherboard broke on a Thursday night, so I phoned Fujitsu Siemens on their geographic telephone number the very next day. The call was answered immediately, and after some basic remote diagnostics (take out the battery etc), I was given a reference number and told a courier would pick it up on Tuesday. Luckily I had backed up my hard drive, acquired a box, packaged the laptop nicely (customers are liable if it gets damaged due to inadequate packaging) and gave it to the porters late Sunday night, because by 11am on Monday UPS had already collected it, ahead of schedule, and left the tracking number with the porters.

Since Andrew used the service, it seems that the UK repair centre has closed. My laptop was sent to Sömmerda Bitronic in Germany, according to UPS' slip. On Bitronic's site you can track your repair's progress. In my case they had to order the part in which added an extra day. They also link to UPS tracking for the return journey. The laptop arrived back today, Tuesday, 8 days after collection, in a brand new box. They include a packing slip detailing what was repaired, and if the laptop had physical or water damage. My screen had been cleaned of all its dust, and most importantly, they didn't format my hard disk, which some other repair centres seem to do as standard practice, regardless of whether its necessary.

I am happy with Fujitsu Siemens service, and their subcontractor. The only minor problem is shipping to Germany adds time to wait to get your machine back.

I've always been somewhat envious when seeing the various sites with nice maps of crime, prices and other things for non-UK regions.

However, over the past week and a bit me and Martin have been changing that. WhoseTurf rolled out of the coding mill some time last week, and it's been improved a lot since, in our quest for useful (and fun) online geographical statistics for the UK.

GeoDjango has been on my list of things to play with properly for a long time, and so it was good fun to get my teeth into it. It works surprisingly well - the admin interface was really helpful when collating all our various area info (and even drawing some districts by hand - not a fun task when your source map is on a different projection).

At the core of WhoseTurf is a load of 'reviews'. Each review has a certain 'fanout' - the radius in which it can affect the ratings - and, accordingly, most of the work in our custom tile renderer is correctly interpolating between the correct review points and their values.

Having circles affecting values rather than the actual discrete shapes of the various UK districts makes for some slight inaccuracy, but for most variables this makes more sense than drawing it discretely (crime doesn't suddenly drop when you change county, for example). Council tax is one of the few exceptions to this, and so accordingly is drawn discretely.

Another clever trick we use is that, because the heatmaps we serve up just consist of tiles of large squares, we simply serve 40x40px images and let the browser scale them up to 256x256. This cuts down on rendering time and bandwidth, makes our cache (Varnish) happier, and also lets me crow on about how clever it is (even though it's reasonably obvious).

It turns out there's some things GeoDjango just can't do - like this massive query that makes correctly-shaped districts out of the various output areas we have (they don't fill the whole space, nor line up perfectly, so otherwise you get holes or hair-thin 'cracks'):

SELECT st_collect(cpoly) FROM (
        SELECT st_simplify(st_makepolygon(st_exteriorring(
        (st_dump(upoly)).geom )), 0.001) AS cpoly FROM (
            SELECT st_union(spoly) AS upoly FROM (
                SELECT st_snaptogrid(mpoly, 0.01) AS spoly
                FROM core_outputarea
                WHERE id LIKE '""" + code + """%%' AND
                st_isvalid(mpoly) = true)
                AS a
            ) AS b
        )
    AS c;

The other small problem was Firefox not having enough JS stack space to load in the border of the Highlands (it has lots of very silly little islands and glens), but I'm willing to let that one slide.

We'll probably be releasing the code behind the thing as open source; I doubt any of it is _that_ transferable, but there is all the code in there to render standard (spherical mercator) tiles using cairo and/or PIL, and if you too want to show lots of heatmaps, and perhaps kill your server, it could be for you.

One last side note: OpenLayers is, unfortunately, still not really useable. Google maps is a) smaller and b) more efficient (drags more smoothly, scroll zoom, although I did have to write our own layers widget). On the flip side, OpenStreetMap is getting more than useable for a site like ours now.

Do you find the GNOME date/time panel applet virtually useless during Oxford termtime? I've written one that displays the Oxford date instead. It's based on Andrew Godwin's oxdate python module, which is inspired by Dom's Oxford Dates perl module.

Download oxdate_1.0_all.deb, which will also install oxdate to your Python packages path, then right click a panel, choose Add to Panel... and select Oxford Date. If the .deb fails, you're not enlightened enough to be using a Debian-based distro, or you want to tinker with the source, its released under GPL v3 and is available here.

If you change the default date/time applet to only show time, and put the applets side-by-side, it'll look something like this:

Next time one of your application or websites goes wrong and the users shout at you, take comfort in the knowledge that even Google sometimes Get It Wrong…

(retrieved at 14:44 GMT on Saturday 31 January 2009)

Edit: As Michael points out in the comments, Oxford users can avoid the mess below by simply disabling hybrid mode.

So there I am, firing up the Advent to read my email between lectures when…

$ sudo vpn-connect oxford
vpnc-connect was built without openssl: Can't do hybrid or cert mode.

Gah? The VPN client has always worked in the past, but following the upgrade to Ubuntu Intrepid, it doesn’t. A quick search lead to a solution which can be summarised as:

$ cd /tmp
$ apt-get source vpnc
$ cd vpnc-0.5.1r275
$ $EDITOR Makefile

Uncomment the lines beginning with OPENSSL
Read this and despair for the future of open source software

$ sudo apt-get build-dep vpnc
$ dpkg-buildpkg

Wonder why it all collapses in a heap
Search and find this.

$ apt-get install libssl-dev
$ dpkg-buildpkg
$ cd ..
$ sudo dpkg -i vpnc_0.5.1r275-1ubuntu1_i386.deb

Wonder why this has changed between Hardy and Intrepid
Seriously consider going back to Windows on this machine.

Sigh.