You're Asking for the Wrong Thing. When Pull Should Be Push.

There was a great discussion on Radio 4 last week about freeing up National Rail Enquires (NRE) departure board data for developers to use. There were some great points made by both sides. But the whole thing was quite black and white and didn’t really go into the more delicate and interesting issues.  I often find in these discussions that people conflate a feed being publicly available and having permission to use that feed. They also conflate a feed with static data. Nonetheless, it’s great to have these discussions out in the open in a civilised and constructive manner.

First, I’d like to make my background clear. In 2008 I worked for a company called Kizoom. Kizoom had been creating mobile applications, mobile websites and websites with train information for nearly 10 years. We’d had a long standing license with NRE to do this. In November 2008 I released MyRail Lite to the Apple AppStore. It displayed departure boards and calling patterns for UK trains. It was a moderate success and many people liked it. It was free too. We removed the application from the store in December 2008. I don’t want to go into the details of that here. It’s not really relevant to this discussion. However, I learnt many things from the episode. I learnt how to make iPhone applications. I learnt that business can be very personal. Personal in terms of making a connection with customers. Personal in terms of business relationships; both good and bad ones. MyRail Lite opened many doors and not long after I got to work on thetrainline’s iPhone application. Something I still do today. So on balance it was a great thing. It heartens me today that people still talk about it.

So, back to the National Rail Enquires data. To be honest, departure boards are not particularly interesting and the scope for doing exciting things is a little limited. It almost feels that once you’ve seen one departure board app, you’ve seen them all. [That’s a bit unfair, as the same does not seem to be true of Twitter clients]. I’m not saying you can’t do interesting things with departure board data, just that there is a far more exciting feed available.  As you may or may not be aware there is another way that NRE could give the data out to developers. Instead of the departure board, or pull feed as I shall call it, there is also a push feed. The push feed is like a fire hose (not sure what this means?) of train movements. As each train moves over sensors, periodically placed along the tracks, an event is generated (For more information see here [Page 9]). As you can imagine there are quite a lot of train movements. At Kizoom we saw the value of the push feed and built a really cool alerting engine with it. Internally we called it Clarence. On the outside we called it the Travel Angel. We could tell you when your train journey was going to be delayed via an SMS. We could even tell you when to get ready to change trains. We could do this because we knew where all the trains were. 

Another more subtle advantage of the push feed is that it gives NRE better load independence with respect to the number of apps in the wild. Each app developer would be given a single push feed connection. It is then your job as a developer to handle the transformation of the feed into interesting information and scale appropriately. So if you want to build a departure board feed then you’ll have to handle the transformations and scalability yourself. NRE are protected as each developer only gets one push feed connection. With the current pull feed, each user of an application is making a connection to NRE’s servers.

Yes, it will be harder to work with. Yes, you won’t be able to simply plug your app into the feed and spit the data out. Yes, you’ll have to think about how you’re going to scale. But the rewards could be great. You could do things like infer problems on the lines, keep historical data and make pretty graphs. You could even create an alerting engine. You could analyse what happens to trains when the weather is bad. Maybe there are new ways to help commuters in these situations? Is it really helpful to tell people the next train is in 95 minutes from Waterloo? What does that really mean?

So to summarise, it’s great that NRE are giving out licences to the departure board feed. But what I’d really like to see is the push feed opened up. Because I believe that with the push feed we’ll really see what can be done with this data. Let’s aim higher and look beyond the departure board.