data, general, programming

Who wants to write a 200 page report that no-one will ever read?

I was at the Data Without Borders DC Data Dive a couple of weeks back, definitely a trip out of my usual worlds of Java and travel IT and visit into something a bit different.

A couple of quotes stuck in my head from that weekend – the first was from one of the participating charities, DC Action for Kids whose representative, in their initial presentation, said that one of their major motivators for taking part in the event was (and I paraphrase)

we wanted to produce something more interesting with a two-hundred page report that no-one will ever read.

One of Saturday’s presenters picked this up and ran with it, showing how they believed that we were in the process of trying to find a new methods of deep data presentation at events like the DC Datadive. While this may be somewhat of an exaggeration, DC Action for Kids achieved their aim, check out the beautiful app that came from their datadive team.

It made me realise what a powerful difference you can make with such an app – reading a 200 page report is usually a pretty tiresome process, but exploring the data set in a visual form is not only a joy, but allows the app writer to guide the end users to the same conclusion from the data that they would like them to draw, while at the same time allowing the user to feel that they are verifying this conclusion themselves. In today’s media landscape where discerning readers place full trust in few, if any, publications, this is a pretty powerful tool! While I believe that app was coded by hand in the usual mix of CSS and HTML held together by Javascript, there’s surely a gap in the market for a powerpoint-style application to generate visual data exploration mini-apps.

The second observation that struck me over the weekend was again a quote that went something along the lines of

In the coming 2 decades, a statistician will be the trendy job to have, in the same way that it software engineer has been the trendy job for the previous two decades

Again, while I feel this is somewhat of an exaggeration (and as a software engineer, I’ll confess to bias), the event certainly outlined to me the power of inference from data… and the importance of keeping your data and keeping your data clean. Single examples from data are rarely sufficient to indicate an extrapolation and a zero value field is not the same as an empty one… they may sound like simple observations but as programmers it’s not unusual for us to treat these single examples as general trends and zero and null as being equivalent. But they’re not, and it’s really easy to mess up your conclusions as a result of these and many other simple but potentially incorrect assumptions. The weekend served as a reminder of these simple values to me – remembering these at coding time can make the difference between a piece of code that lasts 5 years instead of suffering death by maintenance in the first year of its life.

Finally, in addition to my languages-of-the-future predictions of javascript and erlang, I think I’ll have to add R. It’s such a powerful and easy language, I was really blown away by it


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s