I recently wrote a series on Open Data. In those posts I mentioned Adam Jennison and the work he has been doing. He’s written up the talk he gave to the Hull Digital Developer Group and added in his hopes for what he thinks could be possible if Hull City Council and the digital talent in the city invest in working together. So, over to Adam.
‘Geek meet and greet’
Underneath the Bridge by Vaidas M
I attend the regular geek meets run by Hull Digital as often as I can, not only to be able to geek out without the usual look of disdain but also to see how people on the ‘outside’ are working, how they are managing and what they see as the future..
Did I not mention that I am on what the media portray as the ‘darkside’? I am a public sector worker.. and worse than that I am a back office public sector worker, I work in ICT supporting front office workers… Yes I know I am lower than a snakes belly etc.. but hear me out for I feel, nay I believe that we can do good and also help local businesses lead the way.
I asked John Polling, the organiser of the Hull Digital Dev Group if I could follow up on a talk by one of my colleagues titled ‘Open Data’ with a talk of my own titled ‘RESTful APIs am i doing it rite?‘. This would centre around the problems associated with integration, data (vast amounts of data) and open data.
I started the talk off with a quick background to some of the issues that face public sector ICT and how it differs from private sector ICT. Asking for a show of hands from those who had ever worked in the public sector brought just one hand…I thought then that this would be a hard sell to get them to see beyond the headlines..
Discussing the statutory needs that the average Council has to provide along with the evidence/information they need to collect juxtaposed with the constant political pressure (both local and national) that weighs like a strong North Westerly on a unfolded sail I described the massive number of different systems that are present in most Local Govs.
People were shocked.
Most private companies are vertical players, they focus their business on a couple of key systems, they specialise. This makes total business sense, spreading your effort thinly means you are not very good at a lot of things instead of being good at a few. When I pointed out that we can not back out of our commitments to the elderly, to business, to schools, etc etc they’re statutory requirements that we need to fulfil regardless of money pressures, the audience were starting to see the dilemma about public ICT provision.
‘Public Sector Data’
You may ask where am I going with this conversation, well be patient! By providing this overview I set the scene to talk about not only the number of systems we have to overlord but also the MASSIVE amount of data that flows into, around and out of these systems. I have argued for a very long time that the public sector in general is very data rich but information poor.
As a business I would want key information to be available in near real time to be able to make informed, evidence based decisions that ensure my business heads in the right direction. If you can imagine it as akin to a salty seadog skipper making micro adjustments of the wheel of his vessel, feeling the water constantly flowing past the hull: he can feel the current, read the waters and keep his craft [business] from hitting those rocks.
If I were dealing with one, two or even three systems then this information collation would be difficult but doable because the metrics I’m reporting on are relatively limited: cash flow, manufacturing, leads, prospects, etc.
Trogdor (the Burninator) by Joey Day
But multiply those systems by 50 and you start to see the usual demand for the average Local Authority. Metrics ranging from number of children in care, allotment demand, street naming and numbering, dog fouling, monitoring of river flow, etc they all need reporting on.
How to build a dashboard for all this information, how to integrate it and how to make usable information from such a depth of numbers is where the real difficulty lies, as the old maps used to say – ‘DANGER – here be dragons.’
‘Integration – a different way?’
I spoke at length on integration, how the majority of integration was done as an integration ‘path’ with complexity included throughout the process from system A to system B (c,d,e etc). I chatted about how most integration is usually how system A speaks fluent ‘Klingon’ while system B speaks ‘Olde English’. What we do is add a Babel fish in the middle that converts these languages for each other. Adding to the public sector’s dilemma is our requirement to spend the public’s money carefully, this means every xx years we need to go to the market and ask – is this still the right way of solving this problem? Can it be done cheaper/better/differently?
Most of the time the answer is yes…but when you take into account the unravelling of any integration the project costs start to outweigh any improvement.
I proposed a RESTful style wrapper that would help reduce the complexity of integration and allow decoupling of systems from the integration path, well actually I argued you can never remove complexity just reduce its footpath to that of the API wrapper.
090827 Day 68 facts by dabblelicious
If system A can be wrapped with an API that publishes its resources in a standard way (using standards such as ISO date format, BS7666 for address data, etc) then if we replace system A with a new system all we need to do is recreate the API wrapper, test it and then switch it on.. The decoupling of the systems from the integration would allow public ICT to be more reactive to systems design, it would allow us to go to the market and actually get best value for the tax payer.
It would also provide a publish/subscribe model of data flow. This could be human readable (browser based) simple information presented as an automated integrated dashboard to replace the various reports etc that are hand built at great effort.
My proposal is to create complexity out of simplicity. Start small, build up API calls that call other APIs, add in business logic and suddenly you have a system that is intelligent, maintainable and scalable.
Building it on REST methods means that all calls are via http and are therefore cacheable. One of the main demands of REST principles is for the transfer of the state of the resource to be complete and therefore cacheable. This could reduce the load on our main systems whilst still providing up to date information.
The whole concept of RESTful wrappers would require a new mindset and a large investment of effort and commitment but it can be done. One of the main reasons for putting on my talk was to get critique from the audience of private sector developers. These guys live and die by their code, their businesses have to be responsive and their code/applications have to be efficient. Anything less means their business will die. It’s the nautical equivalent of placing Homer Simpson in charge of the ships wheel with a crate of Duff and sharp rocks ahead – something bad WILL happen..
I presented my ideas and asked for their opinions; luckily I’m heading in the right direction and got quality feedback on the difficulty on attempting to introduce a ‘pure’ RESTful setup.
All I have to do now is convince my bosses…
‘Advantages and opportunities’
Which brings me to one of the advantages of moving towards a more open publish/subscribe model of data flow. Not only could we decouple the applications from the integration path but we could decouple any integration hub from the path. And, as a direct result find Open Data at its very heart. It wouldn’t need to be an after thought; it wouldn’t be ‘something we are forced to do’ taking precious resources away from business as usual; it wouldn’t be a waste of time on a vanity project. It would be part and parcel of connecting systems together.
By making Open Data easy to create we could start to gain the advantages that I feel will flow from Open Data usage. In the next section of my talk I described how Open Data could be provided with examples of datasets that show once data is open it can be used by anyone for anything, even to make money – great. #winning
Local Authorities are there to improve the lives of the populace, they are there as a net if people fail but they are also there to support and help business and innovators. Open Data could be a big winner with the latter while helping the former.
Broke by Johnny Grim, on Flickr
Explaining the concept of Open Data took a little while, my audience asked about funding; the response: NO MONEY. We give data, you use it. If you create something others will use that improves their life then great, we have social return. If you make money on the back of it then great, the social return is doubled. At the end of the day – the data is yours, it’s public data for the public.
More importantly, what you create, what you mash together is up to you. If you can make something that is innovative and makes you, I and everyone else stop and think, then that is brilliant, that is one of the drivers for Open Data – innovation. Innovation leads to new ways of thinking, new ways of working and new ways of looking at the World. We all welcome innovation, its full of win.
BTW a big note here for anyone who’s ready to jump down my throat – we are talking about data that’s NOT person centric. Releasing data that contains individual names, address details etc flies in the face of the Data Protection Act and all recommendations on Information Security so will never be released. I hear strange and wonderful variations of the Data Protection Act quoted all the time and through my job have had to understand it very well. I and all my colleagues take Information Security very seriously, in fact we even put away our Sponge Bob coffee mugs when discussing Information Security…it’s that serious.
So what data could we provide?
Well within my talk I used an example of trees. Did you know that we, Hull City Council have all the trees geo-coded within our asset management system, this includes their exact location and the type of tree. It may be a little out of date as we don’t have the funds to keep counting trees.
We also have geo-coded datasets of listed buildings, toilets, public buildings, schools, etc you name it and we will most probably have it.
And most of it could be issued easily and simply, which takes me back to the RESTful wrappers, once they are in place around an internal system it wouldn’t take much effort to create an API ‘OpenData’ resource that GETs the required information on a regular basis. This could then be fed into an automated externally facing system that presents the data in all the formats geeks/nerds love (JSON/YAML/XML/etc) while also being available to humans in something akin to CSV.
‘Hackathon anyone?’
As part of the way forward I suggested that the council jump on the bandwagon and organise a hackathon in conjunction with the local devs where they could use data supplied by us…Pizza to go with the coding would be mandatory.
Pizza Craving. by ecsuecsu, on Flickr
The response was overwhelming, I suspect it had something to do with the offer of free pizza but I also felt my talk hit a nerve. Not because it was awesome, that goes without question ;-), but because it addressed a need at the right time – data has become the life blood of the population. We live in very changeable times but having the right information presented in the right way at the right time allows us to navigate the treacherous waters a little easier.
Providing Open Data in partnership with local developers could be a perfect example of Public/Private partnership and I feel that one without the other will not work. Shouting the mantra of OpenData in a similar way to Steve Ballmers’ notorious ‘Developers, Developers, Developers…’ rant will provide nothing for the public.
It may be open but not in any real way.
For it to be open it has to be accessible and more importantly understandable, it has to have context. We (the public sector) are not the best people to provide that context, we provide the fuel for the vehicles, let the mechanics and engineers create the vehicles, let them build apps.
If in doubt look to Google – search for Open Data, look at data.gov.uk, look at the London datastore, look to dados.gov.br, look to the future.
I look forward to the future with optimism and know collective knowledge is the way forward, the more we learn the more we understand and the less we destroy.
Hull’s Hackday will be announced in the not too distant future, keep an eye on the HullDigital website.