Listening to the interwebs

The other day I saw a tweet that pointed me towards @SavvyCitizens and their Top Ten online resources for getting savvier ahead of the general election. It really is a top list of resources and all but one of them are fully featured, for free.

Council Monitor is the one that you have to pay for to get all its features. Now, for a basic ability to look around the country and see your council’s sentiment rating or look at how it compares to others this is excellent. It can give you at a glance that sort of information. However, to find out anything more you need to pay. And, for me, the costs they’re suggesting make this a little bit more than just Freemium.

The basic subscription of £99 a month gives you insight into your own organisation and allows you to actually see the mentions being made about you. For an extra £100 a month you can add 5 keywords and for the princely sum of £299 a month you can have up to 10. And they’re the special introductory prices.

OK so that’s only about £1,200 a year and no more than £4,000 tops, so what’s the big deal? After all that’s nothing for organisations with budgets in the multi-millions. But it’s this seemingly common attitude to the pennies leaves me thinking that it’s no wonder we have a problem with the pounds. So, perhaps what will follow is not worth your effort but I end up feeling a bit disappointed that we’d rather end up paying for a service than shaping the tools that are out there ourselves*.

Council Monitor is an aggregator of content that can be found ‘out there’ but happens to be housed within a shiny package that allows for comparison across the national picture. I have nothing at all against the shiny but what is most important is actually hearing what is being said, listening to it and then responding. It’s gratifying to know that ours is the council with the most positive mentions in the country but it’s not what’s important.

What’s important is recognising that people are saying things and we need to hear them because our service delivery can be improved by responding to the comments being offered in cyberspace. Not in a terrifying Big Brotheresque fashion but in the way that we’re coming to expect of organisations and companies that are important to us. The way that sees a need and then fills it or hears a criticism and fixes it so that not only are people valued for their contribution but the next person benefits from such a proactive response.

So, on that basis it’s content which is key. It’s not the overview of sentiment, which can be picked up for free but it’s the actual information itself, the stuff that sets you back £99 a month for a single keyword that an organisation wants to hear. Set against Radian6 or one of the other very impressive, and fairly expensive reputation management/social media monitoring services that seems good value.

But we don’t have the budget even for the good value. Dave Briggs flagged up this list of Social Media Monitoring resources which are free and sometimes have similarly shiny interfaces. But we’ve been thinking about how we make something that we can control and that which can pull a variety of different sources into one place. It is still in a prototype stage but the idea of doing something ourselves, whilst it might seem daunting, may actually be preferable.
But first, 2 caveats, and 3 tools:
  • These are not pretty solutions
  • There is a lot of potential to improve them
  • A variety of search engines
  • Yahoo Pipes
  • Netvibes


There is an impressive array of tools with which to interrogate the internet. We identified the following 15 services as being useful for different reasons but it is by no means exhaustive.



Almost all of these use APIs to enable the interrogation of their results from afar. What this means is that we can enter a search term away from the site in question and get a response directly to us or as part of an RSS feed or into an email or a widget.

In order to do this I use Yahoo Pipes. Now, I’ve blogged here and here about how to do this for wildly different purposes and I like Yahoo Pipes. It is certainly quite daunting to begin with and it can be quite temperamental but on the whole it is a very clever environment in which to build tools that can search for information, connect it together and then filter it as necessary. We’ve used it to make pipes for the search engines listed above.

So, using that lovely technology we’ve put together a pipe that looks at SocialMention, and, crucially, for the point I’m hoping to make, it does return sentiment too. At the moment it is pulling two pipes – the Social Mention search and the Sentiment tracker into one.

If it doesn’t work in the page then take a look at one of the following:

SocialMention (all mentions)

SocialMention (sentiment)

SocialMention (mentions and sentiment)

We’ve got pipes for all the search engines we listed above and had wanted to make a single feed from these individual elements but find Pipes cannot cope with this although, if we had there was a recognition of what we could do and a commitment to resourcing it I think we could probably identify some other solutions too. And the beauty of it? Once you have it set up outputting for one search term you can set up more using the same infrastructure (if you want it all lumped together it will accept multiple search terms separated by commas).

We have a dump of all our pipes onto one page. Some of them do not contribute anything to us and will be rooted out; some of them duplicate content and that may mean those two feeds could be merged into one; and being a public page makes tracking and storing activity impossible. In practice this would be a private page accessible to whoever has the responsibility to keep track of the content that had been seen and/or dismissed and that which was still of interest or had not yet been looked at. The Comms & Marketing team are going to be testing it out and exploring how best to use the information and how to process it for the benefit of the organisation.

It is true that we might not be getting the same results as Council Monitor and we might not be able to gauge sentiment elsewhere (although with the right commitment to developing this we could certainly get there). It’s also true that we can’t get trends but that’s just a metric that means nothing if we’re not hearing or responding to the people who are talking about, or to, us.

We’re exploring how we make this better and more useable. I’m moving placements but I’ll look to blog through the technical aspects of making these things happen so that you can make your own sentiment monitoring tools. But, in the meantime, feel free to test ours and see whether they’re useful as an alternative to spending money.


*I feel the same about GovDelivery. What they offer is much more technical and would require more effort to duplicate but, nevertheless, it is essentially publishing content in ways that would not be difficult to fashion yourselves. At least that’s my take on it.

About Benjamin Welby

Hi, I'm Benjamin Welby. I'm a displaced northerner currently living in Croydon, I church with a group of Christians who meet in a Soho nightclub on Wednesdays and I support Bradford City. I've an academic background in History, Politics and International Development. I work for the Government Digital Service but I left my heart in local government. This blog is infrequently updated and may feature any, all or none of these things...