Measuring Impact with Twitter and Bit.ly
For the past month (and ongoing) we have been running an experiment to test what impact the advertising of selected podcasts can have via Twitter. In this short report we look at the initial findings and walk through the methodology and indirect benefits noting that Twitter is not a simple panacea for attracting an audience but implementing processes involving it will help improve access to content and allow for some easy gains to be made in measuring impact.
The Oxford Podcasting service has a twitter feed for making announcements about our podcasts and related news (@OxfordPodcasts). It has been a fairly under utilised tool thus far, and we wanted to see if it could make an appreciable impact on selected podcasts being accessed and downloaded. Prior to our experiment, there had been 43 tweets and we had 1525 followers – a fairly sizable following, though far from superstar status.
Learning to tweet
We adopted a thrice-weekly posting strategy, where we would look for a topical (news related) item on Mondays, an older (>6 months) podcast that had received little attention for Wednesdays, and Friday was for a selected new (<2 weeks) podcast. Twitter has a 140 character posting limit, so there was some initial debate over exactly what to put there. First thoughts were to try and squeeze direct links to the podcast file and a related RSS feed in, leaving about 85 characters to “sell” the podcast with. We realised this wasn’t going to be ideal, and that whilst we were going to use an URL shortening service, we really needed a landing page to give more information on the podcast to point at. As the current web portal is very limited and difficult to redevelop and because the replacement is not yet ready for production usage, we decided that a blog site for the podcasting service would meet our needs.
The Oxford Podcasting blog takes advantage of an OUCS blogging service and allows us to create short pages (blog posts) providing more information on the content we are promoting and links to the file, the web portal and the iTunes U page in the same posting. Many twitter users use something similar to allow them to create tweets of more than 140 characters. As a means of streamlining the process, the blog site allows us to auto-generate a tweet from a published posting, meaning we can write the page and with one click, generate the blog post and the tweet that points to it. Selecting the podcasts for inclusion has been left to mostly skill and judgement with occasional inputs from the Podcasting team, and represents a broad cross section of our content. We have avoided publicising our already popular items as to eliminate as much external “noise” from the data as possible.
Save space, shorten everything
We used a free Bit.ly account as our URL shortening service, as this provides some basic analytics (see post on comparison of analytic systems) and integrates easily with Twitter usage. Using such a service has highlighted a strong use case for having a more capable system in place at Oxford that would allow us to implement what some would call “Business Intelligence” type services – essentially, a centralised tracking system and analytics solution to tie together many disparate portals, locations and hosting arrangements in one easy-to-use setup.
Four shortened URLs were created for each posting. The first was included in the tweet and linked directly to a specific blog post. The next three were then added to the blog post page and linked to the podcast file, the web portal, and the iTunes U page for the podcasting series. Each blog post was given a “decorated” URL for the podcast file which could be used by our (in-development) web log analysis package to identify specific requests to a file. Because we can not presently track links on arrival at iTunes U and at the Web Portal, we have to rely on the data collected by bit.ly to count the number of clicks they received. However, because of the tracking URL, we can compare the bit.ly analytics for the link to the podcast file with our own logfile analysis. Let’s take a look at some of the results now.
The first result
Our first posting (http://blogs.it.ox.ac.uk/podcasting/2011/02/15/lord-norman-foster-lecture-–-performance-sustainability-via-architecture/) on February 15th happens to have been our most popular item (based on total number of clicks) based on data received thus far. The following diagram overlays the bit.ly analytics data onto the different measuring points.
The four bit.ly links are represented as the yellow arrows. Notionally the arrow leading into the Blog page contains visitors who have clicked on the bit.ly link in the tweet. However, this isn’t necessarily accurate, and the limited data does show that several of the bit.ly links have been accessed from locations other than where we placed them (e.g. the link has been copied onto another webpage elsewhere). The numbers (19, 10, 6 & 1) represent the clicks counted by bit.ly, and the three charts near each destination are the bit.ly analytics data.
The data shows clicks along a timeline (the blue bar charts), referrer information in the left piechart and geographical information in the right piechart. Looking at the click timeline for the Blog Page link (top right chart data) we can see that most of the clicks that took people to the blog page occurred within 24 hours of the original tweet, which is to be expected as Twitter has a rather short attention span with most users likely only seeing the tweet once before it is pushed down their displays by newer tweets.
The referrer data offers a basic breakdown as to where the link was when it was clicked (e.g. what website URL). However, as we know from Referrer Analysis in our own log files, this data is often missing and can be vague. This is highlighted by 47% of the clicks being from devices or systems that did not report Referrer information. Of those that did, only 8 of 10 clicks were from visitors looking at our specific Twitter feed (http://twitter.com/oxfordpodcasts) and 2 of 10 came from the Twitter front page (where they display a sample briefly of the latest tweets from worldwide). The second piechart is looking at GeoIP data and reporting that 94% of all the clicks were from addresses in the UK (green slice), with the lone other being from the Philippines.
This same level of information can be found for the three bit.ly links on the blog page and they paint a similar, though not quite matching, story. For example, a crude analysis would suggest that of 1500+ followers, only 19 clicked on the link in the tweet, and of those 19, 6 of them clicked on the link to the Podcast (web) Portal. However, even this limited data shows that that is not quite the case. In this example the Referrer information does suggest that 4 of the 6 (or 67% which sounds more impressive) came from within the blog page (Referrer shows as blogs.it.ox.ac.uk). However, the key clue is the GeoIP data which says 5 clicks were from the UK, but one was from the Czech Republic, not the Philippines.
This disparity is even more apparent when looking at the clicks to the actual podcast File, where 2 clicks came from Columbia, 1 from Australia and one from the Czech Republic (the rest being the UK). Unfortunately we can not tally when these clicks occurred with where they came from, so we can’t say whether the person from the Czech Republic who accessed the file was also the same person who visited the Podcasts Portal. Another problem is that with the numbers being so low, the error factor introduced by our own clicking on the links to test them is significant. In one example, when I accidentally clicked on a link in the bit.ly reporting interface, it was then logged as an access and changed the above figures from 19 to 20. It could be posited that two to three of the UK clicks logged by bit.ly were actually ourselves setting up the experiment. The other postings have all followed a similar trend, with few links recording more than single digit clicks.
Who do we believe?
Another aspect is that bit.ly’s GeoIP data does not tally with our own access log analysis. As I mentioned above, we added a tracking decoration to the URL for the podcast file, this allows us to find specific web server access requests that can only come from this bit.ly URL (or by someone who has followed the bit.ly URL and then reposted that link somewhere else). Whilst our data is a little more fuzzy, the quantities tally proportionally to the bit.ly click counts suggesting they’re in a reasonable accord. However, the GeoIP data does not match. Looking at the Lord Foster podcast example above, our Log Analysis tool has determined that there were 9 computers to access that file using that URL. 5 were from the UK, and indeed, looking at the IP addresses of those computers, they were also from within OUCS and were ourselves. 1 Colombian IP address was detected, but the other 3 IP addresses were all from the USA – not Czech or Australia.
How do we reconcile these differences? That is unclear presently. I am comfortable trusting our GeoIP data having reviewed available databases and selected a respected source from MaxMind, but I can’t suggest that bit.ly haven’t also been similarly diligent and therefore their data would be expected to match ours. It might be that a link was clicked on, and registered at bit.ly, but was cancelled before the request reached our servers – though I’m not convinced by that idea. It is also reasonable that the link might have been harvested by a “bot” and used elsewhere – this I can’t determine from the bit.ly data, but with further digging I may be able to tally using our own analysis tool and database. Either way, with the key values being so small, the margin for error is unreasonably high and not worth dwelling on heavily.
We did discover that the bit.ly links had been found and reposted elsewhere for a couple of our files, as the referrer data showed clicks from other websites than our blogs.oucs site. This and some of the other factors would suggest that our blog posting and blog site attracted as many people to download our podcast content as did our tweeting. We haven’t advertised our blog site at all outside of the tweeting, so this would suggest small amounts of serendipity and/or accidental indexing by Google. Web analytics on our blog site is limited and what there is has yet to be analysed, so we can’t say anything about referrer information now.
So, what do we conclude from this experiment? Well, the absolute number of click-throughs (or goals/conversions as would be measured by Google Analytics users) was very low compared to the number of people that were shown the links. We did have a significant increase in the number of followers for @OxfordPodcasts during this experiment, up 303 to 1828, though the reason for this increase is unclear.
Also, our usage of Twitter was very basic. Twitter is a social media system and to be most effective needs an active community of followers to generate “buzz”. We did not retweet any of our posts to increase exposure, though we know that a couple of followers did retweet a few. We have little information about the makeup of our followers and whether these 1800+ accounts are particularly active, and what their usage patterns are. Our material is not necessarily the most accessible to a popularist audience, but we would imagine that anyone expressing an interest in university produced podcasts to be more than capable of consuming the material. The material is also rather diverse, so this too dilutes a community effect to a degree as the subjects/topics are perhaps too varied. Whilst some of our material was selected to be topical, we did not employ any “hashtags” to mark the tweets, which may have helped to attracted an intended audience who could be monitoring those keywords.
We do feel that using bit.ly has clearly demonstrated a strong use case for a more Oxford centric and data rich “Business Intelligence” system for link generation and monitoring, however, implementing such a system is outside the scope of this project but is now under consideration by the Podcasting Service. The methods used to market this material is applicable to a broader range of collections and projects and with better Tweeting skills (using some of the techniques outlined above) a stronger response could be generated. The ease of which the tracking links were created and the basic reporting they offered is appealing compared to the efforts required to validate/compare again our own log files (though this is rapidly becoming easier). The development of the Podcasting Blog has benefited a range of service needs and thus seen as a boon whilst not being a direct aim of the LfI experiment. Finally, the process has been judged easy enough that it will continue after the conclusion of the LfI project and likely to implement some of the refinements discussed.