by Katie Delahaye Paine—I’ve spent an inordinate amount of time mucking about with measurement products in the last few months. Some of them have made our “best new thing in the world” column. For others, my mucking about has become more muck raking, finding the flaws in some of the more highly touted systems. There is no doubt that technology is improving in its ability to track our whereabouts and behavior (see Welcome to your Destination: How Arrivalist is changing destination marketing). But in certain ways it’s becoming more misleading as well.
When presenting data, there are several very weak links that both newbies and measurement pros need to be aware of:
1. Beware the Great Unmeasured.
All major measurement systems today rely on people’s willingness to enable web cookies, and on individuals being willing to provide you with accurate data. It is very easy to be wowed by the statistics and metrics provided by the latest versions of Adobe Site Catalyst or Google Analytics. But people who do not enable cookies or do not register or provide information are part of the Great Unmeasured. It’s a population of indeterminate size that could add significant errors to your calculations. As you look at all those lovely statistics and numbers, you need to bear in mind that they only represent a percentage of your entire audience. The percentage that is going online, and that has cookies enabled, and perhaps has given you some other piece of information. How does that group compare to the rest of your audience?
2. Social is not the world.
The social media measurement standards at www.smmstandards.org point out that social sentiment is not a proxy for public opinion. It represents only those people who are actively engaged in social media. The number of decisions today that get made based on automated sentiment analysis of a Twitter stream is truly frightening.
3. Without tags, someone else is taking credit for your work.
Tagging — or putting source code of some sort on an ad, press release, or any other piece of content — enables systems like Site Catalyst and Google Analytics and Web Trends to track the performance of that piece of content over time. The problem is that most marketing departments only put tags on the things they pay for. Which means that most of the earned and owned media content gets ignored by these systems. I can’t tell you how many marketing measurement reports I’ve seen that arbitrarily attribute all kinds of revenue to paid programs, despite the fact that earned media was responsible for most of the results. Unless PR and social teams begin to tag all their campaigns and efforts, revenue attribution models will continue to cut PR and social out of the equation.
4. Most content is irrelevant.
A study we did several years ago showed that as much as 40% of social content is created by robots or is spam or other worthless content. Sadly, that number hasn’t improved much. All too often automated social tracking systems pull in data that most PR people wouldn’t consider relevant. A recent test revealed that 90% of the content provided for a sentiment test that was supposed to be promoting a new Toyota vehicle, instead reflected hundreds of people talking about getting rid of their old car. So while they were getting lots of content, none of it had anything to do with the brand messages the company was trying to track. ∞
