This story originally appeared in the early May edition of The Measurement Advisor.
The ‘Paine’ of Measurement
Most of my arguments about data and measurement concern the Myth of the Perfect Measurement System. That’s the fairy tale that measurement of communications can be an absolute and that there is only one way to do it. It’s why so many people have wasted so much time hacking their way through medieval forests of AVEs and ROEs searching for the Holy Grail.
There is no such thing as perfection in a communications measurement system. As my father told a luncheon crowd in St. Louis 45 years ago:
If we can put a man in orbit, why can’t we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: People, human beings with a wide range of choice. Unpredictable, cantankerous, capricious and motivated by innumerable conflicting interests, and conflicting desires.
Okay Dad, so it took me awhile to realize how wise you were. The conclusion that I’ve come to after nearly 30 years of designing and implementing measurement programs is that consistency is more important than perfection. Let’s look at a few examples:
- Is Google Analytics perfect, or 100% accurate? No, at least not according to most web analytics experts. Does it matter? Probably not. If you use the same data month after month in your reporting of results, and it’s all coming from the same Google Analytics account, then your metrics are measuring change over time. Where inaccuracies come into play is if you are using data from Twitter Analytics in one chart and data from Google Analytics on another and saying the two are the same. They are not.
- Does that mean we shouldn’t be using Twitter Analytics? Not at all. If you are only using Twitter Analytics to measure engagement and clicks, and you measure consistently over time, then you will get valuable usable data. Is it perfect? Probably not. But as long as you are consistent, you can make sound decisions based on it.
- Automated sentiment analysis: Good thing or bad thing? I ranted against it for years, knowing that our highly skilled human coders produced far more accurate results than any machine. They weren’t just more accurate in determining messaging and tonality, but they also were great at eliminating duplicates, paid and owned media that didn’t belong in the analysis, and all kinds of other things that clients didn’t find salient to their report. But times change. I’ve since used those trained analysis to test automated sentiment analysis, and, at least in the case of NetBase, they agreed 88% of the time. I’ve also found that badly trained humans are just as inaccurate as a badly trained computer.
In the end, every program has its priorities. While I’ll always believe that trained human analysts provide necessary oversight to every program at some stage, I can accept the reliable consistency of a machine that can tackle a hundred thousand tweets at the blink of an eye. The reality is that for many clients who need fast answers, speed and consistency are far more valuable than precision and conscientiousness.
Today’s world demands data-informed decisions at the drop of a hat. So we all have to be a little less measurement “perfectionistic” and a bit more insistent on consistency. ∞
Thanks to psyche2go.net for the image.
“Consistency”, that is the key point in your post. Even if the measurement tool is inaccurate, as long as that inaccuracy is stable over time the resulting measures can provide relevant metrics for comparison purposes. I think the question for a user of media measurement becomes, “Do you want a highly managed solution, or an automated solution. An automated solution does require the user to pay much closer attention in order to make sure the algorithms and heuristics are correctly interpreting their media results. With “well trained” human analysts, the analysts are managing each mention measured, and therefore the outcome tends to be accurate to the story level. This later methodology relieves the client from much of the oversight necessary to ensure valid results.
You are absolutely right about the need for paying attention, but that is true of any measurement program. A manual system by its nature requires daily attention. An automated solution still requires oversight by SOMEONE. I think automation is a myth. I can’t agree that inaccuracy is acceptable in any form. For example, if the measurement tool consistently delivered AVEs or Earned Media Value– a metric that is inherently inaccurate — that would not be acceptable. But if the tool is delivering only an 80% accurate sentiment score or a “click thru” rate that you can’t verify that thru WebTrends, Omniture or Google Analytics, then its’ better to be consistent than not have that data at all.