This story originally appeared in the early May edition of The Measurement Advisor.
The ‘Paine’ of Measurement
Most of my arguments about data and measurement concern the Myth of the Perfect Measurement System. That’s the fairy tale that measurement of communications can be an absolute and that there is only one way to do it. It’s why so many people have wasted so much time hacking their way through medieval forests of AVEs and ROEs searching for the Holy Grail.
There is no such thing as perfection in a communications measurement system. As my father told a luncheon crowd in St. Louis 45 years ago:
If we can put a man in orbit, why can’t we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: People, human beings with a wide range of choice. Unpredictable, cantankerous, capricious and motivated by innumerable conflicting interests, and conflicting desires.
Okay Dad, so it took me awhile to realize how wise you were. The conclusion that I’ve come to after nearly 30 years of designing and implementing measurement programs is that consistency is more important than perfection. Let’s look at a few examples:
- Is Google Analytics perfect, or 100% accurate? No, at least not according to most web analytics experts. Does it matter? Probably not. If you use the same data month after month in your reporting of results, and it’s all coming from the same Google Analytics account, then your metrics are measuring change over time. Where inaccuracies come into play is if you are using data from Twitter Analytics in one chart and data from Google Analytics on another and saying the two are the same. They are not.
- Does that mean we shouldn’t be using Twitter Analytics? Not at all. If you are only using Twitter Analytics to measure engagement and clicks, and you measure consistently over time, then you will get valuable usable data. Is it perfect? Probably not. But as long as you are consistent, you can make sound decisions based on it.
- Automated sentiment analysis: Good thing or bad thing? I ranted against it for years, knowing that our highly skilled human coders produced far more accurate results than any machine. They weren’t just more accurate in determining messaging and tonality, but they also were great at eliminating duplicates, paid and owned media that didn’t belong in the analysis, and all kinds of other things that clients didn’t find salient to their report. But times change. I’ve since used those trained analysis to test automated sentiment analysis, and, at least in the case of NetBase, they agreed 88% of the time. I’ve also found that badly trained humans are just as inaccurate as a badly trained computer.
In the end, every program has its priorities. While I’ll always believe that trained human analysts provide necessary oversight to every program at some stage, I can accept the reliable consistency of a machine that can tackle a hundred thousand tweets at the blink of an eye. The reality is that for many clients who need fast answers, speed and consistency are far more valuable than precision and conscientiousness.
Today’s world demands data-informed decisions at the drop of a hat. So we all have to be a little less measurement “perfectionistic” and a bit more insistent on consistency. ∞
Thanks to psyche2go.net for the image.