Here at Paine Publishing, we keep a steady supply of data geek jokes for those times when we need a bit of levity to get through those tough days of number crunching. Our favorite is:
“Data are like political prisoners: Torture them long enough and they’ll say anything.”
Now, we’re all for deep dive analyses into data. And there’s nothing we love more than cross tabs. But, unfortunately, as we know, torture doesn’t always lead to the truth. It’s no different with manipulating measurement data. There’s a fine line between data analysis and data misinterpretation.
Here are 4 common ways that you may be mishandling your data:
#1. You are asked for percentages of very small subsets of all the data.
We once delivered a launch report to an agency client that included data from about 250 social and traditional media items. It showed that the desired messages appeared in about 7% of the coverage – not a bad number at all. But it wasn’t what the client expected or wanted to see.
So they asked us to redo the analysis after eliminating all social media, arguing that their outreach efforts had been focused on traditional media. That reduced the data set by half. 150 is still an acceptable amount of data to analyze under the circumstances, and, as it happened, the data didn’t change all that much. The percentage of items containing one or more key messages went up slightly to 10%.
But apparently that still wasn’t an acceptable number. So then we were asked to include only the top tier media at which the outreach was directed. That analysis produced a much higher percentage, and now the agency was able to proudly declare that one-third of all coverage contained a key message. The problem was that for all the effort and energy put into the campaign, the data set was now so reduced that only three media items contained a key message. When you put it that way, it probably won’t get much praise from the corner office.
Lesson #1: The person you’re trying to convince is probably a numbers person, so your numbers better stand up to careful scrutiny.
#2. You are asked to use multipliers.
In those organizations that still believe that impressions count, no matter how they’re counted, inflating impressions is a common problem. For some reason people want to “add weight” for specific media outlets or certain types of stories. The right way to do this is to develop a custom index that you can track over time. The wrong way is to multiply impression counts (which are probably flawed to begin with) by multiplying those from top tier publications by 2 or 3 or whatever number someone dreams up. Don’t do it. The IPR has published a great paper by Mark Weiner and Don Bartholomew on why you should never use multipliers.
Lesson #2: Your top tier list should reflect the degree to which it reaches your target audience. That’s why it’s called top tier. Why do you need multipliers if you are already reaching a high percentage of your target audience?
#3. You’re asked to do a benchmark comparison against the competition, even though the circumstances were vastly different.
Comparing results to the competition is a very powerful and persuasive way to report results. However, in order to be truly comparable you need to make sure you are comparing the same media outlets in a similar time frame and in the same geography. If the competition launched its program the same day that Princess Diana died, then they were lucky to get any coverage at all (true story), and it’s hardly a fair comparison. If you do not have the full data set for the competition because their launch or activity exceeds the time frame of your data, then it’s not a fair comparison. If you are launching into a mature market but the competition was the first to market and had to educate the market as well as promote its products, then it’s not a fair comparison either.
Lesson #3: Compare apples to apples. Otherwise you’re just blundering about in the fruit aisle.
#4. You are asked to include only “the good news.”
If you are given a bunch of data collected from agencies who may or may not have been clipping consistently with your methodology, then you are treading in dangerous waters. If in addition you are told to show just ”the good news,” then you had better expect alligators.
Lesson #4: Beware of unknown waters. Also, if someone wants you to ignore the bad news, then there just might be something hiding in there that could bite you. ∞
Thanks to Ontario Wind Resistance for the image.
