When an organization is setting up a measurement program, the conversation will inevitably come around to: “Do you know of any industry benchmarks we can use to set our goals?”
My answer is invariably: “None that are valid.” I know, because I used to publish them.
You can’t compare tanning lotion to tanks.
A decade or more ago we used to publish the Delahaye Benchmark study, for which we would gather all the results from all our clients into a single database and come up with an overall average for positive, negative, and neutral, and for message content.
I won’t keep you in suspense: It was invariably 3-5% negative, 15-20% positive, and 20% of all articles contained one or more key messages.
The problem with that approach is that it is like blending a vintage Bordeaux, a tasty Chilean Merlot, perhaps a little spicy Gewürztraminer, and some robust Napa Valley Zinfandel. Despite your fine ingredients, you’d get a blend that even the least discriminating wine drinker would agree was undrinkable. You can’t just combine lots of good stuff into one bucket and expect it to still be good. And that’s exactly what you get when you try to benchmark with an average of results across many different industries, cultures, and geographies.
As it turns out, results in Asia were ten times more positive than they were in parts of the world where the press is much less polite and far freer. Needless to say, the % of negatives was significantly higher for chemical companies than it was for non-profits. And it’s a lot easier to get messages across for beauty products than it is for fighter jets. Duh!
Why does everyone ask for these homogenized benchmarks? All it would take to completely destroy the credibility of the PR person using these sorts of numbers would be one good data analyst asking a few pointed questions during his or her presentation.
My best guess is that people just want to know where to start.
Well, the place to start is with benchmarks that your senior leadership will believe. Who do they worry about? What competing organizations keep your CEO up at night? Study them for a month or two—preferably a full quarter.
And if you’re a utility or a government agency and don’t have “competitors,” then study your own results for three months. That will establish where you are, how fast data shifts, and what is a reasonable goal.
Five clips does not a benchmark make.
In another case we were asked for a benchmark versus the competition around a specific product launch. The only trouble was that that the client’s product was only mentioned in about five clips out of 50 total items. One of which was positive. So it would be entirely accurate to say that 20% of their coverage was positive during the competitive launch period. But it would certainly not be meaningful either.
As with any data set, you need enough data, content, survey responses, or whatever data you are gathering to at least pass a common sense test.
Inconsistency, thy fruit is madness.
The other major problem with most benchmarking efforts is inconsistency in methodology. When we were initiating a standards-compliant measurement program for a client we thought we could use the prior year’s data that had been collected and coded by their PR agency. Unfortunately, the agency had used completely different definitions of positive and negative than we were applying. Pretty much anything the agency did was positive; had we used their data, the benchmark would have been set unrealistically high.
Here’s a simple rule of thumb: If you don’t know how data was collected and coded, and/or you can’t replicate it exactly, then you can’t use it as a benchmark.
Three rules for better benchmarking:
- Make it relevant to leadership.
- Gather enough data to make it valid and meaningful.
- Be consistent in the methodology.
∞
Thanks to New Zealand Inland Revenue for the image.
