Measurement Experts Debate the Meaning and Usefulness of ROI
By Katie Delahaye Paine—We recently sent our “Barcelona Principles” slide show (“How to Use the Barcelona Principles to Save Time & Money, Win Budget Battles, Make Better Decisions, and More!”) to my fellow members of the IPR Measurement Commission (IPRMC) thinking that they might find some of the slides useful when explaining the standards. Little did I anticipate the debate that ensued.
My Canadian colleague Fraser Likely, who I respect tremendously and quote frequently, took umbrage with my use of ROI in two of the slides. Since I had borrowed them from the presentations of two other commission members, Mark Weiner, CEO of Prime Research and Tom Nicholson, Managing Director of The Nicholson Group, I copied them on his comments.
What ensued was unusual in these days of fur-flying Twitter fights: A thoughtful, very well-reasoned, incredibly polite, and respectful debate (and not just because it involved a Canadian) on the topic of the misuse of the term “ROI,” the emergence of a need for insight, and the state of the PR profession in general.
They all gave me their consent to publish their discussion, which I share here because I find it to be the perfect summary of the quandary that faces the PR profession and those who measure it almost every day. Senior management wants “ROI,” plus lots of pundits and vendors promise it, but it is almost never calculated correctly. So what’s a communications professional to do? Below you’ll read about several different ways to calculate ROI, and why the “right” way to calculate it is not always obvious (or the most fitting for your purpose).
First, here are the three ROI slides in question. What follows is the discussion. We hope you enjoy it, and would love to hear your thoughts.
Page 32:
Page 43:
Page 45:
Fraser Likely:
I applaud you, Katie, for your stand on AVEs and multipliers and other quasi-scientific measures we’ve used in PR/C for too long now. But, I would criticize you for taking other measures, such as ROI, from other disciplines and then using them incorrectly.
1. ROI has always been used in accounting as an “all in” measure. Your use on slide 32 is not all in. The financial measure on slide 32 is a cost-effectiveness analysis (CEA) measure. The measure is the comparable cost-effectiveness of each of those three channels. Customer spend (either actual or historic) over the cost of each channel. To measure ROI, one needs to cost the role of marketing, comms, sales, customer relations, etc. to get a gross cost, and then factor in sales returns, customer service, etc. to get a net return.
We want our terms used properly, yet we apply other terms from other disciplines and feel free to use them incorrectly.
2. You do this on slide 43. Isn’t “discounted by 2/3s” the same application of a false measure as PR/C’s use of multipliers? Where does 2/3s come from?
And, the ROI calculation on the same slide has to be “all in.” It’s not just the cost of merchandise plus the PR/C costs, it is sales costs, return costs, customer relations costs, etc.
Same argument for slide 45. Yes, one comms channel drove increased sales. But, were there extra sales people there and thus additional costs? Was there enough product there or was there extra cost accrued in delivering “rain check” product? Were people happy with the sales experience, or, if not, did it affect later sales? What was the volume of the returns like? What was the incremental sales volume compared to the total sales volume (thinking that some people were regulars there and didn’t see the placement or stumbled on the store without seeing the placement), since incremental sales should be subtracted from the total?
That is, ROI — as a measure that finance or accounting would recognize — can only be calculated properly after all possible investments/costs are added, a net return calculation, and then, only then, a real dollar credit given to the placement.
Yours with all due respect,
Fraser
Katie Paine:
You and Mark can fight it out over slide 32 – that was taken directly from his book and subsequent presentations. I’m happy to delete it if it isn’t accurate and yield to whomever is left standing at the end of the battle.
For 43, you’ll have to do battle with Tom Nicholson. All the Sears case studies came from him.
Mark and Tom, I will await your responses before deleting the slides. Mark and Tom – I suggest you avoid using a hockey stick as your weapon of choice.
Mark Weiner:
I stand by my definition (which is closer to Fraser’s than he may realize and I don’t consider them to be exclusive). Many other costs and sales drivers ARE included in this case and in others I use to defend PR’s ability to drive ROI. Fraser and I have already run this race many times. I respect him and his opinion.
Tom Nicholson:
Regarding the Sears case studies, Fraser may have a point. ROI is an all in measure and I respect him for pointing it out.
Perhaps we can agree on the term “incremental ROI?”
What we did was assume the same costs in the three markets used for the Fashion Footwear example. Because we had a national ad spend, the weights of the advertising in New York, Chicago, and Los Angeles were the same. Ditto for cost of operating the stores, customer service, etc.
What I was trying to get at was the value of the PR spend in those markets. For that we took the costs of our PR efforts around the public relations efforts that resulted in the placement in the New York Times. If I remember correctly that was about $20,000 (though please don’t quote me on that since I’m doing this from memory).
When we looked at ALL the variables in those markets for the three weeks leading up to the placement and the three weeks following, we had almost identical numbers. The one big difference was the placement in the NYT. If that had been in the national edition we would not have had much to talk about. But, it ran only in the NY/New Jersey edition of the paper. We now could compare that market with Chicago and Los Angeles and with our national numbers for the seven week period in question.
I had the CFO of the footwear business run all the numbers for me so I could be confident that I was not simply finding the number I wanted to find. As an aside, the head of the footwear business initially said the NYT article would have absolutely no impact since Sears customers did not read the NYT. So, we went in with the head of the business pre-disposed to finding no impact.
Instead, his CFO came back with the numbers in the presentation showing about a 20% increase in sales during the week after the article ran, with no other variable to explain it other than the NYT article.
Now, I may not be using ROI exactly as the accountants would, but I felt pretty confident that we had proven the value of a PR placement with the help of someone who spent his lifetime analyzing financials for a major retailer. He never questioned our use of the term, but Fraser may still be right.
Fraser, would you agree to a modification and consider the use of “incremental ROI” for our purposes here?
Let me know if I can provide any additional clarification.
Fraser:
As you know Mark, I don’t have a problem with the concept that PR drives sales. Of course it does.
I have a problem with any supposed ROI calculation that includes just PR costs and just gross sales/revenue data. In finance and accounting, ROI measures actual, net (not gross) returns, and all gross costs from all investment pots (advertising, marketing, sales, customer relations, etc.). My problem is with the use of this finance term in PR/C when it’s used incorrectly. The calculation on slide 32 is not a ROI calculation. It is a cost-effectiveness measure.
I haven’t compared the slide to the examples in the book. So, I take Mark’s word that the book covers the examples in a way similar to the finance definition. (By the way, Mark’s book is about an arm’s length from where I’m sitting in my office right now.)
This race is one we should be running. We in PR/C take terms from other disciplines and use them differently. Or, we take terms and give them different meanings (for example how Internal Comms would use “engagement” and how Social Media would use it {see Conclave}).
Katie:
Fraser, I agree with you in theory. But in practice, if you’re working with an organization that is using a Marketing Mix Model, don’t you pretty much have to calculate returns and costs the way the model is set up?
Tom:
Just looked at the presentation, and noticed that you also used the Oprah example from Sears. In this case we did not have a control as in the Footwear example, so we couldn’t identify the incremental ROI.
Instead, we used a survey to measure purchase intent.
Fraser is right here that ROI is used a little loosely. We only measured the difference in purchase intent before the show and immediately after the show and calculated the difference. Regarding the 2/3 discounting of the numbers (purchase intent increased by $40 million), we looked at our model and decided that while increase in intent is good, most people never follow through on what they intend to do. Either they find something else in the interim, they show up at the store and can’t find what they are looking for, or they change their mind. The 2/3 discount was to discount for what we have typically seen in other surveys of consumer intent vs. actual purchase.
So, we were trying to be accurate with the actual increase in spending we would see following the Oprah placement, but there is no way to accurately isolate that variable and measure it. The best we could do was a pre- and post-survey to get at the impact.
Not perfect. But a valid way to get to the impact. Agree?
Fraser:
The marketing mix model only takes you so far down the value chain most of the time. Yes, you can get all marketing, ad, and PR costs and assign credit (cost vs. effectiveness). You can get all sales revenues — but these are gross revenues. Typically, they don’t include other parallel costs (returns, post sale customer relations, etc.). A true ROI is only calculated when all costs and thus net revenues are calculated.
I like Tom’s term “incremental ROI.” Or, maybe “gross ROI.” But, here we go inventing new terms!
Mark:
I’m afraid that in your scenario, Fraser, the ROI of anything but the enterprise overall would prove nearly impossible. I usually qualify my intentions by saying “return on PR investment,” “PR-ROI,” or “PR’s relative return within the marketing mix.”
I’m afraid that the alternative in most people’s minds would be to do nothing at all… that measuring the PR “All In” just isn’t possible, so why try.
Fraser:
That’s my point, Mark. ROI is at the enterprise level — at least that’s how it has been traditionally with finance and accounting. ALL in. I realize that I may be fighting a (losing) battle. I know full well that Marketing, PR, HR, — at the function level — talk about marketing ROI, PR ROI, HR ROI. One also sees terms like the ROI of a Tweet, or Web ROI or ROI of any and every channel or media. ROI is everywhere!!!!!
We’ve come to a point where PR uses ROI to mean a number of different things (any return over any investment, cost-effectiveness, efficiency).
But, more to your point, which is about credit. How does PR get credit for revenue generation? If ROI is calculated as “ALL in” at the enterprise level after all gross costs/investments and net revenues are known and thus a net return over investment can be calculated, then it is still possible to carve out PR’s actual costs and the NET revenue it generated, compared to other channels, through a MMM type calculation. This calculation would deal with actuals.
My argument is that credit can best be determined after “all in.” Why? There are a number of reasons. Here’s one: In a marcom campaign, traditional media, advertising, and point of sale may each seem to drive 1/3 of all sales and thus a third of the revenue generated, if calculated at the end of the marketing campaign. But, at the ‘all in’ stage, we might learn that there were more product returns coming from the sales generated by advertising than from traditional media and point of sale. Thus, the “actual” return on PR investment is higher at the “all in” ROI stage than it was at the interim marketing ROI stage.
What you are talking about is some type of interim ROI calculation or interim credit calculation — what Tom labeled “incremental ROI” — using interim costs/investments and interim revenue figures. I might be a little mean here, but I think it’s the research firms and the agency folks who have their knickers in a knot more than their in-house clients. The consultants want to show an “immediate” return for the work they did. They don’t want to wait.
To consultants, the problem with the “all in” ROI is that it would occur much after the campaign — after the consultants have downed tools. But, if it is interim or “all in,” it is the same calculation for in-house folks. “All in” uses actuals and my guess is that PR would benefit more if the calculation was made at this stage. Can the calculation be made in-house? Yes, especially if the corporation uses ABC (activity-based costing).
So, I fully support any calculation to give PR credit. Where I believe it should be done is at the “all in” stage. Consultants just have to improve their offering to include the steps after the marketing campaign or after sales.
And, it’s imperative for CCOs to push the need for the calculation at the “all in” stage as well as the calculation of the assignment of credit.
Mark:
To your last point, Fraser, this is how modern models work. Ten years ago when I wrote the book, data and computing power weren’t where they are now and social media was in its infancy. Models were conducted annually so returns, sales costs, and other factors (not ALL the factors in an “all in” computation) were included. Now, the models are more routine, conducted more frequently, and can handle even more data than they could in the past. Plus, as companies have evolved, more of the “all in” factors are computed with similar regularity, frequency, and ability to integrate with other data streams.
So: Am I waffling a little? Perhaps. The data and computing power for “all In” are here. But is it practical and widely applicable? No. And for the purposes of most PR planning and evaluation (or roll it up to “marketing planning and evaluation” or “corporate reputation planning and evaluation”), the practical question is, “does the difference matter as long as we are clear in our definitions?” I think not.
What do you think?
Fraser:
I appreciate that you know more about these models than I do Mark. From what you said, it seems that it is possible — or will be shortly — to regularly perform an “all in” calculation. Then a PR ROI from within the overall enterprise ROI is possible, based on actuals.
In the meantime, should we call what we are doing now PR ROI? I think we should give it a qualifier, like incremental PR ROI, or interim PR ROI, or intermediate PR ROI, or 1st stage PR ROI.
Because, if soon we’ll be able to calculate the “all in” or final ROI, then we‘ll need to differentiate between the two. So, let‘s start by doing it now. And the consultants can teach their clients and the academics can teach their students and all of us can then understand the difference and can apply the terms properly.
This is a communications business, n’est pas?
But, the other point worth noting is that ROI can be calculated on cost savings. This type of calculation is more complex, and definitely has a two-stage ROI process.
Mark:
Thanks Fraser. As you and I have debated in the past, my version of “PR-ROI” includes not only sales/revenues but also cost savings (efficiency) and, for that matter, catastrophic costs avoided (that’s actually the toughest one to quantify but models can also run disaster scenarios).
So maybe another question is “Who needs to know the definition?” PR people get it and it’s defensible.
And let me say also that one of the most confusing aspects of the ROI debate comes from PR service vendors who promise ROI all over the place… the great withheld truth? “Data” does not equal “ROI” (even, given the differences voiced throughout this email exchange, their promise falls far, far short). And that’s even with GOOD data, which many vendors do not provide to begin with!
Don’t get me started…
Wild promises for “Insights” and “ROI” make me ready to jump out a window. Interestingly, I hear more interest in “insights for better decision-making” these days than I hear demand for “ROI.” Not sure why.
Katie:
I think because people realize that ROI is too hard to do right.
Tom:
I think this is a great debate. A few thoughts:
1. Calculating the “all in” number for an enterprise is going to be very difficult to do. When you take all of the revenue and all of the costs and calculate a true ROI, you are not going to even see the impact of any one tool or set of tools. The numbers will be minuscule. As Fraser pointed out, you will need to include things like call center costs, customer relations, back-office support, logistics, merchandise costs, support, etc., etc. I was once told at Sears that my entire PR budget was simply a rounding error. So I doubled it and nobody noticed. My point is that the impact of PR will be virtually impossible to pull out of such an “all in” calculation.
2. When investors calculate an ROI, it is generally a very simple calculation. I pay $1,000 for a stock and sell it for $1200 and my ROI is 20%. No muddying the waters with carrying costs, opportunity costs, etc. If we can keep our calculations clean and simple, we will serve the profession better, I believe. Will we trigger debates on the proper use of an accounting term? Yes, but even my former CFO had no problem with the way we were using ROI. It helped answer the question that really mattered: If I invest more dollars in PR, will I get more than my investment back in additional revenue, avoiding negative crisis and other benefits? On the other hand, if I cut the budget by 50%, will it have a negative impact on revenue and increase the risk of negative events damaging the company’s real value?
3. So my view is that even finance professionals will support a well-reasoned model that produces a legitimate means of evaluating “How much more or how much less should I invest in this function?” There are no perfect models, and searching for the perfect model is like trying to identify why someone gets cancer: Is it genetic, environmental, behavioral, or something entirely random? Way too many variables for even IBM Newton to evaluate with any degree of confidence. The same is true when you try to create the perfect measurement system of something as complex as a national retailer. Way, way too many variable to put in the mix. I tried and gave up when at Sears because I was driving myself and my boss crazy. Despite all the measurement and presentations we did, it really came down to whether the CEO believed we were making a difference or not. One CEO believed in PR strongly; his successor did not and the budgets reflected that. Our arguments and presentations and studies made very little difference.
We were fortunate to have a test case where we could isolate a major placement in the NYT and compare that to actual sales numbers in NY/NJ vs. other markets where the article did not run. That was back in 2000. Fifteen years later, when that same article would be available everywhere via the web, you couldn’t even make the same argument we made back then. Instead, we would be left with the opinion of the business unit head saying, “Sears customers don’t read the NYT, therefore it had no impact.”
Perhaps a better term is needed, but I would point to the way others use ROI (such as investors) as an argument for appropriating the term or something similar when we can actually demonstrate the value of PR or other communication tools using actual numbers.
Some questions are too hard to parse out with definitive proof. A little faith is needed.
For example, you could argue that the bump in footwear sales was a random fluke. Perhaps there was a big cross dressing convention in NY that week and the article had nothing to do with the bump in sales. Unless we could evaluate EVERY variable — even those outside of the organization’s control — you have to take a reasonable step of faith in order to move ahead with any decision.
Mark, your comment about more interest in “insights for better decision-making” gets to the heart of the matter. If we are not measuring so that we can make better decisions, why do it? To make ourselves feel better?
And, if you have enough confidence that your decision is correct, you don’t need more data or more measures. You need to move ahead and demonstrate that the decision was correct or that it isn’t working and you need to adjust course.
I would love to have better measures. I would also like to prevent cancer.
Katie:
I hate to say it, but most people measure only to justify their budgets, which yes, makes them feel better.
The clients that really want insight don’t necessary need or want ROI. They just need to get agreement on what success looks like it and then measure against that metric, factor in some resource time/money. THEN you have enough information to at least have a clue what is cost effective and what isn’t.
In one of the more enlightening moments of my career, I was given some very simple reports with no data in them other than the number of press contacts made and the number of stories published. I produced the chart below (it was the only thing I could figure out to chart from half a dozen 10-page “reports”). But it yielded far more insight and better conversation than a lot of much fancier charts that I’ve generated in my day. At first there was shocked silence, then there was a lot of “yeah buts,” and finally they started analyzing exactly what each office had done, what they had announced, and why one office was so much more efficient than the others. This was not an unsophisticated client; they have a robust media measurement system, a lot of data, and a certified data geek that runs measurement. But this is what insight looks like to them:
Fraser:
Love the chart, Katie.
See, now you’re into management consulting. The chart is all about efficiency and productivity.
Mark:
Well… me too.
The point about “insights” rather than “ROI” speaks to what I’m hearing now in the marketplace. 5-10 years ago, it was all about “proving value” and “ROI.” Now it’s insights. Certain phrases become buzzwords. How often do you hear about “PR as story-telling” nowadays? I met someone the other day who is the “Chief Story Teller” at a Fortune 100 company.
Fraser:
Well, insights gets us back to the issues management period of the late 70s, early 80s when Howard Chase led the charge to have PR branded anew as Issues Management.
The heart of issues management was environmental scanning. PR should lead an environmental scanning program for the organization. Lots of practitioner articles on ES systems in the 1980s. Then, environmental scanning got picked up by the strategic planning folks, as part of an outside-in approach to planning. PR lost its hold on ES.
PR went on to frame ES as stakeholder relations, that simple scanning of stakeholders was not enough and was replaced by building relationships with them.
Then social media hit and scanning is back in because relationships have moved beyond identifiable stakeholders GROUPS to those INDIVIDUALS who self identify, aggregate, and act, seemingly out of the blue. We call scanning “monitoring” now.
Led by Jim Macnamara, the new emphasis is on PR’s ability to lead listening; to develop information, turn it into intelligence, and then analyse to create insight.
Me thinks we are back in the 1970s, just with new terminology.
Of course, I have always counseled that environmental scanning, formative research, listening, monitoring, intelligence, insight… is the #1 priority of any CCO.
Mark:
Reasonably stated. And I think you’ve hit the nail on the head with your point about PR being too small. It’s already a rounding error in the marketing budget; what would happen in the overall “all in” budget? We’d be in the fourth decimal (.000x). Also, for what it’s worth, no one else within the organization is measuring their function based on “all in” measures. I like to be conservative when making claims – especially ROI as I know it – but this would be conservative to a point that it provides no insight for PR planning and evaluation. ∞
