With RFPs flying around us every day, and confused clients throwing up their hands and saying “Help…I can’t figure this all out!” we thought it’d be the perfect time to provide a basic “how to” lesson on selecting the right vendor (as well as a brief review of the latest tools).
First of all, the IPR Measurement Commission Guidelines for Measuring PR has a wonderful checklist of questions to ask anyone you are thinking about hiring for a measurement project. The next question is: “What’s the budget?” There’s no point in putting out an RFP for a research project that will bring back bids larger than your PR budget. There are lots of ways of bringing costs down—mostly by looking at smaller, more focused influencer or Top Tier Media lists—so don’t think you can’t afford measurement. With today’s tools anyone (yes…including YOU!) can afford measurement, but the key is matching your budget with the results that you expect.
You should plan on spending at least 5% of your communications budget measuring your results. Think about it: doesn’t it make sense to spend at least 5% of the budget with which you talk at your customers to actually listening to them? Here are a few general guidelines we use to guesstimate programs.
- Automated content analysis systems have brought the cost of big programs down considerably, but when all is said and done, it still costs money to get your content analyzed by humans, see Katie Paine’s Vendor Selection Chart for the right choice for your project. Assume somewhere between $3 and $10 a clip for content analysis. Can you get it cheaper? Sure. Is with your data be valid and reliable? Probably not. Can you spend more? Absolutely. Most organizations don’t break out their charges on a per clip basis, but it’s not hard to do the math.
- For survey research, assume a minimum of around $75 per completed survey, depending on the complexity of the analysis, the difficulty in reaching people, and the availability of a good list.
- If you are on a really tight budget and want to do the analysis yourself, you still have to get the media items from somewhere. Figure you’re going to spend at least $300 a month on clipping.
- Going international is expensive, no matter how you do it. Adding “Europe” is adding multiple countries, and multiple languages. Do you mean U.K., France, and Germany, or all of the EU, and do you want to include Eastern Europe? Figure that if you’re spending $30K per year to analyze your coverage in one country – the U.S. – then you will probably spend that per country overseas if all the parameters remain the same. Asia always seems to be slightly more expensive than Europe, and Latin America is typically cheaper.
So, once you have a rough idea of budget, you can begin to define what it is you are going to get for your money. Let’s look at it by way of the 3 phases of any research program:
Phase 1: Data collection
For survey research this means crafting a survey and distributing it to the target audience you want to survey. This requires both a list of people to survey and a method to ask them questions. The cheapest route is to use an existing database and a free tool like Survey Monkey. Unfortunately, using your own list is likely to produce biased results, so you may need to go out and purchase a list. Survey Monkey will be happy to sell you a list, as will Survata. Both charge by the number of names you buy. Just remember that for most surveys you will need a minimum of 500 completed surveys if you want statistically significant data.
In social and web analytics data collection means making sure that the data you want to track is set up to be tracked within the system. Is the tracking code in place and working? Do you have access to your web analytics, Twitter Analytics, or Facebook Insights? If you need to measure a metric that isn’t available in the platform, or you are too allergic to math to deal with all the numbers, you may need to go to an outside firm like Unmetric or Simply Measured.
To analyze your media coverage you have to have access to the relevant coverage—and, if you’re looking to measure share of voice—that of your competitors. This is true whether you use a print clipping service like BurrellesLuce, or an online provider like CyberAlert, or you just use Google News Alerts.
If you are doing competitive media coverage it is absolutely critical that your methodology for clip collection is the same for both yourself and the competition. We recently saw a client who was convincingly beating the competition every month in key trade publications—until we looked into how they were collecting clips. Turns out the person responsible for pulling the clips hadn’t been told to look for the competition as well. Needless to say those great results didn’t last…
One thing to remember about services like Factiva or Moreover is that they are not all created equal. There are different levels of service, which translates into different publications being available. So make sure you check to see that your key publications are available through the service. If you want local weeklies that only exist in print form, you have only one choice: BurrellesLuce or a local clipping service.
Phase 2: Media or Data Analysis
In survey research this is where someone has to actually sit down with SPSS, SAS, or Excel and actually crunch numbers to create the analysis and conclusions that have to be drawn. If you go with Survey Monkey, their analysis and reporting features are limited but adequate for most simple surveys. If you need to do extensive cross tabs and correlations, then you’ll want to export the entire data set into Excel.
If the thought of cross tabs and correlations is giving you a panic attack, you’ll want to call in the experts. My go-to source for market research is John Gilfeather at Koski Research. I’ve worked with him for years and trust them implicitly to do excellent research. I’ve also had great luck with Research Now. And of course there are the big guns like Ipsos, Gallup, GFK, Qualtrics, Synovate, TNS, etc.
For web and social analysis if you’re using a service like Unmetric, Simply Measured, Omniture, or Google Analytics, then all you need is someone who knows how to use it to generate the reports you need. Those tools can pretty much crunch data and export in anyway you can imagine.
In media measurement, the choices are varied. Any automated platform just needs a trained operator to get the data out of it that you need. But if you want human analysis of data, (i.e. someone actually sitting down with the posts, articles, and determining if they contain your key messages, if they are positive or negative, whether they mention your brand benefits, etc.) you have several options:
- Option 1: Do it yourself. I’ve never been a huge fan of PR people reading and analyzing their own clips, since we typically do not consume media like normal human beings. Most of us are news junkies and far more sensitive to messaging than regular folks besides, PR people can spot their own key messages from a mile away. But, if all you can afford is someone in-house, then you need to have a very detailed set of reader instructions to ensure consistency between readers.
- Option 2: Go the automated route. According to our research, most automated analysis engines only get sentiment right about 50% of the time. NetBase is the only system I’ve tested against trained human coders that gets it right more than 88% of the time. It can be done, but you can’t just buy a platform and expect it to be accurate straight out of the box.
- Option 3: Outsource it. There are a few companies that have long track records in human coding. Most of the large PR agencies will do this for you. While some outsource it, others have trained analysts on their research teams that can do this, but you will be paying standard agency fees to get the job done. In terms of private companies, CyberAlert, (that absorbed most of the old KDPaine & Partners team), Gorkana (that absorbed Metrica), Carma (that absorbed Report International and Salience), Media Miser, and Millward Brown Precis all tout their human coding capabilities. I’ve worked with all of them and find that as long as you stay on top of it, their results will be accurate.
Phase 3: Making It Meaningful, Finding the “So What?”
As we’ve said before “Data without insight is just trivia.” And nothing is more trivial than a chart without a headline that helps you understand what it means. So analyzing and reporting on the data is just as important as the systems you’ve used to gather that data.
Make sure your vendor’s reporting capabilities match your needs. Most vendors offer fairly standard charting and then charge extra for a more thorough analysis. Plan to spend at least $2,000 for a good report complete with insights and recommendations.
The best approach is to have your vendor prepare the report and then sit down with you and your agency and discuss the results, and then write your conclusions and recommendations. But that doesn’t happen very often. So the next best thing is to brief your vendor on what happened during the period, keep them up to speed on the priorities, and let them do a first draft based on the data. Then work with them to refine it based on those priorities.
Do not use pie charts. Knowing you have the greatest share of ink is nice. Knowing that your share of negative coverage has increased every month this quarter is priceless.
Avoid the “big spike in June” headline. Anyone with eyes can see that you had a big increase in positive coverage in a particular month. What you want to report on is why. What made that trend chart go up or down? You, or whoever you hire, needs to put the results in context, and explain what it is your audience is looking at.
∞
Paine Publishing provides communications measurement training and education. Katie Delahaye Paine provides customized measurement training and services.
