This article is part of The Measurement Advisor’s Special Issue on the Measurement Sherpa, an in-house data wrangler and measurement resource who organizes, queries, and gains insight from data. Our coverage follows the development of a Sherpa’s skill and experience, from newbie to pro, and is organized in three levels:
Sherpa Level One: Getting Out of Base Camp
- Introducing the Measurement Sherpa
- 10 Signs That You Need to Hire a Measurement Sherpa
- 5 Reasons Why You Really Want to Become a Measurement Sherpa
- The First 5 Steps to Take to Become a Measurement Sherpa
- Learning on the Job: Notes from a Sherpa-in-Training
Sherpa Level Two: Climbing the Mountain
- Must-have Equipment for a Measurement Sherpa
- Commonly Confused PR and Social Media Measurement Terms
- 6 Books That Should be in Every Measurement Sherpa’s Library
- Katie Paine’s Advice for Measurement Sherpas: Dos and Don’ts for Reporting Web and Social Media Analytics
Sherpa Level Three: Peak Operations
A Measurement Sherpa’s job is to tend an organization’s data, and to be inquisitive, and insightful with it. There are certain data-related projects that will come up again and again as a Sherpa helps with the communications measurement projects of an organization. Here are six projects that any Measurement Sherpa needs to know how to do:
1. Prepare a measurement budget.
2. Manage an RFP process.
3. Conduct a data validity/accuracy test.
This is a two-part process. First you need to test for accuracy of data: Were the items collected really about the topic or about something unrelated? This requires that you personally review a sample of the data. I recommend looking at about 500 items. Assuming the items are in an Excel data sheet:
- First do a custom sort by date, then by publication, then by subject. This will quickly find any obvious duplicates.
- Then do a search for terms that you know are frequently confused or linked to spam—“Viagra” is a classic. I’ve eliminated 10% of bad data just by looking for Viagra and swear words.
- Next sort by date and check the activity level around a particular announcement. Is the coverage you expected there?
- Now look for dates when you know nothing was going on. Is there data there? Is it about your brand?
Once you’ve ascertained that the data is clean, you need to check the accuracy of the coding. Typically this is done via an intercoder reliability test. Krippendorf’s alpha is the preferred metric. Have trained human coders read each item according to industry-standard coding parameters and following your detailed coding definitions. You will want to achieve a minimum of 85% agreement, which meets academic standards for intercoder reliability.
4. Calculate cost benefit analysis.
You should have the formula for correctly calculating ROI tattooed on your brain. You need to know the total investment as well as the net return, here’s an example:
After I helped launch the first HP LaserJet printer, I was asked to do a promotion to computer dealers who had never before carried a Hewlett-Packard product. We put together a snazzy campaign and went a little over the budget, spending $15K instead of the budgeted $12K. So I was called on the carpet to justify the extra spend. I pulled a list of the dealers who received the promotional piece (none of whom had ordered from us before) and compared it to the list of actual orders in the subsequent 60 days. I plugged in the average revenue HP got from a Laser Printer, multiplied it by the number of orders and calculated that our “over-budget campaign” had generated $1.3 million in incremental revenue, or an 8567% return.
Most of the time getting sales and order data is tough, and not nearly as clear cut as my example. So you may have to come up with some other technique. An excellent alternative is a cost-benefit analysis. You can read Fraser Likely’s excellent paper here. Essentially what you’ll want to do is to take the metrics on two similar campaigns that had different budgets. Calculate a cost per engagement or cost per message communicated for each and compare the two.
5. Search a data base for obvious errors.
6. Know where to find stuff in a hurry.
A key to being a Measurement Sherpa is having all the measurement answers at your fingertips, so get organized. You will constantly be downloading white papers on relevant measurement topics (we hope from Paine Publishing), so make sure they are organized and in a place you can access easily. Keep copies of any reports in easily accessible locations, since the moment people realize you know the answers, they’ll come asking you for them. Make sure you have the raw data available, not just the pretty PowerPoints. You can save a ton of time and money if you can re-analyze old data. ∞
Thanks to The Practical Psychosomaticist for the image.
Comments are closed.