In our last post on the trials and tribulations of creating a Barcelona Principles-compliant measurement program, we covered the challenge of simply getting agreement on goals. See that post for the stories of how our six Barcelona compliance pioneer organizations managed to achieve consensus on how PR contributes to the bottom line.
Now we’re going to get into the nitty gritty of implementation, specifically, where the data for these metrics will come from. All of our Barcelona pioneers started out with an audit of available data. An important step was to make sure that the data they thought was there was in fact “available.”
Step 1. Finding your data—maybe.
In most organizations everyone knows data exists, they just don’t know where it is. And unfortunately, in many organizations people guard their data the way dragons guard gold. For almost every one of our Barcelona pioneers the power of the CMO was required to get people to hand over data. In some cases we hit an institutional brick wall and needed to figure out a path around it. As with one non-profit, where donation and member data was simply not going to ever be available to the communications team. They were, however, able to use Google Analytics to determine how many people went to the donation and volunteer pages, and how many people actually saw the “thank you for giving/signing up” pages. Leadership was willing to accept these as proxy measures in lieu of the actual donation and member data.
In several organizations there was an assumption that data existed when in fact it:
- Had never been collected; or
- Vendor contracts hadn’t yet been signed, hence the data wouldn’t be available for months; or
- Data sources that were once available had been eliminated for budget reasons.
In some cases we had to go back to the drawing board to change, replace, or eliminate specific metrics. In others we had to go out and find the right vendor to match the metrics.
Step 2. Finding the right tool and vendor for your metrics.
Most of our Barcelona pioneers needed to develop new sources of data. For several of them, a key goal was message communication, but their media monitoring companies didn’t provide that service. For another, sophisticated social analytics was critical, but their freebie subscription to a basic social analytics platform wasn’t up to the challenge. At yet another one, awareness was the key goal, but they only conducted awareness research annually. So they needed to find a way to conduct short pulse surveys quarterly.
If the data you need is simply not there yet, you’ll probably have to conduct a vendor search. Which is what our Barcelona pioneers had to do. In any vendor search you start with a standard set of requirements and criteria. Our best advice is to use the The Conclave’s Standard Transparency Table (also called “The Transparency Reporting Template”) as a base, and adapt its list of vendor questions to meet your organization’s specific needs. For more detailed information on handling an RFP process, read this article.
Example: One of our Barcelona pioneers, a large pharma company, began by conducting a vendor search using the Transparency Table. This enabled them to immediately eliminate a number of vendors that didn’t meet their basic requirements. To rate the rest of the vendors we created a set of vendor judging criteria, weighted according to the priorities of the pharma company’s dashboard. The weighting was assigned as follows:
- Accuracy: 5 points
- Completeness of data: 5 points
- Timeliness: 4 points
- Ease of use: 3 points
- Cost: 2 points (Cost was weighted relatively low in this instance because the organization was saving money by eliminating at least 3 duplicative reporting systems.)
- Capability, credibility, stability, culture of the supplier: 1 point
Our next step was to assess each vendor on the above criteria. So we did a “stress test” on each to determine accuracy and completeness of the data, as well as to gauge their ability to deliver on time. We asked each of the vendors to collect and analyze data during the same four week period, providing each of them with the same criteria and search terms. The results were shockingly different, see Figure 1.
Positive sentiment ranged from 7 to 58 percent, while negative varied as much as 25 percent! Some of these differences could be explained by the methodology. The incumbent vendors, Beyond and SDL, used human coding, while UberVu and NetBase used automated coding. Not surprisingly, the automated coding found far more neutral items, while Beyond reported very high positives, and SDL reported the highest negatives.
We were able to examine the raw coded data from three of the companies to help us determine the cause of the variance. The fourth company refused to provide the raw data so they were eliminated from further tests.
To further test the automated vendors, we set up an intercoder reliability test using the standard coding guidelines established by the research of Eisenmann, Geddes & O’Neil. We employed a team of highly-trained human coders with a minimum of five years of complex content analysis experience. The human coders were presented with raw data from a one month period and asked to code for positive, neutral, and negative sentiment based on an agreed-upon standard coding document.
Once the human coding was complete, the team compared the human coding with the coding provided by the three leading vendors. With 89% agreement, NetBase clearly surpassed the other vendors on the accuracy test, see Figure 2.
Our next test was for completeness of data. The company had in place a comprehensive (but very expensive) monitoring system that was used as the benchmark. If a vendor could find everything the incumbent system was finding, it would be an acceptable alternative. Unfortunately, initial results showed that while NetBase excelled in in accuracy, it failed on the completeness of the data test, see Figure 3.
As it turned out, for the purposes of the test, NetBase had only provided a sampling of its full feed. Once it provided the full feed, it was found to get 99.9% of the alerts provided by the incumbent.
Step 3. Doing the Laundry
Once you’ve tracked down your data, you’ll probably not want it all. All data is not good data. It needs to be clean, valid, quality data—and that might be your biggest challenge of all. Read this article for more information on how to clean your data.
To make sure your data is accurate, you may need to do a validity test, as in the following example.
Example: In the case of our tourism Barcelona pioneer, a vendor had been selected based on its ability to deliver a very broad and complete selection of clips. But errors in coding were found during the very first reporting process. Coverage of specific events had been miscoded and the tonality seemed wrong.
Once again we conducted an intercoder reliability test to determine the accuracy of the vendor’s coding. The following charts (Figures 4 and 5) illustrate the discrepancies between the coding of trained humans using the Eisenmann, Geddes & O’Neil guidelines and the relatively untrained coders of the incumbent vendor. “Yes” indicates that the trained coder agreed with the way the incumbent vendor had coded. The red means they disagreed.
Clearly better training was in order. In fact, the client elected to switch to a different vendor.
Lessons learned: Expect the unexpected.
For all six of our Barcelona pioneer companies, the process of locating clean, usable data was filled with unexpected difficulties. “Who knew…?” was the most frequent comment my clients and I would share:
- Who knew how complex some of these platforms would be to implement?
- Who knew what kind of push-back we would get from people who had a large personal stake in the “old reports,” full of activity metrics and zero insight? We had people advocating for old reports even though they admitted they were bogus, just because change was harder to swallow than bad data!
- Who knew how hard it would be to create an “apples to apples” comparison between vendors? The media analysis vendors were bad, but the social analytics vendors were worse.
And here are some lessons we learned the hard way:
- Don’t start a vendor search until you have defined and agreed upon criteria for success.
- Don’t believe any technology will do what it promises until you’ve gotten it to do what you want it to do.
- Don’t assume any data collection process is accurate or consistent unless you have tested it.
- Don’t finalize any standards or metrics until you’ve done the math and tested your theories.
We hope that these examples will make your own journey to Barcelona compliance a little easier. Next issue: Crawling to Compliance, Part 3: What to do with the data. ∞
Thanks to JJ’s Beautiful Mess for the image.
