Your 10 Biggest Mistakes When Hiring a Media Monitoring Vendor

When you’re faced with over 400 vendors of something that looks or sounds like media monitoring or measurement, the first thing you need to do is eliminate most of them. The easiest way to do this is to establish a basic set of qualifying criteria such as:

• Can they collect from all of your required media outlets? In this day and age, any vendor that cannot provide cross-platform collection — news, broadcast, Twitter, blogs, etc. — shouldn’t be on your list. Your customers regularly consume news from an array of sources (digital natives switch media 27 times an hour) so you need to be monitoring whatever or wherever your customers get their information.
• Can they monitor in different languages? There are few organizations today that only operate in a single country or a single language. If you aren’t one of them, you can drop any vendor that can’t meet your language and geographic requirements from your list.
• Have they pledged to support the standards? The standards that have been established by The Conclave and the Coalition are in place to make this process easier, and to make your data more more consistent and more valuable once you get it. Make sure any vendor you talk to has signed the standards pledge.
• Can they provide the items you need in the timeframe you require? If not, scratch them off your list.

It is astounding to me that in these days of RFPs and hawk-like scrutiny from purchasing departments, so many decisions still get made on whether or not someone likes the salesperson. We all understand the importance of personality and relationships in choosing a vendor. But that person you liked so much in the sales process will hand over your account to an account manager as soon as you sign the contract.

Two tools will help you deal with this. First, use the Sources & Methods Transparency Table produced by #smmstandards. This will insure both that you know what’s inside a social media monitoring program, and that you can more easily compare apples to apples between vendors.

Secondly, every vendor-selection process should incorporate a scoring sheet that rates each vendor in your search, so that you have a logical and methodical way of evaluating your choices.

A good place to start is with the Vendor Questionnaire/ Transparency Reporting Template. You’ll need to decide for yourself how important each criteria is, but ultimately you will want to end up with a table that looks like this (click it to see it bigger):

Reporting results based on lousy data is like building a high rise building on a foundation of bad concrete: It will look good just long enough to get everyone to buy into it, and then it will collapse. There is nothing more important to measurement than the accuracy of your data. So how do you check to make sure that your data is accurate? Test, test, and test again. (See our article here for tips on how to clean your data.)

Get your vendors to provide at least a week’s worth of data. Pay them if necessary; it’s worth it to know what kind of data you’ll be dealing with. First, test it for completeness. If you have no monitoring in place, at least run Google Alerts for a week to see what kind of quantity you’re getting. If you have a system in place, compare the test results to what you’ve been getting thus far. Then double check that your key media outlets are represented in the data.

Next, check for duplicates and spam. Chances are about 50% of raw data will fall into one of those two categories. A good vendor will have systems in place to screen for duplicates, spam, and inappropriate content (like wedding announcements, incorrect names or references, and police blotters). If they don’t have the ability to create good filters, you’ll be paying for a lot of stuff that you don’t want and you’ll have to do the cleanup work yourself.

Finally, check for accuracy in the way the incoming data is tagged or coded. Chances are you need data in specific buckets: corporate, product, customer service, etc. Most systems rely on some sort of algorithm to identify these subjects based on computer-automated selection. Check a random selection of items — at least 50 — to see if they are correctly tagged and coded.

If your vendor is providing sentiment or tonality, you will need to conduct a separate test to validate their coding. Select a sample of 50 items and have them read by an intern, or someone who didn’t have anything to do with “placing” the stories. Compare the results to the system. If you don’t find agreement in 80% of the items, you may have to adjust their algorithm. Make sure they can handle this level of customization, or you’ll be in trouble down the line.

In today’s torrent of media, you will be surprised at just how many “alerts” you will get from your monitoring system. So many, in fact, that the system will soon be like the boy who called “Wolf!” too many times, and you will find yourself skipping the alerts.

But the reason you have a monitoring system is to ensure that you aren’t the next Domino’s Pizza or Kenneth Cole. So you need a process to stay on top of them and send them along to the people who will need to handle them. Do not foist that task onto your summer intern. You need to identify your own internal Olivia Pope who has the judgment and background to know how and when to respond.

Whether you are incorporating human coding or relying solely on machine coding — or taking a hybrid approach and combining the two — you still need to have good clear definitions. Make sure that you provide your vendor with information, and make sure they understand it. Any abbreviations, acronyms, or internal code words need to be clearly defined and those definitions should be provided to each vendor.

Part of good definitions is manifested in the search terms you provide to the vendors. Search strings tell their systems what to monitor for. And even more importantly what to exclude. So, for instance, if you are the PR manager for the city of Philadelphia, then you will want to make sure your monitoring excludes the Philadelphia Eagles and traffic reports for Philadelphia Avenue in King of Prussia.

If you’ve been using Google Alerts and doing your own collection, you know how time consuming this process is. So don’t expect someone else to do the same thing better, faster, or more efficiently and think it will cost peanuts. The reality in the monitoring world is that you get what you pay for.

You can pay as little as $5,000 a year, but you’ll have to do most of the work yourself. A monitoring and measurement service that comes with account management can range from $20,000 to $500,000 depending on the number of items collected and analyzed, and the degree of customization. Do not be the client I had once who asked me to prepare a proposal to monitor, measure, and evaluate its media coverage in 10 different countries, with 5 different competitors — and then admit later that the budget was only $25,000.

Don’t make the mistake of believing that all monitoring vendors do the same thing. They do not. Some are designed around vertical markets, some are designed for large enterprises, and most do a few things well. The rest of the promises you can chalk up to good website copy. Some are great at monitoring. Others are great at managing your conversations, but only so-so at monitoring. Others may be great at local coverage, but have no national or international experience. Make sure that the vendors you are talking to have experience in your industry, and are really good at what you need them to do. Again, use the Sources & Methods Transparency Table produced by #smmstandards.

Finally, with the plethora of systems out there it is easy to get caught up in the thrill of pretty charts and sexy metrics. But don’t allow yourself to be distracted from the most important question: “Does the system meet our needs, will it help achieve our goals, and will it enable me to do my job better and more efficiently?” When your answer is Yes, you’ll know you’ve found the right vendor. ∞

image: The Tim Sackett Project