Readers frequently ask me how to measure success as they define the goals and metrics for their departments. And as much as I love examples like the revenue and profit ones that Avinash Kaushik cites in this wonderful piece, they just don’t seem to be relevant to most of the people struggling with measurement.
Primarily, the problem lies with the fact that those people aren’t selling widgets, and they don’t pay for Google Adwords or any other paid advertising. Their communications goals are focused on more esoteric concepts like trust, credibility, compliance, and changing perceptions.
Their “champagne moments,” (i.e., the program achievements that will get their boss to walk into their office and drop a case of champagne on their desk because they’ve done such a wonderful job) are likely to have to do with “be perceived as the most credible source,” or “increase compliance,” or “keep the haters and pitchforks away from our doorstep.”
These concepts are not easy to measure. The process of developing measurement programs for them is often fraught with difficulty. So I thought it might be useful to share my insights and solutions on how to measure success for a few real-life case studies.
Case #1: A regional financial regulatory organization
Primary Target Audiences: Educators, influencers, and business owners residing within the five states in the organization’s jurisdiction.
Champagne Moment Goal: Be perceived as the most credible source of economic information by a majority of constituents.
Challenge: Normally we’d advocate for regular surveys to measure the degree to which people found the organization credible or trusted its data. But in this case, alas, there was neither budget nor appetite for regular polling.
Solution: Brainstorm to come up with acceptable proxies for credibility. In other words, “If people find your content credible, then how would you know?” As it turns out, the organization had quite a lot of data on the extent to which people downloaded their reports, shared or quoted their data, recommended their experts as sources, and invited those experts to speak at conferences. All of which, everyone agreed, would only happen if they found their data credible.
Additionally, we developed an earned media score to quantify credibility in media coverage:
| A Perfect 10 Item |
Score | A Worst Nightmare Item |
Score |
| Item (post, article, etc.) contains a hashtag or message | 3 | Item does not contain a message | 0 |
| Contains a quote from one of our experts, or attribution for our online content or research | 2 | Contains no quote and/or no attribution for our research | -1 |
| Positive (leaves an analyst more likely to support the organization or its positions) | 1.5 | Negative (leaves a reader less likely to support the organization or its positions.) | -3 |
| Contains a desirable visual | 2 | Contains an undesirable visual | -3 |
| Appears in Tier 1 media | 1.5 | Contains a factual error | -3 |
| Total score | 10 | -10 |
Ultimate Measures of Success:
- Quarterly percentage increase in downloads of reports
- Percentage increase in shares and desirable comments in social and traditional media
- Percentage growth in subscribers to newsletter
- Percentage growth in subscribers to blogs
- Percentage growth in followers on social media
- Percentage increase in depth of sessions, i.e., number of sessions per user and number of pages per session
- Percentage improvement in quality score
____________________________________________________________________________
Case #2: A state government organization that manages licenses and other regulations
Primary Target Audiences:
- Internal staff
- Licensees that need approval from the organization
Champagne Moment Goals:
- Be seen as the most responsive government agency.
- Provide a great customer experience that ensures compliance.
Communications Objectives:
- Increase internal engagement with, and buy-in to, the organizational priorities.
- Raise awareness of funding opportunities by 30%.
- Increase by 10 percent the number of licensees that perceive the organization as “responsive.”
Challenge: Little budget for measurement and little appetite for constituent surveys.
Solutions:
- To measure the extent to which employees understand the organization and its priorities, we recommended gamifying questions on these topics using their internal communications channel. So a quarterly knowledge quiz was introduced, with competition between the various organizational teams. The winning teams (those that presumably best understand the priorities) would not only get recognition but a contribution to their favorite charity in their name.
- Ultimately, the point of employee engagement was to increase the satisfaction scores from applicants and other members of the public. So an NPS-type survey could be added to the website to gather feedback.
- To measure the extent that people are aware of the organizations’ funding opportunities, we recommended that they use a combination of web analytics and “touch points,” (i.e., the number of opportunities to have contact with a member of the target audience, whether at an event, through social media, email, or other outreach method). We recommend using touch points as an indicator of engagement rather than reach, which is frequently inflated.
- New and repeat visits to their program’s funding pages were used as an indicator of awareness. Actual online applications were also counted. Events were set up on Google Analytics to track online application downloads and completions as well as sign-ups for additional information.
- For email campaigns we recommended click-throughs, not opens.
- Engagement on social media was also considered an acceptable proxy for touch points, e.g., increase in follower count, shares, and comments.
Ultimate Measures of Success
For employee understanding of priorities:
- Percentage participation in knowledge quizzes
- Percentage correct answers
- Percentage increase in customer satisfaction (NPS) score
For external audiences:
- Percentage increase in new visits of longer than ten seconds to their funding application website
- Percentage increase in completed applications
- Percentage increase in touch points
- Percentage increase in subscribers to email
Case #3: A military organization
Primary Target Audience: Voters
Champagne Moment Goal: Increase support enough to motivate voters and legislators to approve the organization’s budget.
Communications Objectives:
- Reduce the percentage of community opposed to the organization by 10 percent by the end of Q2.
- Increase the number of applications from qualified recruits.
- Increase supportive media coverage by 20 percent by the end of Q2.
Challenges:
- The likelihood that a young person will join the Armed Forces is heavily dependent on the people from whom that young person takes advice. These are typically teachers, coaches, mentors, family, and friends. So it is critical that the military maintain good relationships with community leaders.
- All military branches are ultimately dependent on legislative bodies for budget approval. Which means that influential voters and organizations in the home districts of lawmakers on key committees are highly influential on the future of the military. So relationships with leaders in those districts are key. And thus reducing opposition among legislator’s constituents is key, or else they will want to reduce the military budget.
Solution:
At NATO, for example, a large part of every Public Affairs Officer’s (PAO) job is making sure the communities around NATO bases understand NATO’s role and purpose. Since a good part of the organizational funding comes from host countries, continued support is critical. So the little things that affect community attitudes, like garbage and water and local sports leagues, become a high priority of NATO PAOs in the field.
In order to measure this, NATO set up a measurement system that tracks coverage of NATO in local communities as well as throughout Europe. They developed a custom quality index that takes factors such as “desirable image or photo,” messaging, the reach and importance of the media outlet, and whether the story leaves a reader more or less likely to support NATO’s mission. These quality factors can then be analyzed by program, campaign, and against specific spokespeople and outlets to determine what generates coverage that gets the desired outcomes.
Measures of Success
- Percentage reduction in media coverage that leaves the reader more likely to oppose.
- Percentage increase in media coverage that leaves the reader less likely to oppose.
- Percentage increase in share of favorable quotes from key influencers.
- Percentage increase in opportunities to see messages in key geographies.
- Percentage increase in trust levels by local community.
∞
