By Katie Paine, CEO, Paine Publishing
Measuring the effectiveness of your communications with employees sounds like a wise idea, right? Right! Here are 5 steps to successfully do it:
Step 1: Understand the Environment and Where They Get Information
A thorough, honest, and independent evaluation of existing communications—both official and unofficial sources of news and information—begins the process. Conduct a benchmark audit using the appropriate tools (from the tools section below) or use data that is already available somewhere in the organization. This initial benchmark study might be more or less involved depending on the size and complexity of the organization, what measurement tools are already being used, and what data is already available.
First you need to collect and analyze samples of all the various news bites, rumors, and pieces of electronic as well as non-electronic documentation that regularly bombard your employees. You’re looking for:
How are messages getting through to employees, and what are they?
You’ve probably wondered more than once whether all those memos and emails are being read. Are they getting to people and/or departments in a timely manner? Are they being passed along or automatically deleted? Are they reaching the right people? (This is where BananaTag would come in handy.)
Step 1: Understand the Environment and Where They Get Information
A thorough, honest, and independent evaluation of existing communications—both official and unofficial sources of news and information—begins the process. Conduct a benchmark audit using the appropriate tools (from the tools section below) or use data that is already available somewhere in the organization. This initial benchmark study might be more or less involved depending on the size and complexity of the organization, what measurement tools are already being used, and what data is already available.
First you need to collect and analyze samples of all the various news bites, rumors, and pieces of electronic as well as non-electronic documentation that regularly bombard your employees. You’re looking for:
How are messages getting through to employees, and what are they?
You’ve probably wondered more than once whether all those memos and emails are being read. Are they getting to people and/or departments in a timely manner? Are they being passed along or automatically deleted? Are they reaching the right people? (This is where BananaTag would come in handy.)
We refer to this phase of measurement as internal message analysis. Analyze all outgoing communications, including emails, newsletters, memos, voice mails, videos, speeches, and presentations to determine what messages are being communicated, who is getting the messages, and what they are doing with them (such as deleting them, forwarding them, or saving them for later). More sophisticated clients actually analyze the email traffic to determine connections and networks that are developing. For large organizations, there are systems like Valdis Kreb’s Inflow to map the forwarding and response patterns of email.
While most organizations are naturally concerned about over-surveying employees, a quick survey on email usage generally pays off. JPMorgan Chase analyzed email usage and discovered that by managing email communications more efficiently, the organization could save several million dollars a year. Other important metrics are available from your intranet log files: To what extent are people clicking on various pages on your intranet? To what extent are they downloading?
What channels or vehicles do employees trust?
Of course we’d like employees to learn everything from “official” sources, but that just isn’t going to happen. In today’s media environment, they are just as likely to hear the latest company news on the sidelines of a soccer team or over a beer than to hear it from corporate channels. So you need to use your research to find out the influence of individual sources of information. Chatting around the water cooler or coffeepot is probably a valuable source for some news, but as we increasingly see workers telecommuting there are fewer and fewer opportunities for that kind of casual communication. Instead, employees instant message, Skype, or use Twitter to reach their colleagues. You will probably find that the company newsletter and company email is important for other types of information.
Use either a formal survey or a series of focus groups to find out what channels employees trust. Generally we find peers and immediate supervisors to be the most trustworthy, but this depends very much on the nature of the employee. One study we did revealed that engineers in a telecommunications company only trusted information they received electronically, and they inherently distrusted information delivered in big corporate meetings.
Child development researchers have discovered that children learn in different ways. Some respond more readily to shapes and colors, others learn verbally from words and pictures, while still others learn aurally. To get all the children in a classroom to learn the same thing at the same time requires a cornucopia of teaching tools. Employees are no different. Caterpillar, Inc., the tractor company, learned this lesson with regard to its internal communications program. The organization was accustomed to communicating internally in half a dozen different ways, but found that it could never quite reach everyone all the time; some group of employees always remained uninformed. Consequently, Caterpillar decided to put out the same news in every format and found dramatic increases in employees’ knowledge of corporate messages.
What’s important to them?
Some 40 years ago, in a keynote address to the Advertising Club of St. Louis, Ralph Delahaye Paine (yes, a relative…my father in fact) the editor of Fortune, mused, “If we can put a man in orbit, why can’t we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires.” His conclusion: “You’ll never get your employees to stand up and salute when you walk by, unless you’re in the military. The best you should aim for is that when chatting at the watercooler (he was born in 1906, long before Yammer) the consensus is, ‘Ah hell, this ain’t such a bad place to work.”
In the ensuing years, we have developed increasingly effective methods to measure people’s capriciousness, but the reality remains that humans still hear what they want to hear. What they want to hear is based upon what’s important to them, perhaps what constitutes their most pressing and unfulfilled desires. If I need a new refrigerator because my old one is broken, I will be particularly receptive to news and ads about refrigerators and appliances. The same goes for your employees. If their biggest concerns revolves around the health of the organization and job security, then your messages about vision, values, and health care benefits will hardly register with them. If a number of pregnancies exist in a particular department, then those benefit messages will be picked up on first—you can bet on it.
What do they think today?
Existing perceptions play an enormous role in whether or not employees will receive whatever messages you’re trying to communicate. If you’ve just announced a merger, the assumption will be that layoffs will follow, so unless you address those perceptions first, no other messages will be believed. But if you ask a few of the Grunig Relationship survey questions to test the health and strength of internal relationships, you’ll have a much better idea if they’re likely to listen to what you have to say. And then, given the nature of the relationship, you still need to get them to tell you if they understand the vision and values: Do they have an understanding of what the organization is trying to do?
In some companies, we find that the gap between line workers and management opens to such an extent that internal communicators need to begin the process by educating employees about what purpose the organization exists to fulfill. On the opposite end of the scale, some companies are using open-book management to keep every employee informed of not just the organization, but the actual financial details of the operation as well. You need to understand where your organization and your employees fit in that spectrum.
Step 2: Agree on Clear, Measurable Goals
Now that you understand the environment and your starting point, it is time to get agreement from top management about what you’re trying to accomplish. To do this you need to understand the vision, objectives, and messages that senior management wants to communicate (and what they expect of HR and your communications effort).
First, you probably will want to present to key management the information from your audit so that they understand the context. Even if, for timing or other reasons, a formal presentation may be impractical, you will want to interview key management and determine what their messages and objectives are. Based on a thorough analysis of your preparatory research, write down clear, explicit objectives and get senior management to agree to them. What do they think is important? What do they see as the corporate vision? What do they see as the strategic direction? Is the goal of your internal communications program to increase loyalty and productivity? To decrease employee turnover? To help in recruitment efforts? Is the goal to communicate specific messages?
Remember that what you measure has everything to do with what the goals of your communications efforts are. For example if you are using Twitter for recruitment, getting more followers is not the goal. In fact, finding better qualified, more engaged employees is the desired result. Therefore you need to have systems in place to track not just the quantity of applications, but the quality as well. You probably want to know if using Twitter has resulted in less employee turnover.
If your goal is greater engagement in the company’s mission, you need to determine how engaged employees are today, and then set reasonable goals for how much more engaged you think they should be after your strategy is in place. So a good goal might be: “Get 50% of all employees engaged by [a particular date].” Then get consensus on specifically what it means to be engaged (see our “Four Metrics” article) and how you are going to measure it. Does that mean heavier use of the Intranet as measured by web stats? More knowledge and buy-in with the mission as measured by a survey? Each has different metrics and requires different measurement tools.
Step 3: Define the Criteria of Success
This process involves defining the actual words and numbers to be used as you create your specific, measurable definitions of success. These criteria are numerical, and most often they are percentages or amounts expressed in dollars or numbers of something. Your definition(s) of success might include, for instance:
— My program will increase understanding of the corporate mission and values by X percent, or
— My program will decrease employee turnover by X percent.
This is where you decide what truly defines your success, and where you commit to achieving specific goals. Make sure your goals are achievable.
Step 4: Select Measurement Tools and Collect Data
Once you know what you want to measure, you must select your measurement tools. These tools matter because they will provide you with the data and statistics you’ll use to evaluate or compare programs. Examples of tools include surveys, media content & message analysis, and web analytics.
The tools we generally recommend are discussed below. Please note: They’re all widely available, and many could already be in place within your organization.
Message analysis tools
Internal communications never functions in a vacuum. Employees are just as likely to get news of company developments from local media or gossip at a soccer game as they are from your emails. Therefore it is critical that you monitor local media (especially blogs) to have a complete understanding of what employees are seeing.
You will also want to conduct a survey that will help you determine what the takeaways are from the messages you are trying to communicate. In other words, did they understand the message, did they interpret it correctly? Did it change their morale, their work habits, and their level of understanding? To what extent did the communications affect their outlook toward the company? We recommend quarterly “pulse checks” of employees’ attitudes to determine how perceptions are changing over time.
Focus groups can help you probe employees to discover the real issues that concern them and to discover the specifics you want to measure. If the major messages aren’t getting through, what is? What are the subtle variations between what the head honchos say and what the employees hear?
Content analysis of message boards like Glassdoor can also reveal emerging issues.
Surveys reveal what employees think
For surveys to be statistically valid, every employee needs an equal opportunity to participate. You don’t need everyone to respond; in fact, random sampling is recommended to save costs.
If every employee does not have equal access to computers, email surveys may be problematic. Hence, it’s no surprise that so many companies still rely on using paper surveys. Although paper surveys may be slow and appear antiquated, employees seem remarkably willing to fill them out.
One company developed an ongoing “Who Wants To be a Millionaire” game to test employees’ knowledge and understanding of the messages. Prizes were awarded for the most right answers. The program significantly increased the entire company’s understanding and belief in the key messages.
The key, of course, is to ask the right questions. Make sure that you have a professional researcher craft the survey questionnaire and include demographics by which you can segment the data. Look around any organization’s holiday party and you’ll quickly realize that employees are hardly a single audience. You’ll see long-time employees and newbies, men and women, geeks and marketers, telecommuters and cubicle dwellers, and branch office and main office folks. So don’t measure them as if they’re one group.
Designing a survey is a deceptively difficult job, and if you get it wrong, you probably won’t find out until there’s a pile of worthless paper staring at you in the face. We often see homegrown benchmarking programs fail because the right questions either were not asked or were asked in a way that failed to yield actionable information. Too many employee attitude surveys measure tactics rather than relationships. The right way to measure employee attitude is to use the Grunig Relationship Survey, not just test satisfaction.
Behavioral Measurement Tools
Outcomes are the behaviors that you want to change within your organization. Ideally, your communications efforts are intended to make employees more loyal, more efficient, more productive, and more knowledgeable. So the outcome metrics might be employee retention, performance ratings, turnover, or efficiency ratings. These numbers frequently come from accounting or HR and should play a key role in your internal communications dashboard.
Important outcome metrics are available by studying your intranet’s log files. How long employees spend in each area, what pages they visit, and the extent to which they download the info you provide are all potentially valuable measures of employee behavior.
Increasingly, organizations are using internal blogs and communities as a way to get messages out to employees and to gather feedback from them. Companies like Southwest and GM rely on blogs to establish two-way conversations between management and employees. If you do have a corporate blog, you have an easy way to track employee responses based not just on the direct comments, but also on the volume of traffic, and content sharing with other employees.
Step 5: Analysis Gives you the Answers
After you have collected the data, it needs to be counted, evaluated, and categorized according to type, effectiveness, messages communicated, and the like. Examine what messages are being delivered in what formats. You can use something as simple as an Excel spreadsheet, or a more sophisticated database package like WinCross, SAS, or SPSS to dig a little deeper.
Make sure you analyze your data by segment. At the very least you will want to compare data by: male vs. female, age group, length of time with the company, title/level, exempt vs. non-exempt, and geographic location.
Many organizations standardize on cost per message communicated as a way to compare the efficiency and effectiveness of different programs. This requires dividing your employee communications budget by the number of messages communicated. Another option is to compare the reach and frequency of message communications in various different vehicles (including email, local media, and internal communiqués). You’ll probably want to compare and contrast internal versus external communications vehicles to test the degree to which different media outlets and tactics are successful in communicating your messages.
The point of measurement is not just to generate a folder full of charts and graphs. You need to analyze the data, segmenting it across different employee groups to glean insight, draw conclusions, and make recommendations. The idea is to use your hard-won knowledge to improve the effectiveness of what you are doing. Therefore you want to have results in-hand when actions can be taken, decisions made, and steps taken toward improvement.
In internal communications the very act of surveying raises expectations that things will improve or at least change. Getting back to employees is especially important, not just with results but with specific changes that you will make or recommend because of the research.
Set up a regular schedule for reporting and planning. “Last week” is generally when most companies need the results of their benchmarking studies. Realistically, you need to work backward from when the results will do the most good. If you do your planning in July and get results in January, the results are 6 months too late and you’re dealing with stale news. Because you don’t want a “message du jour,” but do want to engender consistency and continuity, we recommend benchmarking every 12 to 18 months.
Next Steps
The one rule of employee communications is that once you have conducted a survey or asked a question of an employee, they will anticipate change. So whatever you do, don’t keep the data to yourself. Share it. Post it on an internal blog or Intranet site for everyone to see. While your “experts” can do the formal analysis, don’t be afraid to solicit input, comments, and interpretations from all of your employees. You may want to conduct additional focus groups to explain or interpret data that seems perplexing. As soon as possible, communicate any changes that will be occurring as a result of your research. That way you ensure participation in future research. ∞
Thanks to Rik’s Adventures for the image.
