This one of three articles in this month’s issue that report on AMEC’s recent annual Global Measurement and Evaluation Summit:
Technology wasn’t the only interesting topic at the AMEC Summit. The actual research being done around the globe was also fascinating. Here are a few of our favorites:
Great divide: A polarization index
Given our polarized times, Fred Cook, director of USC Center for Public Relations, and Anjani Bentwood, head of analytics and data at Golin, decided a polarization index might be in order. (See their presentation “Polarization and purpose.”)
They applied Zignal Labs AI’s algorithms to coverage of our most polarizing issues, like climate change, racial equality, and police reform, to determine what was fact and what was fabricated. They categorized all coverage into 8 categories, including everything from complex analysis, to a mix of fact and analysis, to opinion, to fake news. They then looked at the partisan leanings of those who reacted or engaged with each topic to gauge the likelihood of any given topic to generate controversy from either the left or the right.
By weighting these factors, a brand considering advocacy can make better-informed choices as to which issues it wants to make a statement on, depending on its target audiences and the degree it wants to generate controversy.
Surveys vs. bots: Can you trust the data?
We’ve all received unsolicited invites to participate in online polls, and the ease of doing so makes it incredibly simple for bots and spammers to join in as well. Do these bots and spammers skew the data? That was the question Courtney Kennedy, director of survey research at the Pew Research Center, took on. (See her presentation “Assessing the risk from online polls from bogus respondents.”)
Kennedy and Pew identified certain patterns that the faux responders followed, and then added specific questions designed to weed them out. They then conducted a survey that mixed standard political opinion questions with an open-ended question, such as “How are you feeling today?” or “In retirement what skill would you like to learn?” The fakes would give gibberish answers, or say, “Don’t know” or “Unsure.” They also compared responses to identify when answers were identical between multiple surveys.
They found that about 7% were fake responders. They then compared the results of the fakes to the real respondents. Did they change the data?
Obviously, a 7-point difference is significant. They found the fakes were much more prone to give positive answers when asked if they approved of a president. 78% of the bogus respondents approved of Donald Trump, compared to just 41% of the other respondents. The same was true of when asked if they approved of Barack Obama.
When asked about specific policies, however, there was little difference between the two groups. The other Aha! moment was that common research quality checks, like how fast a survey was completed, failed to detect bogus responses. Suggesting that researchers need to improve their processes and use more open-ended questions.
Sporting news: The media is even more biased than you thought
Sport New Zealand teamed up with Isentia to analyze 76,000 media items over two years to prove that women were much less represented than men in sports coverage. (See “Using research to drive change: The media, women and sport in New Zealand.”) They found that, of total sports coverage, only 15% was of women’s sports. Only 21% of bylines were female. And women weren’t even being given a voice: 82% of fans quoted in sports media were male, and only 20% were female. Never mind totally sexist findings, like it was much more likely that a coach would comment on a female athlete’s personal life. Or that a source would comment on the appearance of a female.
The one piece of “good” news is that when they showed the data to the media, sports organizations, and interest groups, things did change. The number of females quoted increased by three percentage points. ∞
Image up top thanks to “Using research to drive change: The media, women and sport in New Zealand.”
