If you downloaded the CDC app to report your symptoms (if any) after getting your COVID vaccination, you were self-reporting data. Public health systems use self-reported data all the time.
When the CDC measures whether its programs are effective they need data on our behavior. They could follow us around and take our temperatures, or watch to see if we are lighting up a cigarette. That’s behavioral research. It’s a lot more accurate than self-reported data, but not very practical.
So public health communicators use questionnaires to ask people to self-report on whatever behavior their communications efforts are attempting to curb. Presumably the people they ask have no compelling reason to lie in order to make the data look better or worse.
Where self-reported data starts to fail is when the people or organizations who are self-reporting the data, or the organization that is doing the research, has a vested interest in the outcome. When you gather or work with your own data, it is all too easy for you to consciously or unconsciously nudge it in the direction you would like it to go. And it can’t be easily validated by replication or other research. If you are self-reporting your ESG or DE&I data, for example, you might not want to include data that might lower your score.
The reality is that assembling the data for these types of surveys never gets the scrutiny or is handled with the proper expertise necessary to assure accuracy. Which is why relying on self-reported data is a dangerous practice.
Whenever possible, self-reported data should be triangulated and/or validated by another data collection method.
The most egregious example is the self-reported data collected for the US News and World Report College Rankings. This popular index uses data provided by the colleges themselves, and for years has been rumored to be inflated by distorted or inaccurate data.Â
Recently a Columbia Univerisity math professor questioned his own institution’s rise in the rankings and found that the data was questionable. And last year, a Yale professor challenged the results of Business Week’s MBA programs on similar grounds.
Thousands of dollars are spent on application fees, college visits, and tuition as a result of these rankings. In an age when the value of a higher education is being called into question, it seems like the least we can do is make sure that the data that drives so many decisions is accurate and validated.
For that reason, we name all organizations that rank entities based on self-reported data our Measurement Menace of the Month. Congrats! ∞
