Social Media versus Online Panel Sampling: Not All Respondents are Created Equal

Rockbridge conducted an online survey using respondents from a leading online panel, as well as those recruited from a popular consumer social media site. A comparison of results reveals striking differences in representativeness and quality.

 

Social media has experienced significant growth in the past few years and now appeals to a mainstream audience. For many, this means having to decide whether or not to accept their mom's Facebook friend request, but for market researchers, there is an opportunity to achieve a more inclusive sample by expanding the universe beyond that of client lists and online panels.

To evaluate the effectiveness of social media as a sample source, Rockbridge conducted an online survey using respondents from a leading online panel, as well as those recruited from a popular consumer social media site. A comparison of results reveals striking differences in representativeness and quality.

• Panelists are more likely to have a college degree (68%) than respondents from the social media site (40%). Respondents from both sampling frames have higher educational levels than the national average reported by the Census Bureau.
• Panelists have a higher average annual household income ($87,000 compared to $59,000 for the social media sample). Again, both are higher than the national average.
• Panelists are slightly older (mean age 49 years compared to 45 years for social media). Both are older than the national average.

 

It seems that online panels are a better source for studies that target a more educated and affluent population. If the study requires a target population of less educated, less affluent or younger respondents, the social media source is more representative. The considerable differences between respondents suggest that there is an opportunity to be more inclusive by utilizing both sample sources, which could ultimately help create a more representative sample. Note that in most studies, Rockbridge weights the sample to match the general population characteristics in the Census

Although social media presents an opportunity to sample from a larger and more diverse audience than with online panels alone, it also has a drawback — quality. Certain survey respondents are considered undesirable for research studies, including those who take too many surveys (hyperactive), are not fully paying attention to the questions (inattentive), or provide dishonest answers. We can identify these respondents through quality checks that are part of our SafeSample™ system, a series of diagnostics that detects and flags problem respondents.

In this comparative study, Rockbridge found that the proportion of respondents who failed to pass quality checks was more than twice the rate among the social media sample than the panel sample. It appears that panelists are more committed to thoughtfully responding to questions as part of their agreement to participate in research.

 

 

Social media has clearly matured beyond its college-aged roots. And although the diversity and accessibility of users makes social media particularly attractive as a research tool, there are limitations that must be considered. Through comparative online studies and proprietary methodologies, Rockbridge is making it easier to separate insight from distraction.

For more information on Rockbridge Associates you may contact Gina Woodall, SVP at 703-757-5213 ext. 11 or [email protected], or Charles Colby, President, at 703.757.5213 ext. 12 or [email protected].