The New Dynamics of Online Sample Quality

Editor’s Note: As one of the early pioneers of online sampling (as e-Rewards back then) Dynata has always been a leader in defining best practices.

Nowadays, as the undisputed leader in terms of sheer size in the sample industry, they continue to use their pragmatic experience to set the bar. Months ago, I was speaking with Gary Laben, CEO of Dynata, on the priorities for 2021 and he mentioned a new commitment to driving innovation in data quality while growing their panel assets by redefining the engagement model for respondents. I’ve been waiting to get the details on this agenda, and now the wait is over. Bob Fawson and Jackie Lorch have released a bit of a manifesto outlining the challenges in online sample that we all need to be aware of and Dynata’s plan to address them. 

Sample is the bedrock of the insights industry; without people to engage with we can’t answer the business questions that we are dependent on to address. To experiment, innovate and evolve is the only way to ensure this precious asset is protected. The steps Bob and Jackie outline go far in honoring that responsibility. It will be very interesting to see how the entire industry builds from here!

 

For several years, the insights industry has attempted to satisfy an insatiable demand for data while changing consumer expectations about how they share data have challenged traditional supply models. This situation – along with advances in technology that maximize sample efficiency – has led to a revolution in how sample is bought, sold and managed. Sample buyers are often unaware of these changes. Dynata is on a mission to share the realities of sampling today, so data buyers can make better, more informed choices.

 

New Market Dynamics Changing our Business

• Demand for consumer data exceeds the time people dedicate to sharing data. There are many contemporary options competing with surveys for people’s time, including those driven by the rise of the “gig economy”.
• People are savvier about the value of their data – and much more nervous about sharing information. Corporate IT Security teams regularly warn staff to never share personal information online; and good people lie if forced to reveal something too personal.
• Attempted fraud is on the rise across the loyalty, retail and data sectors as well as our own. (Imperium, the leading provider of data quality and anti-fraud solutions reports a 200% fraud increase in the industry over the past year.)

 

What this means for Data Quality, Research and Sampling

• Sample is bought and sold in an increasingly complex marketplace; there must be more transparency for buyers, including clearing up misunderstandings about how people are recruited (large variations by supplier), contacted (almost never via email), routed to surveys (differently depending on projected revenue, conversion and other factors) and more.
• We’re in competition with other, better, online experiences so the survey-taking experience must improve. Additionally, giving participants control and choice is becoming more important. The direct link between experience and poor-quality data has been known for decades but wasn’t as critical when the supply of participant’s time exceeded demand.
• Quality is no longer binary, good quality/bad quality. Research buyers are defining quality as “fit for purpose” for each project and making tradeoffs such as restrictive quality controls vs reach and representivity, price vs source, and precision vs scale, for example many people won’t share their date of birth, but almost all will share the year they were born.
• Controlling fraud is only one part of the quality picture, and technology is transforming how it’s done.

 

Online Sample Fraud and How We Fight It

In the world of face to face and telephone research, fraud concerns center around fraud by the interviewer and data being falsified. In online sample, the focus is on the participant who gives false information when they join a panel or take a survey.  The motive is almost always the reward – frauds try to predict how they can qualify and complete as many surveys as possible to claim the reward.

Dynata brings an array of solutions to fraud control; using both traditional techniques, and increasingly leveraging machine learning for increased speed and effectiveness. Dynata’s ownership of the industry-wide fraud-prevention experts Imperium, allows us to co-create a roadmap joining Imperium’s tools, metrics, technology and controls and Dynata’s strategy for recruitment, panel management and the participant experience. Together we have invested in the staff and data sharing infrastructure to meaningfully speed Imperium’s product development roadmap, benefitting the entire industry.

Dynata’s strategy is to collect data at each respondent touchpoint, gathering dozens of data points at every interaction and apply that data to managing participant reputation:

• At enrollment: – tools such as the new Imperium tool RegGuard®, Real Mail^TM, Verity® and RelevantID® controls, ThreatMetrix®, device and IP anomaly and reputation checks, plus open-end engagement tests analyzed via machine learning, use multiple data points to confirm identity and look for unlikely patterns.
• In the survey router –digital fingerprinting, geo location clues and a second round of the checks used at enrollment confirm identity and identify suspicious behavior.
• Within the survey – encrypted end links, customer concern feedback links, and Imperium quality score; and a new quality management platform that evaluates performance and behavior inside the survey.
• Advanced techniques for rare targets – including asking people to describe their job in their own words, leveraging machine learning to sort them into relevant categories and confirming their identity via publicly available information about professional credentials.

Dynata’s current focus is on collecting and acting on real-time and predictive data about how people will interact before and within a survey. Specifically, behavior validation when people join a panel, including open end and engagement validations before each survey; and the new Imperium Quality Score, which detects and scores based on items like open end performance, speeding and straight lining within the survey itself — creating a quality score for each and every participant.

With more proprietary panelists than any other provider, and a connection to the Imperium ecosystem, Dynata has masses of data to act on. The more data we have, the more reliably quality management tools can predict quality and the faster we can prevent fraud.

 

The Importance of Participant Experience and Choice

Improving the participant experience is at the forefront of Dynata’s quality efforts. This starts at recruitment, with the right style of communication to engage and retain people long-term. A team of website design experts have reimagined every screen a participant sees, curating the content to fit the style of a particular panel and location.

Support bots offer our participants help and support, and there are educational videos to explain what it’s like to take a survey. We create fun milestone moments and share how panelists’ participation is making a difference.

Just as we expect transparency and control in our relationship with brands, research participants increasingly expect it when they take a survey. To improve quality, we must give people transparency and agency in the survey-taking experience, fully explaining what we expect and offering options at each interaction.

Dynata is experimenting with giving participants choice to improve their experience without impacting data. In the same way we can choose the type of car, length of wait and driver rating in our ride-sharing apps, participants should be able to share data in the time they have available and with a suitable experience. We believe aligning the goals of data buyers and consumers providing data will elevate quality and make our participant engagement model truly sustainable.

 

Quality Fit for Purpose and Tradeoffs

Given that “ordinary people” increasingly opt out when faced with stringent quality controls and hesitate to enter personal information to prove their identity, the concept of “Quality Fit for Purpose” is becoming more important. A general population study for example might use fewer quality controls to provide an easy, frictionless experience and maximize reach and representivity. In contrast, a low-incidence B2B target where rare and specific knowledge is required might employ more stringent controls, narrowing the universe but ensuring people really do have that knowledge.

 

Quality Ingredients

While most quality discussions focus on fraud, a poor-quality sample, questionnaire or screener can be a greater threat to data quality because every participant in the sample has the potential to give poor quality data if they’ve been incorrectly screened in or the experience is bad. Improving quality goes beyond fraud prevention, requiring:

1. A sample source recruited to maximize representivity
2. Fraud prevention at every touch point, employing machine learning and AI
3. A high-quality experience before someone starts the survey
4. Reasonable rewards
5. An accurate screener
6. A straightforward, mobile-friendly, enjoyable survey

 

12 Action Steps to Improve Quality

1. Ask your sample provider how people are sent to your survey and how they were recruited.
2. Discuss quality and price tradeoffs with your provider.
3. Spend as much time designing the screener as the questionnaire; if people are incorrectly screened in, their data quality will be poor.
4. Leverage existing demographic and geographic information to avoid re-asking these questions.
5. Monitor drop rates carefully and discuss any changes with your sample provider. Be aware that low incidence studies can mask high drop rates.
6. Ensure the survey is mobile-compatible and redesign grids.
7. Keep the length under 15 minutes; Dynata’s research clearly shows data quality deteriorating after that point.
8. Always test with people outside your immediate research team and act on their feedback.
9. Use measurements and time periods which don’t overtax memory or encourage wild guesses.
10. Avoid basic errors like overlapping scales, affirmation questions and too many words.
11. Only collect data that has a defined use.
12. Use recommended in-survey quality control questions.

 

Key Takeaways

• The questionnaire design, screener, sample source and sample plan – and the participant experience – are every bit as important as fraud control in delivering data that successfully meets the research need.
• Quality tradeoffs are playing a bigger role in meeting research goals. The strictest quality controls can cause bias by discouraging good people from taking part and may not improve data for broad population studies. Stricter controls are appropriate for narrow targets. There are also cost implications in “quality fit for purpose” considerations. Dynata can guide you in making those decisions.
• Quality control is complex and today’s sampling procedures have changed dramatically in the past five years. Dynata is determined to give researchers the information they need to make the best decisions to ensure high quality data.

 

Dynata’s Commitment

Dynata is:

• Making an investment of tens of millions of dollars in panel quality and capacity in 2021.
• Offering transparency to help researchers know how sampling works today.
• Bringing a data-driven approach to sample efficiency and the participant experience and sharing data with the industry.
• Designing more accurate feasibility tools.
• Giving participants more visibility, choice and control in their survey-taking experience.
• Partnering with our clients to make a difference in quality outcomes.
• Committed to meeting the industry’s demand for reliable data to answer critical questions.

 

 

About the Authors

Bob Fawson, EVP, Business Operations, Dynata

Bob oversees Dynata’s data and research business running the largest first-party panel operation in the industry. Bob is an expert in ensuring global consistency and integrity of data, from its acquisition to its distribution, and has extensive experience managing panel partnerships, data quality, and survey programming.

Jackie Lorch, VP Global Knowledge Management, Dynata

Jackie leads a team of researchers to maximize research quality and develop best practices at Dynata. She is a frequent writer and speaker on topics including research quality and methodology, as well as trends that impact the business of research and marketing and was named one of Research Magazine’s “25 Researchers You Should Know”.