Revolutionizing Survey Data Quality in the Age of Artificial Intelligence

High quality data occupy the pivotal intersection between what an organization needs to know and the decisions its leaders need to make. The urgent challenges of the pandemic drove insights professionals and business stakeholders to seek out new kinds of data and new ways to analyze it, unintentionally creating other intersections occupied by different data types, ranging from traditional surveys to mammoth troves of unstructured data. 

Data silos can lead to conflicting insights, and the apparently growing estrangement of market research from business leaders as discussed in Unmet Needs doesn’t do much to buttress stakeholder confidence the data. Already enflamed uncertainties and doubts are now whipped up by the cross-winds of artificial intelligence concerns, such as those discussed here. In an era in which stakeholders seem to obsess over speed and cost, market researchers hardly need to have them lose confidence in the data, too.

As the age of artificial intelligence envelops us, it would be naive to think that there are any insights professionals or stakeholders who are unaware of the massive battle being fought against survey fraud. Large language models masquerade as respondents at scale, and ChatGPT enables lazy respondents to avoid answering questions that require effort. At its most benign, these activities create noise unless they are eliminated from the final data set, and, at worst, these activities can manipulate results. Either way, they undermine research credibility.

Fortunately, companies like ours, Dynata, are innovating AI-assisted solutions to defeat AI-enabled fraud. Our strategy is to remove fraudulent surveys prior to invitation and to reject inadequately engaged respondents real-time. In our solution, AI checks are used in every panelist interaction, combining to predict future behavior to limit risks. Our approach includes nearly 200 automated checks to identify data that does not represent a good faith effort. These include elements manual checks can miss, such as unusual acceleration or deceleration, atypical mouse movements, copy/paste in open ends, and other tests. 

The benefits of removing fraudulent or poor quality surveys are obvious, but the benefits of removing them real-time might be more subtle. Removing them as early as possible avoids having to reconcile (or pay) any incentives that might have been “earned.” Dealing with survey quality real-time enables you to track your fieldwork progress more accurately. Other benefits include lower costs for higher quality data, faster field timelines, less labor for cleaning data, less error, less bias, and a reduction in good respondents incorrectly tossed by manual-check errors. 

When you evaluate AI-assisted solutions for data quality, ask about the supplier’s wastage rate: the percentage of completes due to unusable surveys that were paid incentives. (For example, Dynata has an industry-leading wastage of just 6% after data quality checks.) Also, ask for the percentage of panelists who were flagged as “poor” and for their accuracy rate. These metrics will help you to understand the quality of the panel itself as well as the level of quality you are likely to get from your survey. 

Insights professionals need to take steps to restore trust in market research data, as well as to continue to reduce costs and timelines without sacrificing quality. AI-assisted solutions can help achieve these goals just as assuredly as some bad actors are using it to thwart them.