Out With the Old, in With the New: The Trouble With Surveys, and the Next Generation of Market Research

Say you’re launching a new product, or testing claims and RTBs, or trying out different packaging–whatever the case, you’re likely engaging with survey-based testing to feel out your market. Are you confident the information you gather will give you an informed, accurate understanding of what your consumers really want? Do you trust your surveys to tell you how to best invest your time and budget? Increasingly, probably not, for a variety of reasons. But what better option do you have other than to make million-dollar bets based on gut feeling?

What I’m about to say isn’t anything new or radical. It’s the elephant in the room market research professionals hesitate to mention: There is an emerging existential crisis with survey-based research. And worse, the industry appears to be stuck as to how to abate it.

Approaching a Very Big Elephant

Online survey-based research is big business. According to a LinkedIn study on the Online Survey Software Market published in October of last year, the industry had a value of $4.4B as of 2021 and is expected to reach $8.5B by 2027. And Exploding Topics reported in a January market trend post that 90% of market researchers say they regularly use online surveys to reach consumers, because researchers believe they are “fast and cost effective”.

[related-article title="Making Surveys Human Matters Now More Than Ever" url="https://www.greenbook.org/insights/research-methodologies/making-surveys-human-matters-now-more-than-ever"]

You’ll notice it’s not because they believe they’re getting accuracy and a deep understanding of their target audience. There are issues with survey completion, sample diversity, respondent authenticity, ability to scale, and consumer tech preferences making research tactics outmoded faster than we can keep up with.

But beyond those things outlined by researchers themselves, the disadvantages of surveys continue to grow in more insidious ways:

·  Response bias from poorly constructed surveys

·  Declining response rates in consumer panels

·  Difficulty engaging hard-to-reach segments inadequately represented in online panels

·  Potential bias introduced by financial incentives to survey respondents

·  Outright survey fraud

And all these things are widely understood to be big problems. Survey fraud–or the bad-faith interference of people and bots to collect incentives–is getting particular attention at the moment. In a surprising combining of forces, The Market Research Society of the UK (MRS) is coordinating efforts with six other market research associations across the globe.

This international coalition has come together to address such risks across the market research and consumer insights industry as a whole. In their announcement about the initiative, they state the need for: “Fraud detection – tracking the prevalence of fraudulent survey completions by humans or bots and outlining best fraud detection and mitigation practices.”

It’s clear that surveys are becoming more and more difficult to effectively create, field, target, and collect clean data as the modern digital landscape expands into more complex shapes. So what does the next generation of market research look like? You might be thinking this is a great moment to leverage the burgeoning sophistication of AI. Or is it?

The Unintended, Unvetted Consequences of Turning to Generative AI

While AI makes it easier and faster to construct online surveys and send them further and farther, it introduces a catastrophic can of worms. Easy and fast doesn’t always mean better–it makes it easy and fast for everyone, including inexperienced researchers to build poorly constructed surveys that introduce unintended bias.

It also makes it easy and fast for more nefarious activities. Bad actors out to collect incentives also use AI to complete these surveys in more sophisticated ways than ever before–with zero added effort–and since these surveys are generated by the very same tools used to game them, the previously established methods of finding, circumventing, and filtering out bot responses are being rendered ineffective. That’s a lot more junk data making its way into your results that’s nearly impossible to identify as junk.

Lots of smart people are experimenting with and evangelizing AI as a solution to survey research problems, and of course we should use every tool we can to try to fix them. Meanwhile, you still have innovations, concepts, new products, and go-to-market messaging to test in more fragmented market segments. Are we just back to making big bets based on gut feeling? 

We Can’t Trust the Algorithms, But Can We Even Trust Ourselves?

 Consumers are still taking surveys, but behavioral science tells us that as humans we are unreliable witnesses to our own behavior.  Asking direct questions around preferences and purchase intent often yields less than reliable predictions of future encounters with a brand or product. Whether consumers mean to or not.

Even industry leaders like the survey platform provider QuestionPro advises its customers to expect the reliability of survey responses to vary based on a number of factors and warns that “poor data quality amounts to an average annual loss of $15 million.  Bad data leads to strategic mistakes, and in no way, it is possible to go back in time and make corrections.”

Getting good data off a survey is a complicated dance, even if it’s perfectly constructed.

Experts in the subject of survey data quality have published extensively on the evidence of declining reliability in survey-based research. Karine Pepin, self-described “Data Fairy”, has published many posts on Linked In about the impact of poor survey data quality.  She has revealed her own research around the factors contributing to the problem (even from well-intended, authentic respondents), and the lengths to which insights teams and their research partners need to go to counteract the consequences.

And so we are led to the only path forward. The real future of market research: The only way to predict with any accuracy how consumers really behave is to stop asking them what they would say, and start watching what they actually do.

Shifting to a Living Digital Laboratory 

 You’ve likely heard of social listening; a recent tactic of leveraging the conversations happening on social spaces to gain real-world insights and trends. Often, the most promising solutions for capturing data this way are based on text analysis or clickstreams. But this leads to you sifting through a lot of noise, hoping you’ll stumble on something worthwhile.

Digital experimentation, on the other hand, harnesses the online social environments your consumers already inhabit (where they actively make decisions about what they do, like, and buy), and allows researchers to build a living laboratory inside those spaces. It makes it possible to deliver a test to exactly the right consumers you wish to target, capturing thousands of signals that represent authentic consumer behavior rather than incentivized responses to limited surveys.

If you were a scientist, you’d start with your hypothesis, build a statistically significant and sound test, do a blind study, look at your results, learn, and iterate. Survey-based market research has attempted to do that but has struggled to scale in a cost-effective or reliable manner.

What if you could grab not dozens or hundreds, but hundreds of thousands of your consumers, in their familiar digital worlds, feed them isolated but virtually infinite test variables, and compare their responses–without them even knowing they’re in a test? That’s digital experimentation, and that’s what we do at Orchard. And that’s how we’re ushering in the next generation of market research.