The Promise of AI for Market Research & Insights

There is no question the world needs to proceed with great caution. That so many knowledgeable AI practitioners are concerned is a red flag. When I think about what AI can offer the field of research, insights, and analytics, I am not as concerned. AI and Machine leaning have been moving quickly but they have also been moving slowly. I recall as a bright-eyed young quant using ID3 and CHAID for the first time in 1995. I could see the promise of then … but it has taken a long time to advance to ChatGPT.

I can understand that people may have concerns about the idea that AI might replace people and jobs. I think that might be true if one defines an occupation narrowly at a task level. The role of the client-side researcher though is that of a director / facilitator of the insight development process, orchestrating and synthesizing a range of evidence sources into the best answer to business questions. With this “meta-analytic” view in mind, I am open to what AI can deliver as opposed to concerned.

If I think about the research process in task-based steps:

1. Issue definition: Understanding and defining the business problem and the customer problem to be solved.
2. Summarizing: Synthesizing what is already known.
3. Research brief: Identifying knowledge gaps, determining research objectives and developing a research design
4. Fieldwork: Developing field guides, research tools and collecting data
5. Analysis: Analyzing data and evaluating results, synthesizing results with other sources and assembling the narrative
6. Knowledge Management: Managing the knowledge in the business.

I can see many different AI applications could help with these individual tasks. I think there are practical and technical reasons why AI cannot do all these steps as one job-lot of tasks and replace the researcher as the center of the process.

There is no question that the skills of the researcher will look very different in terms of use of technology. The skills required to be a good researcher have been continuously evolving over the years but the role of creating and managing knowledge is fundamentally unchanged by AI.

There are more elements to the role of client-side researcher that make the simplistic task-based view above too simplified. Consider:

1. This task list does not even describe the different types of research that follow different processes and methodologies. Proposition development research is different from digital experience prototyping, user testing and market intelligence. It also does not describe the different business issue types, further complicating task automation.
2. Another important dimension of client-side research is facilitation of stakeholder engagement. Providing exposure to customers to develop empathy and understanding of specific problems among stakeholders. This is not in the task automation domain.
3. The most important role of the client-side researcher is the nuanced task of providing assurance and confidence that evidence is as robust as possible, highlighting the interpretation boundaries and understanding the relative strengths and weakness of the various evidence sources. Indeed, as we have learnt through ChatGPT, transparency on how AI reaches conclusions is a weakness.
4. Another common requirement of the client-side researcher is to act as a customer advocate. Acting this role is also outside of the task automation domain.

Upon reflection I am getting more complex business questions to answer as time goes on. What customers do and don’t like, or what they want, or how happy they are seem elementary and easy to answer. More complex questions becoming more common such as such as what would happen if…? How will customers behave in five years? How can we get customers to do something differently? These types of questions are better answered by experiments.

Probably the most interesting observation I have about AI is the way my team of researchers are experimenting with it and thinking about how they can use it. It seems to be appealing to them as a tool to get things done rather than a threat.

Applications of AI I am excited about

Thinking of the day-today challenges of being a client-side researcher, I think the areas that I would most like help from AI are:

Qualitative Research

While there are already AI assisted qual research applications, I am excited to see substantial improvements in:

1. Moderation, transcribing and summarizing interviews and other qualitative research interactions. I can see how you would need to take different approaches to generative prototyping, versus validation versus discovery type purposes.
2. Making outputs of prior qualitative interactions available to other projects in a more systematized fashion. These types of applications are already available, to a degree, but they can be substantially improved.

Comment & sentiment analysis

No doubt one of the simplest use-cases for AI, text and open-ended comment analysis has been “about to get better” for a long time. There have been improvements, but I hope the latest incarnations of AI can do more to improve the quality of these outputs. The explosion of survey platforms and the take up of NPS has left a lot of companies with an abundance of text feedback well beyond their capability to process responses.

Personalization of the research process

Personalization of the Research process for respondents is another area where AI can make a difference. Consumers are asked the same things many times over in the process of research for the purposes of having consistency in data items. Much of this information is not useful for researchers. In many ways, we ask questions on regular tracking surveys just in case we need the time series. I would like to see dynamic intelligent logic used in the execution of surveys to focus on specific topics and questions if required and un-remarkable questions to be omitted without this inconsistency causing analysis issues.

I need to temper my excitement about the application of AI in the client-side research context, however. There are a lot of challenges on the road to adoption. I see three main challenges.

Firstly, that of formats, locations, and permissions. Getting all sources of information in a format and location so that it can be consumed by AI in a way that is compliant with customer privacy provisions and Regulations governing the use of data is a challenge and requires a lot of manual process work. There will always be important sources outside the perimeter.
Secondly getting soon-to-be regulated AI use-cases will no doubt slow down the adoption process and AI might have a branding problem for a while.

Finally, getting AI incorporated into the myriad of tools and platforms used by researchers will no doubt take a great deal of time.

In the interim, I would encourage all researchers to experiment and work out how AI can help them. Stay in the center of the research process, master the technology!