Computer Does Qual: Avoiding AI-Overclaim and False Equivalency

Almost every week a new company appears claiming to deliver ‘qual at scale’ or ‘robust qual’, using AI to ‘deliver insights from thousands of 1-1 qualitative interviews moderated by an AI researcher.’ These claims are coming from tech start-ups and established agencies – all of them I would argue, are guilty of selling a huge false equivalency.

AI facilitates conversational surveys, not in-depth interviews – they are not the same thing. These platforms use AI to ask a mix of scripted and generative open-ended questions in a conversational survey (with varying levels of intelligent adaptiveness). The responses are processed at-scale and generative summaries are produced by the AI platform. To describe this approach as a suitable replacement for qualitative practice is entirely disingenuous.

The art of unstructured interviews is to acknowledge and value participants’ stories as the interviewee shares their personal experiences. Qualitative interviewing requires gaining access to an interviewee’s intimate and private experiences, the interviewer must court the interviewee, enhance the sense of rapport between them and build a sympathetic relationship and a sense of mutual trust. Rapport between interviewer and respondent is created through friendliness, openness, respectful and sympathetic listening, and a learner's attitude on the part of interviewer.

Qualitative practitioners have an inherent understand of how ‘big’ a question is, they know how and when to push for more and when to move on, they can spot an interesting detail to drill down on – they know what it is to be human. ‘In learning about the other we learn about the self. That is, as we treat the other as a human being, we can no longer remain objective, faceless interviewers, but become human beings and must disclose ourselves, learning about ourselves as we try to learn about the other." Fontana and Frey, Handbook of Qualitative Research, 1998

When interacting with a computer (even a really smart one!) – the dynamic is entirely different. There is no social contract between the interviewer and interviewee. Questions are received from a machine that doesn’t have the ability to understand or empathise with what it’s being told – it can only predict a likely follow-up response based on statistical probability. This might work for a bit of light back and forth, but as soon as the machine asks a question that the interviewee feels is repetitive, boring or inappropriate for any number of reasons - the spell is broken.

I recently used one of these platforms to run an internal experiment where we used a short AIenhanced conversational survey to gather opinions from researchers on the role of AI-powered interviewing. In most cases – the AI generated repetitive questions that the interviewees felt had already been answered. Beyond repetitiveness – survey fatigue was also a problem, you simply cannot ask the same number of questions in a conversational survey as you would in an in-depth interview.

Qualitative interviewing is about more than just the ability to ask a lot of relevant questions – it’s about body language, tone, silences, hesitations, enthusiasm, passion, boredom. These are subtle elements that require powers of reasoning, cultural understanding and human empathy that AI simply doesn’t possess. Conversational surveys can capture qualitative information, but they can’t deliver in-depth qualitative understanding – that’s the real difference.

All of this said – I’m not saying that AI-led interviewing isn’t a valid research approach. In fact quite the opposite – I think it’s an incredibly exciting methodology that we should all be experimenting with. Conversational surveys could be a great way to boost coverage of niche audiences in less detail at a reduced cost or to conduct pre or post tasks. It’s something that undoubtably has a place in the research toolbox, but it won’t do the same job as in-depth qual, so it shouldn’t be advertised as such. 

We need a new vocabulary to talk about AI’s role in research – it facilitates new approaches that occupy an area somewhere in the middle of qualitative and quantitative research practice. I would argue that AI allows us to deliver ‘synergetic’ research - it brings elements of qualitative and quantitative research together to deliver new benefits. Synergetic research adds new tools to our collective research toolbox, but it compliments rather than replaces traditional techniques. In the same way that you wouldn’t ask an intern or a junior researcher to conduct and write up analysis for an entire project of in-depth interviews – you shouldn’t rely on AI to do your qualitative research.