As we head into 2024 we wanted to revisit an audience favorite from this past year!

How will generative AI and synthetic audiences shape the landscape of qualitative research?

In this episode, President of Sago, Isaac Rogers joins us to explore the changing face of qualitative research and the need for more engaging and innovative research designs. We discuss how researchers are adopting multi-phase and iterative research designs, including online discussions, video interviews, and digital diaries, to gather more comprehensive insights from participants. We also explore the emergence of generative AI and synthetic audiences and the challenges around data auditing and security measures to combat fraud in this area. Isaac shares his perspective on the challenges and opportunities in the industry, including the need for incentivization, data auditing against fraud, and the potential of computer-generated open-ended responses.

You can reach out to Isaac on LinkedIn.

Many thanks to Isaac for being our guest. Thanks also to our producer, Natalie Pusch; and our editor, James Carlisle.

 

---

Transcript

Lenny: Hello, everybody. It’s Lenny Murphy with another edition of the GreenBook Podcast. Thank you so much for taking time out of your day to spend it with me and my guest. All my guests are special, you know that by now. I sound like a broken record, but you know, some closer to home and today’s guest is one of those. Isaac Rogers, president at Sāgo. Isaac, welcome.

Isaac: Pleasure to be here, Lenny.

Lenny: So, I say this is special because Isaac and I have known each other for way longer than probably either one of us would want to admit.

Isaac: Oh, more than a decade, I think it is safe. I’ve been in the industry 15 years, so I know I’ve known you the majority of that.

Lenny: Yeah, absolutely. So, back in the 20|20 days.

Isaac: Well, actually, I think you were at Rockhopper at the time, I think. And so, that’s how I got introduced to you, so that’s—that really puts it back a long way.

Lenny: You are definitely taking it back to the 15-year mark, for sure. So anyway, it’s a pleasure to have you on. And for our audience, I recently had the opportunity to go to an event with P&G, and Isaac presented some internal data that Sāgo has been collecting around changes they’re seeing in project type and volume and quality that I just thought was really, really interesting and we’re not getting anywhere else, even GRIT, you know? We don’t get that type of information. So, that’s the core of why I wanted Isaac to come because I think that there’s just a lot of great information there to share.

But also, there’s a little bit of a story that I think we can catch up on, and that’s the transformation of Sāgo, Schlesinger to this new entity, which is my long-winded segue, Isaac. Why don’t you introduce yourself to the audience, talk a little bit about Sāgo, and then we can go into kind of the cool stuff that I thought oh, this would be really great to talk about.

Isaac: Hey, thanks, Lenny. So, first of all, you know again, pleasure to be here. Love talking about the market research space, the industry, and what’s going on. I have been fortunate, been in the industry about 15 years now. As you mentioned, I joined a little company called 20|20 Research that grew and became one of the world’s largest digital qual providers.

We then became part of what was at the time Schlesinger Group, which is one of the world’s largest—or at the time, the largest qualitative provider. And we keep growing and growing. We are now one of the world’s most diversified and one of the largest partners to market researchers and brands around the world. So, we do several 100 million dollars worth of research services every year, data collection, quant qual, digital qual, you name it. We are a global organization, do most of our work here in North America and Europe, but with our quant divisions, with our digital qual divisions, we’re doing research all day, every day, for about 2600 clients around the world.

I have the honor and pleasure of being our president, so all of the business units roll up into me and so I keep an eye on what’s going on generally in our quantitative practice, our qualitative practices, and we’ve got some professional services practices as well. So, we are what I would consider to be a very diverse provider. Which is one of the reasons that we started looking at this presentation because we were having conversations with folks about, you know, what was going on right at the tail end of Covid, with qualitative. Specifically, people were thinking more in person, but people kept coming up to me and being like, “What do you think about, you know, what’s going on in qual?” And I would have answers and, you know, provide kind of my, like, general opinion.

And I was actually talking to an industry individual, a person that runs a trade show, and I was like, “Yeah, you know, people keep asking me like, ‘What’s going on in qual,’ and, ‘What’s going on post-Covid?’ And, ‘What are the trends?’” And I was like, “You know, you guys really should do a presentation or something on that.” And she goes, “Well, you know why they’re asking you, right?” And I [laugh] was like, “Well, no.” And she goes, “Well, you’re one of the biggest providers in the world. You should have a point of view on this. Like, you’re probably sitting on a ton of data.”

And it turns out we are. We do probably more—or enable more—qualitative research than anybody else on Earth. And so, I said, “That’s probably not a bad idea.” So, I worked with our team. And I thought it was going to be a quick and easy little data project and we were going to look across all of our in-person research, our digital research, and come up with some trends.

And it turned out to be a much bigger project than I anticipated. We spent a better part of three months digging into our data about what’s going on with respondent engagement, what people are doing in in-person facilities, what people are doing differently post-Covid and we came up with four or five pretty major findings. Started presenting those to the industry in the spring of this year and we’ve had the honor of presenting, I think, four or five different times. And then mentioned to you right before we get started with the recording, and we’ve got clients that are coming to us, like, a—like, today going, “Hey, can we talk a little bit more about that because we’re starting to see some of those trends that you mentioned in your presentation. What should we do different? What’s going on with respondent engagement? Why are things continuing to change post-Covid? We thought things would settle back down at some point.”

And so, it’s still a hot topic for folks even, you know, call it you know, 6, 9, 12 months post-Covid, having a huge impact on the space. They’re still wondering, like, “Where’s all this change coming from? And what’s happening? And more importantly, how do we get in front of it?”

Lenny: That is the issue, isn’t it? How do we get in front of it? Well, let’s dive in, right, rather than me pontificating. So, I think that there was an awful lot of really interesting things that some were, kind of a yeah, we assumed this is what was happening and how things were going to go, some that were pretty darn surprising.

Isaac: Yeah, absolutely. And so, the questions that continue to come up are, so what’s going on with in-person qualitative research? And so, we looked across our entire dataset, every group we’ve done over the past five years, categorized, organized, and came up with several trends. So, the first thing people say is, you know, “Are people really back doing in-person work?” And the answer is actually, yes.

So, in-person research, we believe is somewhere between 35 and 45% back to pre-Covid numbers. So, that is measured on the volume of research that’s occurring in in-person facilities. So, a little less than half of what it was. Now, I think most people nod their heads and go, “Well, that makes, kind of, sense.” Like, just, you know—I think actually, some people are surprised the number is that big.

Actually, the folks I was talking to earlier today, I asked them what they thought it was, and they thought it was maybe 10%, 15% of what it was pre-Covid. But it’s probably closer to that 35 to 45% range. Now, what’s actually interesting is what we do in those facilities has changed pretty dramatically. So pre-Covid, you know, you would sit a focus group of six or eight folks to talk about advertising cat food, whatever the general research objective was, and then, you know, you would do these, what we would call kind of non-traditional or non-typical groups for jury work, clinical research, product habits, and taste tests. And those were, you know, tucked in there, and they were nice, you know, maybe 20 25% of the work, but the lion’s share was, you know, getting six or eight consumers around a table, getting a bunch of physicians around a table or doing IDIs and just having a conversation with them about a product concept or usage behaviors.

That has not come back. So, in that, call it let’s say, 40% back to pre-Covid numbers, a minority of that work is sitting six or eight consumers around a focus group doing IDIs, and in-person research. That has predominantly stayed online. Researchers have realized the benefit of a nationwide audience, frankly you know, you and I are both calling in from home, people don’t love getting in their cars and driving downtown to local research facilities and so it’s actually hard to get folks to participate in in-person research compared to pre-Covid. And what has taken its place is a lot of product testing, a lot of non-traditional work that people have found that, you know, post-Covid, they’re needing to test new products, new product ideas, new concepts, and so they’re bringing that into the facilities. A lot of usability testing.

But the day of the traditional, you know, Mad Men-esque six people around a table, that really isn’t happening in person nearly as much as it used to. I mean, I think that’s probably dropped 70% or more. So, what we do in the facilities has changed. And for the world’s largest focus group facility operator, like, we’re starting to think about our footprint differently, like, you know, the big conference rooms with the creepy glass wall on the side, you still need that, but you also now need almost like auditorium-sized rooms that we can get a hundred people in to do different types of research, and then we need a lot more IDIs than we needed in the past for, you know, product testing, usability testing. So, you know, we think that you’re going to see your facilities over the next five years or so really become kind of small, medium, large rooms, whereas before it was all mediums. And so, it actually has an impact on the physical layout of the in-person facilities.

Lenny: Sure. A lot of test kitchens? Sensory—

Isaac: Test kitchen, yeah.

Lenny: Yeah.

Isaac: Which, you know, are tough because a lot of facilities are in commercial buildings that don’t allow you to have commercial test kitchens. And so, like, it creates all these really interesting logistics challenges. But there’s no doubt that the world has changed in in-person research. It is back. It is back differently, but you know, I get people all the time going, “Oh, people are really doing in-person groups?” And I say, “Yeah, they are just different than they were pre-Covid.”

Lenny: Yeah. Which absolutely aligns with what we’ve seen via GRIT with asking about methodology [and option 00:08:54], right? We saw that flip. And we’ve seen over the past few years and figured when 2020 hit that that was one of those things that would stick, right, just like working from home. But I’m glad that it is back to the level that it is. So, that’s fantastic. You know, because the next great fear would be as we go move into, you know, the metaverse, what would that do? But thankfully, that hasn’t happened yet. At least not at scale.

Isaac: Not at scale. Not at scale. You know, I think it will have an impact at a point, Lenny, no doubt. And it’s coming, but it will take some time.

Lenny: Yeah. Yeah, I agree. All right, so what else that was one that’s—I’m trying to dismiss it—it’s kind of an obvious one when we think about and looking at the data points, but you found some stuff, particularly around incentives.

Isaac: Yeah, yeah. You know, again, from, from, from the thousands and thousands of people we recruit every year, we’ve got a lot of data to rely on. And so, one of the trends that has been picked up by a lot of folks and that we spend a lot of time with our clients consulting on is we have an almost an existential problem with respondent engagement. And I know that sounds like a little Chicken Little, sky is falling, but as eart—like, earlier today, I was looking at some of the attrition numbers in some of our various qualitative and quantitative panels and they are much, much bigger than they ever were pre-Covid, and a few things are driving that.

So, I talked about in my presentation, kind of these three phases around the research world, and it was pre-Covid, Covid, and post-Covid. Pre-Covid, we had these general expectations about what incentives were, and you could get consumers to do online discussion boards, webcam interviews, in-person research at certain rates, and everybody kind of knew what those rates were, and you didn’t even have to ask. And then something funny happened in Covid. Everybody was sitting at home and so they had tons of free time, and so actually, we had this kind of blissful 18 or 24-month period where people, like, were much, much easier to bring into research. And they were, like, so much easier, I don’t think we even recognized it at the point.

We didn’t have incentive pressure, our engagement numbers were super high, our attrition rates were low, and that is kind of qual and quant. And then about a year ago, we saw it shift back super dramatically. So, people all of a sudden, as they were getting out of their home and finding other ways to fill their time and think about things, we have seen a dramatic shift in consumer behavior towards research as a whole. I believe that we have kind of spoiled ourselves during Covid and pre-Covid on how easy it was to bring people into research, and we’re struggling, man. We’re really struggling.

We have found that consumer incentives have gone up 20 to 40%. And that’s a big number, right, when you think about a qualitative budget, to increase that by 20 or 40%. That’s huge. And that’s just on the consumer side. We see some B2B categories where we are paying over $1,000 an hour as an incentive to a B2B respondent.

Because honestly, like, they’ve got all these other ways that they have found that they can either fill their time, make money, and the value prop of research has gotten really, really low. The screeners have gotten longer, especially during Covid. People just through everything but the kitchen sink into their screeners. So, the dropout and abandon rates on qualitative screeners are through the roof. People are demanding more money for getting into those groups.

They’ve got things like TaskRabbit, and Uber, and Lyft that they’re all back doing now that can supplement their income and they’re like, “Why would I go through all this pain and headache to try and get into an online focus group when I can go out and make more money doing something on one of these digital platforms?” And we hear that time and time again and it’s causing a real existential problem for the industry. We’re paying more to bring new panelists in, our panelists are expecting more as an incentive, and unless we get in front of this and do a better job, frankly, of PR about why people would want to participate in research—because a lot of people think it’s the creepiest thing on earth to be asked all these questions—we’re going to run out of qualified respondents at a point, probably not in the too distant future, where it’s just going to get harder and harder and harder to do the research we need to do.

Lenny: Yeah. So folks, this shouldn’t come as a surprise, right? This is why I really wanted you on, Isaac, because you’re seeing it where the rubber hits the road, right? Because of your volume, it’s unmistakable. But this is a problem has been brewing for a hell of a long time, and then we just reached the perfect storm situation.

And it’s just something that we must rethink what our value proposition is to consumers. There’s too much competition across the board. And you said it, I don’t need to echo it back; everything you said, I agree with wholeheartedly. Even though we pitched a fit when California wanted to group research in as part of the gig economy, there was also a part of me was like, well, hell yeah. We do.

Isaac: It—well, that, that’s—we’re competing with it, so, like—

Lenny: Yeah, and so we better start thinking like Uber, TaskRabbit, you know, et cetera, et cetera, as well as TikTok and [laugh] Twitter and Facebook. You know, how do we engage? How do we deliver value? We have to think like marketers because those businesses have been incredibly successful at building the value proposition for consumers to engage and perform tasks for a reward that—whether it’s social or fun or compensation. And we’ve always had this expectation as an industry of, well, no, you just do it because it’s, you know, the right thing to do, right?

And, you know, we fought to even offer incentives. So, maybe you’re old enough in the industry to remember some of those debates of, oh, if you had pay somebody to participate in a survey, then it’s just bullshit, right? It’s like, well, the world has changed and we must adapt.

Isaac: Well, you know, Lenny, it’s the value prop isn’t just about the money that you have to compensate people. I think it’s a part of it, but like, it’s a fun exercise to do to talk to researchers and say, “Hey, like, let me ask you a question. If you took the last quantitative survey that your company fielded or the last qualitative screener, would you make your mother take that?” And everybody laughed. And they’re like, “What do you mean?”

And I’m like, “Well, like, how long was it?” And people will be, like, “20, 30, 40 minutes quant survey.” And I’m like, “Would you encourage your mom to do that?” Like, you know, “You love your mom and everybody loves your mom, and—but would you want a friend or a relative to, like, actually take your survey?” And, like, every—like, almost everybody goes, “No.”

And so, you go, “Well, that is some cognitive dissonance right there, we need to come to grips with.” If you know the people that you love and appreciate in your life, if you wouldn’t put them through the punishment of a 45-question quant survey? How in the hell are you going to get consumers who don’t care about you or your brand or anything else more than, you know, whatever else is going on on TikTok and Facebook, how are they ever going to want to complete the surveys that you have? Complete the screeners that are getting longer and longer and longer. And I mean, the truth is, Lenny, until we can change the tide of this and make these instruments of data collection competitive with something fun and easy, like, watching TikTok videos, we’re going to continue to see this trend go in the [right 00:15:52] direction.

Now, what’s what is heartening is, you know, you and I have obviously been observing and talking about this industry for over a decade, and this has been given lip service, frankly, for most of that time. People are actually starting to take it seriously now. So, you know, we are getting invited—Sāgo is getting invited to sit down with folks. I mean, I did it literally today and say, “Let’s look at this screener that we’ve written.” This one this morning was 29 questions, I think was what the average person was going to get.

And I said how are we going to get this down to 12? Or 14? Because I’m not going to be able to field this study for you—full stop—unless we make it more accessible for those consumers. So, people are starting to take it seriously. And so, that’s heartening. We better hurry up because the pool of engaged respondents today is leaving at pretty dramatic rates, and we got to get in front of it.

Lenny: Yep. Yep. Agree wholeheartedly. And brands are paying attention, too, right? And it’s not just that the usual suspects, you know, P&Gs of the world that have always kind of led the charge, but eh, you know, not much really happened.

Brands are recognizing it. Because they’re seeing in the data, not just from a quality standpoint, but also [unintelligible 00:16:56] you throw away 30% of the sample because it’s bad quality, but then also economically, does that make sense? Like we just paid for, you know, 30% of something we threw away? So, I think folks are ma—are connecting that tissue. Are you seeing that with brands, your direct brand work?

Isaac: Absolutely. And you know, I hear some really positive things from brands that are kind of new for our industry, things like, wait, are we saying that these are our buyers, these are our customers, the people who drive our revenue and we’re going to punish a thousand of them [laugh]? Like, that’s a bad brand experience. And so, I’m hearing that more and more that researchers, CMOs, folks that are in charge of the brand experience for the brand as a whole are going, “It’s not acceptable for us to punish our consumers in this way and put them through surveys.” Like, sure it’s a blinded instrument and all that kind of stuff, but still, like, this just is the wrong thing to do.

So, people are paying much more attention than they did pre-Covid. And I do think the ship has started to turn in the right direction. It is slow, but we’re going to be forced to. And I think it’s a good force, to be honest with you. Like, I tell people all the time, they’re like, “Why are you charging me more for incentives?” I’m like, “I’m not charging you anything.” Like, these [unintelligible 00:18:06] consumers are what—consumers are charging you to play ball. And so, you know, we are coming to grips with it as an industry and I think it’s gotten to the point where it’s going to make us change our behaviors in a positive way.

Lenny: All right, so let’s add a little more insult to injury then on this conversation because here came generative AI. And, you know, and it’s not just a buzz term, is I think about the conversations that I have, both with suppliers and with brands, and particularly with brands right now, they are all-in on that promise that we had with big data back in the day of, you know, we’ll have the who, what, when, where, and how through all of the synthesis of all this data, but we couldn’t quite pull it off because we didn’t have the right technology to synthesize it. Well, now we do. And the implications that we’re seeing right now on creating virtual audiences, you know, virtual respondents, that is huge, huge.

So, if we can’t get people to—well, let me rephrase that and there’s two issues here that we have to come to grips with. One is when we don’t need to ask as many questions because the data is available through lots of other sources, that has very pragmatic implications for the industry, period. So, questionnaires are going to decrease in size, you know, et cetera, et cetera. We get into filling the gaps of information versus trying to ingest a whole lot of information. So, let’s recognize that.

And then secondly, as they recognize the opportunity to build out these virtual audiences and these virtual models if we can’t even deliver people to fill in the gaps of information, it’s only going to drive more reliance upon utilizing AI to fill in the gaps.

Isaac: Yeah, I think you’re right, Lenny. I mean, we obviously at Sāgo have a lot of investments in AI, we’ve got a lot of partnerships, a lot of our own technology, and we are looking across the entire spectrum of our workflows and seeing how could AI or even generative AI help. And I think, you know, the obvious areas are reporting and analysis. And, you know, there’s a lot of good work being done there; we get a lot of tools in our toolkit that help us there and it’s totally, totally assistive. And I think that’s kind of the obvious one.

The next one that you touched on was this idea of what we call synthetic audiences. And you’re right, you do need, in most cases, some genesis data that you need to get from, you know, live, breathing humans at this point, but then you can do a lot with that synthetic data projection and we’re starting to see some really interesting results from that. And that makes people a little queasy, I think, you know, when you think about synthetic audiences and having AI predict and project what a larger audience would say, people get nervous. And I keep coming back to, like, the core question of why we do research. And, you know, we get away from that a lot.

And I think, you know, we’re not in business just to talk to consumers. Like, we’re in business, to actually help inform business decisions, right? And so, should we really care if the results of generative AI are as good or better in whatever measure you have to help aid business decisions? And I don’t think we should really get wrapped around the axle about whether it’s talking to a real human being or not. The businesses we serve, they want to drive revenue, okay? And they want to drive revenue by knowing what consumers want, think, and their behaviors.

And if generative AI can plug that gap as well as talking to a thousand living breathing human beings, I don’t know that as an industry we should care. We should care that the results are accurate in the sense that they drive the right business decisions, but I don’t think we should get all wrapped around the axle about this idea whether it’s coming from a hundred percent live consumers or not. There’s always going to need to be consumers at the table. I don’t believe, in your or my time in this industry—and hopefully, I’ve got a few more decades left in this industry—that we will fully see the consumer just sat down and technology drive everything. The Terminator I don’t think is coming for our industry.

Because, you know, at this point, AI is really good at matching patterns, right? Well, if you don’t have any existing data to train it on, like, if there’s a brand new product category or brand new idea, what is the data set that you need to point that artificial intelligence at? And the answer is, this is really, really tough to do. So, we will always need the consumer to have a seat at the table. The question is, how do we augment their voice? How do we accelerate their voice using this technology?

And I don’t think it’s something we need to be scared of. I think it’s something we need to embrace. And frankly, if we don’t figure it out and embrace it as an industry, we may see the industry pass us by a little bit. We may see technology players who do figure it out and jump into this space and be able to answer business questions better, faster, cheaper than we can as an industry, and that could pose a real challenge for us. But I have faith in this industry and I have faith that we will figure out a way to embed this into our practices over time and be better for it.

Lenny: Yeah, I couldn’t agree more. And so, this idea that we’re talking about, this has been—in a much less sophisticated level—been done within advertising, you know, other areas for a long time, right? The idea of utilizing—political polling. You know, targeting folks off existing data is something that is normal and customary. So, it’s not a leap to think about how we do that now.

But I agree with you that there will always be a need for research to provide more data inputs because things change, just as we just talked about the shift from pre-Covid, Covid, post-Covid, right? And it seems like the world is not letting up on, you know, these pressures that are changing society, changing cultures, changing people individually and how they adapt; it just continues on. And so, those gaps of information will continue to be there and research is the ideal way to do that. And really, I think that the panel companies, companies with great panel assets like Sāgo are going to be vital in helping to do that, not just the field services, right? We still need to ask the questions. But the assets that are being built and organized around not just engagement, but also kind of the data graph, right, of building profiles of information, those are those training sets.

Isaac: That is our belief at Sāgo for sure that, you know, having access to those consumers, business people to help build that training data, to help project the things that nobody’s asked a question about if there is an existing data set, that is going to be a real key fundamental part of our business. And, you know, when we think about building our panel assets and we’ve got, you know, access to millions of consumers and the depth we know about them and how we start building that out, we put a lot of effort behind what data do we need to collect to help train these AI models that might not have been terribly relevant for survey data collection or qualitative collection, but could be relevant in a data training world that’s a very active conversation here at Sāgo.

Lenny: Yeah, and I think—so here’s my prediction; see whether you agree. Now that I think that the generative AI thing has really put into focus the value of data as a whole. And we’re already seeing companies, I think last week, Reddit said, “Nope. We’re putting a moat around our data. You want access to Reddit content for your training sets? You’re going to pay us for that.”

Brands are recognizing that as well. And I think we’re going to see the emergence of proprietary panels and communities as part of that data asset that they’re building moats around to build that internal value. So, I think that’s a really great opportunity for the industry to think about, what does it look like to enable that type of data asset at the core, that is then fed by research, right, that’s focused on all the business issues that we already think about to fill in the gaps of information? So, what do you think? Am I full of it, or uh—

Isaac: Oh no, you’re not full—Lenny, that’s already happening. Like, you know, we have a fairly large amount of business these days coming from working with brands on proprietary datasets, visual data, textual data, all the things that are relevant to their brand, and they’re starting to build these datasets. And in some cases, you know, we are helping them collect the data. And they don’t even know what they’re going to use it for yet, right, but they know that they’re going to need it

And so, we have several major client projects underway right now doing exactly that. So, I don’t think that’s, like, I’m going to take that bet and I’ll be on your side of it because that’s not going to happen; it’s already starting to happen because they are realizing that this, especially, hybrid proprietary data is something that they need to be collecting now. And, you know, you used the Reddit example of, you know, creating these walled gardens. I think it’s a good one. So, I’m a big Reddit fan. I am on there a lot, and I’ve watched this kind of interestingly because I think it’s got an implication for our business.

In market research, it’s been an extraordinary amount of coopetition, right? So, especially on the quantitative side, there’s a lot of collaboration to get sample filled, there’s a lot of collaboration about opening up panels, there’s exchanges. We’re starting to see those things close down. We’re starting to see the collaboration really be less open and much more focused. And so, you’re picking more premium partners to work with, you’re aligning your datasets more.

And I think it’s for two reasons. Number one, I think the openness that we were sharing, consumer access, we realized, whoa, whoa, whoa, like, we were giving away some of our proprietary secrets in some ways. And number two, we weren’t really in a good position to start building these deeper datasets, unless you have this first-party data access. And so, we are at Sāgo, we have a lot of work and effort in this space to make sure that we’ve got the right first-party premium datasets for our customers. And then also, as we work with partners, we’re kind of narrowing that partner list to work with folks who we can align datasets, we can understand things on similar levels. And I think it’s making us run a smarter, better business.

Lenny: That’s great to hear, not just because I like being told that I’m right. But the—[laugh]—although I make my living by… you know, pontificating and then being right, so it’s good to have that validation [laugh]. But I think it’s a positive vision for the industry. I mean, I remember giving talks on this idea ten years ago and thinking, you know, “Look, it’s going to be scary. It’s going to be weird. It’s going to—but we will get there.”

What I couldn’t understand at that point was what was the technology that would unlock it. So, it’s great that we’re there now and that companies like you were leading that charge. So, I want to get back to a couple of the data points that you brought up in your report. So, what were some of the other findings that you think that the industry needs to hear that maybe they’re not thinking about?

Isaac: Well, you know, we talked about I think one of the big existential one is how we keep respondents engaged in our research. I think another thing that we have seen, specifically looking at digital qualitative is, going into Covid, as an industry, we were a little bit lazy about how we thought about research design: very cookie cutter. And you know, that’s—you know, we’re all trying to run scalable businesses, and so it’s easier to repeat success than it is to go do something new every day. And I get that. I understand that.

But what the trap I think we fell into pre-Covid was, we really weren’t thinking about these digital platforms, online video interviews, digital communities, and there are robust ways we should have. We kind of took what worked in in-person research and we kind of took that discussion guide and we kind of just kind of moved it online. What we’ve seen now, now that Covid is forced everybody to adopt, now you’ve got competition in the space. So, before you had research firms that were good at digital qual and you had firms that just never touched it and there was kind of two different camps. Now, everybody has to do it, and so now everybody has to compete with each other.

And so, what does that mean? How do you leverage these tools to really drive value for your end client to help inform those business decisions? So, we’re seeing a lot more iterative research where it’s multi-phase research. And we’ve been talking about this for years, Lenny. Like, you know, we talked about hybrid research and mixed methodology research and people are like, “Yeah, yeah, yeah, whatever.”

And it was always, like, 5 or 10% of the work out there. Now, it’s probably about 35 or 40% of the work where it’s truly either hybrid quant-qual research or iterative multi-phase research is easily, like, 35 or 40%. I’ll tell you on the iterative designs in digital qual, we see above 50%, meaning it’s not just an online 3-day discussion, it’s not just a set of video interviews, we are combining those methodologies so that I can talk to Lenny in an online digital diary and learn about Lenny, learn about his practices and what you’re doing on your farm there and all the different things and the seeds and whatever my research is, and then I can get Lenny on camera a couple of weeks later and ask you some follow up questions. And then I can bring Lenny into an online group discussion a week later with people who are, you know, similar people to Lenny and talk in deeper detail about the fertilizer used or the seeds you used. And so, I get so much more juice from the squeeze from Lenny.

And one of the reasons we didn’t do that before was we thought well, don’t we want fresh respondents every single time and for every phase of research? And what we’re finding is that when you’ve got these multi-phase approaches—which again, have exploded post-Covid—there’s so much more Lenny that I can get in a single way to talk [about it 00:30:53]. Like, Lenny, you and I are talking right now on this podcast. I can actually see you in this version, so we’re doing, like, a video interview—which Sāgo does millions of every year—[which is 00:31:02], that’s great, but you know what, there might be ways that Lenny shows up differently or answers differently when I send him out with a smartphone to go take photos in his garage or take photos of something or take a video of how he, you know, prepares a vegetable bed. And I might learn something totally new that you would never tell me in a one-on-one conversation.

And then if I get Lenny in a group of people who are talking about this subject and finding that, you know, they agree on certain things or disagree on certain things, you might bring up things that you would have never brought up in a one-on-one video interview. And so, we’re finding there’s so many more layers to Lenny. There are so many more layers to that consumer than you can get in a single serving. And that’s why these multi-phase designs I think become such a growth area for our industry in the last couple of years.

Lenny: I am multilayered [crosstalk 00:31:45]—

Isaac: Oh, don’t we know? Don’t we-

Lenny: —thank you.

Isaac: —know?

Lenny: [laugh]. And back to our previous point, too, the technology is making that easier now, right? I mean, before what you’re describing, there was a whole lot of work involved to make that usable.

Isaac: Oh, it’s a click of a button now. And, like, like—and, you know, I still find researchers who are blown away and I’m like, “Hey look, you’re doing an online discussion, our platform, click this button and Linda gets invited to a video interview tomorrow night.” And they’re like, “Wait, is it that easy?” And I’m like, “Yeah, and it’s been that easy for a long time,” but now we see the value in it. And it’s not just about being able to get more data out of Lenny. It’s about learning more about you and learning in different ways and, honestly, providing a richer understanding of who you are as a consumer than just in a, you know, in a 60-minute online video interview.

And, Lenny, I can tell you a story [unintelligible 00:32:35] this is not just a digital story; this is an in-person story and I could tell you all sorts of stories about when you get people together in an in-person event and then you talk to him on a digital scale afterwards, you follow up with them two weeks later, you’ll learn things that they didn’t tell you in the focus group. You’ll learn things that they wouldn’t have shared in front of a group of three or four individuals. And so, none of those methodologies are wrong at all. They all have pluses and minuses, but the combination of multi-method gives you that ability to understand multiple dimensions of Lenny. And it honestly answers a more robust business question about what this consumer is thinking.

Lenny: Yeah. Yeah, agreed. And I would also say that I think—not to keep going back to it, but I think it’s relevant—that all these changes we’re talking about usher in a golden age of qual. I think that research as a whole is going to look and feel experientially be far more qualitative in the way we think about it from a discussion standpoint—I’m not talking about sample size; just the, you know, the techniques of engagement—than it will be quantitative.

Isaac: I agree. And you know, as, you know, the president of Sāgo who does both qual and quant, I can’t, like—I love both of those methodologies. That being said, it is a very common refrain in our industry that, you know, qual may be in a resurgence at this point for a bunch of different reasons. Over-reliance on quantitative data, difficulty in getting the right audiences, you know, pick your poison, there’s lots of reasons why people may be incorporating qualitative more. But you know, for me, the beauty of qual is—well there lots of things I appreciate about qual, but one of the things a friend of mine once told me, “You know, Isaac, we don’t tell our kids bedtime facts, we tell them bedtime stories.”

And the story sticks with you, right? And you get such an amazing story from your consumers from qualitative. You get all the data in the world you want from quant. You can make stories from quant but there is nothing like seeing the written word of your consumer, seeing somebody on video, seeing how they actually go plant that garden, and it just brings it to life. And again, I could tell you, I could sit here for the next four hours and tell you stories about brand managers from, you know, very well-known, respected brands who question results from qualitative because they’re like, “Well, we’ve never seen people do that. We’ve never heard people do that.”

And I’m like, “Well, why not?” And they’re like, “Well, you know, we don’t do a lot of qual. We had no idea people would do this.” We were doing a study one time on appliances in homes without microwaves and we had a bunch of IDIs, and—it was a multi-method study. It was pre-Covid. It was a kind of an early multi-method study. And so, we’d actually gone on shopping trips with these individuals, and then we did online at-home diaries. And we had a respondent who had aluminum foil around their microwave. Okay? And—

Lenny: But not in—

Isaac: Not in.

Lenny: —hopefully [laugh].

Isaac: [crosstalk 00:35:14] it was in a photo that was captured in our [unintelligible 00:35:16] platform. And so, the researcher came back to us and said, “Hey, we got to probe what was going on there.” And turns out this person had a belief—true or not true, I’m not sure—that there was some sort of radio emission. She had a brand new child at home and so she said, “You know, better safe than sorry. I don’t know. It’s maybe weird, [maybe 00:35:33] new protected mother, you know. She got this little baby at home. [unintelligible 00:35:36] I’ve—yeah, I’ve started putting microwave around my microwave—or rather aluminum foil around my microwave, just to—just in case.”

And you know, what’s funny? Like, the client was like, “Oh, my gosh. We got this person that has this weird kind of thing.” And so, guess what we did, we actually asked people in that same group on our online discussion, said, “Does somebody else have that same fear?” It turns out there were two people in that group of thirty who did the same exact thing.

And about half the audience said, “Yeah, I could see how like—you know, now we’ve got questions about it.” And so, here’s this brand manager from this very well-respected appliance manufacturer going, “Oh, my God. I had no idea that people felt this way about this type of appliance. I’ve never asked that question in a survey and I only learned it because we observed it in a photo and then you blow it up and you find out, oh, my gosh, it’s not just this one person. There’s a whole kind of undercurrent of this concern that we’ve really never tapped into.”

So that, I think, is the beauty of qualitative is that you can explore in places that you didn’t even anticipate. And you know, with quant being a little bit more on rails and being able to give you large projected datasets, the beauty of qual is you can really explore with the consumer like you can’t in other methodologies.

Lenny: Yeah. And what a great story. And as somebody whose wife is made—buy lots of Shungite to put all around all of the—it’s a stone—

Isaac: Okay, that was a new one for me.

Lenny: It’s a stone that absorbs EMF radiation. And we have Shungite all around the house to try and [crosstalk 00:37:00]—

Isaac: There you, there you, there you go, Lenny. You should—

Lenny: EMF.

Isaac: You should have been in our qual board discussion because you would have been, you would have been a great audience for this brand manager.

Lenny: It’s an interesting question, right? So, better safe than sorry. But [laugh] as I sit here with my WiFi router, right? But that’s a great point. I remember doing research for Kimberly Clark back in the Rockhopper days on cleaning cloths and discovered that a huge audience for them for these clean—this—I forget this specific type, but it was very lush microfiber, you know—wasn’t janitorial; it was car collectors.

Isaac: Oh, right.

Lenny: —because they—it didn’t leave scratches, it was great for polishing, you know, their car babies. And it launched a whole other product category. And that was done through qualitative research to identify. Because we found out that janitors were taking them home, right? And—rather than throwing them away, they were repurposing them so they could clean their cars.

Isaac: Makes perfect sense. And you know, those are the kind of things that I think qualitative can really help you uncover. And one of the reasons I love this industry is—because I’m always just totally curious about the human condition and finding people who have these, you know, interesting uses for baby wipes. Like, it makes me want to go do it myself, right? And so, you know, consumers are just these fascinating animals.

And I love being in an industry where we get to where we get to be really in service to them. Because at the end of the day, we’re trying to make better products and services for them by learning about what they do. And if Kimberly Clark wanted to make an automotive product that worked better than the competing products and they learned that in research, I think that we made the world a better place because of it.

Lenny: Yep. Absolutely, absolutely agree, right, wholeheartedly. So, I want to be conscious of your time as well as the audience because you and I could go on and on about this type of stuff. So, you did some quantitative research as well. Was there anything—or looked at your quant data—anything else that popped out of that, that we want to make sure that we—

Isaac: Yeah, so we’re—we are, we are still in the middle of analyzing all of all the data, Lenny, and we are finding similar themes on the quantitative side of our business. [respondent 00:39:06] engagement: just as big, if not bigger, challenge in the quantitative world. Incentive rates haven’t changed nearly as much, and I think it’s a conversation we need to start opening up. And as a matter of fact, our findings, which will come out a little bit later in July, I believe, we’re going to talk about the incentivization problem that we have in quant. In qual, it’s just so much more obvious because the consumers will tell you, “No, I won’t do that for $75.” Whereas in quant they will just click away and so you don’t know. Did they abandon the survey because it wasn’t worth their time or did they abandon it because you got too many stupid questions?

And so qual, we’re just faced with it more obviously. And so, in quant, I do think we need to change the value prop in the exact same way that we’re talking about in qual. It’s just a less obvious problem. And we’re going to shine some light on that with the data we’re doing. We’re also finding in the quantitative side that one of the, kind of, more frightening things I think that people think about generative AI is well, you know, what is that going to do for fraud and how’s that going to increase fraud?

And let me tell you, Lenny, it has had an impact already and it will continue to. In our data, we have found that there is an increasing amount of leverage of ChatGPT and similar methodologies to aid and abet fraudsters. And so, the folks out in the industry who are there to combat fraud, folks like Sāgo who put in all sorts of measures and controls, we’re now having to make that part of our control mechanism. And so, in our findings, we’re going to talk about how much more you have to lean into your data auditing, and how do you—you lean into your data security because it is becoming much more challenging to leverage things like open-ends as a telltale sign of fraud. Now, open-ends from ChatGPT look as good or better than what you and I would write it in an open-ended box.

And so, you know, we jokingly say, like, well, maybe that maybe the answer for detecting fraud in open-endeds is looking for people who use, you know, full, complete sentences and [laugh] good grammar because the computers don’t get that wrong and the humans actually do. It’s probably actually not a bad indicator. But yeah, it’s going to be a whole new category of fraud detection is going to be around ChatGPT. And again, like that sounds like a negative. Let’s turn it into a positive.

Like, in some weird way, if that open-ended that a computer generated can enhance the business question and answer it in a way that’s better than human, like, I’ve got this real, like, puzzling question in my mind, like, should we embrace it? Like, if I can ask a thousand Lennys or I can ask a computer and get the same or better answer from a computer than I get from Lennys on these questions, like, we’re not a business just to talk to Lennys, we’re in business to help drive business decisions. And so, like, I think, rather than be afraid of it, we need to control it, obviously, and understand it, but we also I think, need to embrace it because it can actually make our work better.

Lenny: Yeah. Right, there with you. And, yeah, we can go off on another tangent on that, but I do want to put the point on it that we talked about this idea of synthetic sample or virtual respondents. The use case seems to be early exploratory kind of concept, understanding audiences at a basic level. Hypothesis, right, hypothesis-forming, which I think is a logical use case.

And kind of to your point, if we can get—if we have the data and the system can give us answers that help point us in the right direction to new questions, then I think it’s a useful tool. It may be one we have to be careful of, so I personally, I keep struggling with utilizing the ChatGPT-type tools because I don’t want to be lazy. I see that tendency in myself. There’s [laugh] actually a presentation earlier this week and we were looking at a platform called Decktopus that ChatGPT creates a whole PowerPoint deck. Literally. Or something related to—I don’t know if it was ChatGPT, but open—generative AI—a whole PowerPoint deck in seconds.

And it’s like, man… because I hate making PowerPoint [laugh] decks, right? But for me, it’s like, I don’t want to get too lazy, right? I want to make sure that I stay sharp. And those are just the types of things we’re going to have to continue to work through is these tools because they’re there, they add value—obviously—and we’re—keep working through them. And they can be used badly, to your point, around the fraud. So, interesting times. And I’m so glad that you guys have so much volume flowing through you that you can get an idea on what’s happening. So—

Isaac: It’s an honor to be in that position. And you know, one of the ways that we’re trying to help—I don’t want to say, “Give back,” but, like, make sure the industry has some visibility into these things is these reports that we’re doing to, you know, just show trends and things. Because at the end of the day, like we want, you know, the industry to know what’s going on, so we’re just more than happy to help.

Lenny: Yeah. Always appreciate that about the DNA of the company as a whole, right? When you were Schlesinger, you did the same thing. At Sāgo, you’re doing it. It’s just thrilling. And by the way, for an audience who doesn’t know the story, right, this is a company that started literally at a dinner table.

Isaac: That’s right.

Lenny: Right? Doing focus groups in Steve’s mom’s house.

Isaac: That’s right. That’s right. I’ve actually met researchers who debriefed their clients in Steve’s childhood bedroom [laugh].

Lenny: [laugh]. So, what a success story with that, too, right? I mean, you obviously have been inside, but I’ve been privileged to have some views over the years of what was going on and it’s just always been so cool. Just, like, a great story. So, for entrepreneurs out there, right, all these things we’re talking about, they are opportunities as well, and you guys are an example of… literally [laugh] you know, starting it in your house and creating now one of the largest research companies in the world.

Isaac: Yeah. And you know, and on that note, innovation, like you know, keeping an eye to the future because, you know, we have been able to grow and thrive and change because the DNA of this organization is that we are looking to what’s going to come next. If we had stuck in, you know, Steve’s mom’s kitchen table, we would have never grown. If we would have stuck doing just in person, we would have never grown. If we had stuck not getting into quantitative research, not getting into digital qual, not getting into global professional services, we would have not become the company we are today, so you’ve always got to keep an eye on the future. And I think that’s why you know, folks like you educating people on what the future is going to look like, helps firms like us place those bets and really move the industry forward.

Lenny: Oh, great. I love it when we have a mutual admiration society. So, anything else that you want to share with the audience, Isaac?

Isaac: You know, Lenny, I think this has been a fantastic session. I really thank you for hosting me. I do want to point out, we will be releasing our quantitative analysis report, kind of the state of the qualitative industry, sometime mid-summer, so stay tuned for that. And hopefully, maybe you’ll have me back.

Lenny: We’d love to. It’s a great conversation. I know that there’ll be many more, definitely, and let us know—GreenBook—when you release those reports.

Isaac: We’ll do, Lenny.

Lenny: Yep. And I think that’s it for today. So, big shout-out: thank you, Isaac for your time. Thank you to Natalie, our producer, thank you to James, our editor, and thank you to you, our listeners for taking time out of your day to spend it with us. That’s it. We’ll see you on the next edition of the GreenBook Podcast. Bye-bye.

 

*Originally published July 20, 2023