What’s The Big Deal About Agile Research?

 

By Ron Sellers

So what’s the big deal about agile research?  Two good recent posts on this topic (Jeffrey Henning from February 5 and Matt Warta from March 9) describe the benefits of speed and iterative processes in research, detailing their views on how agile research can be of benefit to clients.

In between reading these two posts, I checked the comments section on a blog post I had written late last year, detailing my attempts to send a couple of RFPs to phone field centers.  The post noted that although I gave vendors a week to respond, a number of them still responded late, which I found unimpressive.  The following was left in the comments section:

“A week to respond to two RFPs is not enough time for a thoughtful review, asking questions, and/or scheduling a call to respond, and formulating a thoughtful response.”

That comment really made me understand why agile research is such a hot topic:  two researchers are promoting that they can complete an entire project in less than a week, while a third is saying it should take more than that just to respond to an RFP for the fieldwork.

Like most of the “newer” or “nextgen” research techniques, inevitably some researchers (not Jeffrey or Matt) are promoting agile research as the solution to every problem, while others are dismissing it or calling it snake oil.  But is agile research really new?  Is it really unique?  Is it really a “different” kind of research?

Part of the answer, of course, depends on how “agile research” is defined.  It can include microsurveys, “instant” MROCs, survey automation, changing the questions or the stimuli during the project, and other elements.  But consider a few things “from the good ol’ days” of research:

• In traditional qualitative, we have long changed the stimuli or the questions during the project. Graphic designers in the back room would make changes to logos so we could eliminate problem elements in the designs.  Clients would add questions as we learned some new things.  We would eliminate product or name options as it became clear there were serious, unfixable problems with them.
• Omnibus studies have been providing a “quick consumer read” for decades. For many years, there have been a variety of omnibus studies with large, representative consumer samples, where you can get full findings back in a matter of days.
• In 2013, I wrote a blog post about my experience designing and conducting a 600-person study with a full questionnaire (including three open-ended questions) from start to finish in under two weeks…in 2010. Of course, way back then that approach didn’t have a special name or its own brand – it was just a flexible response to a good client that needed something fast.

I guess a lot of us did “agile research” years ago without even knowing it.  And that’s an important point – the concept of “agile research” isn’t new.  Today, it’s just codified and cleverly branded, and there are a few suppliers specializing in the approach who are making it even faster and more efficient.

The concept itself is really something that traditional researchers have been able to do for years, but they’ve largely chosen not to, for three main reasons:

1. Methodological rigor (carefully constructed sample frames, multiple recontact efforts, etc.) and representative samples were of greater importance.
2. Clients generally wanted more, not less – more questions, more respondents, more analysis. More and agile are not best friends.
3. There was no real client demand for it, since everyone understood that research took time.

The last point, of course, has changed significantly in today’s business climate.  Agile research has gained popularity because of a shift in business priorities – more and more, clients are favoring speed over detail, depth, projectability, methodological rigor, and other factors.  When you streamline something, it means certain elements have to be restricted or eliminated.  Agile research is about less, not more.

That doesn’t mean agile research is a bad thing – sometimes, less time has greater importance than more questions, more respondents, or more analysis.  The biggest concern is if less time means less quality.  Are samples properly balanced?  Do clients even care what the sample sources are?  Are qualitative participants properly recruited?  Are the samples mostly skimming professional respondents who sit at their computers all day taking surveys, so the field can close in two hours?  Are the same people being invited to “instant groups” over and over because they make themselves readily available?

Most importantly, do clients come to believe that all research should be this cheap and easy, and downplay the depth of insight and discovery that comes from other forms of research?  Do we eventually just throw methodological rigor out the window?  As Matt Warta from GutCheck says, agile research “is meant to be directional in nature,” but we all can tell the horror stories about the client who watched eight out of ten people in a focus group choose Logo B as their favorite and then insists that 80% of her target market favors Logo B, making projections in her business case based on that “fact.”

In fairness, there are drawbacks and dangers to all research approaches, and every one of them can be misused by over-eager or under-informed end users.  Questions such as these should be asked about every methodology and approach, not just agile research.

I tend to look at agile research from two perspectives.  The first is understanding what you’re really getting.  You’re generally not getting large sample sizes, which also means relatively little ability to evaluate key subgroups within the data.  You’re not getting a lot of questions (and in some cases, you have significant limits on just how customized your questions can be).  You’re not getting deep analysis or detail.  Instead, you’re getting speed and flexibility, usually at a fairly low cost.  The balance between speed and flexibility versus the importance of detail and rigor will vary with every information need.

The second perspective is understanding what traditional research can learn from agile research.  Does every study require a lengthy questionnaire or a large sample size?  Once a qualitative respondent points out that one of your potential product names means “flatulent” in French, is there really a reason to keep testing it in thirty more interviews?  Can we plan for iterative waves of testing in a quantitative study, adapting as we learn?  Does it really require a couple of weeks to provide a bid on fieldwork?

Agile research is, like most other methodologies, another tool in our ever-widening tool box.  It is not needed in many cases, nor is it even possible.  What if I tell you my target market is 25,000 subscribers to my statewide magazine and the only contact information I have on them is their mailing address?  Agile research that.  Or that it’s going to take me three months to come up with the five product names I want to test and have legal vet all of them, and adding any more to the mix will take another three months?   There goes the value of flexibility and iterative testing.  Yes, those are two projects I’m working on for clients right now.  And no, agile research would not be relevant for either one – but it (or portions of the concept) may be relevant for others.

In concept if not in exact execution, traditional researchers have been conducting agile research for a long time – so there is no reason to be afraid of it or see it as a threat.  We need to be ready to employ it when it’s the best choice for a particular information need.  We also need to understand it well enough so we can explain the advantages and drawbacks to clients, as well as when it may or may not be appropriate for a project.

Finally, we need to adapt the best parts of it to improve the other types of work we do (as the old saying goes, stealing content from one person is plagiarism; stealing content from many persons is research).  Because to stay relevant, even if you’re not conducting agile research, you certainly need to be an agile researcher.