5 Reasons Online Studies Fail

By Ray Fischer, CEO, Aha! Online Research Platform

Let’s admit it: many market researchers are either uncomfortable with online qual or they don’t get the results they expect, and therefore shy away from it.  That’s too bad, because the new wave of online qual tools and techniques is producing incredible insights for clients who have discovered the benefits of both technology, and adopting best-practices based on years of learning. Based on my experience over the last 12 years as an online qual researcher, I have seen five reasons why most online qual studies do not deliver on expectations.  And all of them are fixable…with the right amount of experience and skills.  Here is my take:

1. Not Enough Experience

Online qual can be a bit of a black box if you have not used the method before.  It is a bit like skydiving – you might want a guide attached to you on your first jump or two, but after that you’ll feel like an old pro.  In those early studies, make sure your platform provider is committed to your success; they should offer study design consulting and a dedicated project manager to share best practices along the way.  The same is true if you are new to online or are simply trying a different platform.  All platforms are not the same, nor are the services and support they offer.

2. A Boring Activity/Discussion Guide

The discussion (or activity) guide needs to be clear, concise, and dynamic.  Go beyond a battery of open-ended questions and use the variety of projective techniques that modern platforms offer.  A study that goes beyond open-ends and mixes in respondent video, collages, perceptual maps, social activities, and storytelling will make things SO much more interesting to your respondents and your clients.

3. Committee Approach to Study Design

Avoid the committee approach where everyone gets to add in everything they could ever want to learn in one study.  Don’t let it become a free-for-all.  I’ve seen these more than a few times: you create a mountain of unstructured data loaded with redundancy and irrelevance, ultimately detracting from your objectives.  Not only will the data haul yield insufficient results, it will also bore your respondents.  A key sign from respondents that the committee has left its mark is when you see comments from respondents like “I just answered that question…3 times!”  Stick to your guns and assure clients that the insights will come out if the questions and projective exercises are well thought out and diverse.

4. Lack of Communication

I firmly believe that communication with respondents – from the beginning of the recruiting process through the completion of the study – is key.  Be clear with respondents in the screener sharing exactly what the study is about, why they are important to the research, how much time is expected of them, how many days the study will take, and activities they are required to complete in what time frame.  Moderators – send a morning note to everyone each day of the study giving them group encouragement and letting them know what they are doing on that particular day.  Send at least one probe to all respondents on day 1 telling them, personally, how much you appreciate their contributions.

5. Insufficient Incentives
Nothing will discourage a respondent more than doing a lot more work than they anticipated when recruited.  Typically, a multi-day study should require a respondent to commit at least 30 minutes per day. If the study is interesting and well-designed, respondents will often spend a lot more time sharing because they want to, not because they have to.  Typical incentives for online qualitative tend to be $100 for 3 days, $125 for 4, and so on.  Store trips including video and/or pictures with added open- and closed-ends should include an additional $25+.  Of course numbers can vary, but these are pretty tried and true guidelines. I have heard a few clients who tend to use $.50-$1 per minute that they expect the respondent to engage.  I encourage higher incentives if the budget will allow.

After reading this you may think a few of these points are a blinding glimpse of the obvious.  And they are.  The lessons learned are pretty straightforward:  Keep it simple. Pay attention to the basics of good research. Work with an online qual platform that is intuitive and user-friendly, and most importantly, is supported by seasoned consumer researchers.  With a skilled team guiding you through the process, you should EXPECT better results with your next online study.