Paradigm Shift: From recall to occurrence

Image Credit: Saad Faruque

The human capacity for recall is subject to a number of major psychological flaws. The only way to close the gap between what testers do on your site and what they tell you they did on your site is to listen to their thoughts in real-time.

Recall is, quite simply, the retrieval of information stored in the memory. People rely on recall all the time, reaching into the brain’s vast catalogs to answer questions like: Where did I leave my phone charger? Which gas station has the best prices? What filename did I use to save my document under?

In researching our customers, we frequently rely on recall-based feedback from regular users, recruited testers, and focus group members, through surveys, direct questioning, or other methods. We ask things like: What did you think of the registration process? Did you have any trouble finding the contact information? How did the landing page make you feel about our brand?

The problem is that methods like this force users back to their mental caches for the answers, and this exercise is fraught with psychological flaws. A number of factors exert influence on the accuracy of human recall:

1. Attention – If the respondent’s attention was divided at the time they encoded that memory, their recall later on will be hampered. So for example, if your survey respondent was using your website in one tab, and shopping for Mother’s Day gifts in another, and listening to music, their recollection of the experience is almost sure to be a little mixed-up.
2. Motivation – When no incentive exists for the respondent, their feedback is likely to be less accurate. Alternatively, when an incentive does exist but emphasizes completion over accuracy, respondents are more likely to provide inaccurate, and ultimately counterproductive, information.
3. Primacy & Recency Effects – People tend to be worse at remembering the middle elements of a series (or an experience) than they are at remembering the first and last bits. In other words, between the very first impression and the most recent memory, things tend to get hazy. This also makes it easy to overestimate the importance of early- and late-occurring problems in the user flow relative to other issues.
4. Interference – Recall suffers when time has elapsed between the encoding and the remembering, especially if other actions were taken in the meantime. If respondents aren’t filling out your survey right after they finish using your product, everything they do in between affects their ability to respond accurately.
5. Context & State Dependency – Recollection is impaired when the respondent is in a different environment or mental state than when they encoded that memory. If they used your product at work, for example, but filled out your questionnaire at home in front of the TV, their answers may not reflect their actual experience.

In addition to all these, respondents may also be affected by issues related to response formats or settings. The framing of the question, for example, can have an enormous influence on people’s answers. If your respondents are participants in a focus group, their feedback may be manipulated by social pressures; people give in to conformity and bandwagoning, they lie to avoid appearing inferior or incompetent, they answer questions the way they believe they are “supposed” to, feeding the administrator what he or she wants to hear.

The human mind does endless twisting and turning, self-deceiving, and overcompensating to satisfy the situation – and as a result, almost none of your recall-based feedback is left untouched by one or another distortions.

There are ways to control for some of these issues; problems like division of attention of context dependency might be diminished by confining testers to a controlled environment, but at the cost of losing a genuine, true-to-life look at the customer’s experience. Primacy, recency, and interference could be combatted by asking testers questions at the end of each individual task, but at the cost of obstructing the natural flow.

Shifting to Occurrence

The only way to close the gap between what people do or feel and what they remember doing and feeling, and what they tell you they did and felt, is to look at their occurrent thoughts – that is, the thoughts that are in their mind at the exact moment of their actions.

By eliminating the need for recall, we eliminate the flawed mental filters, the forgetfulness for middle elements, the transitions between mindsets and environments. Verbalization of the thought process, as it happens, becomes the vehicle for higher-quality feedback.

This is something that comes naturally to people: almost all of us talk to ourselves at times, especially when alone. Now companies are harnessing this natural instinct to create the next generation of customer research. Remote user testing taps into testers’ screens and microphones to capture real-time experiences, allowing researchers to listen to their real-time, occurrent thoughts and get the accurate, unadulterated picture.

Keeping things remote has a number of benefits too:

• Testers provide feedback in a comfortable, real-life environment – in their own home or office or other frequented space, using their own device that they’re familiar with.
• Testers are not affected by a compulsion to please the researcher or render the “right” answer since there is no personal connection or interaction between the two.
• Unhindered by matters of logistics, researchers can access testers anywhere in the world, who are able to provide feedback on their own time.

For all of these reasons, there has been a palpable shift from recall-reliant feedback methods towards occurrence-based methods, and the young remote user testing industry has seen massive growth in the past few years – evidence that on a large scale, companies and researchers are rethinking how to get the best data.

The problems with recall have long been known to psychology, but a change in paradigms can only occur when there is something to switch to. Now, with services that connect researchers to users around the world and help them understand what people do and why they do it, recall-centric thinking can and should be laid to rest. After all, why keep peering through a cognitive kaleidoscope when you could be peeking into your customers’ heads?

Tim Rotolo is a UX Researcher at TryMyUI, a remote usability testing company based in the California bay area. You can reach him at [email protected] or follow @trymyui on Twitter.