The Questions Behind the Answers: Why Great Research Begins Long Before the Data

Best Practices Are Meant to be Challenged

Customer researchers are trained to be skeptical.

We question sampling strategies. We debate statistical significance. We examine response bias, test for reliability, investigate unexpected findings, and spend countless hours cleaning data before we’re willing to trust the story it appears to tell. We’ve built an entire discipline around evaluating whether our evidence deserves our confidence.

That’s exactly as it should be.

Organizations don’t collect customer feedback simply to populate dashboards or produce quarterly reports. They collect it because product roadmaps, customer experience investments, marketing strategies, operational improvements, and countless other decisions depend on what that feedback reveals. When millions—or even billions—of dollars ride on those decisions, rigor isn’t optional. It’s a professional responsibility.

Yet there’s an interesting inconsistency in how we apply that rigor.

We’re exceptionally good at scrutinizing data.

We’re far less disciplined about scrutinizing the surveys, interview guides, Voice of the Customer programs, and other feedback systems that produce it.

That isn’t because researchers don’t care about methodology. Quite the opposite. Survey design, sampling, questionnaire construction, and interviewing techniques have benefited from decades of methodological refinement. We have no shortage of best practices.

The question is whether we’ve become a little too comfortable with them.

Think about the last research program you inherited. Maybe it was an NPS tracker. Maybe it was an employee listening program. Maybe it was a customer satisfaction survey that had been running for years. Chances are it followed accepted best practices, had been refined over time, and was considered a trusted source of customer insight.

Did anyone ask whether it was still the right tool for the questions the business was trying to answer?

Or was that assumption inherited along with the survey itself?

There’s an important difference.

This isn’t an argument against best practices. They’re one of the profession’s greatest strengths. They represent decades of accumulated learning and have helped researchers avoid countless methodological mistakes.

But best practices should be the beginning of our thinking—not the end of it.

Customer expectations change. Products evolve. Markets shift. Organizations mature. The questions leaders need answered today are rarely identical to the ones they were asking five years ago. Yet the surveys, interview guides, and customer listening programs those organizations rely on often evolve much more slowly. We add a question. Remove another. Refresh the branding. Update the reporting cadence. The instrument changes incrementally, and over time it’s easy to assume it remains fit for purpose simply because it continues producing data.

Somewhere along the way, our skepticism became downstream.

We rigorously challenge the findings.

We rarely challenge the mechanism that produced them.

That’s the blind spot.

We’ve developed sophisticated disciplines for validating data quality. We know how to evaluate reliability, validity, significance, and bias. Those practices have made customer research immeasurably stronger.

What we haven’t developed with the same consistency is the habit of periodically asking whether the systems generating that data still deserve the confidence we’ve placed in them.

If organizations are willing to make consequential decisions based on customer feedback—and they should be—then perhaps it’s time we subjected the systems that generate that feedback to the same level of thoughtful scrutiny we’ve long applied to the findings themselves.

That’s the conversation I’d like to have.

The Survey Everyone Trusted

The idea for this article didn’t come from a conference presentation or a journal article.

It came from a survey I had no reason to question.

Early in my career, I inherited a mature Voice of the Customer program that looked exactly like what most organizations hope to build. The survey had been in place for years. Leadership trusted it. It followed accepted best practices, produced consistent reporting, and had become a reliable source of customer insight across the business.

From everything I could see, the program was working.

So when something in the data didn’t make sense, I never questioned the survey.

I questioned myself.

I assumed I hadn’t looked deeply enough. I segmented the data, compared customer groups, searched for patterns in open-ended responses, and revisited the analysis from different angles. Like most researchers, my instinct was to believe the answers were already in the data. My job was simply to uncover them.

The problem was a familiar one.

Customer Satisfaction scores consistently suggested customers were having positive experiences. Net Promoter Score, however, told a noticeably different story. Customers seemed satisfied enough with individual interactions, yet far less willing to recommend the company to someone else.

Neither metric appeared to be wrong.

They simply weren’t pointing in the same direction.

For weeks, I treated that discrepancy as an analytical problem. Maybe there was a segment I hadn’t isolated. Maybe one metric was more sensitive than the other. Maybe the answer was buried somewhere in the comments.

The survey itself never entered the conversation.

Looking back, I’m not sure why it would have.

It was an established program. It followed accepted best practices. It had produced years of dependable reporting, and leadership trusted the findings it generated. I inherited that confidence along with the survey itself.

The turning point came when I stopped asking which metric I should trust and started asking a different question altogether.

What if the survey wasn’t designed to answer the question we were asking of it?

That question changed everything.

When I revisited the questionnaire, I realized the survey was doing exactly what it had been designed to do. It measured transactional satisfaction exceptionally well. It helped the organization monitor operational performance and identify where individual customer interactions were improving or declining.

The problem wasn’t the survey.

The problem was that the business had started asking it a different question.

Leadership wanted to understand customer advocacy—why customers would recommend us, remain loyal, or deepen their relationship with the company. The survey, however, had been designed to evaluate transactional experiences. Those are related questions, but they aren’t the same question.

The instrument hadn’t failed.

Our expectations of it had changed.

That realization permanently changed the way I approach customer research.

Today, when I inherit a survey, an interview guide, or an established customer listening program, I don’t begin by asking what the data are telling me.

I begin one step earlier.

What question was this designed to answer—and is that still the question the business needs answered today?

What We Already Know—But Rarely Apply

After that experience, I assumed I had uncovered a flaw in the survey.

Instead, I discovered something more interesting.

There wasn’t a gap in the science.

There was a gap between what the science has been telling us for decades and how we often manage our research programs in practice.

Behavioral researchers have long understood that surveys don’t simply capture customer experiences—they shape how those experiences are recalled, interpreted, and expressed. The questions we ask influence the answers we receive.

Gerald Zaltman summarizes this idea with a deceptively simple phrase: questions beget answers.

At first glance, it sounds almost self-evident. Of course questions produce answers. But that’s not really what Zaltman is saying. His point is that different questions don’t simply produce more information—they produce different information. They direct attention toward different memories, different emotions, and different interpretations of the same underlying experience.

One example from his research illustrates this beautifully.

Consumers were first asked a familiar question:

“What do you think of Mercedes-Benz?”

The responses were overwhelmingly positive.

Researchers then asked those same consumers a different question:

“What do you think Mercedes-Benz thinks of you?”

The respondents hadn’t changed.

The brand hadn’t changed.

Only the question had changed.

Yet an entirely different set of thoughts and feelings emerged.

Neither question was better.

Neither was more “correct.”

Each simply made different aspects of the customer experience visible.

That distinction matters.

Because it reminds us that surveys don’t just collect information. They determine which information becomes available to the organization in the first place.

Decades of work by researchers such as Norbert Schwarz and Roger Tourangeau reinforces this same conclusion. Responding to a survey isn’t a passive act of remembering. It’s an active cognitive process. People interpret questions, search their memories, construct judgments, and fit those judgments into the response options we’ve provided. Every design decision—from question wording to sequencing to response scales—has the potential to influence the evidence we ultimately analyze.

None of this is new.

It’s foundational knowledge within customer research.

Which raises an interesting question.

If we already understand that the design of a survey influences the data it produces, why do we so rarely revisit those design decisions once a feedback program becomes established?

We routinely validate datasets.

We routinely challenge findings.

We routinely revisit our analyses.

Perhaps we should become just as intentional about periodically questioning the systems that produce them.

Because if the science has already shown us that the questions matter, then the next logical step isn’t simply designing better questions.

It’s making sure we’re still asking the right ones.

When Best Practices Stop Being Best

Every profession depends on best practices. Customer research is no exception.

Best practices help us write clearer questions, reduce bias, improve response quality, and build more reliable research programs. They represent decades of methodological learning, and our field is stronger because of them.

This is not an argument against best practices.

It is an argument against treating them as permanent truths.

That distinction matters because the longer a survey, tracker, or Voice of the Customer program remains in place, the easier it becomes to confuse familiarity with fitness. The program keeps running. Dashboards keep updating. Reports keep circulating. Leadership keeps using the findings. Over time, the feedback mechanism becomes part of the organization’s infrastructure.

And infrastructure has a way of becoming invisible.

When we inherit an established survey, our instinct is often to ask whether it follows best practices. Are the questions clear? Are the scales consistent? Is the survey too long? Are we avoiding leading or double-barreled questions? Those are important questions, but they are not the only ones.

A more fundamental question is whether the survey is still the right mechanism for the decisions the business is asking it to inform.

A survey can be exceptionally well designed while no longer being exceptionally well aligned.

Imagine a company that has used the same customer satisfaction survey for years to monitor service quality. The instrument may still produce clean, consistent, useful data. It may follow every accepted principle of good survey design. But what happens when leadership begins asking a different kind of question? What if they are no longer asking only whether customers are satisfied with a transaction, but why loyal customers are quietly leaving, why advocacy is declining, or whether the brand still feels relevant in a changing market?

Those are not simply new analyses.

They are new questions.

And new questions sometimes require different ways of listening.

That does not mean the original survey has failed. It may still be doing exactly what it was built to do. The issue is that the business may now be asking it to do something else.

That is where best practices can become misleading. Not because they are wrong, but because they can give a feedback mechanism legitimacy long after its original purpose has changed. A program can remain methodologically sound while becoming strategically misaligned.

That is why periodic review should go beyond wording, length, and response rates. It should ask:

If we were building this listening program today, knowing what we know now, would we build it the same way?

Not because the answer will always be no.

Because even when the answer is yes, the organization has strengthened its confidence by questioning the mechanism rather than merely inheriting it.

That, to me, is the difference between maintaining a research program and governing one.

Maintenance keeps the program running.

Governance asks whether it still deserves to run in its current form.

That may be the next evolution of best practice in customer research: not replacing established methods, but periodically asking whether they remain the best fit for the questions we need answered now.

Making Curiosity Part of the Process

Every mature profession develops routines.

Those routines make us more efficient. They create consistency, improve quality, and allow organizations to rely on established ways of working. Customer research is no different. We develop tracking studies, Voice of the Customer programs, interview guides, and reporting cadences that become part of the organization’s operating rhythm.

Over time, those routines become trusted.

That’s a good thing.

But trust has an unintended consequence.

The more familiar a research program becomes, the less likely we are to step back and ask whether it still deserves the confidence we’ve placed in it.

That’s why I don’t believe this conversation is ultimately about surveys.

It’s about professional curiosity.

The strongest researchers I’ve worked with all share a common habit. They remain curious long after everyone else has become comfortable. They don’t assume an established program remains effective simply because it continues producing data. They periodically revisit the assumptions that program was built upon and ask whether those assumptions still hold.

That kind of curiosity doesn’t require a complete redesign.

In fact, most of the time it probably won’t lead to one.

Sometimes the review will simply reaffirm that the existing listening program remains exactly the right tool for the questions the organization is trying to answer.

That’s a valuable outcome.

Confidence is far more meaningful when it has been earned than when it has simply been inherited.

Other times, however, those conversations may reveal something else.

The business has changed.

Customers have changed.

Leadership has begun asking broader or more strategic questions than the original program was ever designed to answer.

When that happens, the goal isn’t to force the survey to produce new answers.

The goal is to recognize that new questions sometimes require new ways of listening.

To me, that’s what distinguishes maintenance from stewardship.

Maintenance keeps research programs running.

Stewardship ensures they continue serving the purpose they were created to fulfill.

It’s a subtle shift, but an important one.

Instead of asking whether a survey is well designed, stewardship asks whether it is still the right mechanism for cultivating the understanding the organization needs today.

That’s the habit I’d like to see become more common within our profession.

Not annual overhauls.

Not constant redesign.

Simply a willingness to pause, revisit our assumptions, and extend the same professional skepticism to our listening systems that we’ve long applied to the data they produce.

Because great research doesn’t begin with better analysis.

It begins with remaining curious about the questions that make analysis possible in the first place.

The Question Before the Dashboard

The next time you review a dashboard, sit in on a research debrief, or prepare to present findings to stakeholders, try asking one question before discussing the results.

What confidence do we have in the system that produced this evidence?

Not because you expect the answer to be unsettling.

Because it’s a question worth asking.

Customer researchers are exceptionally good at interrogating data. We challenge findings, investigate anomalies, test reliability, and debate interpretation. Those habits have made our profession stronger, and they should remain at the center of good research.

But perhaps our professional skepticism shouldn’t begin once the data have been collected.

Perhaps it should begin earlier.

Long before a respondent answers the first question, a series of decisions has already been made. Someone decided what was worth asking. Someone decided how customers would be invited to describe their experiences. Someone determined what would—and wouldn’t—become visible once the results appeared in a dashboard.

Those decisions deserve scrutiny, too.

Not because they’re inherently flawed.

Because they’re consequential.

Every survey, interview guide, Voice of the Customer program, and customer listening initiative is built around assumptions. Assumptions about the business, about customers, and about the questions that matter most. At the moment those programs are created, those assumptions are often thoughtful and well founded.

But assumptions have a shelf life.

Organizations evolve.

Customers evolve.

Markets evolve.

The questions we need answered evolve.

Our listening systems should evolve with them.

That’s why I believe the real opportunity isn’t simply to build better surveys.

It’s to become more intentional about periodically questioning the systems we’ve come to trust.

Not as an exercise in criticism.

As an exercise in stewardship.

Because confidence is strongest when it has been earned—not simply inherited.

If this article has a single takeaway, I hope it isn’t that researchers should become more skeptical of their data.

I hope it’s that we become more curious about the questions that produce it.

Because before we trust the dashboard…

Perhaps the most important question isn’t what the data are telling us.

It’s whether we’re still asking the questions today’s decisions require.