Respondent Engagement: An Emerging Data Quality Challenge
Respondent Engagement: An Emerging Data Quality Challenge
In market research, we’re trained to look for inconsistencies between what people say and what they actually do. It’s a familiar tension.
Consumers tell us they always choose based on quality…until price promotions drive their behavior. They describe thoughtful decision-making…until we observe how quickly they move through a real purchase. Researchers everywhere are nodding their heads, because this is very familiar.
We’ve long understood that gap isn’t deception. It’s human nature.
People don’t always have direct access to the “why” behind their behavior. They rationalize after the fact, using System 2 thinking to explain decisions that were largely driven by faster, more intuitive processes.
That’s a known, and manageable, limitation of research. But today, there’s a different issue emerging. And it’s more subtle.
From “Say vs. Do” to “Respond vs. Engage”
Traditionally, the challenge has been:
People don’t always do what they say.
Now, we’re increasingly seeing a parallel issue:
People don’t always think about what they say.
In other words, we have an emerging issue with generation further complicating our existing challenge of interpretation.
Many respondents aren’t trying to mislead. They’re not attempting to game the system or inject bad data. They’re simply tired, distracted, rushing to complete a task, or focused on earning an incentive as efficiently as possible.
And in that state, something important changes.

When Effort Drops, So Does Meaning
Survey responses assume a certain level of cognitive engagement. We expect that respondents read questions fully. Consider their answers. React with some level of reflection or emotional processing…
But in reality, engagement varies dramatically and often declines over the course of a survey.
When that happens, behaviors shift:
- Questions are skimmed rather than read
- Answers are selected quickly, not thoughtfully
- Grid responses become patterned or automatic
- Open-ends become shorter, more generic, or mechanically constructed
Individually, these moments seem minor. But collectively, they create something much bigger:
Data that is directionally plausible but behaviorally hollow.

The Danger of “Technically Valid” Data
From a systems perspective, this data often looks fine. Respondents pass quality checks, complete in a timely manner, and provide coherent, usable answers. There are no obvious errors or red flags. Which makes this type of issue harder to detect and easier to overlook. But the impact is real.
Because when engagement drops, two things happen simultaneously:
- Signal weakens — True attitudes and distinctions become less clear
- Noise increases — Random or habitual responses fill the gaps
Over time, this leads to flatter differences between concepts or ideas, artificial consistency across respondents, and inflated or muted metrics that don’t reflect real-world variability.
The data isn’t “wrong,” but it’s no longer fully representative of real human thinking. I’ve shared lots of thoughts about System 1 and System 2 Thinking… what we’re observing here is what I call “System 0” thinking.
Why This Matters More Than Ever
As an industry, we’ve made surveys more efficient. Faster to field. Easier to complete. More scalable. But those gains come with trade-offs.
Tighter timelines, longer surveys, and greater respondent demand all put pressure on engagement. At the same time, incentives remain transactional. Participants are rarely motivated to contribute—they’re motivated to complete.
Which means we’re often designing research environments where speed is rewarded more than thoughtfulness. And when that happens, we shouldn’t be surprised by the behaviors we see
Data Quality Isn’t Just a Fraud Problem
When we talk about data quality today, the conversation often centers on fraud. And while that’s important, it can obscure a quieter, more pervasive issue: Poor-quality data from perfectly legitimate respondents.
These are real people who meet targeting criteria, but their level of engagement doesn’t match the assumptions built into the research design. And that misalignment creates risk.
Because the conclusions we draw depend not just on who responded—but on how they responded.
Designing for Real Engagement
This is where research moves beyond detection—and into design. If engagement is variable, then data quality depends on how well we:
- Match survey length to respondent tolerance
- Structure questions to sustain attention
- Reduce cognitive overload across the experience
- Create moments that re-engage rather than fatigue
- Balance rigor with respondent reality
In other words…
Better data comes from designing for how people actually behave—not how we hope they behave.

A More Honest View of Data Quality
Not all data issues are the result of bad actors. Many are the result of understandable human behavior in environments that don’t fully account for it. That’s what makes this challenge more complex—and more important.
Unlike traditional “say vs. do” gaps, this isn’t something we can adjust for in interpretation alone. It’s something we have to address upstream:
- How we design studies.
- How we assess quality.
- How we define what “clean” data actually means
Final Thought
For years, we’ve accepted that people don’t always say what they do.
Now we have to accept something equally important:
People don’t always answer in a way that reflects what they truly think—especially when we haven’t earned their full attention.
And if we ignore that reality, we risk building insights on data that looks solid on the surface…but isn’t fully grounded underneath.
JUSTIN SUTTON
CO-FOUNDER
CATAPULT INSIGHTS







