Respondent Engagement: An Emerging Data Quality Challenge

Respondent Engagement: An Emerging Data Quality Challenge

Respondent Engagement: An Emerging Data Quality Challenge

In market research, we’re trained to look for inconsistencies between what people say and what they actually do. It’s a familiar tension.

Consumers tell us they always choose based on quality…until price promotions drive their behavior. They describe thoughtful decision-making…until we observe how quickly they move through a real purchase. Researchers everywhere are nodding their heads, because this is very familiar.

We’ve long understood that gap isn’t deception. It’s human nature.

People don’t always have direct access to the “why” behind their behavior. They rationalize after the fact, using System 2 thinking to explain decisions that were largely driven by faster, more intuitive processes.

That’s a known, and manageable, limitation of research. But today, there’s a different issue emerging. And it’s more subtle.

From “Say vs. Do” to “Respond vs. Engage”

Traditionally, the challenge has been:

People don’t always do what they say.

Now, we’re increasingly seeing a parallel issue:

People don’t always think about what they say.

In other words, we have an emerging issue with generation further complicating our existing challenge of interpretation.

Many respondents aren’t trying to mislead. They’re not attempting to game the system or inject bad data. They’re simply tired, distracted, rushing to complete a task, or focused on earning an incentive as efficiently as possible.

And in that state, something important changes.

When Effort Drops, So Does Meaning

Survey responses assume a certain level of cognitive engagement. We expect that respondents read questions fully. Consider their answers. React with some level of reflection or emotional processing…

But in reality, engagement varies dramatically and often declines over the course of a survey.

When that happens, behaviors shift:

  • Questions are skimmed rather than read
  • Answers are selected quickly, not thoughtfully
  • Grid responses become patterned or automatic
  • Open-ends become shorter, more generic, or mechanically constructed

Individually, these moments seem minor. But collectively, they create something much bigger:

Data that is directionally plausible but behaviorally hollow.

The Danger of “Technically Valid” Data

From a systems perspective, this data often looks fine. Respondents pass quality checks, complete in a timely manner, and provide coherent, usable answers. There are no obvious errors or red flags. Which makes this type of issue harder to detect and easier to overlook. But the impact is real.

Because when engagement drops, two things happen simultaneously:

  1. Signal weakens — True attitudes and distinctions become less clear
  2. Noise increases — Random or habitual responses fill the gaps

Over time, this leads to flatter differences between concepts or ideas, artificial consistency across respondents, and inflated or muted metrics that don’t reflect real-world variability.

The data isn’t “wrong,” but it’s no longer fully representative of real human thinking. I’ve shared lots of thoughts about System 1 and System 2 Thinking… what we’re observing here is what I call “System 0” thinking

Why This Matters More Than Ever

As an industry, we’ve made surveys more efficient. Faster to field. Easier to complete. More scalable. But those gains come with trade-offs.

Tighter timelines, longer surveys, and greater respondent demand all put pressure on engagement. At the same time, incentives remain transactional. Participants are rarely motivated to contribute—they’re motivated to complete.

Which means we’re often designing research environments where speed is rewarded more than thoughtfulness. And when that happens, we shouldn’t be surprised by the behaviors we see

Data Quality Isn’t Just a Fraud Problem

When we talk about data quality today, the conversation often centers on fraud. And while that’s important, it can obscure a quieter, more pervasive issue: Poor-quality data from perfectly legitimate respondents.

These are real people who meet targeting criteria, but their level of engagement doesn’t match the assumptions built into the research design. And that misalignment creates risk.

Because the conclusions we draw depend not just on who responded—but on how they responded.

Designing for Real Engagement

This is where research moves beyond detection—and into design. If engagement is variable, then data quality depends on how well we:

  • Match survey length to respondent tolerance
  • Structure questions to sustain attention
  • Reduce cognitive overload across the experience
  • Create moments that re-engage rather than fatigue
  • Balance rigor with respondent reality

In other words…
Better data comes from designing for how people actually behave—not how we hope they behave.

    Survey Experience Design

    A More Honest View of Data Quality

    Not all data issues are the result of bad actors. Many are the result of understandable human behavior in environments that don’t fully account for it. That’s what makes this challenge more complex—and more important.

    Unlike traditional “say vs. do” gaps, this isn’t something we can adjust for in interpretation alone. It’s something we have to address upstream:

    • How we design studies.
    • How we assess quality.
    • How we define what “clean” data actually means

       

    Final Thought

    For years, we’ve accepted that people don’t always say what they do.

    Now we have to accept something equally important:

    People don’t always answer in a way that reflects what they truly think—especially when we haven’t earned their full attention.

    And if we ignore that reality, we risk building insights on data that looks solid on the surface…but isn’t fully grounded underneath.

    JUSTIN SUTTON

    CO-FOUNDER
    CATAPULT INSIGHTS

    Why System 1 Insights Are Becoming the New Competitive Advantage in Market Research

    Why System 1 Insights Are Becoming the New Competitive Advantage in Market Research

    Why System 1 Insights Are Becoming the New Competitive Advantage in Market Research

    For years, market research has been built around a simple assumption:
    If you ask people the right questions, they’ll tell you why they do what they do.

    That assumption is becoming increasingly fragile.

    Today’s consumers are making thousands of decisions a day, most of them quickly, emotionally, and with limited attention. They’re not slowing down to carefully weigh trade offs. They’re reacting, responding, defaulting, and moving on.

    And the brands that understand how those decisions really happen are gaining a clear competitive edge.

    The Reality: Most Decisions Aren’t Rational – They’re Intuitive

    Behavioral science describes two modes of thinking:

    • System 1: Fast, automatic, emotional, intuitive 
    • System 2: Slow, reflexive, analytical, deliberate

    While both matter, the uncomfortable truth for researchers is this: Most real world decisions are made in System 1—not System 2.

    Whether someone is choosing a morning coffee, noticing food in a display case, or scanning a store environment, their first judgment happens almost instantly—before logic, explanation, or justification enters the picture.

    By the time we ask people why they chose something, System 2 is already working backwards to create a story.

    Traditional Research Model Limitations

    Much of traditional research is optimized for System 2:

    • Surveys that ask people to rank, explain, or evaluate
    • Focus groups that rely on conscious reflection
    • Concept tests that assume deliberate comparison

    These tools aren’t wrong—but they’re incomplete. They often capture:

    • What people think they do
    • What sounds reasonable
    • What feels socially acceptable

    What they miss are the gut reactions, subconscious emotional cues, environmental triggers, and friction that people can’t easily articulate. And as markets become more competitive and timelines shorten, this gap matters more than ever. 

    Why System 1 Insights Are Becoming a Competitive Advantage

    Organizations that intentionally design research to access System 1 are seeing tangible benefits.

    1. Better prediction of actual behavior
      System 1 insights surface what people really notice, instinctively trust, or instantly reject. That makes them far more predictive of in market behavior than post rationalized explanations.
       
    2. Clearer emotional differentiation
      Brands rarely win because they say the most. They win because they feel right. System 1 research uncovers the emotional signals that create affinity, confidence, and ease, often before respondents can name them.
       
    3. Faster, more confident decision making 
      When teams understand intuitive reactions early, they spend less time debating subjective opinions internally and more time aligning around evidence that reflects how consumers experience reality.
       
    4. Reduced risk in innovation
      Many product, design, and messaging failures don’t fail because they’re illogical—they fail because something quietly “doesn’t feel right.” System 1 insight helps teams spot and correct that friction before launch.

    What System 1 Research Looks Like in Practice

    Accessing System 1 isn’t about asking better “why” questions. It’s about changing how, when, and where insight is captured.

    It often includes:

    • In context observation
    • Fast reaction tasks
    • Naturalistic exposure
    • Environmental and behavioral cues
    • Real world testing vs. artificial evaluation

    The goal is to start with reality as consumers experience it, not as we ask them to explain it.

    Why System 1 Alone Isn’t Enough (And Why That Matters)

    System 1 is powerful—but it isn’t sufficient by itself.

    Intuition tells us what is happening.
    It doesn’t always explain why or how much it matters.

    That’s where System 2 comes in.

    The strongest insight organizations don’t choose between fast thinking and slow thinking—they sequence them intentionally. They let System 1 reveal truth, then use System 2 to validate, prioritize, and scale decisions with confidence.

    And that’s where modern market research is headed. In today’s markets, the competitive advantage doesn’t belong to the brand with the most data.

    It belongs to the one that understands how people actually decide.

    JUSTIN SUTTON

    CO-FOUNDER
    CATAPULT INSIGHTS

    Tales From The Road: Inside the Aisles of The Home Depot

    Tales From The Road: Inside the Aisles of The Home Depot

    Tales From The Road: Inside the Aisles of The Home Depot

    There’s a moment in every piece of fieldwork when the clipboard disappears, the guide fades to the background, and the person in front of you steps fully into view. During a recent study at The Home Depot, that moment happened again and again. 

    We were there to understand something that seems straightforward on the surface: when shopper journeys reach the store’s aisle, what elements impact product selection?

    But as with most human questions, the truth lives in the shadows. It’s in the pauses, habits, and improvised rituals that real shopper behavior is revealed. Our mission: observe those rituals up close and in person at a full-scale mock Home Depot store in Atlanta, a place designed to let people shop like they would in a real store… only with more control over the stimuli.

    Aisles That Tell Their Own Story

    The facility we use is uncanny in the best way — the fluorescent glow, the metal shelves, the rows of products lined up for inspection. Everything looks familiar… but the stakes are different. Instead of a Saturday morning errand, participants were tackling an unspoken challenge: Watching people move through the aisle was like watching a choreography they didn’t know they’d learned

    What actually matters to me when I’m standing here, trying to choose something
    for my home?

    Watching people move through the aisle was like watching a choreography they didn’t know they’d learned:

    • The instinctive reach for a brand they trust
    • The quick skim of packaging
    • The way fingers trace a logo they recognize
    • The hesitation when facing a wall of options that all promise to be “better,” “healthier,” “approved,” “efficient”

    These in aisle micro behaviors are where product messages are supposed to shine. But the truth is more layered.

    Home Depot

    Product Messages: Loud on Paper, Quiet in the Aisles

    Our client sought to understand exactly where certain product messages do (or don’t) play a role in decision-making — trust, purchase intent, differentiation, etc. But the work revealed something more textured.

    Consumers often treated messages a bit like background music:
    They knew it was there. They noticed when something felt “official.” But unless the messaging tied directly to something personal — health, safety, or quality, especially for more educated or cautious shoppers — the message alone rarely changed their path.

    For professionals, it was different. Years in the trades had shaped a quick-scan practicality:
    “Will this work? Will it last? Will my client call me back with problems?” Some messages could help answer those questions, but only if the meaning behind them was clear, credible, and relevant to what the pro actually does in the real world

    The Human Texture of a Mock Store

    What stuck with the team wasn’t just the insights — it was the people.

    The young couple genuinely trying to learn, whispering to each other in the aisle as if making a decision right there would remodel their whole home.

    The seasoned pro who walked the aisle like he’d been here 10,000 times — because he had — guided by muscle memory until something new jarred him into focus.

    The single mom who told us she just wanted whatever would “keep things working so I don’t have to call anybody.”

    These weren’t just respondents. They were storytellers revealing the subtle emotional weight that their decisions can carry.

    Behind the Glass: The Catapult Insights Crew at Work

    From “hallway full” recruitment updates and balancing gender and ethnicity mix to late-night discussions about whether homework text was clear enough, this project had the familiar blend of logistics, improvisation, and shared purpose.

    There was the moment (there always is) when the first few participants arrived and you can feel whether the study will “sing,” and this one did. The mock aisle brought people’s natural habits to life. And our triads, a format that often reveals tension, agreement, and humor in equal measure, gave us rich layers of meaning to bring home.

    What Atlanta Taught Us

    Physical design is emotional. And business strategy is rational. Renovating thousands of stores requires you to honor both truths.

    System 1 tells you how a space really makes people feel.

    System 2 tells you how to improve it.

    If Petaluma showed how reuse systems rely on people being willing to change simple habits, this project taught us something equally human: people crave confidence when they make decisions that touch their homes, their health, or their clients’ trust.

    Messages matter in the aisle — but only when they become felt meaning, not just printed badges.

    People don’t walk into the aisle thinking about standards and codes.

    They walk in thinking about:

    • “Will this make my life better?”
    • “Can I trust this?”
    • “Is this the right choice for my family?”
    • “Is this going to work the way it should?”

    And in those moments, the magic happens when a message becomes more than noise — when it becomes a cue of reassurance at exactly the moment someone needs it.

    Another Road, Another Story

    Every Tale From The Road is different. But this one was a window into how everyday people navigate choices that feel small until they’re not.

    The mock aisle was just a stage. The real story was the people who walked through it.

    JUSTIN SUTTON

    CO-FOUNDER
    CATAPULT INSIGHTS

    Creativity Is How Great Research Earns Its Power

    Creativity Is How Great Research Earns Its Power

    Creativity Is How Great Research Earns Its Power

    In consumer insights, creativity is often misunderstood. It’s sometimes framed as a risk, a flourish, or something you add after the “real” rigorous research is done. But in practice, creativity, when applied intentionally, is one of the most practical tools we have. 

    It’s how insights work becomes more relevant, resilient, and valuable to the business.

    Creativity Drives Relevance, Not Just Novelty

    Brands don’t struggle because they lack data. They struggle because the data doesn’t quite map to the decisions they need to make.   

    Creative research design can help close that gap. When you allow yourself to rethink the setting, sequence, or combination of methodologies, you’re better equipped to meet people in real contexts, uncover emotional undercurrents, and capture nuance that standardized approaches often miss.

    The restult is research that feels truer to the way people actually think, feel, and behave. 

    Creative Execution Builds Stronger Stakeholder Buy-In

    One of the most underappreciated benefits of creativity in insights isn’t methodological—it’s organizational. 

    Creative execution can make insights easier to absorb. Immersive experiences, interactive workshops, layered storytelling, and participatory moments don’t just make findings more engaging; they make them feel like second nature. 

    Stakeholders develop confidence, alignment, and momentum when they can experience the insights as opposed to simply reading about them. That’s what turns insight into action rather than another deck that quietly fades into the background.

    Creativity Takes You Further

    Some business challenges demand more than just solid answers; they require new perspective. Creative approaches can push beyond the familiar to uncover deeper insights, unspoken tensions, and reframed opportunities that standard methods can miss.

    We’ve found that traditional approaches can usually get you 90% of the way there. But when you layer in some creative elements – whether that be in the design, interpretation/analysis, or socialization, you can get closer to 100%. You aren’t abandoning the rigor of traditional methods, you’re extending it. 

    The Payoff: Insights Work That Travels

    Beyond looking impressive, the benefit of creative research is that it lasts.

    Creative approaches are more likely to produce insights that travel across teams, influence multiple decisions, and remain relevant beyond a single moment in time. They help insights teams shift from order‑takers to strategic partners and from reporters of findings to catalysts for change.

    For brand‑side insights professionals, that’s the real win. Creativity isn’t a departure from rigor. It’s how rigor becomes meaningful, especially when the questions are hard, the context is complex, and the impact truly matters. 

    Drop us a note at hello@catapultinsights.com if you want to chat about how getting creative can take your insights to the next level.

    Jill Miller

    CO-FOUNDER
    CATAPULT INSIGHTS

    Design Decisions Need Both System 1 and System 2: Lessons From Restaurant Format Research

    Design Decisions Need Both System 1 and System 2: Lessons From Restaurant Format Research

    Design Decisions Need Both System 1 and System 2: Lessons From Restaurant Format Research

    Renovating a single restaurant is expensive. Renovating hundreds—or thousands—is a bet on the future of your brand. Once construction begins, there’s no quick pivot. That’s why the smartest organizations lean heavily on consumer insights long before materials are ordered or walls come down. 

    But here’s the tricky part: consumers react in emotions and don’t speak in blueprints. They rationalize in hindsight. They say one thing and feel another. And if you only listen to one layer of feedback, you risk making decisions based on a partial truth.

    That’s where the partnership of System 1 and System 2 insights becomes essential.

    The Challenge With Customer Feedback: Two Systems, One Decision

    When people walk into a store, restaurant, or any environment their brains immediately fire off rapid, intuitive judgments:

    • Does this place feel welcoming?
    • Do I know where to go?
    • Am I on display?
    • Does it feel cramped?

    That’s System 1. Fast, emotional, subconscious. It’s the source of gut reactions and first impressions. And when it comes to physical spaces, System 1 often drives behavior more than any verbal explanation ever will.

    Then comes System 2, the slow, reflective layer. This is where articulate their reasoning:

    • “The lighting felt too harsh.”
    • “The seating layout made it difficult to have a conversation.”
    • “It was confusing to know where the line started.”

    System 2 is what customers tell you when you ask. System 1 is what they show you through the way they behave, react, and feel.

    If you only listen to one, you’ll miss the truth.

    Why Both Systems Matter for High-Cost Decisions

    When the stakes are low, brands can tweak a few things and learn as they go. But remodeling an entire system of restaurants requires a high level of confidence. You need to understand what people say PLUS what actually shapes their behavior when no one is watching.

    This creates a unique research responsibility:

    System 1 tells you the emotional consequences.
    It reveals the anxieties, the comforts, the excitement, the subtle signals your space sends within seconds.

    System 2 tells you the rational interpretation.
    It helps you understand why the space works (or doesn’t) and gives clues to trace emotional responses back to tangible design elements.

    The magic happens when you interpret the two together:
    System 1 identifies the feeling. System 2 helps you identify the cause.

    How We Bring the Two Systems Together: A Real‑World Example

    In a recent restaurant design research project, we began with virtual ethnographies—participants shared their experiences and reactions in real time, before they could edit them or detect what the project was about.

    This gave us raw System 1 data:

    • Spontaneous comments about lighting and noise
    • Micro expressions of discomfort or delight
    • The hesitation before choosing a seat
    • The subconscious scanning for privacy, comfort, or crowding

    Then we did in restaurant interviews, where the same participants had more time to reflect. We asked them to walk through the space, narrate their thoughts, react to specific design elements, and describe their emotions in a more deliberate way.

    This gave us rich System 2 clarity:

    • Why certain seating zones felt inviting or off putting
    • Subconscious associations at the root of feelings and behaviors
    • How design influenced the perception of brand modernity
    • What design choices elevated or diminished the dining experience

    Careful sequencing is vital to capturing unbiased reactions from each System, and when we overlaid the two, patterns emerged — patterns a single method alone would never have surfaced.

    From Insights to Actionable Design Hypotheses

    Blending both systems allowed us to translate raw human reactions into structured design hypotheses. And these hypotheses weren’t guesses. They were emotionally grounded and rationally validated.

    Then came the exciting part: iteration with the design team. Because insights were so clear, we could easily identify the design deltas holding a restaurant back – and the elements that feel essential to the brand identity.

    The Payoff: Confident Decisions at Scale

    By the end of the project, the organization wasn’t relying on abstract preference data. They were using a layered, human centered understanding of:

    • Which formats to champion
    • Which to evolve
    • Which design principles should guide future builds

    And most importantly, they had the confidence to invest at scale—because decisions were built on the full truth of customer experience.

    The Bottom Line

    Physical design is emotional. And business strategy is rational. Renovating thousands of stores requires you to honor both truths.

    System 1 tells you how a space really makes people feel.

    System 2 tells you how to improve it.

    When you combine them thoughtfully, you don’t just build better spaces—you build better outcomes, for customers and the business.

    Drop us a note at hello@catapultinsights.com if you want to learn more about how you can leverage system 1 and 2 insights to elevate your design strategies.

    JUSTIN SUTTON

    CO-FOUNDER
    CATAPULT INSIGHTS

    Insights Don’t Always Come Easy: Why Creativity Matters

    Insights Don’t Always Come Easy: Why Creativity Matters

    Insights Don’t Always Come Easy: Why Creativity Matters

    Consumer insights leaders are constantly under pressure to deliver clarity, confidence, and direction. Stakeholders want answers yesterday. Budgets are tightening. Risk tolerance is shrinking. And the business questions we face grow more complex each year.

    But here’s the truth:

    There is no playbook for the hardest, most meaningful questions

    And the best insights—the ones that spark action, alignment, and transformation—rarely emerge from standard methods alone. They come from the work: the digging, the reframing, the imagination, the courage to craft approaches that don’t yet exist. 

    At Catapult Insights, we’ve learned firsthand that great insights don’t always come easy, and sometimes you need to be creative.

    Read on to learn about our point of view—and a peek behind the curtain—on how creativity elevates the insights function and drives real impact for brand teams.

    Complex Business Questions Require Creative Methodologies

    Leading organizations aren’t merely asking “Which concept wins?” or “What do people prefer?”

    They’re asking:

    • How do emotional mindsets share behavior along a nonlinear journey? 
    • How can we influence attachment in categories where consumers aren’t actively thinking?
    • How do we build a segmentation with personas that teams can, and actually want to, use?

    Often, these aren’t solvable with a single method or a linear design. That’s why innovation in research methodology matters. We’ve witnessed this first-hand how creative combinations of approaches can drive breakthroughs.

    Think:

    Creativity in research must be purposeful, not aimless. It’s about architecting a design that truly matches the complexity of the problem—which often means creating something new when the obvious options fall short. 

    When Constraints Tighten, Creativity Sharpens

    Budget restrictions. Compressed timelines. High stakes audiences. Risk averse leadership. We’ve all been there.

    Sometimes the constraints feel like they box you in, but we’ve found that sometimes they can do the opposite. They force clarity. They spark inventive thinking. 

    Reflecting back, some of our most effective insights programs have been built under constraints.

    For example: 

    • Limited budget? Leverage hybrid designs or unlock value from tools you already own.
    • Little time? Use fast thinking approaches or iterative cycles instead of long single-phase studies.
    • High visibility or risk? Build methodologies that bring stakeholders along on a learning journey rather than keeping insights development behind a curtain, so stakeholders feel both confident and connected. 
    • Internal pressure to “stand out”? Create immersive deliverables, interactive workshops, or mindset-based storytelling that travels through the organization and live beyond a single project. 

    Research doesn’t have to be boundless to be brilliant. It has to be intentional, well designed, and aligned to the decision at hand

    Creativity without Recklessness: Why Experience Matters

    Our POV is simple: You can balance creativity with risk when you have the experience to know where the guardrails are.

    Experience tells you:

    • When “scrappy” is smart — and when it’s not
    • When new thinking will open eyes and when it will just confuse stakeholders
    • How to build credibility with diverse internal audiences
    • How to amplify insights through activation, not just publication

    It’s important to be familiar with and recognize these limits when designing creative approaches to protect the validity of the insights you gather. Otherwise, you can put your insights, and your clients at risk. 

    Why This Matters More Than Ever For Insights Teams

    Brand-side insights leaders are under pressure to:

    • Demonstrate strategic value
    • Deliver influence beyond the research
    • Build organizational readiness for action
    • Connect data points into clear, resonant stories

    And all of that requires creativity.

    As my colleagues and Human Behavior experts, Justin and Debbie, recently discussed, the most powerful insights rarely come from tightly controlled environments. They come when you loosen your control and layer methodologies that capture both what people do and what it means to them.

    Creativity is the bridge between data and meaning. Between insights and activation. Between knowing and doing. 

    Creativity Isn’t a Bonus

    It’s a requirement for modern insights leadership, especially when the questions are tough, the stakes are high, or the path forward is unclear. 

    And the insights that don’t come easy? Those are the ones worth chasing.

    Drop us a note at hello@catapultinsights.com if you’re interested in getting creative to solve your biggest business challenges. 

    Jill Miller

    CO-FOUNDER
    CATAPULT INSIGHTS

    Subscribe to our Newsletter

    * indicates required

    Intuit Mailchimp