Are We Cleaning Too Much? The Missing Signals in Post-Survey Review

Every researcher faces difficult post-survey cleaning decisions. Is a respondent fraudulent, or simply disengaged? Is a poor-quality open-end evidence of fraud, or just a respondent having a bad day? 

We make the best decisions we can with the data in front of us. But by the time we reach post-survey review, the real-time technical and behavioral signals that could help us understand a respondent’s quality are often fragmented across different systems, leaving researchers to make judgment calls without the complete picture. 

What are Those Missing Signals? 

They come from real-time fraud detection systems and fall into three categories: technical, behavioral, and historical signals. They capture everything from device manipulation and duplicate respondents to suspicious in-survey behavior and a respondent’s track record across the research ecosystem. When strong enough, these signals block a respondent upfront. But often, they raise suspicion without meeting the threshold for removal. In those cases, the respondent enters the survey—but not as a blank slate. They bring all of that context with them.

That information is gold. It tells us so much about a respondent before we even look at their survey answers.

Yet too often, researchers don't revisit these signals during post-survey review. Instead, we make judgment calls based on experience and intuition—which are important—but without the complete picture. And the less context we have, the harder it is to make the right call.

Bringing the Signals Back 

The next step is to bring these signals back into the post-survey review process. This is particularly valuable for open-end evaluation, where assessing quality can be subjective and time-consuming. Bringing these signals together, however, adds complexity to an already cumbersome manual process. Tools that help scale this approach and enable researchers to move at the pace of research will be critical to making it practical.

 

To explore this approach, we developed an internal AI-powered tool built as a lightweight application powered by Claude AI. Our prototype evaluates open-ended responses, assigns a single quality score (good, questionable, or bad) along with a brief explanation, and incorporates additional quality signals to recommend whether a respondent should be kept or removed using a predefined decision framework.  For example, a seemingly strong open-end response may still be removed if the respondent is detected as using a virtual machine, while a questionable open-end response may be retained if the respondent has a strong historical trust score. The goal is to minimize false positives by considering the full context and giving respondents the benefit of the doubt when the evidence does not clearly indicate poor quality or fraud.

The Payoff Isn't Just Automation 

While automation makes the process faster and more scalable, an integrated data quality checker does more than save time. It improves how researchers evaluate, prioritize, and make cleaning decisions.

	Before	After
Review	Manual open-end review	AI-assisted review
Evidence	Technical, behavioral, and analytical signals stored in separate systems	Unified respondent assessment
Consistency	Can vary by researcher	Standardized, transparent evaluation across respondents
Efficiency	Time-consuming,  respondent-by-respondent review	Faster review, focused on respondents requiring attention
Decision Making	Researcher judgment call	Evidence-based recommendation with full context

Why This Matters

Over-cleaning has real consequences: it shrinks the pool of good participants, introduces bias, stretches fieldwork timelines, and can delay results. The opportunity is to move from isolated data quality checks to a more holistic review process that helps researchers make smarter, more balanced decisions about data quality. By combining technical, behavioral, historical, and now analytical signals, we can shift data cleaning from educated guesswork to evidence-based decision-making, giving researchers greater confidence in the quality of the data—and ultimately, the insights we deliver.