When Bad Online Sample Looks Good Enough

The answer looked human. The typing trail didn’t.

In a previous market research role, I spent close to two years cleaning and validating survey data. Most bad responses were easy to recognize: irrelevant open ends, obvious contradictions, straight-lining, suspicious IPs, or people racing through a survey for the incentive.

The harder cases were the ones that looked normal.

On one study, we had recently added a check that recorded whether respondents actually typed into an open-ended question. Pasting was disabled. Some responses were polished, on-topic and grammatically clean, yet showed zero keystrokes.

None of that proved fraud on its own. But one supplier had an unusual concentration of these responses on a single study, an expensive one that was already hard to fill.

Somewhere around there, the usual distinction between “good” and “bad” data started to feel less useful to me. The real problem lived in the grey zone in between.

Fraud Detection Is Not the Same as Fraud Resolution

Research teams already have a long list of tools for finding questionable respondents: IP and geolocation checks, reCAPTCHA scores, speed checks, straight-lining flags, logic tests, crosstabs, open-end review and image validation.

Those checks matter. I used many of them myself. But none of them works as a universal truth machine.

A low reCAPTCHA score can be suspicious without being definitive. An IP address can look unusual for reasons that have nothing to do with fraud. A short open-ended answer might just mean someone was lazy, not lying. And a respondent can fail one consistency check while still being a real person telling the truth.

Data cleaning turns into a judgment call very quickly. Detecting suspicious behavior was never really the hard part. The harder part is deciding how much evidence is enough to remove a respondent, and that’s where business reality comes in. 

“Passed the Minimum Checks” Can Become a Practical Standard 

Hard-to-reach audiences are expensive. Replacements take time. Suppliers may need to justify or replace rejected completes. Removing a large share of a sample can mean extending fieldwork, missing quota or absorbing additional cost.

I saw projects where the evidence was strong enough that large numbers of respondents were removed. In one case, respondents who claimed to own a product were asked to upload a photo. Some uploads appeared AI-generated, some were generic internet images, and some were duplicates. Compared with anything in the written survey answers, those photos were a far clearer signal of a problem.

I also saw the opposite situation: responses that looked questionable to me but still passed enough checks to remain in the data.

The pressure was rarely dramatic. Nobody needed to say, “Ignore this.” More often, the logic was that the response was not clearly gibberish, not clearly impossible, and not failing enough indicators to justify removal.

In other words, it was “good enough.” The phrase bothered me more the longer I sat with it.

I want to be fair here: most people I worked with cared about quality. One colleague removed a large amount of poor-quality data and accepted the cost rather than push weak responses through. It’s worth saying, because this was never really a story about a handful of bad actors inside research companies.

Quality has a cost. Every team eventually has to decide when that cost is worth paying.

Incentives Shape Respondent Behavior 

Some survey fraud is crude. Some is adaptive.

Brand-awareness questions are a good example. In tracking studies, respondents may be screened out if they don’t know enough relevant brands. Over time, a paid panelist can learn that claiming awareness of more brands increases the chance of qualifying.

This doesn’t mean every highly aware respondent is faking it. It means researchers have to think harder about how their own incentive structures shape the answers they get back.

The same applies to speed checks. If a respondent, script or automated process can slow down enough to avoid being flagged, completion time stops being a sufficient quality test. Each new guardrail changes behaviour, and the next generation of bad responses adapts around it.

The cleanest-looking data, ironically, can be exactly the data that deserves the most scrutiny.

AI is Now on Both Sides of the Problem 

AI complicates this further: it can help produce convincing open-ended answers and images that look plausible at first glance. At the same time, research teams are increasingly using AI-assisted tools to clean, code, summarize and validate survey data.

I helped build an automated cleaning tool myself, and the technology was genuinely useful. Which is exactly why I think the industry needs to be careful about leaning on it too hard.

Automation can make good processes faster, but it can also make weak inputs easier to process at scale. If the underlying sample is already questionable, faster cleaning doesn’t solve the original judgment problem. It just moves it downstream.

AI already belongs in research. There’s not much left to debate there. What’s still unresolved is where accountable human review needs to sit.

What Research Buyers Should Ask 

Most buyers never see the individual responses that analysts argue over. They see the final chart, the topline, the dashboard, or the “X in 10 consumers” claim.

I wasn’t involved in client communications, so I can’t say what every buyer was or wasn’t told. But I do think buyers should ask more specific questions about how their sample was validated.

At minimum:

• Where did the sample come from, and were multiple suppliers used? 
• What proportion of responses was removed, and for what reasons? 
• Which fraud and quality checks were applied? 
• Were open-ended answers reviewed manually? 
• Were uploaded images or other ownership claims validated? 
• Were any segments unusually difficult or expensive to fill? 
• Would the main findings materially change if borderline responses were excluded? 

Those questions won’t eliminate fraud. What they can do is put the quality decisions behind the final number back in view.

The biggest risk in online research usually isn’t the obviously fake response. It’s the one that passes enough checks, looks clean enough, and moves forward because nothing better is available to replace it.

It’s the part of the process research buyers need to learn to see.