Why Online Surveys Need Smarter Quality Assurance Now

Discover how online surveys can maintain their credibility through coherence checks and behavioral analysis.

A recent scandal has shaken the insights industry: Two U.S. market research companies have been accused of systematic online survey fraud, resulting in millions in damages. Industry reactions have varied widely ranging from alarmist views framing it as the beginning of the end for online surveys and a pivot toward synthetic data, to overly optimistic responses downplaying the case as an isolated incident, insisting that the majority of the industry continues to uphold high quality standards.

Regardless of these differing views, this court case highlights the causes and contributing factors behind the growing issue of poor data quality in online surveys, factors that had already been examined here prior to the recent scandal.

First, professional online survey fraudsters are no longer just located in developing countries. They are also increasingly based in developed markets such as the United States.

Second, fraud has become highly decentralized, making the traditional image of centralized Chinese "click farms" outdated and insufficient to describe today’s threat landscape.

Third, prominent industry brands, membership in industry associations and certifications from reputable organizations offer no guarantee of protection for clients. While these credentials may signal a formal or perhaps former commitment to quality, they cannot ensure or enforce it.

Finally, survey fraud is often carried out by highly skilled individuals, trained by other fraudsters, to bypass even advanced security measures and quality control checks.

So What’s New?

According to the indictment, a key insight from this scandal is that it wasn’t external fraud networks, but individuals within the two implicated market research companies themselves who actively recruited and trained the survey fraudsters. Their involvement included detailed instructions on how to manipulate survey screener questions to avoid detection.

The vulnerability of online surveys is further underscored by a recent study showing that modern bots can now bypass a wide range of commonly used quality checks including open-ended questions. This is particularly alarming, as open-ended response analysis has long been considered one of the most reliable tools for detecting more sophisticated fraud.

But does this mean critics are right to claim that online surveys have lost their legitimacy because professional fraud is now undetectable? We don’t believe so. What we observe is a landscape divided into three groups of researchers and clients, with the first two, in our experience, unfortunately still representing the majority:

1. Those who apply or demand no, or only very limited, quality assurance measures.
2. Those who use or require basic checks such as IP checks, speeders, and straightliner detection.
3. And those who employ or demand advanced methods, such as quality checks of open-ended responses.

It is this third group of researchers and clients that offers a path forward because they not only recognize the risks but are also willing to invest in tools to address them. However, to succeed, they must be equipped with quality checks that are truly up to the task. Many of the methods that have become standard in recent years are no longer sufficient. That’s why we introduced the following two new approaches this year.

Coherence Checks

The Achilles’ heel of professional survey fraud, whether human or automated, is that each question is typically answered in isolation, without regard for consistency across the entire survey. Coherence checks address this vulnerability by shifting the focus from analyzing individual responses to assessing the consistency of the complete interview. This involves evaluating how well answers align from the initial screener questions to the final demographic ones.

Incoherence across responses can signal inattentiveness or fraudulent behavior. Our findings show that contradictions are often not overt; rather, doubts about plausibility tend to accumulate gradually over the course of a questionnaire. Once these doubts surpass a certain threshold, the interview should be excluded.

The key advantage of coherence checks is that subtle inconsistencies, which are missed by traditional, question-level quality control, can be detected using artificial intelligence. This process can be fully automated and applied in real time, allowing to assess the plausibility of each interview by the end of the survey. Moreover, the approach is questionnaire-agnostic: it does not depend on open-ended responses or trap questions, making it broadly applicable across studies.

However, developing a reliable coherence check presents several challenges. It requires continuous effort to evaluate and, when necessary, replace the underlying model. As new AI models frequently emerge, each must be tested against established benchmarks to determine whether it offers improved performance. This process relies on up-to-date, manually validated reference data - so-called “ground truths” - which must span a diverse range of research topics to ensure robustness. Additionally, the selected model must deliver results with sufficient speed and at a cost that is economically viable for real-world application.

Combined Content - and Behavior-Based Analysis of Open-Ended Responses: 

In recent years, responses to open-ended questions have proven to be one of the most reliable indicators for distinguishing between high- and low-quality interviews. Even when chatbots were used to generate answers, they were often detectable due to overly polished phrasing, excessive length, or the linguistic patterns typical of AI-generated text.

Today, however, we are seeing a shift: Fraudsters are now using more sophisticated prompts designed to mimic the informal, imperfect style of answers to open ends from real respondents. These AI-generated responses may deliberately include spelling errors, inconsistent punctuation, irregular capitalization, and colloquial language making it particularly difficult to reliably detect fraud in shorter responses.

This challenge is not unique to us. OpenAI, for example, discontinued its AI text classifier due to its limited accuracy in distinguishing human-written from AI-generated content. To minimize false positives in our own detection efforts, we now apply a strict threshold: A 99.9% confidence level and a minimum response length of 100 characters before flagging a response as AI-generated. While this approach has improved accuracy, it also limits the reliability of detecting chatbot-generated fraud in short answers, increasing the risk of false negatives.

To close this gap, we combine content analysis with behavioral tracking. This method evaluates the authenticity of a respondent’s input behavior when answering open-ended questions. One of the key challenges was developing an approach that works reliably across all input devices whether someone is swiping on a mobile phone or typing on a desktop keyboard. What has proven effective in this context is analyzing variations in input timing patterns, as neither fast nor slow typing alone is a reliable indicator of inauthentic behavior.

Another important challenge in this approach was determining what constitutes fraud. For instance, if a respondent pastes a text but significantly modifies or expands it manually, should that be flagged as fraudulent? To handle such nuances, we developed a scoring system ranging from 0 to 100, where higher scores indicate authentic human input behavior. A score of 0 typically signals artificial, non-human input.

Conclusion

We are far from powerless in the fight against professional survey fraud. Innovative and effective solutions continue to emerge among them coherence checks and the combined content- and behavior-based analysis of open-ended responses, as described above. In our experience, these two methods together form a highly robust dual barrier, making it exceptionally difficult for fraudsters to go undetected.