Continuous Improvement and a Holistic Approach to Data Quality

Respondent-level fraud has become a massive issue in survey research, and the research here confirms that every part of the research ecosystem has serious concerns about it. Bad actors are actually building literal businesses based on fake responses for incentives through the use of bots, survey farms, and other illicit practices. Fraudulent data leads to incorrect insights, which leads to the loss of time, competitiveness, revenue, and reputation. Our industry has a vital need to detect and eliminate these respondents from the data, ideally before they have even entered the survey.

Not raised as clearly in this report is that respondent disengagement in surveys is an even bigger issue. Actual human respondents who are not intentionally trying to game the survey may be so poorly engaged that it reaches the point that their data should not be used. Respondents might be distracted, multitasking, or sick but sometimes the survey experience itself drives poor engagement. Badly written surveys, complicated design, a failure to optimize for mobile devices with their small screens, and overly long instruments are common causes. The challenge is determining where the line in the sand on engagement should be drawn and only removing respondents who are not providing sufficiently faithful responses.

To combat these concerns, it is vital to take a holistic approach to respondent data quality. For example, Dynata’s solution includes using a machine learning tool of our own design to review all behavioral data available about our panelists - from signing up to the panel to taking surveys to receiving awards. The tool, QualityScore™ uses over 175 data points which are fed into our ML in real-time based on in-survey behaviors and passive data to provide a score that determines whether the survey being produced is valid for use. This includes the more typical checks like speeding, strange open-ends, and straight-lining, but also detects more nuanced behaviors such as slowing down and speeding up, unusual keyboard behaviors, illogical within survey behavior, the presence of cutting/pasting of text, and many others. 

Fraudsters don’t sit still and neither can our industry. That is why it is so important to have dynamic learning systems that continuously improve. By feeding new data into the models, we can refine their ability to identify high-quality respondents and adapt to evolving threats as fraudsters also evolve.