Why the Sampling Ecosystem Sets Up Honest Participants for Failure

The challenges with data quality in market research are not new, and it is clear that they won't simply disappear without proactive mitigation. With this in mind, the Global Data Quality partnership that coordinates data quality efforts across major international market research associations signals a serious commitment to finally addressing the crisis.

When it comes to improving data quality, the participant is key: we can't conduct good research without a high-quality sample. While there are small steps researchers can take to ensure we’re working with good-quality data, we remain at the mercy of the larger ecosystem when it comes to sampling and the intricate process that takes place behind the scenes.

Unfortunately, this system is designed to attract exactly the type of participant we wish to avoid: the professional respondents and bad actors who have turned survey participation into a career. Even if the system succeeds in recruiting a greater number of ordinary individuals into panels, they are on a path toward failure.

Rewarding quantity over quality

To examine the problem more closely, let’s take a look at a potential root cause: Earning-Per-Click (EPC).  

EPC measures how much sample suppliers in a marketplace, such as Lucid, or Cint, can expect to earn per survey. Naturally, suppliers want to maximize profits so they direct traffic to surveys with higher EPC which are easier to monetize, like surveys with a higher incidence rate, shorter interview length (LOI), and lower drop-off rate. Surveys that don't meet these criteria struggle as they are likely to be pushed down the suppliers’ priority list.  While increasing the cost-per-interview (CPI) can help, it's rarely enough to make up for poor survey metrics.

Likewise, individuals who qualify more often, complete surveys more efficiently, and promptly proceed to the next one, earn more in the long term. Coupled with abysmal incentives on a per-survey basis, this system has created the perfect environment for professional participants and bad actors to flourish. It has been easy for them to learn that quantity is rewarded over quality. This survey behavior is the polar opposite of how ordinary, infrequent survey-takers approach surveys.

Ordinary survey-takers are set up for failure

While individuals with questionable intent have figured out how to profit from surveys, people who only sporadically participate in surveys are at a real disadvantage. The way ordinary people take surveys goes against the EPC mantra, which makes surveys harder to monetize for sample suppliers.

How do ordinary people take surveys? Compared to online panelists, infrequent survey-takers haven’t been conditioned to bend the truth to qualify for surveys. They’re less likely to say “yes” to satisfier questions, or to click on a large number of attributes to maximize their chance of being selected for the survey. This results in a lower incidence rate, which is not attractive to sample suppliers.

They also read more carefully and do not benefit from familiarity with common survey questions, which would enable them to skim through a survey while still grasping the essence of the questions. In fact, it takes them approximately 20% longer to complete a survey, relative to an online panelist. Despite the fact that a longer LOI could imply that participants are more thoughtful and attentive, it hurts the EPC and ultimately impacts traffic to the survey.

Finally, ordinary people have less patience than online panelists. They are, indeed, twice as likely to drop off. One could argue that they drop off before the quality of their answers suffers, which is a good thing. However, high drop-off rates negatively impact the EPC and, consequently, the traffic sent to your survey.

Reimagining survey monetization to protect the good participants

Much of the discussion about improving data quality revolves around the participant experience and incentives. Though critical, the participant experience doesn't exist in a vacuum, so to improve data quality, the sampling ecosystem needs to start rewarding quality.

Offering higher incentives may help us attract higher quality participants, but it also runs the risk of attracting more fraudulent individuals, potentially resulting in increased costs without improving data quality. Without any change to our approach, we will continue to reward the same dishonest participants, all the while failing to retain the ordinary individuals we work so hard to recruit.

To see sustainable improvements in data quality, we need to design a system that supports the type of survey behavior we want to promote and reward: More thoughtful and honest responses, fewer serial survey-takers. 

Aligning survey metrics with human behavior 

The current emphasis on EPC metrics in survey research prioritizes quantity over quality, leading to issues with data integrity. Honest, ordinary survey participants, who provide more thoughtful and genuine responses, face disadvantages in this system.

This approach goes against the overall goal of improving data quality. If we aim to attract and retain better-quality participants, we need a fundamental shift in the sampling ecosystem, focusing on rewarding quality survey behavior that aligns more closely with how ordinary people take surveys.   

While such a radical change may seem daunting, it is crucial to developing a sustainable and thriving industry.