The High Cost of Cheap Sample

By Rob Berger

“At least it will give us directional information. I mean, how wrong can it be?” I’ve heard those remarks and many like them bantered about when people rationalize using cheap sample sources.

The fact is, bad or “cheap” sample can give you information that is dead wrong. So wrong that the information it provides is directional—it’s just pointing in the wrong direction.

We’re into the third year of a study where we test the reliability and validity of a popular and well known consumer survey service. We compare the findings of that research with the same data tracked by the Pew Research Center, who are well known for their methodological rigor. The question we are tracking is about the use of social media sites and apps.

The consumer survey results and the Pew Research Center findings could hardly be more different. For starters, the consumer survey suggests social media usage is half the level that Pew and many other sources say it is at.

The consumer survey source says, for example, that 39% of online Americans are using Facebook. Pew puts that number at 79%. For other social media sites the differences are equally stark. Furthermore the consumer survey data would suggest great volatility in the use of social media—with vast surges and declines. Pew shows a slow steady growth.

Why this discrepancy? We believe a lot of it has to do with why people are completing the survey. The consumer survey source obtains respondents by working with publishers to intercept people who are seeking to access “premium content” on their sites. Potential respondents are asked to answer a few questions in order to get access to content. In a whitepaper on this topic we consider a host of potential reasons, but the one that seems to be the most important is respondent motivation.

These people are not taking the survey because they want to. They are doing it to get to their desired content. They don’t have any stake in how they answer—in fact, the question is just a nuisance. When you treat people like that, it is no surprise that the data they give you may be inaccurate. Why would they bother? After all, they have just been frustrated in their pursuit of something they want.

That’s why we take respondent engagement so seriously. Whether it be on our Market Communities—recruited to be representative of the US and Canadian populations—or our client’s specially recruited Insight Communities, we take care to ensure respondents know that their opinion is valued.

We’ve researched why people respond, and we work hard to let respondents know their opinion is important to us and that their input makes a difference. We strive to provide them with feedback on what we’re learning and we try to expose them to interesting new things.

When we invite people to join our communities we mean community in the fullest sense of the word. That’s an important part of what allows us to collect accurate and consistent information to inform our client’s decision making.

People conduct research to help them make better decisions. When the research is wrong, they make bad, even terrible, decisions. In those cases the data is hurting rather than helping them. That makes cheap sample very expensive.

To learn more, download our whitepaper The High Cost of Cheap Sample: Evaluating the Reliability and Validity of a Publisher-driven Online Sample Source.