Still More Dirty Little Secrets of Online Panels

Here are two critical numbers to consider: 46% and 90%.

Grey Matter Research and Harmon Research teamed up for the new study Still More Dirty Little Secrets of Online Panels. We fielded a pretty typical online questionnaire with five of the ten largest panel providers in the industry. But we set up a variety of traps, tests, and quality control measures, and the results were pretty disturbing.

First, the proportion of respondents we tossed out of the study for having serious problems was 46%. Just to be clear, these weren’t respondents who had one mistake or a verbatim that was too short. Nearly half of our respondents either had a problem that was so egregious they obviously needed to go, or they had multiple problems – as in four or more in a ten-minute questionnaire.

Second, Harmon Research fields surveys with tens of thousands of online panel respondents each month. In their experience, at least 90% of researchers are not taking sufficient steps to ensure online panel quality. Most throw out respondents with gibberish open-ends (e.g. “kukn;lkjij;lk”) and eliminate people who are completing 15-minute questionnaires in three minutes, but in Harmon’s observation:

• 25% don’t toss out respondents with bogus verbatims beyond obvious gibberish (e.g. “It’s my family’s favorite brand” when asked how they plan to change their shopping habits due to COVID-19)
• 50% don’t identify and remove straightliners
• 75% don’t include red herring questions in their questionnaires (e.g. “Please mark the box all the way to the left”)
• 95% don’t include obvious fake brands in awareness or usage questions

 

This isn’t news

Researchers have been aware for a long time that disengaged and fraudulent respondents (including bots and click farms) are a problem in online panel surveys. But we wanted to quantify just how much of a problem this is, and exactly what impact it may have on your data. Still More Dirty Little Secrets of Online Panels answers both questions.

We’ve already stated that nearly half the panel respondents couldn’t pass our quality check. But just what type of impact are these bogus respondents having on your survey data?

We took the 880 respondents we eliminated from the study and compared them with the respondents we kept. Here are just a few examples of the differences between the two groups:

• Among our valid respondents, 15% claimed brand awareness of the charity watchdog organization Charity Navigator. Among the bogus respondents, it was 58%. That’s 287% inflation of brand awareness among our many fraudulent respondents. (Are you starting to worry about your last brand study?)
• When shown a concept statement with 203 words and asked to read it carefully before answering some questions, the average valid respondent spent 80 seconds on it. The average bogus respondent spent 11 seconds. (Uh-oh – how did that affect your last messaging, advertising, or concept test?)
• Among our valid respondents, 28% felt the U.S. should strictly limit immigration. Among the bogus respondents, it was inflated to 62% as a bunch of them straightlined “agree strongly” on a series of five statements on the topic. Again, that’s 121% inflation of what the number should be. (Think that might have some impact on political or public policy research?)
• Not only did 62% of our bogus respondents agree that “The US should have no limits on how many immigrants we accept each year,” but 70% of them also agreed that “The US should strictly limit the number of immigrants we accept each year.” Huh?
• In a brand familiarity question listing a number of financial institutions, 3% of our valid respondents told us they were very or somewhat familiar with either LiveWire Bank or Generosity Bank. Problem is, neither one of those brands exists (proving that even carefully screened, engaged respondents can make a mistake). However, among our bogus respondents, 37% claimed familiarity with one or both fake brands. Unfortunately, they also claimed levels of familiarity with Citibank that were 44% higher than for our valid respondents, with Regions that were 167% higher, and with M&T Bank that were 400% higher. (Are you really starting to worry about your last brand study? Maybe you should…)
• Huntington is a financial services company with locations in seven states. Among valid respondents, 78% of the people who claimed strong familiarity with the brand lived in one of those states. Among our bogus respondents, only 25% who claimed strong familiarity lived within the bank’s footprint. Some people from outside their footprint may be familiar with the brand because they live close to a market that has branches, they see regional advertising for the brand, they used to live in one of those states, etc. But three out of four living outside the footprint? That’s the very definition of dubious data.

 

This leads to two questions for your online panels.

First, are you comfortable with nearly half your data being utter trash? If you’ve read this far, I’ll assume the answer is no.

Second, and most importantly, what are you going to do about it?

The responsibility does not rest only with panel companies. They are under intense pressure from clients to provide data faster and cheaper every day, and that does nothing to foster quality. Tens of thousands of researchers and marketers rely on online panel research, and that’s not likely to change any time soon. The problem is that many clients are either 1) in charge of the data collection themselves and not taking the necessary steps to ensure respondent quality, or 2) handing the data collection off to a vendor and just assuming that the vendor is responsibly taking those critical steps. Probably more than nine out of ten are not.

The key is that this is not something that can be solved after the data is collected. It has to be addressed before the study reaches the field. Quality assurance must be baked into the study design. You need programming instructions that terminate people from the questionnaire when there are egregious problems (and ways to identify those egregious problems). You need questionnaire design that includes traps and measures to determine whether a respondent is valid or not. Then during and after the field, you need an intensive data review. If you’re not comprehensively handling it before, during, and after the field, you’re not really handling it at all.

Different types of studies may require different types of quality measures. Speeding problems are more obvious on a 15-minute questionnaire than a 5-minute questionnaire; screen timers don’t work well on short questions; straightlining isn’t an issue on questionnaires with no grids.

We also have found some quality measures to be consistently much better than others. For instance, we rarely use red herring questions because they’re too easily identified by bogus respondents and even bots. Our latest study showed many of the people who correctly answered the red herring questions were tossed out for other major problems.

 

What can be done?

There are various ways you can combat the problem of bogus respondents, and the best formula is for multiple methods to work together in concert. This is covered more in-depth in the full report (which is available without cost – contact [email protected]). It’s not perfect, but it’s far better than getting brand awareness figures that are inflated by 400% or getting critical feedback on a concept respondents didn’t even read. Unfortunately, there’s a really good chance you’re getting a lot of that in your panel studies.