Fixing the Ship while Sailing It: Slow Change in Survey World

Market research has a survey data quality problem. We have been grappling with this since at least 2022, and it seems things are getting worse. My first hypothesis was that we would see a decline in the percentage of buyers and suppliers using online surveys. We clearly do not see this in the data, as 95 percent of research buyers and 99 percent of full-service firms reported using the methodology, and there is little change from last year. With that said, face-to-face interviews are up among buyers. What we don’t know is whether individual buyers are commissioning fewer online surveys and moving studies to face-to-face.

Another hypothesis was that we would see an uptick in microsurveys. These surveys are easier on respondents and could help with the issue of panelist engagement via a better survey experience.  We do see this to some extent. A majority of buyers are using microsurveys at least occasionally, and only 23 percent of research buyers are not likely to use them or are not familiar with them. Among full-service firms, the use of microsurveys is up 9 percentage points from five years ago. Twenty-one percent of full-service firms says they do not currently but probably will use them in the future. I would like this trend to continue.

Another fact is that technology has enabled a blending of quant and qual in online surveys. This could be a good thing if a focus on data quality means there are fewer acceptable survey takers in the ecosystem. My hypothesis here is that we would see growing interest in the use of chatbots, which represent an AI-based tool that allows qual research at scale. The data shows mixed results. Research buyers had a drop-off in the use of chatbots from last year and a stable rate across five years. However, 36 percent think they will start using chatbots in the future. 

Transparency around data quality metrics is key, as is client education. Buyers need to know what questions to ask to be more discerning consumers. What is your panel’s average acceptance rate? If your panel says they verify panelists’ identities, what does that really mean? (You’re probably in for a surprise.) Secondly, if you think you can spot survey fraud just by staring really hard at your Excel spreadsheet, you are woefully mistaken and at risk for all of the problems that bad data can cause. 

Tech isn’t a panacea, but along with best practices, you can reduce risk significantly. It’s time for shorter or more conversational surveys and smaller samples from suppliers who check all your security boxes. It’s time to leave the router behind. By thinking small, we can do big things, and not only keep surveys afloat but help them thrive. It’s not too late.