[GRIT COMMENTARY] Sample Quality: Did You Know?

2016 Q3-Q4 GRIT Report commentary by Melanie Courtright, Executive Vice President, Global Client Services, Research Now, sharing her thoughts on the current state of the market research industry.

In the last twelve months, we as an industry have talked a lot about mobile and its impact on survey data and participation. We’ve also discussed #mrx innovation and changes in the macro landscape with integrated data, passive and behavioral measurement, privacy changes, and automation. We’ve even discussed how the traditional agency-buyer relationships and value propositions are being challenged. 

During that time, there have been some quiet, but important, changes taking place in the sampling and sample quality space that you may not be aware of, or fully appreciate.  So how about a quick “Did You Know” exercise?

1. Did you know that there are only a couple of true double opt in panels left that are investing in building a predictable set of known people who are deeply profiled and validated? Many firms today are taking a “traffic generation” approach to sampling, and prescreening heavily to support survey work. They are also going to small ponds and fishing heavily from them. This makes a difference in the incidence rates you see, and in the inherent bias present beginning at even the first question in your study. 

2. Did you know that every company now uses yield management technology, or routers, to direct people to surveys, and that each of these routers is different? Parallel versus serial routers and priority-based versus random are two things you should be asking your partners about. Some excellent research exists that covers how to use a router without introducing bias, and the most important aspect is randomization of assignment and constant testing, along with ensuring sufficient depth of participants included and breadth of research topics to complete. 

3. Did you know that there are many ways to sample a study, and if you don’t tell your partner which you prefer, they will generally pick the most efficient one? You can balance on invitations, balance on survey starts, and balance on survey completes. The last two are the most effective, and the choice depends on your study objectives. Once you decide on that, you also need to decide on how much, or how little, targeting to permit. 

4. Did you know that targeting and prescreening are NOT the same, but they both change the incidence you will see in your survey? Targeting is based on questions or behaviors from a previous interaction, and pre-screening is based on real time questions. Targeting has less bias than prescreening, because there it is less likely to affect their current cognitive thought process. 

Just one more, lest you experience a piece from me that doesn’t continue to sound the mobile alarm. 

5.  Did you know that we haven’t actually made much progress on mobile design elements as an industry? Research Now has been assessing surveys across ten mobile factors and coding them into four groups, which have not changed much after YEARS of discussions, all while the participants have long since evolved.  As an industry, we get a failing grade on addressing this issue.





Mobile Incompatible




Mobile Possible




Mobile Friendly




Mobile Optimized




Our jobs are busy and our industry is changing. That can make it hard to keep up with everything going on around us. But in market research, everything matters. Every change yields ripples that affect the data. And since quality is best measured by how well the final data tells us the truth we need to make decisions, we need to know it all. Make sure you partner with a company that has great research as its guiding principle, and gives you the information you need to make good decisions about research. 

This content was originally published by Research Now . Visit their website at www.researchnow.com.

Company profile

Research Now SSI

Research Now SSI

Plano, Texas, United States of America
(214) 365 5000
About Research Now SSI:
Research Now SSI is the leading global provider of first-party consumer and professional data based on extensive, proprietary market research panels.

other content shared by Research Now SSI