White Paper:
Non-Response Bias In Survey Sampling
by TRC
Market research accounts for many scenarios to ensure high quality of data. One of the most overlooked problems is non-response bias. TRC describes ways to reduce its effects through survey design and data adjustment in this white paper.
Introduction
As market researchers, we use data gathered from surveys to make informed decisions and give recommendations to clients on ways to improve their sales and standing in the marketplace. Whether these recommendations are to invest resources to increase satisfaction with customer support or to introduce a new product to the market, we need to feel confident that the results from many crosstabs and multivariate analyses are speaking the truth. The confidence we have in our results stems from the quality of our data. In today’s market research industry, we do a lot to help ensure our data meet certain standards: screener questions target the specific audience we want, online panels take many steps to ensure their samples contain the target we need, we weight respondents to match specific population demographics, etc. However, one of the most over-looked problems is that of non-response bias.
Non-Response Bias
In data collection, there are two types of non-response: item and unit non-response. Item non-response occurs when certain questions in a survey are not answered by a respondent. Unit non-response takes place when a randomly sampled individual cannot be contacted or refuses to participate in a survey. The bias occurs when answers to questions differ among the observed and non-respondent items or units. A general formula for measuring bias is:
Bias = P ( O – N )
where:
P is the proportion of non-respondents from the targeted sample (i.e. non-response rate)
O is the answer based on observed responses
N is the answer based on non-respondents only
Bias is calculated as the product of two components: non-response rate and the difference between the observed and non-respondent answers. Increasing either of the two components will lead to an increase in bias. Since it’s quite difficult and often impractical to design a survey to impact the difference between the observed and non-respondent answers, researchers will often focus their attention on reducing the non-response rate in order to reduce bias.
The first (and possibly most important) step in reducing non-response bias is to create a properly designed survey. Whether it be online or by phone, the design of the survey can have a large impact on whether a respondent chooses to partake in the survey, and to what extent they complete the survey. Having a personable yet professional introduction, interesting survey content, short survey length, clear and concise wording, practical and appealing incentives, placing multiple follow-up calls or email reminders on non-respondents, and being mindful of the time, day, or season that the survey is fielded all can impact the non-response rate. Even after designing a great survey, both item and unit non-response are likely to exist.
There are many ways to deal with item non-response, with case deletion and mean replacement being the most popular. Due to researchers’ familiarity with item non-response, the remainder of this paper focuses on the less talked about subject of unit non-response bias. Our next steps are to measure and then adjust for any bias in the data.
Identifying Unit Non-Response Bias
Comparing Initial and Late Respondents
As mentioned earlier, following up on non-respondents is an excellent way to reduce the non-response rate, but it also gives us some additional information. Because late-respondents, or those that respond after several attempts, are theorized to have some similarities with non-respondents, one approach would be to compare scores on key metrics from both the initial respondents and the late respondents. Any differences would be considered an estimate of non-response bias. Of course, we need to keep in mind that these similarities with the non-respondents and the differences from the initial responders do not always pan out.
Comparing Survey Results to Known Population Parameters
Instead of comparing initial and late responders, we can compare the demographic profiles (e.g. age, gender, race, and income) of our respondents to some reliable external source. One option for this would be U.S. Census data for our intended target population. If the comparison results in clear differences, we conclude that these differences indicate that we may have non-response bias in our data. While this method allows for the comparison of data gathered on respondents to population totals, it lacks the ability to measure differences on key variables of interest. In order to evaluate key variables, a popular technique is to weight the survey respondents based on census totals and compare the weighted and un-weighted results. If they differ, we may conclude non-response bias is present, assuming that the demographic or database variables have an association with the response rate.
Using Known Database Variables to Identify Bias
Suppose that along with contact information for the entire sampled group, we have some additional demographic and database (e.g. tenure with company, purchase quantity) variables. One option utilizing this extra information is to examine non-response rates over different sub-groups of the population using the demographic variables. Differences indicate the possible presence of bias in the data. Another option is to use the database variables and compare statistics among the responders and non-responders where any differences give evidence of non-response bias. Both of these methods fail to focus on key survey variables and assume that the demographic or database variables are correlated with non-response bias.
Adjusting for Unit Non-Response Bias
The above methods give the researcher a sense of whether bias exists, but do not provide a way to deal with it. The following procedures both measure and adjust for non-response bias.
Weighting-Class Adjustments
Suppose again that additional demographic or database variables are available for all members of the targeted sample group. These variables are used to create sub-groups containing respondents and non-respondents. Weights are then calculated based on the proportions in each sub-group and applied to the respondents to reflect the total sample population. Comparisons on key variables are then observed between the unadjusted and weighting-class adjusted respondents. If clear differences are detected, then non-response bias is assumed to be at fault and the weighting-class adjustments are used as they provide results with less bias. Poststratification is another technique similar to weighting-class adjustment, except that the procedure uses population counts instead of the total sample counts. The downside to these techniques is that they assume that the differences between respondents and non-respondents are captured in the subgroups, and that there is no rule of thumb for comparing adjustments to determine which to use.
Other Adjustment Techniques
A couple of other techniques exist to adjust for non-response bias such as propensity models, which require some information (e.g. demographics) be known for the entire sampled group, or calibration methods, which make the use of auxiliary population data like from a census. Both of these methods are extensions to techniques previously discussed, and the interested reader is encouraged to research more on these topics in detail.
Conclusion
The purpose of this paper was to make the reader aware of non-response bias, describe ways to reduce its effects in the design stage before fielding a survey, and explain the ways to measure and adjust datasets for non-response bias. Because post-survey adjustments are merely estimated “fixes” to the problem, the most effective way to reduce non-response bias is to reduce non-response rates through properly designed studies. Then the adjustments on the back-end will help reduce, but not eliminate, non-response bias.
This white paper was provided by TRC, a full-service market research provider located in Fort Washington, PA.
Other content shared by TRC
Better Questions For Segmentation: Use of MAX-DIFF
by Rajan Sambandam, TRC
Using Maximum Difference Scaling as a method in designing surveys may ensure more useful results in your market research. It is a comparative method based on importance that sidesteps the problems associated with traditional importance scales. TRC explains the mechanics behind this method through a detailed example in this white paper. Read Article »
Database Scoring with Object Based Segmentation
by Rajan Sambandam, TRC
Segmentation created from company databases are often lacking the rich segmentation schemes formed by attitudinal surveys. A new approach is Object based segmentation that uses database variables at the basis for forming attitudinal segments, leaving both markets classifiable with clear demographic segments. TRC compares traditional segmentation analysis with Object based. Read Article »
Asymmetry in Product Features: Use of the Kano Method
by Rajan Sambandam, TRC
The presence or absence of product features strongly affect consumer satisfaction with the design. Comparing these features using asymmetry analysis can help identify satisfiers and dissatisfiers from among the features of a product. The Kano method is similar but results in categorizing each respondent's answers. TRC presents this essential method of deciding new product features in detail. Read Article »
Conjoint Analysis versus Self-Explicated Method: A Comparison
by Rajan Sambandam, TRC
Determining feature importance in a product can be divided into two techniques - top-down methods where a customer evaluates the whole product at once, and bottum-up methods where features are evaluated individually or in sets. The former method, Conjoint Analysis, is more common while the latter method, Self-Explicated Method, is not widely used but has practical advantages. TRC compares the two methods in this white paper. Read Article »
Product Configurator
by Rajan Sambandam, TRC
To help customers purchase the right product, companies often use product configurators - tools that let customers design their purchase before ordering. This method is employed as a market research technique, similar to conjoint analysis but without some of the constrictions. This white paper from TRC explains an appropriate use of the product configurator method. Read Article »
Market Segmentation: One Method, Four Examples
by Rajan Sambandam, TRC
Effective market segmentation requires an understanding of the market and the skilled art of finding the appropriate segments. TRC gives four examples of this method's application with results. Read Article »
How to Measure the Value of a Brand
by Rajan Sambandam, TRC
Brand name evokes an inherent value; finding a way to reliably measure that value is crucial in determining product development. A technique called discrete choice conjoint analysis is described in this paper by TRC. Read Article »
Deriving Value from Research: the Use of Conjoint Analysis for Product Development
by Rajan Sambandam, TRC
Marketing research has been used by firms over the last several decades to provide information for decision making. Over time, increasingly sophisticated statistical methods have been developed and deployed in the service of this goal. This article focuses on one such method - conjoint analysis - and its application to product development. Read Article »
Cluster Analysis Gets Complicated
by Rajan Sambandam, TRC
Segmentation studies using cluster analysis have become commonplace. However, the data may be affected by collinearity, which can have a strong impact and affect the results of the analysis unless addressed. This article investigates what level presents a problem, why it's a problem, and how to get around it. Simulated data allows a clear demonstration of the issue without clouding it with extraneous factors. Read Article »
Identifying Feature Importance: A Comparison of Methods
by TRC
Understanding what customers want is fundamental to the new product development process as well as to the process of keeping existing products fresh and relevant. To be successful in this area we need to be able to correctly identify what features are important to consumers. Feature importance can be measured using a variety of methods of differing effectiveness. In this paper we will deal with the following methods: Importance Scales, Pick data, Pairwise Comparisons, and Max-Diff.
Read Article »
Monadic Price Testing vs. Price Laddering
by TRC
Compares two popular pricing methods to understand the difference in take rate information. Read Article »
New Product Development: Stages and Methods
by Rajan Sambandam, TRC
TRC identifies the best methods for each stage of the product development process, from Idea Generation through Feature Development, Product Development and Product Testing. Read Article »
Understand Choice in Banking: Use of Discrete Choice Conjoint Analysis
by TRC
Conjoint analysis provides incentive for survey respondents to determine which features must not be omitted in their final purchase. The method closely mirrors decision-making in the real world, and as shown by TRC in this white paper, is applicable to many situations including how customers choose their bank. Read Article »
Want better product ideas? Try smart incentives
by Rajan Samandam, TRC
Idea generation from survey respondents is strongly dependent on incentive. Introducing competition strengthens the quantity and quality of creative responses. TRC provides examples of smart incentives in this white paper. Read Article »
An alternative method of reporting customer satisfaction scores
by Rajan Sambandam and George Hausser of TRC
Though customer satisfaction evaluations are widely used, reporting of these scores has varied from one study to another. This is likely the result of each method’s advantages and disadvantages, as well as the personal preferences and habits of the researcher. In this article we review various reporting methods and outline our method with an example. Read Article »
Identifying the Key Drivers of Brand Image
by TRC
Measuring brand image requires looking at direct effects as well as indirect effects of a company's performance. TRC compares traditional multiple regression with SatiscanTM, a method that can review all possible path models. Read Article »
Improving Call Satisfaction: A Case Study
by TRC
TRC presents a case study of analyzing and improving a call center as an on-going data collection process. Read Article »
Improving Claim Satisfaction: A Case Study
by TRC
A case study on applying full-service market research to help an insurance company improve their client satisfaction with claim handling. Read Article »
Segmentation Success
by Michael Sosnowski, TRC
This paper explains the basic building blocks of the segmentation process and its implementation. Read Article »
Survey of Analysis Methods Part I
by Rajan Sambandam, TRC
Practical marketing research deals with two major problems: identifying key drivers and developing segments. In this two-part series TRC looks at key driver analysis and segmentation. Read Article »
Survey of Analysis Methods Part II
by Rajan Sambandam, TRC
This is Part II of a series looking at aspects of practical marketing research: identifying key drivers and developing segments. This content describes specific segmentation methods: cluster analysis, neural networks, self-organizing map (SOM), and mixture models. Included is a discussion on ideas for developing good segments. Read Article »
Validating Satiscan Using A Split Sample Approach
by TRC
TRC's SatiscanTM model is tested for validity using call center data and a split sample approach. This shows that SatiscanTM produces similar models when run on random halves of an energy industry dataset. Read Article »
Satiscan and Regression Analysis: A Comparison
by TRC
The comparison shows the advantages of SatiscanTM, an analytical method from TRC, over regression in identifying the correct and cost efficient action steps. Read Article »
TURF: New Methods for Implementation
by Westley Ritz, TRC
TURF is a long-established and quite useful marketing research tool, but not everyone is familiar with how it works, or with the latest developments that can make TURF even more effective. The purposes of this paper are twofold: (1) to explain the technique and (2) to describe the latest methods for implementation.
Read Article »
Product Configuration: A Research Approach for the Times
by Rajan Sambandam & Pankaj Kumar, TRC
The marketplace has shifted in the last decade with the ability of consumers to configure the product they want. This white paper explains the basics of configuration, an approach that mimics the real world of customer driven product design to obtain insight into consumer decision-making. Read Article »
Product Configuration: Evidence for Effectiveness
by Rajan Sambandam & Pankaj Kumar, TRC
This white paper looks at the examples from one product configuration study, the kinds of information that can be derived and the possibilities provided by statistical analysis. Read Article »
New Product Research: A Dynamic Approach to Feature Prioritization
by Pankaj Kumar, Westley Ritz and Rajan Sambandam of TRC
Feature prioritization is a very common new product research problem. Over the last few years, the most popular technique has been Max-Diff. However, as the number of features increases it becomes difficult to use. Bracket is a tournament-based approach that produces Max-Diff like results and can easily prioritize fifty or more features. Read Article »
Doing More with Less: Getting Greater Value from Mobile Quant
by TRC
What “more with less” means with respect to mobile MR, and examples from traditional online studies to challenge existing assumptions about what will and will not work on a mobile device. Read Article »
How to measure the value of a brand?
by TRC
Knowing how to price your product that you can optimize your ROI is key. This video explains various ways to measure the value of a brand and talks about a discrete choice conjoint technique as a perfect approach to measuring the value of a brand. Read Article »
Product Configuration with Michael Sosnowski
by TRC
Consider a person who wants to buy a personal computer. The customer can select exactly the combination desired, subject to a price constraint. Would it be possible to use such a process for research? Read Article »
How to Improve Your Market Segmentation
by TRC
Bob Hull from TRC talks about a market research technique for market segmentation and ways of improving them. Read Article »
Rich Raquet Market Research Consulting
by TRC
Rich Raquet is introducing TRC, a research & analytics firm, specializing in new product research, conjoint, segmentation, brand equity, sat & loyalty. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 1 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 1 of 4](/images/library/TRC_kumar_p1.jpg "[Video] Building Realistic Models of Choice in Practice Part 1 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 2 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 2 of 4](/images/library/TRC_kumar_p2.jpg "[Video] Building Realistic Models of Choice in Practice Part 2 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 3 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 3 of 4](/images/library/TRC_kumar_p3.jpg "[Video] Building Realistic Models of Choice in Practice Part 3 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 4 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 4 of 4](/images/library/TRC_kumar_p4.jpg "[Video] Building Realistic Models of Choice in Practice Part 4 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
