Article:
New Product Research: A Dynamic Approach to Feature Prioritization
by Pankaj Kumar, Westley Ritz and Rajan Sambandam of TRC
Feature prioritization is a very common new product research problem. Over the last few years, the most popular technique has been Max-Diff. However, as the number of features increases it becomes difficult to use. Bracket is a tournament-based approach that produces Max-Diff like results and can easily prioritize fifty or more features.
Have you had this problem? You need to measure preference for the features in your product/service but there are just so many of them it seems like an impossible task. Using conventional approaches you would have asked about the importance of each feature on a scale. But we all know how that story goes. With no other constraints, respondents don’t have an incentive to say that anything is unimportant. You could use constraint-based methods like constant sum scales, but cannot realistically deal with more than a handful of features at a time. Over the last few years, the most popular technique to address this kind of feature prioritization problem has been Max-Diff (see white paper on Max-Diff). But using Max-Diff when there are more than a dozen attributes becomes a real chore. So what can you do when you have dozens of features that need to be efficiently culled? Let’s first start with a look at a standard Max-Diff approach.
Standard Max-Diff
If there are ten features to prioritize, the Max-Diff algorithm can be set up such that respondents see grids of 3-5 features at a time, perhaps 8-10 grids in total. In each grid a respondent would indicate the feature that is most important (or some other relevant metric) and the one that is least important. At that point the respondent is done and the analysis of the data is conducted with Hierarchical Bayesian estimation to identify not only the rank ordering of the features but also the distances between them. The really neat outcome is that this information is available for each individual respondent, allowing further cutting and filtering of the results.
But when the number of features to be tested increases, so does the number of grids. Since features have to be shown multiple times in this approach, the task quickly becomes monotonous and seemingly mindless for the respondent. When you have 30 or more features, it is hard not to sympathize with the respondent’s plight and at some point the quality of data obtained will come under question. So what can we do about that?
A New Approach to New Product Development
We use a dynamic approach (called BracketTM) that uses a tournament structure to successively eliminate the “losing” features, thus making the task more engaging and cognitively challenging. The first round is similar to a Max-Diff task in that respondents will see a series of grids with a few features in each one and indicate their preference. The losing features (in this case those that are not preferred) fall away and the winners live to compete in the next round and so on, till we narrow the features down to each respondent’s final set.
There are several advantages to this approach. Most important is that it allows us to handle large numbers of features (upwards of 50) without running into respondent fatigue as the elimination process whittles down unimportant features very rapidly. What is also significant is that the process is more engaging to respondents as they don’t have to keep choosing between features they have no interest in, as in a regular Max-Diff task. Since only the preferred features move through the rounds of the tournament, the task becomes more engaging and difficult (in a good way) encouraging a more cognitive response than usual. Lastly, the most important features for each respondent are dynamically identified in real time allowing follow-up questions to be seamlessly asked in the same survey.
Sounds like a winner, but how do we know that we are actually getting good quality data on the back end? We ran tests to confirm this and here is what we found.
A BracketTM Example
The subject in this case was how movie-goers make decisions about which movie to see and where to see it. One can imagine many such factors: the stars, the director, the theater location, the show timing, etc. We imagined 18 of them and constructed a study where one cell of respondents was provided a standard Max-Diff task, while another cell was provided a BracketTM task.
Why choose only 18 features? We could have gone with, say, 30 or 40 features. But to make the methods comparable we needed a number that both methods could handle comfortably. The higher we went, the more the test would have been unfair to Max-Diff, so we chose to go with something in the neighborhood of 20.
In the Max-Diff case respondents saw 14 grids of four features and picked the features that were most (and least) important to them in choosing to go to a movie. In the BracketTM case, respondents saw six grids of three features in the first round, two grids of three features in the second round and one grid in the third round. For validation purposes we asked four sets of choice questions set up like conjoint tasks. In each set respondents saw three profiles comprising the features tested in the study. They were asked which profile they would pick. So a person putting a lot of weight on a movie with their favorite stars would be expected to be more likely to pick the profile that offered that option. That was it for the data collection part.
Both datasets were analyzed using Hierarchical Bayesian estimation. The analysis is pretty standard in the Max-Diff case but is more nuanced in the case of BracketTM. Nevertheless, feature preferences for individual respondents can be obtained in both cases. The question then is how do they compare, and that is where the validation tasks come into play.
Validation
We know which profile options respondents picked in the holdout validation tasks in the study. Based on the individual level utilities (preferences for features) that we got from the analysis we can also make a prediction of which options they would pick. Comparing the two tells how well we did in estimating respondent preferences. So what did we find?
Before we get there, it should be noted that in these kinds of comparisons Max-Diff should be expected to do better as it gathers more data. Each respondent in the Max-Diff cell is providing preference information on each feature approximately three times, whereas in the BracketTM cell the winning features get a second and perhaps third look but a good number don’t get more than one look. So, in theory, it should be expected that Max-Diff will provide more accurate preference scores and thus validation.
What we found surprised us. The validation hit rate for Max-Diff was 47% and for BracketTM it was 45% (these numbers are in the expected range given the type of validation task and the domain). In other words, there was almost no difference between the more rigorous Max-Diff approach and the less rigorous, but more engaging, BracketTM approach. How could this be? We further investigated the issue by examining carefully the information from each round of BracketTM and found that information respondents provided in the very first round were highly useful. Think about that for a second. In the first round, respondents see each feature just once in random groupings of three. But the information they provide there, combined with the Hierarchical Bayesian analysis is capable of producing a very robust foundation on which the next two rounds can sit. This tells us that BracketTM is capable of producing very nearly Max-Diff type information content.
As mentioned before, we deliberately set the test up to be fair to Max-Diff and therefore chose to use only 18 features. Our contention is that when there are far more features BracketTM will be the only useable method as Max-Diff would be too tedious. This validation study, well, validates that notion by establishing the robustness of BracketTM. However, it also raises the question of whether BracketTM could be used in cases where there are fewer features to test and we are hard pressed to argue against that.
We have applied this approach to several studies with feature sets varying from the teens to the fifties and have found it possible to elicit individual level preferences while keeping respondents engaged. We don’t see why even larger feature sets cannot be practically prioritized.
Another advantage we have learned by applying BracketTM is that the tournament-based approach is particularly well suited in situations where the features are not well distinguished from each other. In such cases standard Max-Diff tasks become even more difficult for respondents, whereas BracketTM tasks are well suited to help respondents focus on the most important ones. Consider message testing where many of the messages being tested may be only subtly different from each other. As we found out in a test, regular Max-Diff can lead to a confusing mess, whereas BracketTM can provide quite clear results.
In Conclusion
Feature prioritization is a very common new product research problem. However, as the number of features increases into the teens and beyond it becomes difficult to use state-of-the-art methods like Max-Diff without substantially increasing the tedium quotient of the study. BracketTM is a tournament-based approach that produces Max-Diff like results and can easily prioritize fifty or more features.
This content was provided by TRC, a market research company located near Philadelphia, PA. Visit their website at www.trchome.com.
Other content shared by TRC
Better Questions For Segmentation: Use of MAX-DIFF
by Rajan Sambandam, TRC
Using Maximum Difference Scaling as a method in designing surveys may ensure more useful results in your market research. It is a comparative method based on importance that sidesteps the problems associated with traditional importance scales. TRC explains the mechanics behind this method through a detailed example in this white paper. Read Article »
Database Scoring with Object Based Segmentation
by Rajan Sambandam, TRC
Segmentation created from company databases are often lacking the rich segmentation schemes formed by attitudinal surveys. A new approach is Object based segmentation that uses database variables at the basis for forming attitudinal segments, leaving both markets classifiable with clear demographic segments. TRC compares traditional segmentation analysis with Object based. Read Article »
Asymmetry in Product Features: Use of the Kano Method
by Rajan Sambandam, TRC
The presence or absence of product features strongly affect consumer satisfaction with the design. Comparing these features using asymmetry analysis can help identify satisfiers and dissatisfiers from among the features of a product. The Kano method is similar but results in categorizing each respondent's answers. TRC presents this essential method of deciding new product features in detail. Read Article »
Conjoint Analysis versus Self-Explicated Method: A Comparison
by Rajan Sambandam, TRC
Determining feature importance in a product can be divided into two techniques - top-down methods where a customer evaluates the whole product at once, and bottum-up methods where features are evaluated individually or in sets. The former method, Conjoint Analysis, is more common while the latter method, Self-Explicated Method, is not widely used but has practical advantages. TRC compares the two methods in this white paper. Read Article »
Product Configurator
by Rajan Sambandam, TRC
To help customers purchase the right product, companies often use product configurators - tools that let customers design their purchase before ordering. This method is employed as a market research technique, similar to conjoint analysis but without some of the constrictions. This white paper from TRC explains an appropriate use of the product configurator method. Read Article »
Market Segmentation: One Method, Four Examples
by Rajan Sambandam, TRC
Effective market segmentation requires an understanding of the market and the skilled art of finding the appropriate segments. TRC gives four examples of this method's application with results. Read Article »
How to Measure the Value of a Brand
by Rajan Sambandam, TRC
Brand name evokes an inherent value; finding a way to reliably measure that value is crucial in determining product development. A technique called discrete choice conjoint analysis is described in this paper by TRC. Read Article »
Deriving Value from Research: the Use of Conjoint Analysis for Product Development
by Rajan Sambandam, TRC
Marketing research has been used by firms over the last several decades to provide information for decision making. Over time, increasingly sophisticated statistical methods have been developed and deployed in the service of this goal. This article focuses on one such method - conjoint analysis - and its application to product development. Read Article »
Cluster Analysis Gets Complicated
by Rajan Sambandam, TRC
Segmentation studies using cluster analysis have become commonplace. However, the data may be affected by collinearity, which can have a strong impact and affect the results of the analysis unless addressed. This article investigates what level presents a problem, why it's a problem, and how to get around it. Simulated data allows a clear demonstration of the issue without clouding it with extraneous factors. Read Article »
Identifying Feature Importance: A Comparison of Methods
by TRC
Understanding what customers want is fundamental to the new product development process as well as to the process of keeping existing products fresh and relevant. To be successful in this area we need to be able to correctly identify what features are important to consumers. Feature importance can be measured using a variety of methods of differing effectiveness. In this paper we will deal with the following methods: Importance Scales, Pick data, Pairwise Comparisons, and Max-Diff.
Read Article »
Monadic Price Testing vs. Price Laddering
by TRC
Compares two popular pricing methods to understand the difference in take rate information. Read Article »
New Product Development: Stages and Methods
by Rajan Sambandam, TRC
TRC identifies the best methods for each stage of the product development process, from Idea Generation through Feature Development, Product Development and Product Testing. Read Article »
Understand Choice in Banking: Use of Discrete Choice Conjoint Analysis
by TRC
Conjoint analysis provides incentive for survey respondents to determine which features must not be omitted in their final purchase. The method closely mirrors decision-making in the real world, and as shown by TRC in this white paper, is applicable to many situations including how customers choose their bank. Read Article »
Want better product ideas? Try smart incentives
by Rajan Samandam, TRC
Idea generation from survey respondents is strongly dependent on incentive. Introducing competition strengthens the quantity and quality of creative responses. TRC provides examples of smart incentives in this white paper. Read Article »
An alternative method of reporting customer satisfaction scores
by Rajan Sambandam and George Hausser of TRC
Though customer satisfaction evaluations are widely used, reporting of these scores has varied from one study to another. This is likely the result of each method’s advantages and disadvantages, as well as the personal preferences and habits of the researcher. In this article we review various reporting methods and outline our method with an example. Read Article »
Identifying the Key Drivers of Brand Image
by TRC
Measuring brand image requires looking at direct effects as well as indirect effects of a company's performance. TRC compares traditional multiple regression with SatiscanTM, a method that can review all possible path models. Read Article »
Improving Call Satisfaction: A Case Study
by TRC
TRC presents a case study of analyzing and improving a call center as an on-going data collection process. Read Article »
Improving Claim Satisfaction: A Case Study
by TRC
A case study on applying full-service market research to help an insurance company improve their client satisfaction with claim handling. Read Article »
Non-Response Bias In Survey Sampling
by TRC
Market research accounts for many scenarios to ensure high quality of data. One of the most overlooked problems is non-response bias. TRC describes ways to reduce its effects through survey design and data adjustment in this white paper. Read Article »
Segmentation Success
by Michael Sosnowski, TRC
This paper explains the basic building blocks of the segmentation process and its implementation. Read Article »
Survey of Analysis Methods Part I
by Rajan Sambandam, TRC
Practical marketing research deals with two major problems: identifying key drivers and developing segments. In this two-part series TRC looks at key driver analysis and segmentation. Read Article »
Survey of Analysis Methods Part II
by Rajan Sambandam, TRC
This is Part II of a series looking at aspects of practical marketing research: identifying key drivers and developing segments. This content describes specific segmentation methods: cluster analysis, neural networks, self-organizing map (SOM), and mixture models. Included is a discussion on ideas for developing good segments. Read Article »
Validating Satiscan Using A Split Sample Approach
by TRC
TRC's SatiscanTM model is tested for validity using call center data and a split sample approach. This shows that SatiscanTM produces similar models when run on random halves of an energy industry dataset. Read Article »
Satiscan and Regression Analysis: A Comparison
by TRC
The comparison shows the advantages of SatiscanTM, an analytical method from TRC, over regression in identifying the correct and cost efficient action steps. Read Article »
TURF: New Methods for Implementation
by Westley Ritz, TRC
TURF is a long-established and quite useful marketing research tool, but not everyone is familiar with how it works, or with the latest developments that can make TURF even more effective. The purposes of this paper are twofold: (1) to explain the technique and (2) to describe the latest methods for implementation.
Read Article »
Product Configuration: A Research Approach for the Times
by Rajan Sambandam & Pankaj Kumar, TRC
The marketplace has shifted in the last decade with the ability of consumers to configure the product they want. This white paper explains the basics of configuration, an approach that mimics the real world of customer driven product design to obtain insight into consumer decision-making. Read Article »
Product Configuration: Evidence for Effectiveness
by Rajan Sambandam & Pankaj Kumar, TRC
This white paper looks at the examples from one product configuration study, the kinds of information that can be derived and the possibilities provided by statistical analysis. Read Article »
Doing More with Less: Getting Greater Value from Mobile Quant
by TRC
What “more with less” means with respect to mobile MR, and examples from traditional online studies to challenge existing assumptions about what will and will not work on a mobile device. Read Article »
How to measure the value of a brand?
by TRC
Knowing how to price your product that you can optimize your ROI is key. This video explains various ways to measure the value of a brand and talks about a discrete choice conjoint technique as a perfect approach to measuring the value of a brand. Read Article »
Product Configuration with Michael Sosnowski
by TRC
Consider a person who wants to buy a personal computer. The customer can select exactly the combination desired, subject to a price constraint. Would it be possible to use such a process for research? Read Article »
How to Improve Your Market Segmentation
by TRC
Bob Hull from TRC talks about a market research technique for market segmentation and ways of improving them. Read Article »
Rich Raquet Market Research Consulting
by TRC
Rich Raquet is introducing TRC, a research & analytics firm, specializing in new product research, conjoint, segmentation, brand equity, sat & loyalty. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 1 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 1 of 4](/images/library/TRC_kumar_p1.jpg "[Video] Building Realistic Models of Choice in Practice Part 1 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 2 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 2 of 4](/images/library/TRC_kumar_p2.jpg "[Video] Building Realistic Models of Choice in Practice Part 2 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 3 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 3 of 4](/images/library/TRC_kumar_p3.jpg "[Video] Building Realistic Models of Choice in Practice Part 3 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
[Video] Building Realistic Models of Choice in Practice Part 4 of 4
by TRC
![[Video] Building Realistic Models of Choice in Practice Part 4 of 4](/images/library/TRC_kumar_p4.jpg "[Video] Building Realistic Models of Choice in Practice Part 4 of 4")
Most choice research and modeling studies in practice fail to account for variations in choice processes. We discuss some of these issues in this presentation and show how we have made advancements in choice research design and modeling through some practical examples. Read Article »
