Prediction Markets for Concept Testing

This white paper describes prediction markets, an innovative way to improve the speed and accuracy of online concept tests. You will learn why prediction markets deliver accurate concept tests more quickly than traditional methods, how prediction markets work, why they are more cost effective, and when you should utilize a prediction market.

 

What if there were a better way to do concept tests?

Approximately $24.6 billion is spent on market research annually(8) – much of it on new product, packaging, and advertising concept testing. Accurate concept tests make smart market researchers seem like time travelers, pulling the future forward and revealing which new products, creative designs, and product development ideas will pay off.

But as you know, predictive accuracy comes with an investment—often a substantial one. Fortunately, we’re at a turning point regarding best practices in concept testing. Both academic and empirical evidence mount in favor of harnessing what bestselling author James Surowiecki has named “the wisdom of crowds” through the intelligent use of prediction markets.

This paper is your guide to the latest research, evidence, and methods regarding more effective and efficient concept tests. We reveal how and why prediction markets are becoming a preferred research method for accurate concept testing and why forward thinking corporations like Google, Hewlett Packard, and Eli Lily rely on it for strategy and direction. We look at iCE, the on-demand prediction market from Infosurv, and discuss how it frees companies to engage in faster, more effective concept testing.

Of course, new methods in market research, just as in all scientific endeavors, have to be rigorously tested and validiated before they become mainstream. Once you read this paper, we think you’ll agree that the balance is tipping in favor of prediction markets and you should consider a prediction market for your next concept test. Let’s take a look at why.

 

The development of prediction markets

So just what is a prediction market?

Prediction markets are speculative markets created for the purpose of making predictions. Assets are created whose final cash value is tied to a particular event or parameter. The current market prices can then be interpreted as predictions of the probability of the event or the expected value of the parameter. (2)

A well-known example of a prediction market is the New York Stock Exchange (NYSE), where investors trade equity shares in public corporations. The share price of a corporation may be interpreted as a predicted value of their future earnings – an extremely accurate prediction in fact.

When celebrated New Yorker financial columnist James Surowiecki wrote the book, The Wisdom of Crowds, he shattered the conventional wisdom that a small group of experts is smarter than the masses. He pulled from a wide variety of sources to deliver the idea that the aggregate wisdom of a crowd is better than a poll of a trusted few or even the deliberation of an expert elite.

In fact, his research, which is corroborated by real life examples like the NYSE as well as academic literature, reveals that if the aggregate is properly assembled, the outcomes are actually more accurate than any other research method.(4)

His gives example after example of how diverse groups of regular people routinely exhibit precision and accuracy in their collective wisdom. For example, in the story that opens his book, Surowiecki details how 800 Scottish fair goers estimated the weight of an ox. Of course the intelligence and oxen expertise of the fair goers varied, but in aggregate their collective wisdom was impressive. As a group, their average guess of the ox’s weight was 1197 pounds – the ox actually weighed in at just one pound over their guess.

In another example from the book, Surowiecki describes the case of the naval vessel Scorpion, which disappeared. Salvage experts defined a search and rescue zone 20 miles wide—hopelessly large for a successful recover. Non experts, meanwhile, competed over a bottle of Chivas Regal to guess the sunken ship’s location. The aggregated estimate of the group of Scotch aficionados was 227 feet from where the Scorpion was actually found.

Group consensus exhibits uncannily accurate prediction when reasonable diversity, independence of group members, and decentralization are present.

Prediction markets provide the platform for crowds to exhibit their wisdom

So how do you provide “a crowd” with the structured ability to deliver its insight? One answer lies in the prediction market—a constructed opportunity for crowds to quickly exhibit their collective wisdom. Prediction markets have proven themselves over and over in recent years. For example, one of the most famous markets, the Iowa Electronic Market, out predicts Gallup polls regarding presidential elections 75% of the time. In a lesser known example, the Hollywood Stock Exchange (HSX) prediction market allows anyone to wager play money on who wins the Academy Awards. In 2000, a group of Wall Street Journal reporters went head to head against the predictions of the HSX when they interviewed 356 Academy judges on their vote before the awards were announced. The WSJ accurately predicted 5 out of 6 awards; the HSX prediction was 100% accurate (6 of 6) and didn’t cost a dime.

The uncanny, but well documented, ability of prediction markets to provide accurate insight has recently gained notoriety in the defense industry. According to the May 6, 2007 paper, “The Securities Trading of Concepts,” which details concept testing research from MIT that was sponsored by the Office of Naval Research (DARPA),

Our trading experiments show that the market prices of securities designed to represent product attributes and features are remarkably efficient and accurate measures of preferences, even with relatively few traders in the market . . . [and] may offer a particularly efficient screen mechanism for firms developing new products and services, and deciding where to invest additional product development dollars. (5)

 

The anatomy of high performance prediction markets

Researchers are particularly interested in the prediction market approach because it ensures that personal accountability backs each answer, in much the same way a bet or a stock trade is intentional and personal for the participant. Furthermore, the market itself provides a decentralized framework of values through which decisions are weighed. And add to that, the group of traders self-selects their level of participation and valuation. These attributes make prediction markets much different than polls or surveys.

Across the many empirical and academic approaches to prediction markets, it has been found that three key elements must be present to produce accurate results:

1. Offers in the market must be clearly understood and enforceable


2. There must be an incentive to motivate traders


3. Information discovery and sharing must take place (6)

On Infosurv’s proprietary prediction market known as the Infosurv Concept Exchange (iCE), participants are invited to buy shares in virtual “stocks” representing new product, packaging, design or creative concepts. Traders are rewarded according to their prediction prowess, with real-world cash prizes going to those traders who make the best predictions. Ample details about each stock in the iCE market are provided so traders can make the most informed trading decisions possible.

 

How prediction markets improve concept testing

More accurate identification of “winner” and “loser” concepts

In validation studies prediction markets have proven more accurate at predicting the success of new concepts than traditional monadic concept tests. Prediction markets also tend to produce more polarized ratings of the concepts tested, making it easier to differentiate between not only good and bad concepts, but also between good and great concepts.

Speed

It can take up to 4-6 weeks to design, field, and analyze a traditional concept test. Turn-around time on iCE concept tests is much less – just 1 week from project kick-off to presentation of final results. iCE projects are completed more quickly due to ease of setup, respondent recruitment, and data analysis.

Cost savings

Costs are lower with iCE than traditional concept tests because a "general population" sample can be used for most iCE markets. This always comes as a shock to market researchers since a traditional concept test typically targets respondents within a specific market. However, respondent targeting isn’t usually necessary on iCE since participants have an incentive to “self-select” only for markets they think they can win. Nobody likes to lose money, real or virtual. Respondents therefore only participate in studies dealing with topics and market segments they are familiar with, even though they may not be in that segment.

Also, iCE doesn’t ask respondents “Would you buy this concept?” but instead “Would others buy this concept?” iCE harnesses a phenomena known as “the wisdom of crowds” to allow non-experts to answer expert questions often more accurately than the experts themselves.

Better respondent engagement

One of the chief concerns in the market research industry today is respondent engagement. Since surveys are often long and uninteresting, respondents have the tendency to rush through just to collect their incentive. iCE respondents, on the other hand, view participating in the marketplace as an entertaining game with both intrinsic and extrinsic rewards. Since respondents can win real world prizes with their iCE dollar winnings, they have ample motivation to remain engaged in the process.

Easy respondent recruitment

iCE respondents are recruited just like survey respondents. Top survey panel providers are used to recruit a statistically valid sample of "general population" consumers to participate in every iCE market. Though specific demographic subgroups may also be targeted, it's usually not necessary. Alternately, a client’s own customers or employees may be used as iCE respondents.

No risk of misrepresented or “professional” survey takers

Other major concerns in the market research industry today are respondents who misrepresent themselves to qualify for a survey, and “professional” survey respondents who participate in lots of survey panels and have a disproportionate voice in samples. iCE avoids both of these issues. Since iCE respondents rarely have to qualify for a particular study based on demographic or behavioral criteria, they have no reason to misrepresent themselves. In addition, “professional” respondents are not a concern because if some iCE participants are disproportionately active in our markets, the validity of the market’s conclusions is only enhanced.

Qualitative concept feedback

iCE respondents are given a mechanism for explaining why they buy or sell shares in certain concepts, allowing clients to collect qualitative feedback, which answers the question, “Why?” This feedback can be used to explain the behavior of the iCE market and help the client create more innovative and successful future concepts.

 

When to use prediction markets versus more traditional methods

This is an excerpt - click here to download the full white paper (pdf).

Jared Heyman is the founder and president of Infosurv, a full-service market research firm specializing in design, administration, and analysis of online and telephone employee, customer, and market research studies. Visit the company website at www.infosurv.com.