When Stat Testing Is like a Head Fake

I have never been a big fan of stat testing because there is nothing sacrosanct to me about a 90% confidence interval, especially when 50% of ad campaigns fail to show sales lift and 80% of new products fail.

However, now I have a new reason not to like it…in certain situations stat testing can lead to exactly the wrong business decision. 

When you have a low-priced item, like many CPG, it is easy to see that you could stat test advertising testing results between exposed and unexposed consumers and find a statistical difference. Yet that difference might not be big enough to imply that the tested advertising plan is likely to generate positive profit return.

Conversely, on a high profit item, like solar panels or boats, which typically have really low conversion rates, you could fail to find statistical significance at measured differences that would be highly profitable.

Two ingredients are necessary to fix this disconnect.

1. Have the marketer assert the economic value of a 1 percentage point movement in the outcome metric you are testing against.

2. Instead of stat testing, provide 4 numbers:

1. The probability of the advertising generating negative profit vs. positive profit

2. The expected loss if negative profit and the expected gain if positive profit.

With these four numbers, the marketer can make fully informed decisions. If they are risk seeking, they can place more emphasis on the expected value of profit return. If they are evaluating advertising alternatives, the expected profit return, which can be easily calculated from these 4 numbers, gives them the expected value of their alternatives.

Imagine a scenario where the probability of a success for an ad campaign that is budgeted for $1MM is only 60%. That obviously fails a stat sig test. Yet, say the expected profit if the campaign works is $5MM and the expected loss if it fails, is $1MM because it is capped at the ad spending level.  The expected return is $2.4MM on an $1MM investment. Who doesn’t like that bet?  Yet, stat sig approaches will tell you not to run the campaign.

How can you get to these numbers to make a better informed decision?  I like running distributions via monte carlo simulation.  You can generate, say, 1000 simulations, and see from the data what the average profit is, what it is among those runs that were profit positive vs. negative…it’s all in the simulated data. If the percentage conversion rates are really low, you might choose to model this as counting data from Gamma distributions because the profit function is not likely to be symmetric. You might even want to use a Bayesian prior if you have a strong expectation that, say, one audience SHOULD outperform another based on Movable Middle theory. These are choices on a case-by-case basis, but the big thing is this…

Stop doing stat testing that is divorced from financial analysis.  You will get more meaningful answers and will connect with the CFO better when budget setting time comes around.

Caveat: There are motivations other than profit where stat testing is, of course, still needed.  Two examples…product testing…you better be 99.9% sure a product will not harm the user. Another…if you are making an ad claim that will go before the NAD (National Advertising Division, is a self-regulatory body that monitors advertising for truth and accuracy), you need to meet their standards. Having served as an expert witness, I can tell you there are NAD rules you have to follow to get a favorable review.