Stop Worshipping Significance: Rethinking Research for Business Decisions

People are obsessed with large samples and statistical significance. But think about it — that’s often not the point.

In the pure sense, quant research does mean large samples and statistical significance. That’s where it came from, and why it was originally valued. But over time, quant became widely adopted across use cases for another reason: time and cost efficiency, plus simplicity. It became the default not because every use case requires statistical significance, but because quant fit constraints better than qual.

That’s the problem. Businesses now default to large-N quant even when the actual decision doesn’t call for it. The result is wasted time, wasted money, and evidence that looks rigorous but doesn’t match the decision at hand.

The Decision Lifecycle

Every research project has two fundamentally different types of decisions:

1. Steering Decisions — Early, iterative, often reversible. These are about direction, not proof. You’re deciding which creative route to explore, which feature to refine, which customer pain point to prioritize. Getting these wrong is cheap — the cost of delay is higher than the cost of error.
2. Commitment Decisions — Late, high-stakes, harder to reverse. These are final go/no-go calls: launch or don’t, set the price, approve the claim, roll the campaign. These do require stronger evidence because the cost of being wrong is high.

The mistake most companies make: applying the same evidentiary standard to both types. They drag steering decisions through the mud of statistical significance, wasting time when speed and iteration matter most.

The Framework: Cost of Error × Reversibility

Imagine a simple 2×2:

• X-axis: Reversibility of the decision (easy ↔ hard to roll back)
• Y-axis: Cost of being wrong (low ↔ high)
• Bottom-left (low cost, reversible): Rapid qual, small-N quant, AI-moderated steerability. Act fast, learn fast.
• Top-right (high cost, irreversible): Large-N quant, sequential testing, statistical significance. Here, rigor is justified.
• Other quadrants: Blended approaches, depending on risk appetite and timing.

The key insight: Every project moves through multiple quadrants. Early = steerability. Final = commitment. If you treat them the same, you’re either over-engineering or under-protecting.

Business Function Examples

Different functions simply have different “funnels” of decision types:

• Marketing & Branding: 90% steering (creative iteration, messaging tweaks), 10% commitment (final campaign roll-out).
• Product & Innovation: Early is all steerability (concept iteration, prototype testing). Launch decisions lean commitment.
• Operations & CX: Nearly everything is reversible. Steerability dominates.
• Strategy & Pricing: Skews heavily toward commitment (claims, price points, brand architecture). Here, rigor matters more.

The point: don’t default to quant with large samples just because it feels scientific. Match the method to the decision type.

Steelmanning the Other Side

When does significance matter?

• Regulatory and legal contexts (on-pack claims, compliance).
• High-spend moves (new brand architecture, pricing shifts).
• Organizational alignment (executives need a hard number to commit).

Yes, there are moments where p-values are worth the wait. But know that you’re buying ceremony and political alignment — not necessarily better decision quality.

Why This Matters

Research has been broken by confusing “proof” with “progress.” In reality:

• Most of your time should be spent on steerability decisions.
• Only a small fraction of your budget should go to significance.

The companies that win don’t wait for proof. They steer faster, with eyes open to risk.

Closing Thought

In business, what matters isn’t p < 0.05. It’s whether you can make the right call, fast enough, with confidence in the risks you’re taking.