There was a time when founders built companies. Today, many seem busy building experiments and if you want to sound strategic, just sprinkle the words “We’re A/B testing it”.
The headline. The button. The colour. The font. The emoji. The CTA.
Conceptually, A/B testing sounds incredibly smart. Data-driven. Scientific. Objective.
But here’s the uncomfortable question:
What exactly are you testing… and on whom?
Many early-stage startups celebrate that Version B got a 7% better conversion.
Out of 84 visitors.
Most of whom came from friends, LinkedIn followers, curious competitors, or random traffic that was never going to become customers anyway.
That’s like launching a consumer product for the entire Indian market after conducting research on a sample size of 100 people standing outside one shopping mall.
The numbers may be statistically significant.
The decision certainly isn’t.
The irony is that A/B testing was never designed for companies trying to discover product-market fit.
It was designed for businesses already operating at scale.
When millions of users visit your platform every month, even a 1% improvement translates into crores of additional revenue. At that scale, testing isn’t optional, it is responsible management.
But startups often imitate the tools without understanding the context.
Big companies optimise certainty.
Startups are still searching for it.
A founder with 500 monthly visitors doesn’t have an A/B testing problem.
He has a customer understanding problem.
Talk to ten customers.
Watch them use your product.
Listen to why they don’t buy.
That insight is worth far more than discovering that a blue button outperformed a green one by 3%.
Even history reminds us that transformative ideas weren’t always democratically validated.
When Steve Jobs approved Apple’s iconic 1984 commercial, he didn’t run endless A/B tests across multiple television edits. It was conviction backed by deep customer understanding and an instinct for where the world was heading.
The first iPhone wasn’t born because a dashboard showed people preferred touchscreen keyboards over physical ones.
Some decisions cannot emerge from spreadsheets. They emerge from clarity.
Yes, gather feedback.
Yes, study behaviour.
Yes, measure outcomes.
But don’t outsource your judgement to analytics.
Because if every strategic decision requires statistical validation, perhaps the missing variable isn’t the data.
It’s the founder.
Data should sharpen conviction.
It should never replace it.
So ask yourself:
Are you truly building your company on scientific, data-driven decision-making… or are you conveniently using A/B testing to sound smart while avoiding to take responsibility for the decision?