Big Lever and Small Optimisation

Why I Hate The “41 Shades of Blue”

There’s this famous story:

Google once ran an experiment of 41 different shades of link click colour.

One color stood out with a somewhat impressively higher CTR.

Google used this color henceforth and made $200 Million more.

I fucking hate this story.

I’m not saying Google engineers is dumb and did not do a proper split test. I trust the result to be legit and the best color did bring a higher CTR. But it’s a typical correlation =/= causation fallacy. More importantly, it creates the illusion of small optimisation bringing huge result.

In reality, small optimisation almost never brings meaningful result. For small optimisation to actually bring result, it requires the below hypothesis:

1. The whole project actually have product-market fit (imagine Microsoft Phone keep iterating on their home button colour when their whole OS sucks)
2. The thing you’re optimising on actually matters (imagine Google went to optimise for their ‘I’m Feeling Lucky’ wording)
3. The optimisation methodology have no bias (which I believe Google engineers is smart enough. But how about average startup teams?)

It goes beyond just small optimisation. If your product have any area that have room for optimisation, it’s a red flag. The immediate example I can think of are social apps that uses the user’s first name’s initial letter as their default icon. The fact that you as the creator need to build this part out, you’re doomed.

Instead, go for the big lever. Challenge all assumptions. Change things up. Make bold bets.

Don’t test 41 shades of color. Go build a new mobile OS. Go build a state-of-the-art LLM. Go build a robot. That’s big lever.

Optimisation is a scam. It’s created by Silicon Valley product manager to keep the engineering team running.

Chase order of magnitude change.