A/B Testing: Can you iterate your way to great products?
Jeff Atwood’s (@codinghorror) blog about the movie “Groundhog Day” got me thinking much more deeply about A/B testing, a powerful tool in a set of tools that enable a new generation of nimble startups to compete effectively against the big Internet players.
For example, Match.com is stuck on a decade-old architecture and would be hard-pressed to create dozens of unique traffic funnels and ruthlessly tune them to extract optimal performance from each acquisition channel. Hence, the window is open for new competitors who can use these tools to develop products from the get-go.
But what are the limits of iterative optimization?
Split testing is, in my view, no substitute for product vision.
Human beings are probably the ultimate result of great product development through split testing – evolution is one giant iterative experiment. Unfortunately for Internet startups it took millions of years for life as we know it to develop, and even if we reduce the test cycle time a few orders of magnitude, the “development cycle” is beyond our comprehension.
A/B testing has demonstrated its advantage most clearly in efforts to optimize customer acquisition. Google made this approach table stakes by designing AdWords as a meritocracy of ad performance, and by offering all its AdWords customers easy split testing as a free feature. Write as much different ad copy as you like – Google will automatically test all the variants and promote the winners. This kind of metrics, instant feedback, and optimization has proven highly addictive to marketers – obvious now, innovative when it was implemented.
Trying to A/B test your way to a magical product is like waiting for 100 monkeys to type Shakespeare’s “Hamlet” – theoretically they will eventually put the letters in the right order, but by that time we’ll all be dead.
A less obvious reason for success is that all the iteration does not take place on your website but rather on the AdWords site. Writing 10 unique versions of the homepage and showing each to a slice of visitors would be a very big deal for a lot of people at your company. But an effective CEO is happy to let the marketing team run whatever ad copy converts customers best – after all, the customer experience doesn’t really start until the visitor gets to your website.
The best companies, however, go beyond customer acquisition: they can now evolve their products and user experiences iteratively and scientifically, right on their sites, practically on the fly. No more debate as to whether the button should be blue or red, whether it’s worth it to put the TRUSTe logo on the checkout page – we’ll ask the users and find the answer, definitively.
Does such a thing as a product visionary exist anymore? Or is it just a false idol from an age before reason? In my view, the question for 2010 is whether GREAT products can be created through A/B testing.
The case for the prosecution would seem to be Zynga. If ever there was a sector that would value product vision and creative expression, it would be online social gaming. Yet Zynga seems to be replacing the majority of the old-school creative innovation in product development with scientific process, and in the process growing revenue and profit at a mind-bending rate. I can only imagine the panic that must exist at EA today, as its execs stare at the gulf in the rate of product innovation and iteration between their team and Zynga’s crew. And every minute they stare at the chasm, it gets a little wider.
If Zynga can create games that retain and engage players over months and years, it becomes a lot harder to justify any other type of product development. The jury is out, but we’ll learn a lot more over the next year, and Facebook is helping us keep the experiment pure by fighting Zynga’s (smart) attempts to leverage its scale to build a platform that can win through market dominance rather than solely relying on the great execution that put Zynga in front.
Despite my overly rational and scientific bent, I’m secretly hoping that the product visionary still has a big role in the future. The Zynga process feels a bit like fast food: just as the McDonald’s engineers can find the chemical combination that strikes my taste buds in such a way that a corn byproduct tastes like chicken, the Zynga team can find the perfect psychological stimulus to keep me coming back to the game.
But like the Chicken McNugget, FarmVille, while tasty, offers very little in the way of sustenance.