A/B Testing: Can you iterate your way to great products?

Jeff Atwood’s (@codinghorror) blog about the movie “Groundhog Day” got me thinking much more deeply about A/B testing, a powerful tool in a set of tools that enable a new generation of nimble startups to compete effectively against the big Internet players.

For example, Match.com is stuck on a decade-old architecture and would be hard-pressed to create dozens of unique traffic funnels and ruthlessly tune them to extract optimal performance from each acquisition channel. Hence, the window is open for new competitors who can use these tools to develop products from the get-go.

But what are the limits of iterative optimization?

Split testing is, in my view, no substitute for product vision.

Human beings are probably the ultimate result of great product development through split testing – evolution is one giant iterative experiment. Unfortunately for Internet startups it took millions of years for life as we know it to develop, and even if we reduce the test cycle time a few orders of magnitude, the “development cycle” is beyond our comprehension.

A/B testing has demonstrated its advantage most clearly in efforts to optimize customer acquisition. Google made this approach table stakes by designing AdWords as a meritocracy of ad performance, and by offering all its AdWords customers easy split testing as a free feature. Write as much different ad copy as you like – Google will automatically test all the variants and promote the winners. This kind of metrics, instant feedback, and optimization has proven highly addictive to marketers – obvious now, innovative when it was implemented.

Trying to A/B test your way to a magical product is like waiting for 100 monkeys to type Shakespeare’s “Hamlet” – theoretically they will eventually put the letters in the right order, but by that time we’ll all be dead.

A less obvious reason for success is that all the iteration does not take place on your website but rather on the AdWords site. Writing 10 unique versions of the homepage and showing each to a slice of visitors would be a very big deal for a lot of people at your company. But an effective CEO is happy to let the marketing team run whatever ad copy converts customers best – after all, the customer experience doesn’t really start until the visitor gets to your website.

The best companies, however, go beyond customer acquisition: they can now evolve their products and user experiences iteratively and scientifically, right on their sites, practically on the fly. No more debate as to whether the button should be blue or red, whether it’s worth it to put the TRUSTe logo on the checkout page – we’ll ask the users and find the answer, definitively.

Does such a thing as a product visionary exist anymore? Or is it just a false idol from an age before reason? In my view, the question for 2010 is whether GREAT products can be created through A/B testing.

The case for the prosecution would seem to be Zynga. If ever there was a sector that would value product vision and creative expression, it would be online social gaming. Yet Zynga seems to be replacing the majority of the old-school creative innovation in product development with scientific process, and in the process growing revenue and profit at a mind-bending rate. I can only imagine the panic that must exist at EA today, as its execs stare at the gulf in the rate of product innovation and iteration between their team and Zynga’s crew. And every minute they stare at the chasm, it gets a little wider.

If Zynga can create games that retain and engage players over months and years, it becomes a lot harder to justify any other type of product development. The jury is out, but we’ll learn a lot more over the next year, and Facebook is helping us keep the experiment pure by fighting Zynga’s (smart) attempts to leverage its scale to build a platform that can win through market dominance rather than solely relying on the great execution that put Zynga in front.

Despite my overly rational and scientific bent, I’m secretly hoping that the product visionary still has a big role in the future. The Zynga process feels a bit like fast food: just as the McDonald’s engineers can find the chemical combination that strikes my taste buds in such a way that a corn byproduct tastes like chicken, the Zynga team can find the perfect psychological stimulus to keep me coming back to the game.

But like the Chicken McNugget, FarmVille, while tasty, offers very little in the way of sustenance.

Want to share this post?

Click to Tweet: A/B Testing: Can you iterate your way to great products? via @jdh http://www.joshhannah.com/2010/07/ab-testing-can-you-iterate-your-way-to-great-products/
Like what you've read? Click here to subscribe to this blog!
Written by Josh Hannah
Josh Hannah joined Matrix Partners after a career as a serial entrepreneur (Betfair, eHow, wikiHow.) Read more about Josh.
  • http://twitter.com/yannthesaint Yann Ngongang

    Josh, great point! A/B iteration leads to the local minima problem in machine learning: you conquer the hill but miss the mountain.

    SiValley is drinking the cool aid of the customer development frameworks which risks focusing a lot of smart & creative people on marginal improvements to existing platforms instead bold bets. If Henri Ford et all stuck to A/B testing, we'd be riding horse buggies on 101.

  • http://www.matrixpartners.com josh

    Agreed. Good as a tool but not as your entire approach to product. Highly addictive though, in its quantitative certainty, and in many companies can drive out product vision.

  • Long Ouyang

    Science doesn't have to preclude vision. A/B testing gives you information about the solution space – you can act however you want based on the information, which means you can ignore it too.

    And the common complaint about local minima is uninspired. You don't have to stop A/B testing when you arrive at a good result – you can probe further if you're interested in the causal model that drives everything.

  • http://davidcancel.com/ David Cancel

    Hi Josh,

    Thanks for the good post and contribution to the current Anti-A/B testing trend.

    I'm a big proponent of A/B Testing ( having created the site abtests.com ) but like any other tool it can and is often misused. I gave a talk recently called Data-Driven Startups ( http://davidcancel.com/data-driven-startups/ ) at a Lean Startup Meetup that goes into this issue, as well as the whole lean startup/customer development trend

    I won't repeat myself but will include a couple of excerpts from my presentation:

    “There are *NO* repeatable patterns for startup success. *None*. Stop looking for one.” (Slide 14)

    “Analytics & A/B Testing are also *not* the answer” (Slide 23)

    “Multivariate Testing is *Useless*.” (Slide 26)

    “The point of testing is to prove or disprove assumptions *quickly*”.


  • http://dlw.myopenid.com/ Dan Weinreb

    A/B testing is a fine tool when used as directed. Sure, nobody ever created a brilliant site or idea buy starting with a blank web site and then doing A/B testing from there. But A/B testing is a good way to do what A/B testing is good for: improving, optimizing, and tweaking something that is good already but could be incrementally improved.

    If you want to succeed, allocating 100% of your resources to the “big idea” and 0% to detail and (relentless) improvement is not usually the best way to go. A/B isn't “good” or “bad”; it's another tool in the toolbox.

From the Blog

The Fax Machine Died When I Wasn’t Looking

Of course, I’ve known the fax machine’s days were numbered for a while now. The fact of it’s death...

An Ad-Blocking Compromise

The debate over ad-blocking, spurred by the OS-level support for ad blockers in iOS9, has the feel of a religious war to me. People...