A news article on Ajaxian led me to this post about this article (PDF) from Microsoft. It talks about A/B testing and how important it is to success on the web. The article even cites the legendary Avinash Kaushik whom I had the pleasure of working with at Intuit. The best line in the article is: “The fewer the facts, the stronger the opinion” — Arnold Glasow. Avinash is credited with the term HiPO, which stands for Highest Paid Opinion.

This is something I truly think cripples most design projects. However, this article misses a crucial point. A/B testing is not free. You need a decent level of traffic and you need to produce each variant. These variants have details and decisions that can make or break a test. The difference between 4% conversion and 8% conversion is often a minor adjustment in the design.

This is an argument that I have had with Avinash. he once told me, “You don’t know anything, only the audience knows, so we test.” My point was, “Not all tests are created equally.” What is the point of creating A/B variants if all they do is fail against the control. Additionally, who is best trained to compete against the control? At my previous company, it was usually the HiPO who got to decide what would be tested.

In the worst cases, there was not even A/B testing. It was purely “put it up”. In the case of the one redesign, the actual statistics tanked after the redesign. I thought I could fix the design to make it perform better but was not given the opportunity. This may have something to do with my total lack of tact. Yes, it sadly is true, I piss people off.

At an internal UX Conference, the CEO claimed that the design teams didn’t “own” anything and that we were there to help the business leaders who had the final say on the User Experience. I believe this thinking is extremely flawed. The design team are the ones trained in User Experience design, so why aren’t they being delegated to on the subject? My proposal at that meeting was this:

A/B testing is critical and any company who can possibly do it, should mandate it for all substantive changes. The business leaders should define the business priorities, such as which products/prices, and which audience and which metrics they are most interested in. The business leaders should define success in a measurable way. User Experience Design teams should be responsible for producing the designs to test based on those business requirements. If the test fails to beat the control then it is the designers fault. If they repeatedly fail to beat the control, then you should consider a different designer or design team. That is their job. They must beat the control. They should be trusted to do their job but they need A/B testing to confirm that they are doing right by the company.

All to often, the HiPO changes the design based on personal untrained opinion. The marketing department chimes in. The CEO, the CMO, the VP of This Department, the Director of That Department, the engineers, everyone has an opinion. It’s design by committee at it’s worst and it happens everywhere. The designer is put in the horrible position of executing a poor design based on compromise and consensus.

I once told a HiPo. If I want Yellow and you want Blue, and the audience likes either; then we shouldn’t compromise on Green.  No one likes Green.

Trust your designers and don’t force them to compromise. Then A/B test the hell out of them and say, “Your bonus is based on the success/failure of these metrics.” You would be surprised how quickly they will push the needle.