In 2005, I worked for a year at Intuit and they claimed to be Customer-Driven. This came from a specific story. Originally the company produced Quicken, a personal finance tool. When they watched their customers, they realized that many of them were using it for business purposes. They realized this market and produced QuickBooks, which turned into the category king of small business accounting software.

The customer DROVE them to build the software. They took their cues from the customer. They weren’t data driven or creativity driven or any other kind of driven.

At early Apple, one might say it they were “Steve-Driven“. Steve Jobs had a vision and was a force of nature. He DROVE the company towards his vision. Then he was driven out.

Don Normal talks about the process after Steve departed as “Human-Centered“. Certainly better than Customer-Driven, but still had problems.

Norman describes Apple’s design method back then as “a well-structured process” and says he is still proud of it. But he is quick to point out its shortcomings.

“It was a consultative process,” he says; many different points of view and impressions were solicited. But “this can lead to a lack of cohesion in the product.” Then Steve came back and famously said, “You can’t just ask customers what they want and then try to give that to them. By the time you get it built, they’ll want something new.” —Steve Jobs, 2005

This was around the same time Don Norman changed his tune and was quoted, “I prefer design by experts – by people who know what they are doing” —Don Norman, 2005 Actually, I heard Norman speak at a conference and he said that Jobs fired all the data analysts and scientists and hired designers. He said that after Jobs did that, the products improved and sales went up.

I call this model Design-Driven. Jef Raskin said it well when he was quoted, “As far as the customer is concerned, the interface is the product.”—Jef Raskin, 2001

In the last few years, there has been this new popular phrase called Data-Driven. For me it originates with this quote, “If we have data, let’s look at data. If all we have are opinions, let’s go with mine.” -Jim Barksdale, former CEO of Netscape. Jim was basically denying any form of expertise to be a factor in the decision.

“Data” can be anything. My opinion is data. Your opinion is data. That’s not what Jim was saying. He was saying that unless you have something resembling a scientific mathematically-provable study, then it’s all just opinion. I believe this is wrong-headed and led to poor product decisions. Remember Netscape? Many poor decisions in that company’s history.

It’s very tempting (especially for engineers) to believe in a data driven world. What can you hold in your hand? What can you see with your own eyes? Prove it. This is totally aligned to an engineering point of view.

If you have ever been to a doctor with a problem you probably have direct experience with the lack of science that most medicine entails. Sure, there are amazing things happening in medicine, but more often than not the doctor has no real idea what is wrong with you. Example: It took a year and 4 doctors and 1 surgery to realize my wrist pain was a torn ligament.

Doctors are not data driven. They are Experience-Driven. Doctors spend an enormous amount of time studying, training, interning, and practicing to be a doctor. They literally say “Practice Medicine”. They read studies for sure, but doctors make their decisions based mainly on their experience. Keep in mind, a doctor’s decisions are literally life-death struggles. No one (usually) dies when a product manager chooses a poor path for a product.

Most professions are Experience Driven.  Here are some problems I see with the Data Driven approach.

Statistical Confidence
A huge majority of tests I have seen lack basic statistical confidence in the results. Great post on A/A testing demonstrates this. What this means is that if you test a hypothesis, you can get a result that is actually just random. Flip a coin 10 times and you should get 5 heads and 5 tails, right? I will bet you a dollar that you don’t get that result. Randomness is a powerful force.

Replication Crisis
In the academic world, there has recently been an existential earthquake. According to a 2016 poll of 1,500 scientists reported in the journal Nature, 70% of them had failed to reproduce at least one other scientist’s experiment (50% had failed to reproduce one of their own experiments).

This means that the majority of FACTS that you think you know, based on DATA and SCIENCE are just flatly untrue. Randomness is the nemesis of being Data Driven. It will screw up your results more often than not.

We are living in a world of “Fake News” partially because we can’t believe data anymore. This is incredibly stressful and makes decision-making difficult.

Pre-Testing Politics
At Intuit, Avinash Kaushik was an extreme proponent of testing. My simple question, however, bothered him. I asked, “Who decides what we test?” The problem is that you can’t test everything. There are audience limits and analyst limits and designer limits. We have to pick and choose what we test. In most companies, the decision of what to test is political and completely opinion based.

If I test two flavors of cookies, A) Shit-flavored and B) Puke-Flavored, I will end up with a winner. That doesn’t mean they are good cookies or deserve to win. Our presidential politics are often described as choosing between two bad choices exactly for this reason.

Luckily Avinash allowed me to add test variants without committee approval. I think he was surprised how often my version would win the test. It wasn’t surprising to me though. It’s not hard to beat a shit-flavored cookie or website designed by a committee.

Confirmation Bias
Test results are rarely black and white. They are often interpret-able. People will find the part of the study that agrees with their point of view and emphasize that. We want to confirm that we are smart and prescient. We want to believe that our opinions are, in fact, correct and true. I saw this first hand when a VP would cite a minor result in a test to overturn the clear fact that another choice was superior in most ways.

This was extremely depressing because it showed that Data-Driven was just a smoke screen for HiPPO. (Avinash’s acronym) If the executive wants data, but really just wants data that backs up their own opinion, then the results are no better than opinions in the first place.

So what do we do?

I believe there is a better way. I think being Data-Driven doesn’t actually work in the real-world. Here are elements to a new model I just made up called Team-Driven.

Collaboration
I don’t think people work best alone. I don’t think you need one super genius to make all the decisions. It’s disenfranchising to the team and yields results that often are sub-optimal. Pair-programming, pair-designing, Co-owners…people work better as teams. This kind of collaboration requires trust and Radical Candor. (Im listening to that book in the car and think it’s pretty good.)

Important: If you have co-owners of a project, they need to have proven to work well together.  Great things happen in that case. It also bolsters camaraderie and higher productivity.

Iteration
Don’t try and build the right thing on the first try. Actually, build it once and throw it away. Your final version will be much, much better. Most things in programming, products, and design improve with iterations. Plan for it. Stop being in such a rush. Raise more money to iterate. Have a culture where improvements are embraced. It’s not a fail to iterate, it’s a way to learn. I hate the term “Fail Fast“.  How about “Learn Fast” instead? Learning is part of iterating. Do more retrospectives and learn and pour that into the next iteration.  Yes, you need to ship, so don’t iterate forever and ever. Zero is the wrong amount of iteration.

Respect for Expertise
An engineer is trained to understand computer science. A designer is trained to understand user experience. A marketer spent years learning how to generate demand. Good ideas can come from anywhere, but experts should be given some latitude to do their jobs. There is nothing more depressing than someone who has no experience whatsoever in your field making decisions for you.

Important: Collaboration and Respect for Expertise go together. It’s not blind trust. Everyone is expected to participate.

Values/Culture
A good team understands their own style. All too often values are “mom and apple pie” meaningless phrases. A true value is one where you are strongly guided about decisions based on the values. I can blog about this more another day. This post is already too long.

Ok, how do I end this thing…

Data-Driven is flawed. Team-Driven is the way I would do it if I had my own startup. How would you do it?