Design is decisions. How do you make those decisions? There are many competing schools of thought. I’ll break them down and give you my experience with them all.
Focus Groups
There is a famous story from Sony.
Sony Corporation asked teenagers in a focus group which color portable radio/CD player they preferred–black or yellow? The overwhelming response was “yellow.” At the end of the three-hour session, the teenagers were told to pick out their choice of a free boom box as a gift for participating in the focus group. The majority walked out with a black radio. The key to client intention is not what people say, it’s what they do.
I have seen this many times in other forms. The reality is that people are terrible at communicating their own future/past behavior, likes/dislikes or even how they currently use something. I have often described user research as Dian Fossey observing the gorillas. The best thing to do is to watch their behavior, not to talk to them.
At Marketo, I attended a focus group that was conducted by the VP of Marketing. The attendees were people who would likely use the product. (The target market) At the time, we had nothing to show them. The VP asked one question that I thought was pure brilliance, “What was the most awesome thing you have ever done in Marketing.” The answers told me everything I needed to know to design the “flavor” of the application.
Quick aside of one of the answers: (From a Director of Marketing)
Back in the day, our company was really tiny. We put an ad in a magazine that said we were 28% better than the big (Fortune 500) competitor in the space. The ad wasn’t doing much good. We received a letter from the big competitors lawyers saying, “Cease and Desist! or we will sue your asses!” (Paraphrased)
Well, I took every dollar of marketing money I could muster and we put the ad everywhere, and we upped the percentage to 78%! The ad didn’t do much for sales this time either.
However, the big company sued us. As you might imagine, we had a PR field day. (bright eyes, bright smile) We got so much attention because of the lawsuit that sales went through the roof. We built the company off that attention. (Stretches back with hands behind head)
His most awesome thing in Marketing centered around a shrewd decision he made. It had nothing to do with emails or technology. It was a personal choice that showed him to be clever and prescient. I realized that the system had to give this kind of person direct controls over the flow of events. They didn’t want a system that did marketing for them, they wanted something to help them do it themselves.
Anyway, the point is that the focus group was successful because I was learning about their personalities and their quirks. I didn’t try to say, “What kind of system should I build?”. Focus groups are great for discovery and exploration.
On the flip side, Focus Groups are terrible at validation. Unfortunately, this is the way most people use them. They show them screenshots and ask them if they like it. They try and validate whether their product is going to make them money or not. This is a deadly strategy with terrible results. Liking a product in a focus group doesn’t mean they will choose your product when it comes time to sign a check. Even liking a product in the group doesn’t mean you will actually like the product later on.
Generally, I think focus groups have rotten ROI. Just go visit some people and see how they do their jobs. A good focus group is a whole event. You need to plan it, write good questions, get food and wine, etc etc. If you have all that extra energy to put this together then maybe you should work on something else more important? However, if you do it right, you can get fantastic results. I know this is a mixed review, but I have seen far extremes in this field.
Whatya think?