I think research is one of the most misunderstood and complex parts of the tech design world. There are so many angles, so many facets, and of course, so many opinions. It all adds up to a confusing hairball that is hard to talk about. I will do my best to unravel and break it down.

Domain Research

Becoming Subject Matter Expert (SME) is a key step in being true partners with product management and engineering. If you, the designer, do not understand the details of any given product, then how can they possibly trust you to make decisions? You must understand the details the way the engineers and product managers do.

When looking at some random design, I might ask, “What does this toggle do?” If the answers is “I don’t know, the PM/Eng said we needed it.”, then I know the design should not be trusted. Designers should not be shrugging their shoulders when asked about the details.

SME or Domain research is not about showing UI designs to users. It’s also not about task completion rates or any other usability metric. It is purely understanding the domain of the product or feature and all of the details.

Never put anything in your design spec if you don’t completely understand it. (Rule #1)

At Treasure Data (SaaS for database engineers), I required each designer to take a SQL tutorial online as part of their onboarding. How can they talk about databases if they didn’t under the basic structure and language of that world?

If you get anything from me or this blog, I hope it is this; Designers must always be subject matter experts of their products.

Persona Research

This kind of research is not about the product or particular feature at all. It is about the people, their environments, and their goals. About Face by Alan Cooper (book) does an excellent job of detailing this process. To accomplish this task, you mainly rely on interviews, focus groups, and surveys.

Persona Research is essential for new products or when you are introduced to new personas that you don’t have experience with. Once you understand the people and their psychology, you make personas and move on. This is not the kind of research that should be done over and over, but it is wise to refresh it every few (3-5?) years.

I generally suggest sticky personas to make them easier to talk about. Also, if your executives don’t use them in regular conversations, they will die on the vine. Personas are only useful if you actually use them in decision making.

Competitive Analysis

Whether this is done by the design team, PM or Product Marketing, it is wise to understand how the rest of the world approaches any particular product or feature. Sometimes you want to copy the pattern that everyone uses and sometimes you explicitly want to do the opposite and create differentiation. It’s important to know when to do each. PMs will often just want to copy the competition. In my opinion, most companies don’t emphasize differentiation enough.

A key mental exercise when looking at competitive products is to understand what is good or bad about them and why they chose that approach. Take screenshots and videos and keep them in a folder for future consideration. This kind of research helps PMs and Executives trust the judgement of the design team.

Early UI Exploration

This is the one I think can be most misused. A designer will make a design and then show it to users before it is implemented. This can be done in interviews, focus groups and even surveys. The problem is that decisions are being made based on anecdata (anecdotal data). I believe this kind of research should be exploratory only. Here are some reasons you can’t truly validate a UI by just showing it to people.

1. Hawthorne Effect

People acted differently when they knew they were being watched.

Henry A. Landsberger – 1958

The Harthorne effect is a devious little psychological reality. I have seen it a thousand times myself. People will give answers they think they are supposed to give rather than the truth. They say they like something when they clearly don’t. They say they will buy when they obviously won’t. They say the opposites too! Never trust what users say when they are not anonymous. A related effect is Stockholm Syndrome.

Pay attention to what users do, not what they say

Jakob Nielsen- 1958

2. Recency Bias

Also known as the Serial Position effect. I have seen it happen decade after decade in every situation imaginable. Sales will lose a deal and the reason the customer gave as the reason will turn into the biggest priority. I remember one time Sales was adamant that A/B testing was a key feature for Marketo. We built it and 97% of users never touched it. 3% loved it. Was this the best use of time?

Additionally, when you interview many people over months, most people fall victim to overvaluing the most recent interviews. It is rare to keep track evenly.

3. Confirmation Bias

“The HiPPO is the Highest Paid Person’s Opinion”

Avinash Kaushik – 2006

Since the beginning of business, executives have used data to build the case for their own opinions. This is the dirty secret of all research. Humans are not logical beings. A PM, Executive, or even a designer, will cherry pick data they think best supports their point of view. When people buy a TV or car, they generally want to look at reviews that say it is a good purchase. Confirmation bias and cognitive dissonance are tightly related. We don’t want to believe we are wrong. This is basic human psychology.

Lack of Statistical Confidence

When I first learned about DOE (Design of Experiments), I was introduced to the concept of statistical confidence and the controlling of variables. These are mathematical models to ensure your research isn’t random. If your sample size is too small then you can end up overvaluing minority or even fringe opinions. Most UX Research has low sample percentages.

The right way

If early UI Research is strictly exploratory, then it is just fresh eyes on your design, which is great. However, if you are trying to validate the design is “good”, then you are likely making mistakes.

I think it’s imperative to show your work to many people as you design. Your early explorations should not be a secret. Show your work in short 60 second videos. Meet with various people including sales engineers and show your work. Make it clear that it is early and iterating daily. This is the best way to get good input to make the designs even better.

Usability Research

Usability research happens after a feature is built and can be tested with much higher sample sizes. There are a few ways this happens. One is by watching users in a controlled environment. Intuit in 2006 were masters of this. Unfortunately, you had the same problems described above.

If you have enough traffic and dedicated analysts, you can do true A/B testing with controlled variables. Unfortunately, most B2B applications do not have this luxury. Most applications need to get the feature out the door and move on to the next thing.

The last method is to use quantitative techniques to see what the user is doing. Tools like Pendo, Gainsight PX, and Amplitude are used to monitor user traffic. The biggest problem here is that few designers or product managers spend the time to become proficient in these tools. Additionally, they take significant effort to maintain. I have had serious issues in the past with inaccurate counting of users. I’ll make a post in the future about this kind of analytics.

My best advice here is to think about usability on a scale of 1-100. Most software is less than 50 on my usability benchmark. (No scientific method for that judgement.) However, I feel I can usually design a 75-80 feature without much effort. Getting it from 80-90 takes much more effort than going from 0-80. It’s exponentially harder as you climb up the scale.

I have ranted plenty about how designers are not being taught how to get a desktop application feature to 80% usability. Again, another blog.

Summary

If you work on a product that you KNOW is unusable (You all know who you are!) then I would say that additional research isn’t going to fix your problems. I have seen many companies try to research their way to a better UI and it never works.

I remember Don Norman giving a speech 15 years ago. He was responsible for the research arm of Apple while Steve Jobs was off doing his other companies. he described the best research facilities in the world. When Steve Jobs returned to Apple, they were near bankrupt. One of his first steps was to fire Don Norman and his whole team. He did not replace them with different researchers. He brought in artists and designers and changed the culture to be design driven. In 10 years they created the iMac, the iPod, the iPhone, and the largest tech company ever created.

None of this is to say research is not important. Designers need to do all of the kinds of research above. How can you design something if you don’t understand it?

I will end how I began. Research is largely misunderstood and often misused. I hope this unravels a bit of it.