All Articles

Articles

Cover Image

10 Bad User Research Practices You Will Want to Avoid
April 30, 2021

Some might think user research is as simple as watching people perform a few tasks on a website or asking them a few questions, but user research is definitely a not walk in the park.

When done right research leads to evidence-based decisions and the design of great services and products. However, there are a lot of ways user research can go wrong that will lead to low-quality data, wrong conclusions and ultimately to the development of a product destined to fail.

Let’s go through some of the mistakes that can arise when planning and conducting research.

1. Setting unclear research objectives

Before you get into timeframes, budgets and recruitment you need to make sure you understand what you want to find out. The “why” of your research. The questions you want to see answered. Objectives will, of course, depend on which stage of the product development you are in — discovery, testing and validating or post-launch.

Good research questions are specific, actionable and practical and very importantly — research questions are not interview questions. They drive the entire research project so they should be written effectively. Let’s take a look at a few examples:

  • How do people currently use the favourites feature on our website?
  • What actions do our customers take when they’re thinking about buying a new fridge?

Objectives will help you to determine which research method you will need to use to get the answers you need and from then your tasks and interview questions.

2. Choosing the wrong research method

As we’ve seen before, the first question you should ask is — what do you want to know? Do you need to find out who are your users or perhaps how are they using the product? Once you know what you’re trying to learn, you can start thinking about how to get the answers you need.

User research can be done at any stage of the design and development process and there’s a wide range of research methods, the difficulty might be to know which to use and when.

Let’s say you’re in the discovery stage of your project. It might be useful to conduct a field study or interview stakeholders. If you’re in the testing stage then you should consider conducting usability tests (in-person or remotely) or an accessibility evaluation.

Some methods might be more appropriate than others. It will depend on factors like time constraints, the type of product or service or the goals and types of insight you want to get. Most projects will actually benefit from multiple research methods so the right method should be based on the project’s circumstances.

At NNGroup’s website you’ll find a comprehensive chart with all UX methods and activities in various project stages.

3. Not designing the right tasks

In a usability-testing session, a facilitator asks participants to perform a series of tasks. Those tasks are actions a real user would typically carry out in a real context of use. The tasks we ask participants to perform during a usability test can seriously affect the results. They must be pivotal to the service and represent business value.

Observing the users flow through the design gives us more actionable insights than asking them if they like it or not. With the right tasks, you’ll be able to identify whether the interface causes frustration or confusion, where users encounter problems or hurdles to overcome.

In order to engage participants you should provide realistic task scenarios. That means that instead of telling users to do something with no explanation — “Buy a pair of running shoes in XSports website”, for example — it would be better to provide the user with some context for the action that explains why they are doing that — “Imagine you want to start running with a group of friends two times a week and you need a better pair of running shoes. You don’t want to spend more than 50€. Go to XSports website and show me how you would buy them.”

4. Recruiting the wrong participants

One very common mistake when recruiting research participants is to rely solely on demographics (such as age, gender and income). However, to obtain the most insightful, actionable data, participants should represent the actual user group of the final service or product as closely as possible. Recruiting the wrong users can lead to capturing the wrong opinions, needs and problems.

If you’re designing a university app, then you should conduct research with the ones who will actually use it — the students. If you’re developing a system to be used internally by the employees of a company, you should take a sample of its future users — the workers.

That’s why it’s extremely important to find and recruit representative users. You should determine which user groups to include in the testing and identify their characteristics in order to get the right screener (a survey used to qualify potential participants).

5. Not carrying out a pilot test

When you’re working on a short timeline, it might be tempting to skip a pilot test. However, it’s crucial to do it as it will help to assess if the test is too long, if the tasks are understandable, if the order of the tasks makes sense or if questions are repetitive.

A pilot test will help to uncover any problems with the test script or the prototype that you can then fix before real testing begins. Ideally you would conduct the pilot test with a real participant but it’s perfectly fine to ask a co-worker to play the part of the participant.

6. Juggling note-taking and facilitation

Facilitating is a demanding job. You have to know what to ask next, make the participant feel comfortable and talkative, maintain eye-contact, help them to stay on track with the tasks, decide whether and when to ask probing questions.

The ideal is to have a note taker observing the test session to ensure the facilitator can be 100% focused on guiding the participant through the process and that no insight is forgotten.

7. Failing to record research sessions

Even a good note taker can miss things. It’s hard capture all insights during a user interview and it becomes even harder in a usability testing session when you’re not only supposed to note down the user’s opinions but also their behaviour when using the website or app.

Does the user tap or swipe? Where do they click first? Do they hesitate? The best way to ensure none of those details escapes is to record user’s actions by recording the screen and their hands, not forgetting to also record their voice.

Important to know! In some countries, specially in Europe where data protection laws are quite strict, it’s mandatory to inform participants beforehand about filming, sound recording and eye-tracking. They should be asked to sign a data collection consent before starting the test session.

8. Conducting biased user research

One great example of a biased user research is asking leading questions. A leading question encourages the person to give a particular answer. They should be avoided in research as they influence the way participants think and imply what kind of information you expect to confirm. Here’s an example:

If you ask a participant — “What do you dislike about feature X?” — you’re prompting them to think negatively about that particular feature and consequently include negative aspects in their answer.

One the other hand, if you ask — “How good is feature X?” — you’re prompting the participant to answer only in relation to the feature’s usefulness.

User research is about uncovering truthful and honest feedback from users, not leading them to tell what you want to hear. The solution here is striving to remain neutral and ask open, non-leading questions instead.

If you ask — “What do you think about feature X?” — you’re not suggesting anything about the feature or prompting the participant to think about it in any particular way.

Then this thing called confirmation bias. People seek out the data that supports their own opinion and confirms their assumptions while discarding what goes against it. This is quite a dangerous bias in user research. You might discard pain points that users are struggling with because those points don’t fit your existing assumptions. For example, if you designed a navigation system that looks pretty logical to you, if you hear users complaining about it you might tend to devaluate their feedback. 

9. Asking users what they want and blindly trusting their feedback 

In user research it is not a good idea to relying too much on what participants say. To understand users needs, more than simply asking them what they want, you should understand them, their tasks, tools and environment. You can do that by observing them while they perform their daily tasks in their natural environment. While observing users is key it is also important to understand what them are doing by asking them.

Most users have no clue of what they want. Sometimes they don’t know what they want until they actually accomplish a given task in the product. So instead of asking what they want and need you should be able to get that information by observing users completing their tasks, discuss what they are trying to achieve and ask them about the difficulties they come across.

10. Rushing through analysis and not documenting your findings

After spending time and money on research, teams might feel tempted to rush through the results analysis and jumping in to making the necessary changes to move on to the next step in the project. They might end up simply having a meeting to talk about what you’ve found. The problem is a meeting will simple not do the trick.

Analysing and making recommendations takes some time, specially if there’s a large number of participants, if the test script comprises several tasks and questions or if the user interface is complex. The research project plan should include a reasonable amount of time to analyse all the findings. If the team is eager to know about results, you can present them a document with the initial findings, and then move on to creating a deliverable documenting all findings and recommendations, as sometimes they are difficult to describe without visuals. That document will ensure that even the people who couldn’t attend the meeting will know the results and there will be a permanent record of your findings, guaranteeing that information will not get lost.

Gathering reliable insights to build great experiences

More and more organisations are focused on creating great experiences for their users. User research plays a big part in that. All insights gathered can provide input and information to help the team make the right decisions when designing a product or service.

However, some teams still can’t get the best out of user research. Maybe because they get caught up by time constrains, they don’t have the necessary resources, they lack in-house capability or they don’t have enough money to hire a research partner. That might lead them to conduct research that doesn’t produce useful and actionable outcomes.

Spotting the most common mistakes in user research will help us learn how we can plan and conduct better user research that leads to reliable data and informed design decisions.

Want to know more about how to conduct research that provides meaningful insights? We strongly recommend checking out these books:

Need some help with your research project?

Xperienz has a team of experienced UX researchers ready to help you uncover useful insights about your users’ needs, behaviours and motivations. We apply several qualitative and quantitative research methods, including usability testing, ethnography research, surveys, diary studies, collaborative workshops and remote research.

Tell me more

Related Articles