"You're on Mute" - Lessons Learned After a Year of Conducting Remote User Research
July 21, 2021
Back in March 2020, the circumstances forced us to quickly adapt to a new mode of working — remotely. That change couldn’t slow our user research work down. Unfortunately, the research setup we were used to, with our own fully-equipped research lab, could no longer be an option so we had to adapt and reframe to a new remote reality.
Throughout the year we conducted diary studies, individual and collective interviews, and usability tested prototypes for both national and international projects. We ended up learning a lot with each one, perfecting and refining our remote methods, tools and tasks.
Now, after more than one year of engaging with users remotely, we want to reflect on the pitfalls of remote user research, share some of the lessons we learned and reflect on what’s going to be “the next normal” after Covid’s impact.
1 — No control over participant’s environment
We were used to conducting research in our in-house research lab. We knew we had good internet connection and all equipment ready and working for the session. The research room was a quiet place. We knew we weren’t going to be interrupted by a door bell ringing or a dog barking. We had almost full control over the environment. When moving to remote research we simply lost control over the conditions in which research takes place. Participants can join in from anywhere, from their home, to their workplace to a tree shade in the middle of the countryside (yup, true story 😅).
Lesson Learned #1
It’s pretty much impossible to control where our participants will join us from, the noises they have in the background (often hammering noises from construction work nearby or kids wanting attention), their wi-fi connection, etc. In such cases, the facilitator has to assess the situation and decide whether or not the interview has the necessary conditions to continue.
Even if you try to validate the participant’s conditions beforehand, there is no guarantee that on the research session day they are not elsewhere or with different conditions. The best thing to do is validate right before the beginning of the session.
One of our participants joined the interview in the middle of the countryside, under a tree, with his computer on top of the car hood. Of course, his internet connection was not ideal and we had to call off the test.
Sometimes asking the participant to move closer to the router or change to mobile data might solve the situation. Other times their internet connection is so poor it’s impossible to move on with the session.
No matter the issue, the facilitator’s decision should be based on how the research will be impacted, on how long the issue seems like is going to last, and if the session is going to provide the necessary insights.
2 — Participants might be tech-savvy or not so much
Research participants have different backgrounds and ages so they might or not be at ease with video conferencing tools. Fortunately, the increase of remote work over the past year and the need to find new ways to keep in touch with friends and family has made people more used to tools like Zoom, Microsoft Teams or Google Meet. They’ll likely know how to install the necessary app, join the call and share their screen.
In one of the sessions we conducted, a participant was having trouble using Zoom and who was there to help him with the technical setup? His young daughter, around 11 or 12 years-old, who clearly knew more about it than he did.
Lesson Learned #2
In the first remote research sessions we conducted, we used to start with a set of introductory questions to make the participant feel more comfortable. Only then we moved to the prototype testing, when we asked them to share their screen so we could see what they were doing in the website or app we were testing. However, we’ve rapidly come to the conclusion that before making any questions it would be best to take a quick tech check, where we ask participants to share their screen. This allows us to make sure that everything’s working and that they’ll know how to do it when the prototype testing begins.
In the recruitment phase, the technical requirements for the session must be clearly explained. We share the Zoom link to the interview beforehand, inform participants whether they should join with their computer or smartphone or if they’re supposed to share their screen or not.
3 — Time management becomes more challenging
Participants running late, technical problems, distractions, background noises are just some of the things that might cause a remote research session to take a bit longer than a normal in-lab session.
Also, not everyone is a talker. There are those participants who only answer exactly what you ask them and often need a bit more prompting to share the information you’re looking for. Then there are those chatty ones who provide feedback freely with little prompting, and even go off-topic sometimes. In this last case we risk the session being longer than anticipated. The facilitator has the somehow difficult task to find ways to make the participant focus on what’s being asked or might even have the need to cut off a few questions to stay within schedule.
Lesson learned #3
When you have a full-day of remote sessions, it’s important that you make room for at least a 30 to 45-minute break between them. This will ensure that any delay with a session will not affect the next one and that the facilitator and the note taker can have some resting breaks in-between sessions.
In order to save some time during the session we also ask participants to fill the consent form beforehand.
4 — Take your script for a test drive
Imagine you’re 10 minutes into the remote session, you’ve done a few introductory questions, and you’re now moving on to the prototype testing. As the participant tries to accomplish the tasks you realise some feature in the prototype is not working properly or that you’re making irrelevant or repetitive questions.
That could all be avoid with a pilot test. It will help to identify whether you’re asking the right questions, if your prototype is working properly, and if your tasks fit the time you have for the session. It can easily be done with a colleague over a video call.
Lesson Learned #4
Sometimes we’re the ones writing the test script, other times the client sends us the test script for us to translate and apply. In either situation we always conduct a pilot test. It has helped us to remove repetitive questions and, most importantly, check if the prototype is working properly or not. Running it a couple of days before the sessions gives us enough time to get back to the client with any changes or clarifications needed.
5 — Always take notes
Don’t rely solely on video or audio recordings. Technology has its set backs and might not always work as expected. The old good pen and paper (or an Excel doc) will be your best ally to write down what you hear and observe. The better your notes, the less time you’ll spend reviewing the session recordings. You might only need to refer back to them if you missed any detail or want to confirm something.
Lesson Learned #5
We always have a note taker backing up the facilitator. This way the facilitator can focus entirely on conducting the test, do the necessary follow-up questions and assist the participant with any technical difficulties.
Our preferred tool — Zoom
There’s a wide variety of remote video conferencing and collaboration tools but we ended up realising Zoom was a good fit for us. Instead of a Zoom Meeting we found out it would be better to create a Zoom Webinar for our usability tests and user interviews. The facilitator starts the webinar and enters as a host. Then the participant enters as a panelist, which allows them to share their video, audio and screen. All team members (like the note taker) and the client team, enter the webinar as attendees. They don’t have their video or audio on but can see and hear everything, without disrupting the session.
One of the things we found out was that the client team would sometimes end up talking with each other using the Zoom chat. The problem here is that the participant will actually be able to read those messages, which can make them feel more uncomfortable or deviate their attention during the session. The solution then is to change the settings to “Allow attendees to chat with no one”.
One of the Zoom features that turned out to be a huge plus for our team is language interpretation. We’ve had a few research projects with foreign clients that wanted to test the Portuguese and Brazilian markets so having a live interpreter allows them to observe and understand what’s being said. You can easily enable this feature when scheduling a webinar by adding an interpreter and selecting the language they’ll translate. Whoever is observing can select the audio channel of the language they want to hear. They also have the option to mute the original audio or hearing it in a lower volume with their chosen language.
How UX research is moving to the “next normal” after Covid’s impact
As things gradually open up globally, returning to face-to-face research might be too soon but eventually things will get back to normal. The pandemic will undoubtedly influence how people decide about the activities they do and the places they go for the next months, or even years.
The UX Matters report “Going Back to Face-to-Face Research After COVID-19” found that, as of June 2020, 71.8% of UK participants would be comfortable doing in-person research. Older people are less willing to participate whereas key workers are the most likely group to want to participante in face-to-face research. They concluded that even if people are currently uncomfortable with in-person research, they’re likely to be ready to participate soon.
Covid-19 risks vary considerably from location to location, and sometimes even from one week to the other, so the return face-to-face research might be a closer reality for some countries than for others.
For a safe return to our user research labs we’ll have to provide a safe environment based on the recommendations of local health officials. That will probably include a detailed hygiene protocol (cleaning and disinfection of facilities and hardware), personal protective equipment (face masks), social distancing and screening questions specific to Covid-19 health and safety.
User research is possibly more relevant now than ever. More people are now grocery shopping online, using their bank’s app, ordering their dinner through food delivery apps. Globally, 49% of consumers shop online more now than they did pre-COVID-19.
Businesses are then seeing a new users increase, who are possibly less digitally experienced, older, with different needs and wants. For that reason it’s crucial that businesses understand these new users and build the best experience to retain them as customers as the rise of online activities is likely to outlast the pandemic.
For now, remote research methods seem to be the lowest-risk option for both participants and researchers. Although it doesn’t replace in-person research entirely, remote user research is a fast and reliable way for businesses to get the insights they need to build better products and services.
Planning a remote research project?
Using screen and voice recording software, Xperienz can run remote interviews in Portugal or anywhere the world. We can moderate the sessions, develop unmoderated testing and build effective surveys.
We’ll recruit users to match your screener, ask the right questions, analyse and evaluate all the findings, and come up with an action plan to improve your product.Tell me more
"You're on Mute" - Lessons Learned After a Year of Conducting Remote User Research
After more than one year of engaging with users remotely, we want to reflect on the pitfalls of remote user research, share some of the lessons we learned and reflect on what’s going to be “the next normal” after Covid’s impact.
Quick & Dirty User Research
Tight timescales and budgets are no excuses to ditch user research altogether, specially when we all know it’s essential to make sure you deliver easy-to-use products. Quick and dirty research is a great way to get user insights fast and on a budget.
How bad metrics are hurting your business and your users’ experience
Businesses are deceiving themselves and annoying their customers as a consequence. They do so when they apply biased surveys only expecting to confirm what they want to hear.
UX Writing — Create better experiences with better content
Imagine a website or an app with no words. If it wasn’t for the logo, would you be able tell what this page is about? Would you know which button to click? Where navigation would take you? What you’re supposed to write in the search bar? No matter how good-looking an interface is, without words users will simply not be able to accomplish any tasks in it.
10 Bad User Research Practices You Will Want to Avoid
Some might think user research is as simple as watching people perform a few tasks on a website or asking them a few questions, but user research is definitely not walk in the park. Let’s go through some of the mistakes that can arise when planning and conducting research.
UXLx Masters — Wrap-up
From 10 to 13 February attendees from 25 countries and 14 world-renowned UX experts joined online for 3 days of learning. The programme included 12 live masterclasses, 2 keynotes, 2 live podcasts, and more.
Remote UX Research — our selection of the best online tools to conduct it
As a company that focus on UX research and design, we gathered some of the best tools to conduct remote research and combined them, with our personal knowledge, in this article.
Health Habits during the Lockdown
Xperienz, along with 15 other agencies from the global network of user research companies UX Fellows, conducted an intercultural study in 15 different countries about health and wellbeing during the lockdown caused by the current pandemic situation.