How to perform Usability testing?

Preparing a test plan

The first step is to create a test plan. It is a very important step to determine what we are testing, what we are going to give to our users. Before preparing a test plan we need to:

  • Define what areas to concentrate on
  • Determine potential usability issues
  • Write a test plan
  • Determine what tasks will be tested

Website Usability Testing: How To Get Started Today | Hotjar Blog

A Test plan should include
  1. Objective
  2. Test task (scenario)
  3. End state (answers)

Well-formed task scenarios make smoother tests. Here are some quick tips for you to ponder over-

  • You(motivation) X…
  • What is… which X… how can…
  • Use specifics to see if they get (e.g. is X available?)
  • According to X(our app) what…
  • End with a question

Sequence the task in the order of easy (orientation tasks), difficult and moderate so that users don’t get frustrated at the beginning of the test. This can happen in case they face any difficulty in understanding their role.

Interact with participants before test starts

  • Explain to your participants that the objective is to test the software and not the participants’ intelligence. This is important as the participants may feel that their ability is being tested.
  • Explain how the test material and record will be used
  • Encourage participants to think aloud while using the product
  • Prepare a pre-test questionnaire and a post-test questionnaire

Why Usability Testing is important for your Web or Mobile App? - Zignuts Technolab

Prepare a checklist for Usability Testing

  • Success rates
  • Time on task
  • Errors made in performing the task
  • Confusion(unexpected user actions)
  • System features used / not used
  • System bugs or failures

During the test

  • Record techniques and search patterns participants employ when attempting to work through a difficulty, for recording we can use software like ScreenFlow or silverback (mac only)
  • If participants are thinking aloud, record assumptions and inferences being made.
  • Do not interrupt participants unless absolutely necessary
  • If participants need help provide some responses:
    1. Provide encouragement or hint
    2. Give general hints before specific hints
    3. Record the number of hints given
  • Watch carefully for the signs of stress in participants:
    • Sitting for a long time doing nothing
    • Blaming themselves for the problems
    • Flipping through documentation without really reading them
  • Provide short breaks when needed
  • Maintain a positive attitude, no matter what happens
  • Use a monotone tonality with users i.e. never let users know that you are super excited or your body language indicate them that they are doing bad and things like that.
  • In case users ask questions to try to get your consent if they are doing it right never give them direct answers instead use words like “Okay, uh huh”. Instead, ask reverse questions like “Is it what you were looking for”
  • Let them struggle and don’t over moderate

After the test

  • Hold a final interview with the participants and tell them what has been learned in the test, and if you noticed any discomfort in users or they gave a signal of confusion through “think aloud” during the test ask them more about such things.
  • Provide a follow-up questionnaire that asks participants to evaluate the product or tasks performed. The post-test questionnaire is generally used to gauge what users think of their performance. We should also include the questions like “What is the one thing that you would change…?”.
  • If video recording is required then get a written permission and respect participant’s privacy
  • Do not generalize the opinion unless 5-8 users indicate the same problem

Analyzing the collected data

  • Video Data
    • In less formal studies, video can be very useful for informally showing managers or disbelieving system designers exactly what problems encounter.
    • In more formal studies two types of analysis can be performed: Task-based analysis and Performance-based analysis-
      Task-Based Analysis is used to determine how user tackled the task given where the major difficulties lie and what can be done
      Performance analysis is used to obtain clearly defined performance measure from the data collected (task timing, frequency of correct task completion, frequency of errors, productive vs. unproductive time)
  • Analyzing Questionnaire
    • The quantitative type of questionnaire is often used when statistical data is required to emphasize the results of the test.
    • These questionnaires have some form of rating scale associated with it.
    • There are a number of different rating scales-
      • Three-point scale i.e. “Do you know how to copy a text” YES, NO, DON’T KNOW
      • Six point scale i.e. “Rate on the following scale” 1 to 6 from “very useful” to “of no use
      • Likert scale i.e. “The help function in this system gives solutions to complex problems”, “Strongly Agree, Agree, Slightly Agree, Neutral, Slightly Disagree, Disagree, Strongly Disagree”
      • Semantic differential scale

Once the questionnaire has been given to the selected population, the responses obtained on different rating scales are converted into numerical values and statistical analysis is performed.
Usually, mean and standard deviation from the mean are the main statistics used in the analysis of most survey data.
The questionnaire has to be designed as easy as possible and keep it short. Try to aim for no more than 2 sides of the paper

Strategies for analyzing the test results

Deciding upon the strategy to adopt will depend on the circumstances of the usability testing, including the time available and main objectives of the testing.

Top-down Analysis

Top-down analysis begins with an overall assessment of the user interface from the questionnaire, summarizing the major strengths and weaknesses.
This overall assessment can be collected from the results of the set of general questions on system usability

General questions on the system

Please provide your views on the usability of the website by answering the questions below. There are no right or wrong answers.

  • What are the best aspects of the system for the users?
  • What are the worst aspects of the system for the users?
  • Are there any parts of the system that you found confusing and difficult to fully understand?
  • Were there any aspects of the system that you found particularly irritating although they did not cause major problems?
  • What were the most common mistakes you made during using the system?
  • What changes you would like to the system to make it better from a user’s point of view?
  • Is there anything else about the system that you would like to add?

The answers to the general questions may highlight particular aspects of the interface causing problems, and a more detailed investigation may then be carried out to examine other instances where those aspects have been highlighted within the questionnaire.

Bottom-up analysis

A bottom-up analysis is a more detailed analysis that investigates the responses to each question within the questionnaire. This type of analysis will enable a more comprehensive picture of the interface to be generated. It is likely to yield a highly detailed specification of aspects of the interface requiring improvement, amendment, addition etc. A detailed analysis of each of the criterion-based selection will enable a summary to be drawn from the interface in terms of each of the criteria.

Importance of Usability Testing

When Lyla was asked if she could interview an author on her colleague’s behalf, she jumped at the opportunity. Lyla was interning with an uptown magazine for a role that deemed her ‘unfit’ for any important tasks such as interviewing a popular author. Getting an opportunity to talk to her favorite author was like a jackpot. She thought she is finally gripping onto a childhood dream. She thought she finally can brag about meeting Paula Hawkins. She wanted to be careful about not screwing this golden opportunity, so she double-checked everything on her list.
Notepad, check. Pen, check. Camera, check. Questions, check. Recorder, check! Oh wait! I have to check for a new update, just in case I am outdated.
Which Usability Testing Method Should I Use in 2021? | PlaybookUX

She updated her app quickly and reached the coffee shop before time. She was excited, she was anxious. It’s an easy-peasy task, you just have to ask questions, record them and pour it out later– she kept repeating this in her mind.
But the moment Paula Hawkins walked in, she started feeling the breathlessness. Part excited, part nervous, she could feel her palms sweating. After exchanging some pleasantries they kicked off with the interview.

“Shall we start?”
“Yes, of course. Let me just..umm… Start this thing”
*she clicked on the record button and started talking, simultaneously taking notes.*

Unaware of what was happening after she clicked on that little red button, she went on with the interview, taking down notes as per her whims. Ignorant of the new updated feature — which now needed ‘long press and hold to start recording’, she finished the interview, smiling and pleased with herself. But as she looked down, in order to ‘STOP’ the recorder, she lost the fanatic feel of doing a good job!

A little hover over the button said “long press and hold to start recording” which was an obvious slip.
Game over. She probably lost her mind or her job. Or both. But that’s none of our concern. Right? She should have been more attentive.

Lyla is a fake person. She did not meet Paula Hawkins ever. But the incident is real. Only, it happened with an acquaintance and I figured out it wouldn’t be fair to name her. This incident forced me to shine some light on the importance of usability testing and why it’s more important to pay attention to user’s behavior while they interact with the product.

They say, never talk to your users, they hardly know what they want. But they missed out on the summary-

“To design the best UX, pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior. Users do not know what they want.”

Usability testing- What is it?

What is usability testing? - Quora

Usability testing is like black-box testing of an application to ascertain if the product built is convenient to use and easy to learn.

These are methods of testing and observing the behavior of the users to find out what works and what doesn’t work. Users are given specific tasks to complete and when they are at work, observers watch their body language, facial expressions, emotions and encourage them to “think aloud” i.e. speak up whatever comes to their mind while using the product. Doing this exercise we can get qualitative and quantitative data and figure out usability issues with a product.

So, why usability testing is important?

Website Usability Testing: How To Get Started Today | Hotjar Blog

Doing usability testing the right way, at right time with the right set of people reduces the risk of building the wrong product; thereby saving time, money and other precious resources. In other words, if done at an early stage when the product is at paper prototyping stage, it finds the problems when they are easy and cheap to fix. And when done on a product which has attained maturity it helps to understand user’s success rate and time spent to complete a task. There are hundreds and thousands of cases when usability testing proved to be a good exercise in terms of ROI.

For example, a slight tweak in design suggested by usability testing for Mac’s UI, the company got 90% fewer support calls.

Need more clarity about why usability testing is a good idea? Here you go-
#1. To check if product meet user’s expectations
#2. Matches business decisions to real-world use
#3. Removes flaws in the product
#4. Allows you to see how successful users are with their tasks
#5. Useful for getting user reactions and feedback about the product

What are the types of data that we can get as a result of our analysis?

Two types of data results received are — quantitative and qualitative. Usability testing is largely a qualitative research technique and is not driven by statistics like surveys where lots of people participate. Usability testing is done using a small set of people, usually five to seven.

Qualitative methods are very useful to test the stress response of the users like their body language, movement of hands, expressions on the face and squinting eyes especially doing a test on a mobile device.

The metrics we get after usability testing can be quantitative as well. For example, time spent on doing a task, success and failure rates and also the efforts, like how many clicks a user needs in order to complete a task.

Is there a need to record all the metrics obtained from a usability testing?

Yes, keeping a record of the metrics is very important. Why? Because usability testing is not just for designers to understand how to make better designs but it is also an important tool to influence the rest of the stakeholders like clients, their sales/support team, project managers, developers, other designers etc.

Every stakeholder involved may have a different point of view for a design decision. Being subjective by nature, design decision often leads to long debates among stakeholders. Most often design decisions are influenced by a person who holds the highest position among fellow stakeholder or has superior oratory skills.

In short, metrics help us in iterating and validating design concepts. It gives objectivity to design debates and it helps in taking fact-based design decisions.

At what phase of the design process usability testing is recommended?

When it comes to usability testing there are two terms often referred by big names of the UX industry (like Jacob Nielsen) and these terms are Summative Test and Formative Test.

These tests are done at different stages of the design process. They are as explained below:
Formative tests are low-fidelity tests (to gain quick insights)-
#1. During the very initial development phase using paper prototypes
#2. It can be done anywhere and a formal lab is not required
#3. It can be done just between a moderator and a participant

The results from a formative test may include-
#a. Users’ comments in the form of “Thinking Aloud” narrative i.e. their emotions, confusion sources and their reasons for actions.

Summative tests are high fidelity tests (to capture metrics)-
#1. These are carried out at the end of the development stage
#2. At this stage usability of a product is validated
#3. This gives an answer to the question “How usable the product is?”
#4. This gives a comparison against competitor products
#5. Conducted in usability labs or remotely using many tools available where users can do the test using their computers or mobile phones

The results from summative tests may include-
#a. User’s success rate to achieve a goal
#b. The time spent on completing a task

How many users are required to conduct the testing?

“Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests you can afford”
Jacob Nielsen

“It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.”
Laura Faulkner

Who among them is right?

It depends on what type of test we are doing and where we are doing it. For example, if we are doing a low-fidelity formative testing we can do away with a small sample size. But if we are doing a summative testing we need a bigger sample size. In the type of testing where we are comparing our site to competitor’s website by using an online tool which is cheap and fast, we can use a large sample size. But we should keep in mind that these online tools like UserTesting or Loop11, they don’t capture metrics. It’s us who has to be aware of how all the participants did it.

So how to prepare a test plan?

That is certainly a good question. You have an inquisitive mind, I must say. But don’t you think that will be too much to digest in one bite?
Still, if you are really into it, give these things a thought-
-Decide what areas to concentrate on
-Determine potential usability issues
-Determine what tasks you want to test

error: Content is protected !!