MarTech » Customer experience » User Experience Smackdown: Usability Testing Vs. User Testing

User Experience Smackdown: Usability Testing Vs. User Testing

What's the difference between user testing and usability testing? Contributor Shari Thurow explains the distinction.

Shari Thurow on November 21, 2014 at 9:38 am | Reading time: 8 minutes

Chat with MarTechBot

Usability testing vs. user testing - image

Throughout my user experience (UX) career, I have encountered many people who honestly, truly believe that their company conducts usability tests on a regular basis. But when I probe for details, I quickly learn the truth.

For example, I was hired to produce wireframes and advanced prototypes for the health section of a major brand’s website. After conducting the appropriate card sort tests to determine the optimal global, local, and contextual navigation, I presented my usability-tested wireframes to the new health manager.

The health manager essentially trashed all of my wireframes and test results because she claimed to have regularly conducted A/B and multivariate tests on another health website with great success. I have seen the end result over the years. Here’s a summary:

The site’s information architecture and corresponding navigation system is based on date and pagination. In reality, patients and patient caregivers do not organize and health information by date and page number.
Supplemental navigation is provided “By Topic” — but topics are not organized in a way that makes sense to users. The topic labels are paginated, and topic labels are not presented in any particular order. Heck, even alphabetizing the topics would make them easier to locate and discover.
Not surprisingly, the health site does not rank well at all (on Google, Yahoo, and Bing) for most of their targeted keyword phrases.

In another scenario, I was at an event where I provided site reviews for usability and the user experience. One attendee claimed that his company worked with multiple firms that offered user testing. However, when I looked at the website, it only had one page. There was no site navigation…only a form to fill out. The page did not provide any benefit to users for filling out the form.

I realized the problem: the term user testing. Somehow, throughout the years, the term user testing has become synonymous with usability testing. Well, Marketing Land readers, I am here to tell you that they are not the same thing. Just because a person or company conducts A/B tests, multivariate tests, and focus groups does not mean that the company is a usability firm.

That previous sentence is a rather bold statement, isn’t it?

List Of Usability Tests

Usability is about contextual task completion. Website usability isn’t about one’s personal opinion. Website usability is not about a “coolness” or “wow” factor. It’s not about forcing your personal beliefs or design preferences onto users. It’s about task completion.

The ISO defines usability as:

The effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments.

As I mentioned in Conversion Optimization: Measuring Usability In The User Experience (UX) – Part 3:

User satisfaction is directly related to task completion. If users can complete their assigned tasks quickly and easily, they often report higher satisfaction, especially if there is an element of delight in the interaction. If users have a difficult time completing their tasks or cannot complete them at all, they often report low satisfaction.

To determine user mental models and the effectiveness of a website or application, usability professionals conduct ongoing usability tests. I don’t use the term testers when I refer to usability test participants.

Reason? What is being evaluated is the website or application, not the person participating in the usability test. If a test participant has a difficult time completing a task, the problem lies in the interface. Perhaps the labels are confusing. Perhaps the functionality is inconsistent from page to page, screen to screen.

Below is a short list of usability tests that professional usability firms typically conduct:

USABILITY TEST	MEASURES	FORMATIVE or SUMMATIVE	NOTES
Brand Perception Test	Brand message	Formative	Identifies the feelings and attributes associated with a website or application. Done with mock-ups of the visual brand. Generally a document that asks test participants to circle the adjectives that best describe each mock-up. Another variation is to show test participants two screenshots. Of the two, which is more (the following adjectives are examples): Professional, reliable, dependable Cool, hip, cutting edge, state-of-the-art Sophisticated, polished, professional Warm, fuzzy, friendly Should be done separately from performance tests.
Closed Card Sort Test	Organization of content; whether or not navigation labels make sense to users; user mental models	Formative	Show test participants a website’s pre-defined navigation or a group of categories. Ask test participants where they would go to find ________________. Focus on: Number correct Amount of time to answer Confidence in answer Often used after an Open Card Sort Test to confirm whether or not the navigational structure is self-evident. Another version of this test is called a Tree Test or Treejack Test.
Expectancy Test	Initial mental model	Formative	Early prototype test that evaluates users’ initial mental models of an interface. The facilitator instructs test participants, “Without using the mouse, keyboard, or hand, please tell us what you think [this] does.” The facilitator can also ask, “What would you expect to find under each category?” Accept all test participants answers without indicating whether or not the answer is “correct”.
Eye-tracking	Actual usage	Formative	Usually used on a detailed design. Answers the questions: What attracts and holds users’ attention? How long did they focus on a specific area of the screen? What is the order in which users looked at items on a screen? Evaluate what test participants scan vs. what they read. Research shows that where test participants look depends on what you have asked them. Please see Eyetracking Studies – 7 Traps to Avoid.
5-Second Usability Test	Initial mental model, aboutness, keyword focus	Formative	Used to evaluate individual content pages, not home pages or category pages. Present a scenario and task to test participants. Show test participants the web page for only 5 seconds. Remove the page it by either covering it up or switching to another window. Ask participants to write down everything they remember about the page. When they finish jotting down their recollections, ask 2 or 3 useful questions to assess whether or not users accomplished the task.
Free Exploration Test	Actual usage	Summative	Use when navigation is not an issue. Observe users at work on a detailed website design or application. For example: “I will give you ~five minutes to freely explore this website. You may go anywhere you like on the site. Please remember to speak aloud as you do so. I will tell you when the five minutes are up.”
Functional Salience Test	Importance of functions	Summative	Give test participants a list of possible functions. Instruct participants to only select 3 functions that are the most important to them.
Open Card Sort Test	Organization of content	Formative	Test participants are asked organize a list of items into categories that make sense to them. Test participants actually name and describe the categories. See the Open Card Sort Example below.
Performance Test	Navigation design	Formative	Observe and listen to test participants as they interact with your website or application. Assign test participants a task with a scenario. Indicate: 2=Completed 1=Difficult or close 0=Fail Identify potential roadblocks encountered and any perceived error corrections.
Performance Test	Organization of content, layout, task flow, controls and content	Summative	Evaluates the navigation design, labeling, and functionality on an advanced prototype: a working website or application under realistic conditions. Present individual scenarios and tasks to test participants. Measure success rate, time on task, and user satisfaction. See how test participants completed each task. Identify potential roadblocks encountered and any perceived error corrections, though there should be less (or zero) of these in summative tests.
Sample-of-One Test	Roadblocks or errors in interface layout, labels, content organization, affordance, and functionality	Summative	A type of performance test primarily used to identify roadblocks and errors. Larry Constantine coined this type of test. Uses only one test participant who fits the primary persona/profile. Test past comfort zone. As with all usability tests, the facilitator must be careful to not ask leading questions or lead the test participant in any way.
Visual Affordance Test	Clickability/Tappability	Formative	Show test participants a web page. Give them a printed version of the web page. Ask them to circle every item on the web page that they believe is clickable. Option: After circling the clickable items, give test participants another printed version of the web page. Ask them to circle every item that they believe is NOT clickable.

Open Card Sort Test Example

An open card sort test is often used by both information architects and usability professionals to determine the organization, labeling, and prioritization of content. A skilled usability professional knows how to measure the results. Here’s a good article: 10 Things to Know About Card Sorting.

Why am I on a rant about usability testing vs. user testing? I listen to numerous sales pitches. I hear people claim that their site does ongoing testing without specifying the types of tests that are conducted. I observe people using the wrong methods to collect the wrong data, such as using keyword research tools to determine a website’s information architecture.

Use this list to help you determine whether or not you are working with a true usability professional or organization. Usability is a critical facet of the user and searcher experience. You can delight or entertain your website users all you want, but if they can’t complete their desired tasks? If they can’t locate or discover desired content? That is not much of an experience.

Okay, end of rant.

Contributing authors are invited to create content for MarTech and are chosen for their expertise and contribution to the martech community. Our contributors work under the oversight of the editorial staff and contributions are checked for quality and relevance to our readers. The opinions they express are their own.

Add MarTech to your Google News feed.