Creating Effective User Testing Questions

Creating Effective User Testing Questions

If you have great questions to ask your users, you will get great answers to apply to your testing goal. If you work for a business that has limited resources, getting the best feedback is important the first round testing, thus saving money.

Understand the Goal

Every user test needs to have at least one goal to be effective. If you work for a company, ask the product owner, product manager, or leader what the goal for the test is. If you work by yourself, come up with a goal that will allow you to compare and measure the results.

Once you have the goal(s), understand what that goal means. For example, if your goal is to determine if there are any usability issues on an interface, you can use quantitative data to ask the user if a particular task was easy or difficult.

Think Like a User

The best advice for writing great questions is to put yourself in your user’s shoes. Pretend you are a user that takes things literally and have never seen your product nor have any idea how it works. Ask yourself these questions:

  • How did I get to this product?
  • What is this product for?
  • What am I supposed to do?
  • What will I get?

Knowing the answers to the questions will help you write an introduction for your user to get in the mindset of your test.

Example:
You were browsing your social media feed and see a title that looks interesting, so you click on it and come to our website.

Use a Prototype or Mockup as Reference

Always have access to a prototype or mockup when writing the questions for the user test. Having a visual of the interface in front of you will allow for you to see the path you want the user to take, and this will easily allow you to write the questions and the actual tasks you want them to perform.

Readily having the mockup or prototype available will also prevent you from accidentally skipping over an important step or feature of the interface when writing the questions.

Write Questions with Clarity

Here are some great quick tips to help you write clear questions.

Use Layman’s Terms

Please don’t use industry jargon when writing the questions! Your users will not understand the question clearly and will not provide good feedback.

Example:
No: What are your thoughts on the interstitial on the application?
Yes: What are your thoughts on the pop-up with the discount information?

One Question at a Time

Keep the question focused to one thing. Do not pack in several things into a single question as this gets too difficult for the user.

Example:
No: What is your opinion on how the search works and the way the results are given to you?
Yes: What is your opinion on how the search works?

Unbiased Feedback

Write questions in a way so you won’t get biased feedback from your users. Write your question as neutral as possible.

Example:
Wrong: Was it hard to find the search box on the site?
Right: How easy or difficult was it to find the search bar on the site?

Get the Responses You Need

When writing your questions, you have two types of feedback based on the types of questions you write:

1.) Quantitative (can be easily measured)
2.) Qualitative (influenced by feelings or emotions)

To achieve the user testing goal ultimately, it is ideal to have a healthy mix of both quantitative and qualitative feedback from your users.

Quantitative Questions

Use the following types of systems to write your questions for easy to measure results from your users.

  • Rating Scale (levels of difficulty, clarity)
  • Yes/No (completion of task)
  • Multiple Choice (to prevent a wide range of answers)

Qualitative Questions

Qualitative questions are best if you ask for the user to talk out loud what they are thinking or to write their answer and talk out loud as they do. Your best feedback will come because the user gets to speak their mind and express areas on your prototype that may be confusing or cause friction.

Examples:

  • Find a class that is convenient for you. Talk out loud as you demonstrate this.
  • Is there any other way you would find a class on this page?
  • Was there anything that seemed confusing or difficult while you were searching for a class? Please explain out loud as you type your answer.
  • What frustrated you most about this site?

Avoid User Test Frustration and Issues

If you have ever conducted a user test and when going over the results find that there was confusion on the instructions or questions, there is nothing more frustrating than seeing your user wander away from the page or area you want feedback on.

Here are some helpful tips to keep your users on track and get the results you need.

This is Just a Test

Sometimes you have to let your users know that your prototype or mockup isn’t functional yet. Doing so will keep them from asking questions about links that don’t work or getting frustrated when they can’t move further along on your interface.

Example Statement:
What you will be testing is a prototype, an idea where not everything is 100% functional or clickable, so as you explore you may find things that don’t work yet.

Unexpected Results from Free Reign Testers

If you give your users free reign to click around on your prototype (if it is functional), there is the possibility that your user may move on to the next question while on the page you did not intend to test. It is important to communicate with your user to return to the specific page you want them to be when moving on to the next question by simply adding a statement (and link if you really want to be nice) to the beginning of your next question.

Example:
Be sure you are on the home page before continuing to the next question.

Allow the User to Vent

Sometimes, you can really get awesome feedback from allowing your user to vent, or tell you what frustrated them the most on your interface. The best way to do this is to allow them to type in their answer rather than talk out loud. The user will be more intentional about what they write and provide insightful feedback.

Example:
What frustrated you most about this site (remember it’s a prototype, so items may not be clickable)?

Summing Up

To get the best possible results from user testing, clear, concise, easy to understand questions are essential to get the feedback you want to achieve your goal. Being able to think like your user is a huge plus when writing effective questions.

Carefully crafted qualitative and quantitative user test questions are the framework for clear feedback responses that can help you see if you achieved your user testing goal or not.

Most importantly, being able to avoid user frustration while taking the test will result in honest, helpful, and effective feedback that you can bring back to your business unit, stakeholders, leaders, or product owners.

FREE User Testing Results Template for Sketch

Free User Testing Results Template

As a user experience geek, I love coming up with ways to save me time when I do the numerous user tests. I have created a simple user testing results templates in Sketch format.  It is very customizable including symbols and easy to change donut charts. This is perfect for stakeholders, marketers, developers, and creative team members.

Leave your comments to suggest any improvements.

Preview

Preview User Testing Results Template for Sketch

Preview User Testing Results Template for Sketch

Download

 

 

 

How To Plan for Visually Disabled Users on Computers and Mobile Devices: Color Blindness, Low Vision, and Blindness

WebAIM

Did you know that there is about 7.4 million people with vision disorders? These vision disorders include but is not limited to:

  • Color blindness
  • Low Vision (associated with illnesses and age)
  • Blindness

I am going to touch on the very basics of these vision disabilities and so that you will know what to avoid in your interface, and most importantly, what to do in your interface.

Color Blindness

In the world, color blindness affects around 1 in 12 men (8%) and 1 in 200 women (0.5%).

People with color blindness have the inability to process red and green or blue and yellow. This can cause issues when using colors to indicate important states or statistics in a app or website.

Some possible issues people with color blindness may deal with:

  • Traffic lights
  • Picking out clothes that match
  • Being rejected from a job assignment due to color deficiency
  • Buying ripe bananas at the grocery store
  • Making sure meat is thoroughly cooked
  • Not being able to tell difference between vacant (green) and occupied (red)
  • Colored maps and graphics (i.e. weather radars)

Examples

If you love making charts or read them yourselves, check out this Google Analytics chart (normal vision at top and color blindness below it). It’s difficult to see which one is 25 to 34 and 18 to 24 years old.

Normal Vision reading charts is great.
Color Blindness causes issues with red and green colors, not able to differentiate labels.

People that are color blind may also have issues seeing traffic light colors and rely heavily on position of those colors for their action.

Left to Right: Normal Vision, Tritanopia (Missing Blue), and Protanopia (Missing Red) Color Blindness

What to Avoid in the Interface

What can you do to accommodate 8+% of the United States population that visit the web or use an app? There are a bunch of color combinations that should be avoided if at all possible. The colors on the chart below are if viewed by normal vision.

Avoid pairing green with red, grey, blue, brown or black. Avoid pairing blue with purple, gray, and yellow.

Here’s a more comprehensive color chart of colors to avoid. The two color combinations above the main block translates into the solid color block to color blind users.

How To Improve the Interface

  • Use colors and symbols to indicate the meaning of something.
  • Use shade and tone variations instead of multiple colors in your designs.
  • Use contrasting colors and tones.
  • Consider using textured background patterns to differentiate similar colors.

Low Vision

Did you know about 135 million people around the world have low vision?

A person with low vision will have difficulty with daily activities such as looking at websites, reading, and driving. Even with glasses, contact lenses, medicine, or surgery, their vision may not be that good. Typically we think of elderly people that are associated with this disability, but it affects any age.

Low vision issues may affect anyone with the following.

  • Glaucoma, Diabetes, Macular degeneration, or Cataracts
  • Older monitors with poor screen resolution
  • Mobile devices with smaller screens
  • Accessing the web via phone or computer with poorly lit screens or washed out by sunlight
  • Migraine sufferers that are sensitive to light (screen light adjustment)

According to WebAIM, the most important rule for low vision is how perceivable something is. People with low vision issues can’t see content that is small, does not enlarge well, or does not have enough contrast.

Some possible issues people with low vision may deal with:

  • Recognizing the faces of friends and relatives
  • Reading, cooking, or sewing
  • Seeing because the lights don’t seem as bright as usual
  • Crossing the street or reading signs

What to Avoid in the Interface

Embedded Text in Images
People with low vision use screen magnifier programs to view the web. Avoid text that is embedded within graphics (unless it’s a vector image). Why? If the user zooms in, the text will be rasterized and sometimes be not legible.

Left side is text in an image, while the right side is that text zoomed in.

Bad Contrast with Readable Elements
Avoid bad contrast of background and overlaying text. For example, having a black background with blue, white, or very dark gray text would not be a great idea. Same goes for red background with blue or green text. Yuck!

Examples of bad contrasts between background and overlaying text.

How to Improve the Interface

  • Pull most if not all important text out of images
  • Make sure the contrast between backgrounds and overlaying text are distinct. ( dark background and light text vs. light background and dark text).
  • Text in images should be large, clear, and distinct (see both items above).
  • Allow people to customize their contrast settings by not overriding backgrounds and main body text with stylesheets.

Blindness

Users that are blind can’t rely on using a monitor or mouse to access the web. They use screen readers and keyboards to get the information they need.

Blind users heavily rely on keyboard accessibility and screen readers. Because of this, relying on mouse-only events could be problematic.

Brailliant Braille Display helps users to read from the computer (courtesy of Humanware)

What to Avoid in the Interface

  • Images without alt tags
  • Improperly named links without title attributes
  • Too much non-relevant information at the top of a website
  • Lack of instructions for filling out a form
  • Avoid iframe content, as this can disorientate the user
  • Avoid flash content (use HTML5 player instead)
  • Image maps — not all screen readers can interpret these
  • Avoid using javascript within links so there’s no question where it goes
    <a href=”javascript:history.go(-1)”>See Info about Flowers</a>

How to Improve the Interface

Use skip to content links at the top of the page. Skipping over navigation and sidebars with secondary content can be very helpful for those wanting to get to information quickly.
<a href=”#content”>Skip to Main Content</a>
<a name=”content”></a>

Use proper labels on form fields.
<label for=”firstname”>First Name</label>
<input type=”text” id=”firstname”>

Name links carefully so that the user knows what it will do when clicked.
Never use “Click Here” or “Learn More” because these do not explain what the link will do, however, use something more descriptive such as “Download the Latest Version of Chrome”

Provide alt text for all images.
Keep it brief and to the point. If the image is meaningless, your alt text doesn’t need to be too detailed, however, if the image conveys an action, be sure to be concise. If you are still stuck in the 90’s and want to use a graphic for spacing, you should still use alt text, but designate it as empty by adding a space in between the quotes.

If using tables on your site, provide column and row headers using the <th> tag. Make sure that tables (don’t forget merged cells) make sense when read row by row from left to right. Also don’t forget to add a summary and caption to your table.

In Conclusion

I hope you find this information about visual disabilities helpful so that whenever you plan on building a new web app or website, you can also plan to make it accessible to a larger range of audience.

If you care about your users, you may want to try putting yourself in their shoes briefly by using a screen reader, changing up the colors of your monitor, or even trying to enlarge fonts on a screen (that includes mobile devices as well).

Remember, larger audience equals more engagement, and more engagement equals more opportunities!

Coming soon! How to Plan for Cognitive Disabled Users on Computers and Mobile Devices


How to Plan for Disabled Users on Computers and Mobile Devices (Overview)

How to Plan for Disabled Users on Computers and Mobile Devices (Overview)

I get it that you are busy. You may not have thought about accessibility because you are thinking about #allthethings required to get your website or app to a place where it’s deliverable and usable. Totally understandable. That’s why I am breaking all this down into several digestible parts for you to take a look at when you have time.

Accessibility is constantly on my mind all the time, but not because it’s the right thing to do. It’s because I want to. I am reminded of accessibility when my brother is attempting to use his iPhone. My brother (an adult) has cognitive disorders that keeps him from being able to access information easily on the web. I cringe when he opens an app and tries to use it but usually ends up getting help most of the time.

According to the United States Census Bureau, as of March 2015, there were 321.4 million people in the U.S. (dang, that’s a lot!) Did you know most studies find 64.3 million (or 20%) of the entire United States population has some kind of disability? That is a lot of people with disabilities that may issues accessing your web site or mobile app.

Disabilities at a Glance

Are you a designer, user experience, or developer that has missed planning and prepping a product for people with disabilities? It’s easy to forget unless you know someone or if you are included in the population of these disabilities mentioned below.

There are five (5) main disability groups according to the Web Accessibility Initiative:

Physical

There are 36.2 million people in the U.S. with physical disabilities.

Examples of physical disabilities (layman’s terms for now):

  • Missing body parts
  • Joint inflammation, damage, and pain
  • Lack of hand or eye coordination
  • Weak or degeneration of muscles
  • Injury to bones, joints, tendons, tissues
  • Twitches, spasms, or involuntary movements
  • Paralysis (partial or full)

Hearing

There are 40.3 million people in the U.S. with hearing disabilities.

Examples of hearing disabilities:

  • Hard of hearing
  • Deafness

Cognitive

There are 16 million people in the U.S. with cognitive disabilities.

Examples of cognitive disabilities:

  • Inability to focus on single tasks
  • Social and interaction communication issues
  • Impairment of intelligence
  • Difficulty focusing, processing, or understanding information
  • Short or long-term memory issues
  • Visual flickering or audio signals at certain frequencies or patterns

Speech

There are 7.5 million people in the U.S. with speech disabilities.

Examples of speech disabilities:

  • Inconsistent articulation and speech sounds
  • Fast speaking rate, incorrect rhythm, intonation, etc.
  • Weakness or complete paralysis of muscles related to speech
  • Difficulty to make certain sounds or patterns of sounds
  • Repetition of individual sounds or entire words and phrases
  • Inability to speak (from anxiety, brain injuries, or inability to hear and learn speech)

Vision

There are 21.7 million people in the U.S. with vision disabilities.

Examples of vision disabilities:

  • Difficulty distinguishing between colors
  • Poor, tunnel, central field loss, and clouded vision
  • Blindness

In Conclusion

I hope just by those statistics above and basic disability information above has piqued your interest in accessibility. 😉

In the coming articles, I will break down each of these five disability groups into digestable parts with examples of what to avoid when working on a user interface.

Up Next: How to Plan for Visually Disabled Users on Computers and Mobile Devices