Skip to main content

Planning the sessions

White paper: conducting user evaluations with people with disabilities


What do we mean by "evaluations"?

For the purposes of this paper, the terms "user evaluation", "user testing" or "user studies" are taken to be sessions in which users are asked to interact with and use a product to identify any usability and accessibility shortcomings in that product. Ideally, suggestions for possible re-designs will also be generated. These sessions can be called many things, perhaps the most common name is "user testing". However, when working with the specific user communities addressed in this report, it is very important to ensure that the users are fully reassured that they are in no way being 'tested'. An important step towards achieving this is to ensure that the sessions themselves are not described as "user testing" sessions, hence this paper recommends use of the term "evaluations" when describing such sessions to prospective participants.

Equally, the term "tester" should be avoided to refer to the person running the sessions, and "Evaluator" should also be avoided since it is ambiguous and may also be interpreted to refer to the participant (the user). Consequently, the term "session administrators" should be used to denote those who plan, organise, run and observe the product evaluation sessions.

Which disabilities should be included in user evaluations?

The types of disabilities to include in evaluations are contingent on the product being evaluated and the known target audience. Time and funding may limit ability to hold sessions with disabled participants. It is sometimes not necessary to conduct evaluation sessions with every type of disability, but rather focus on those PwD who will be most impacted if your product is not accessible. For instance, if designing software with no auditory cues, it is not necessary to include deaf or hard of hearing users. The following table shows the kinds of difficulties users with specific impairments may encounter and the Assistive Technology that they may use.

Vision
Issues: Cannot use the mouse for input, cannot see the screen or may need magnification and color contrast.
Assistive Technology: Screen readers, Braille displays and screen magnifiers.
Hearing
Issues: Cannot hear audio, video, system alerts or alarms.
Assistive Technology: Closed captioning, transcripts, ShowSounds - a Windows accessibility option.
Motor
Issues: Limited or no use of hands, limited range, speed and strength.
Assistive Technology: Alternate input (e.g. voice), access keys, latches that are easy to reach and manipulate, also single switches as alternatives to standard point and click devices, on-screen keyboards, operating system (OS) based keyboard filtering (e.g. StickyKeys in Windows).
Cognitive/learning
Issues: Difficulty reading and comprehending information, difficulty writing.
Assistive Technology: Spell checkers, word prediction aids, reading/writing comprehension aids.

Here is an example of how users with different disabilities may be affected by a common graphical user interface activity, namely clicking on an on-screen button:

Consider a small button or icon on a software interface. Someone who is blind would not be able to see the button or locate it. Someone with low vision may be able to see that there is a button, but not be able to recognize the type of button or read the text on it. Someone with a motor impairment may be able to see it and recognize it, but may not be able to position the mouse pointer or keyboard focus over it. Someone with a cognitive impairment or learning disability may be able to see, recognize [note - assuming that it is not described solely by words and that the learning disability is not a literary one] and activate the button, but may not know what it does.

Next, consider a small button on a piece of hardware. Many of the same issues apply. Blind users may not be able to see or locate the button; low vision users may not be able to recognize it; motor impaired users activate it and cognitive impaired users know what it does.

The easiest way of identifying which users may be affected is to break down the interaction into its common component steps and think about what the user has to be able to do to complete that step, e.g.:

Software scenario example:
A dialogue box has popped up on the screen and you need to cancel it.

Steps:

  1. Recognize that a dialogue box has appeared - either see it (vision) or hear the notification sound (hearing)
  2. Find the available buttons to choose between (vision)
  3. Distinguish between the buttons to choose correct one (vision, cognitive/learning)
  4. Decide which one to activate (cognitive/learning)
  5. Activate it (vision, motor)

Hardware scenario example:
The ATM is prompting you "Which account do you wish to withdraw your cash from?" You have to press the relevant (hardware) button to proceed.

Steps:

  1. Read the prompt - (vision, cognitive/learning)
  2. Find the available buttons to choose between (vision, maybe motor)
  3. Distinguish between the buttons to choose correct one (vision, cognitive/learning)
  4. Decide which one to activate (cognitive/learning)
  5. Activate it (vision, motor)

The similarities between software and hardware activation of a button should be obvious.

Note that each of the principal impairment classifications (sensory, motor and cognitive) could potentially encounter a 'showstopper' accessibility difficulty in these scenarios. Also note that the use of assistive technology requires these steps to be gone through again and amended accordingly. For example, the use of screen-reader software in the software scenario removes the vision demands on the user, but replaces them with hearing ones.

Clearly evaluations with only one user group will only identify the issues for other users of that same group. If you only have resources to conduct evaluations with one or two users, one option is to find users with a range of impairments. Older adults, for example, may have limited dexterity (motor impairment) through arthritis, as well as low vision from age-related macular degeneration and slight hearing loss.

Keep in mind some users may require highly specialized or customized solutions to meet their needs. Such solutions may not necessarily be applicable or beneficial to users with lesser impairments. It is preferred practice to find users who should be able to use the product but are experiencing or are likely to experience unnecessary difficulties in doing so.

How many participants should be included and when to engage?

This too is a question with no definite answer. Given resource constraints, it is generally suitable to conduct evaluations with anywhere from three to five users, whether disabled or not. Practitioners suggest that a sampling size of five users often uncovers the majority of a product's usability issues.

Include one subject with a disability in each round of user evaluations, understanding that evaluations should be carried out throughout the product's life cycle. This ensures that required feedback is gathered along the way without having all users with disabilities participate simultaneously.

The use of personas

The use of personas during the design process is a helpful way of making users more real to User Experience practitioners and designers. These user profiles are fictional individuals representative of the target user group the product is being designed for, in turn they help put a name and face to the user and ensure they are being considered at every phase of design. While user group profiles are general, personas are specific, the more detailed a persona is the more useful it becomes to developers and designers.

Often personas have a name and a picture associated with them to help make them realistic. They also include attributes such as demographics, family background, personal and career goals, attitudes, expectations, educational history, job experience, and other "human" characteristics that developers and designers can relate to.

Much work has been done in defining user groups for products, and recently accessibility characteristics have begun to be included in the personas created for these groups. The same attributes listed above are included along with information on the type of disability, the symptoms and conditions associated with that disability, and the assistive technology used. Including accessibility in personas ensures that users with disabilities and their unique requirements are also considered during design. Since members of your target user group are exactly who should be included in your user evaluation sessions, it is beneficial to examine these personas prior to conducting your sessions.

How long should the sessions run?

Session length depends on the type of disability a participant has. Plan on more time for evaluations with users with disabilities as it may take participants longer to complete tasks than users without disabilities. A reduced task list is also recommended, as fatigue can be a factor with disabled participants. Indeed, it may take much more energy for a user with tremors to click through a Web application or prototype. Likewise, it can be exhausting for a screen reader user to listen to a busy interface be read aloud again and again.

Plan for fewer tasks and reserve a few tasks to add to the list, if time permits. Sessions should always be designed with this degree of modularity in mind, so tasks can be removed or added as time permits. Some participants may quickly accomplish all tasks while other participants may not but be considerate of energy levels nonetheless.

User location versus lab environments - pros and cons

There are advantages and disadvantages to conducting evaluations with users in a usability lab environment compared with at a user location. Likewise, there are pros and cons to running a remote user session in the participant's own work environment. These differentiators are even more significant when conducting evaluation sessions with users with disabilities.

Lab environment pros

A lab environment may be preferred for conducting evaluations for a variety of reasons.

It is easier to control the environment for evaluations in a lab. Session administrators can control interruptions and other aspects of the environment more easily, such as lighting and furniture positioning so session administrators have more room to maneuver and take notes. Administrators also have access to better recording equipment in a lab setting, with more advanced labs equipped with digital recording technology to capture the user's computer screen and movements as well as high quality audio. With more control comes a better chance of gathering consistent and meaningful data.

Lab environment cons

There are also drawbacks to evaluations in a lab environment:

  • Participant travel to and from the lab may be difficult, and serve as a hindrance to recruiting.
  • Lab environments need to be considered for physical accessibility as well - for example, ensuring there is an accessible restroom close to the evaluation rooms.
  • Your lab may need to limit participants to those who use the assistive technologies that your lab has acquired. The corollary of this is that your lab may need to acquire more assistive technologies.
  • The largest detractor from conducting evaluations in a lab environment is the simple fact that it is impossible to perfectly recreate a user's actual working environment in a lab setting.
    • The most accurate usability feedback is gathered when watching users using an interface in their own work space. With users with disabilities, this is even more of a concern.
    • Users with disabilities often have intricate, highly personalized set ups in their home and work environments. They may have personalized AT available at home, but it is not available to them in a lab. This may have a negative impact on their performance and, therefore, the accuracy of results if their hardware and software set ups cannot be recreated in the lab.

There are two types of remote evaluations that should be considered when you wish for a user to be in their own work environment for the session:

  1. Evaluations at a user's location with the administrator present at that location
  2. Remote evaluations with the administrator staying in the lab and monitoring the user in their environment

Evaluations at a user's location

Evaluations at a user's location allow the user to use the technology they are familiar with and this ensures that an inaccurate lab setup does not contribute to their difficulties during the usability session. It may be important to travel to a user's home or office and witness them using your product in their own environment, even more so for users with disabilities.

Examples of possible locations include the user's:

  • home
  • workplace
  • local day-care center
  • local school, college or university

Unfortunately it is difficult to conduct user sessions in a remote environment for a number of reasons:

  • Sessions at a user's location usually cannot accommodate many observers.
    Session administrator roles must be determined beforehand as will your protocol for handling interruptions, such as a doorbell or phone call that must be answered. Some users will not appreciate a large audience, so it is recommended only two session administrators be present for these sessions, one to sit with the user, and one to stay off to the side and take notes.
  • Liability issues abound when conducting sessions at a user's home.
    It is preferred and advised that you conduct sessions at a user's work place or rehabilitation center. If you must conduct a session at a user's home always bring another session administrator, never go alone.
  • It is often harder to record a session at user's location.
    Consideration will need to be given to using a tape recorder or portable digital video camera to capture user comments and the computer's audio and screen display respectively.
  • The administrator has much less control over the evaluation environment in a user's location.
    This includes lighting and requirements for the session such as a working Internet connection; much verification needs to take place with the user before the session to ensure the availability of these resources.
  • Travel arrangements must be made.
    Administrators will also need to plan and prepare for travel to and from the user's location when conducting remote sessions.

Remote evaluations via monitoring

Another form of evaluation involves the session administrator staying in the lab and monitoring the user in their work environment. Materials required for the evaluation sessions, such as URLs, can be sent to the user ahead of time. On the day of the session, users and session administrators can connect to Web conferencing sites, such as IBM E-meetings, and dial in to a toll free conference phone number. Session administrators can then listen to screen reader output over the phone and follow along to what the user is hearing as they navigate the product.

Think aloud protocol should always be followed, and one should note the session administrator will not have the benefit of seeing the user and his expressions during the session.

Resource constraints - the use of accessibility consultants

When limits in time, funding, or resources restrict administrators from running actual user sessions, the use of an accessibility consultant is advisable. Input from accessibility consultants cannot take the place of the feedback gathered from an actual user, so running sessions is preferred. However, if the evaluations session administrators are under a deadline or new to accessibility, these experts can provide advice on how to improve the interface and make it more usable and accessible in the interim. As with any usability evaluation, if it is deemed necessary to incorporate an accessibility consultant, notify and engage them early.

Recruiting participants with disabilities

Recruiting users with disabilities takes time, particularly because this user group is small; participants must be typical users of the product in addition to having a disability. Best results for recruiting are usually achieved if session administrators can reach potential users and request contact either by e-mail or phone. Organizations usually have electronic bulletin boards or e-mail lists established to communicate with their members. Often they will be accommodating and post a recruiting notice or forward an invitation to their constituents.

Here is a sample e-mail invitation excerpt to potential participants:

  • IBM's User-Centered Design team is interested in gathering feedback from blind or low-vision users of IBM Data Management products. These products include DB2, IMS, Content Manager, and WebSphere. If you have experience with any of these products and are interested in participating in usability sessions concerning the accessibility of future versions, please contact Joe Sacco directly at 408-463-2050 or at jsacco@us.ibm.com.

    IBM has always placed an emphasis on the accessibility of our products. You can help us make our products easier for people with disabilities to use by participating in these sessions. We look forward to hearing from you.

You should also consider each participant's assistive technology skills by asking them a few questions or by including a required list of skills in the invitation. For example, find out which assistive technologies they use (screen reader, magnifier, single switch, larger text size, color contrast, voice recognition, etc) and how skilled they are at using the technology (expert, beginner, average, and number of years used). Expert users may give you lots of good usability and accessibility feedback, but you may need to filter out feedback that is too advanced for most users or too costly to implement. Beginners may not provide you with sufficient feedback. You may need to limit participants to those who use the assistive technologies that your lab can provide for the user evaluations.

Good sources of local users include charitable organizations, social clubs, support groups, and vocational rehabilitation offices, which can be found in most towns and cities. Most local colleges and universities have specific offices for students with disabilities and these too can be an excellent resource for recruiting computer savvy participants. General resources for recruiting also include newsgroups, forums, and non-profit organizations. Specific disability groups have their own organizations established.

Contacting users with specific disabilities

There are numerous resources available to assist in contacting users with specific disabilities. The following table lists examples of some organisations that may be able to help in locating potential users. Please note, though, that this list is representative only and that there are many other organizations that may also be useful to contact.

Disability Resource
All American Association of People with Disabilities
Blind and Low vision A complete listing of mailing lists for blind organizations

American Foundation for the Blind

National Federation of the Blind

Royal National Institute for the Blind
Deaf or Hard of Hearing Gallaudet University , the world's only university in which all programs and services are designed to accommodate deaf and hard-of-hearing students, offers programs in technology, digital media, computer science, information systems, and more. The college currently has over 2000 deaf and hard-of-hearing students

National Technical Institute for the Deaf (NTID), one of eight colleges of Rochester Institute of Technology, is the world's first and largest technological college for students who are deaf and hard of hearing. Currently, it has about 1,400 deaf and hard-of-hearing students (90% of those are cross-registered students at RIT)

National Association of the Deaf (NAD)

Royal National Institute for the Deaf (RNID)
Motor Impaired American Parkinson's Disease Association

United Cerebral Palsy

Muscular Dystrophy Campaign
Cognitive/ learning impaired International Dyslexia Association

Muscular Dystrophy Campaign

Alzheimer's Disease Education & Referral Center
Older Adults* SeniorNet

AARP

National Institute on Aging

University of the 3rd Age

*Note: Older adults also commonly exhibit some degree of motor, vision and hearing impairment, albeit frequently minor. It is tempting to simply find the nearest residential home for older adults or people with severe disabilities as these are a ready source of multiple potential users in one place. Remember that the older the users you recruit, the more there will be an imbalance in the population between men and women as women typically live longer than men. You will need to decide whether it is better to recruit equal numbers of men and women, or proportionally more women to closer reflect the actual population distribution. If you have a business partnership with a company or organization that has employees with disabilities who will potentially use your software, you could inquire about involving those users in your evaluations.