Skip to main content

Before you begin

White paper: conducting user evaluations with people with disabilities

This paper references IBM internal, external, and worldwide laws, guidelines, and regulations. Refer to the Resources section of this paper for a listing of resources for these laws, mandates, and regulations.



Heuristic evaluation criteria

Heuristic evaluation is a structured method for conducting expert evaluations. It is sometimes used in user centered design projects as a replacement for or a supplement to conducting testing with actual users where there are schedule or cost challenges associated with bringing in actual users. In a heuristic evaluation, usability or product experts perform a systematic evaluation of the user interface guided by explicitly stated usability principles, known as heuristics. More than one expert usually conducts the evaluation, since different evaluators find different problems. While heuristic evaluation is a quick, economical approach to uncovering usability problems, it does not uncover all issues found by direct testing with users, nor does user testing uncover all problems discovered by experts. Since the methods are complementary, both should be used in iterative combination when possible.

Heuristic evaluation is also a useful method for uncovering accessibility issues. It is a good idea to perform a heuristic evaluation (or consult with an accessibility expert) before conducting user sessions, in order to catch major accessibility problems and handle them prior to the user session. It would be wasteful if a blind user were brought in for a session only to find that main parts of the product were not keyboard accessible. This is a barrier which could easily be discovered ahead of time with a heuristic evaluation.

A heuristic evaluation is also useful for defining potential problem areas on which to focus during user testing sessions. For example, an evaluator may find that while a tree control used in the user interface is keyboard accessible, it seems difficult to navigate with a screen reader. User evaluation can then focus on understanding how much of a usability problem it is and getting insights for what would make the navigation intuitive and usable.

The evaluators who conduct an accessibility-focused heuristic evaluation should have knowledge of particular disability issues and relevant assistive technology. Ideally, the evaluators would have knowledge of different disability issues. Most likely the evaluators will be members of the user engineering team who have focused on accessibility issues and could be considered to have at least an intermediate level of knowledge and skill in the accessibility area.

Even though the accessibility heuristics are useful for uncovering accessibility and usability issues in a minimum of time and for a minimum cost, their effectiveness is largely dependent on the skills and experience of the person conducting the assessment.

In an ideal situation, the heuristic assessment should be performed by someone who is very familiar with both PwD and the assessment methods. However, since there is currently no way of 'standardizing' the person doing the assessment, the results of the assessment equally cannot be regarded as 'standardized. Thus, heuristic evaluations are not appropriate for assessing accessibility compliance at this time. Refer to the Resources section for a listing of resources on heuristic evaluation.

Assistive Technology test tools

Assistive Technology (AT) test tools, such as screen readers, screen magnifiers, and accessibility checking tools are important in the validation of accessibility for web pages and products. It is important to use these types of tools to validate the product(s) prior to usability evaluations with PwD. An AT can appear to read the correct product information but that does not necessarily mean the software or document is truly accessible or can pass an accessibility compliance test. So evaluations with the AT test tools along with an AT provide 'checks and balance'- type testing. It is also important to note that IBM's aDesigner is a disability simulator that helps Web designers ensure that their pages are accessible and usable by the visually impaired. This tool is free and available from alphaWorks.

Assistive Technology tools

There are several assistive technology tools generally used in the industry. The tools listed below are examples of assistive technology tools that could be used by people with disabilities and could be used for usability evaluations. These tools include:

  • Screen readers
  • Screen magnifiers
  • Accessibility checking tools

Screen readers

A screen reader is used to replace the visual display viewed on a monitor for those with visual disabilities. Synthesized voice output is produced for text and controls displayed on the computer screen as well as for keystrokes entered via the keyboard.

Screen magnifiers

Screen magnifiers are used by people with low vision to access information displayed on a computer monitor. These tools usually can enlarge information and images by pre-determined increments as well as provide the flexibility to enlarge user-specified portions of the screen. These tools also often allow for high contrast display and enhanced cursor tracking options.

Accessibility checking tools

There are multitudes of accessibility checking tools currently available to help you determine your product or Web site's base accessibility compliance. These tools are an excellent way to familiarize yourself with accessibility issues prior to conducting user evaluations with people with disabilities.

Education on evaluations with Assistive Technologies

There are several possible sources of information depending on your level of experience, and can help you improve your understanding of accessibility issues. The suggestions below are based on the information published on the IBM Web site. Use them as a guide to prepare for evaluations.

Suggestion

  1. Explore the IBM Human Ability and Accessibility Center (AC) Web site.
  2. Thoroughly read the IBM AC Developer Guidelines to understand accessibility requirements. This includes reading through the checklists and techniques to determine what should be implemented and tested.
  3. Review the Accessibility resources.