Having planned the user evaluation sessions, the next stage is to conduct them.
It's a good idea to be familiar with the assistive technology that will be used during your usability session. Realize it will take some time to become familiar with these tools, for instance many have shortcut key combinations that will need to be learned to enable efficient use. However, the more familiar you are with the technology being used, the better you will be able to follow the user's actions during the session and know what behavior can be attributed to the AT and what behavior can be contributed to the interface being evaluated. You may be able to get a demo of the technology by a salesperson with the AT company. If working with a large organization, you may be able to get a demo by a co-worker who is a user or access a lab with the assistive technology.
It's a good idea to prepare all materials, including all forms such as honorarium and participant agreement, ahead of time. These should be written in clear and simple language. Include directions to your site as well with these forms if the user is coming to you. This may include train/bus directions or whatever the person needs. Discuss this ahead of time with the participant. You may need to make arrangements with the participant to pick them up from the bus stop and bring them to the lab or have them use a local pickup service. Instant messaging technology can be an effective way to communicate with the participant as you finalize plans prior to the session. All materials should be available in both hard copy and accessible electronic format and you should consider sending both formats to the user ahead of time so he or she has ample time to read and agree to them before the day of the evaluation session. The hard copy format ensures that if the user does not have a printer, they at least have a copy to sign and bring with them or show somebody else. Likewise, electronic transmission to the person's home ensures that any specialized formats such as large print or Braille are available to the user.
It's important as with all usability activities, to include feedback on accessibility and ease of use for users with disabilities early in the design cycle. Evaluating early prototypes with users with disabilities is encouraged, though more time will need to be invested for these sessions than is typical. Software such as PowerPoint can be used to design screen mock-ups that can still be read with a screen reader for instance and also used for screen magnifying simulations. If you do use PowerPoint for this purpose you must ensure your presentation can be read via a screen reader. Paper prototypes can also be used with a Wizard of Oz approach. For early prototype evaluations with blind or low-vision users, the session administrator should design audio scripts. Instead of swapping pieces of paper in and out, the session administrator would read the scripts aloud depending on the user's actions to simulate what a screen reader would read. For users with motor impairments, on screen prototypes are recommended over paper so feedback can be gathered on issues such as button positioning and size, link proximity and size. Reporting issues and providing recommendations is similar to the procedure that would be followed for any usability session.
Before conducting any evaluations with users with disabilities, you should ensure that the software or hardware complies with appropriate accessibility standards. This will ensure that the software is keyboard operable and compatible with assistive technology and that the hardware can be operated by users with motor impairments.
Pilot sessions are an important part of preparing for a usability session. Pilots should be run at least one week prior to the day of the actual session and can be run with either internal users or users who were not selected to participate. Pilot sessions will help ensure all discrepancies have been worked out of your task list as well as ensure the assistive technology you have set up in your lab environment is configured appropriately before your actual usability sessions take place. For this reason, it is important that pilot sessions are run with participants with disabilities representative of the users being included in your actual session.
Consider the following points when preparing a lab environment for users with disabilities.
Vision impairments are extremely common. For many people they can be corrected using glasses, contact lens or even laser surgery. However, these kinds of correction cannot rectify all visual impairments. For example, glaucoma causes damage to the optic nerve through excessive pressure being placed on it. Cataracts involve a clouding of the lens of the eye and while this can be corrected by surgery, users are often unaware of the problem until it is quite severe. Macular degeneration is the most common source of vision impairment in older adults and arises from a general decay of the retina. It is accompanied by a loss of the central field of view. Glaucoma, cataracts and macular degeneration are often associated with the ageing process, although cataracts can be congenital. Diabetic retinopathy is another common age-related cause of vision loss.
Other kinds of vision impairment can be present from birth or acquired through trauma (for example, blow to the head). These can include:
- Retinitis Pigmentosa (RP) the most common cause of visual impairment in younger people. RP has several unique features including an inability to see clearly in conditions of rapidly changing lighting, excessive glare and dimly lit surroundings (night-blindness).
- Tunnel vision a loss of peripheral field of vision, leading to a narrow field of view. This can sometimes be only an inch or so in size. RP is a common cause of tunnel vision.
- Nystagmus uncontrolled eyeball movement, leading to an inability to focus on one point.
- High degree myopia severe or degenerative short-sightedness that cannot be corrected satisfactorily by traditional means.
- Retinal detachment where the retina is damaged and detaches from the eyeball, leading to cloudy vision.
All of these conditions can be associated with both blindness and low vision. It is not the cause of the impairment that is important, but the severity of it. A simplistic difference between the two is that low vision users have some useful sight, whereas blind users do not. A more complete list is available. Note that this does not mean that blind users have no useful sight. Many can still differentiate light from dark and general shapes, but that is often about all they can do. Very few users have absolutely no sight. Some blind users will have had some useful sight earlier in their life that degenerated while others will never have had any useful sight. Consequently the use of visual imagery (i.e. descriptions such as "it looks like a tree" or "it is purple") will be understood by some blind users, but not others. The following points should be considered when preparing a lab environment for users with vision impairments:
- Ensure the evaluation environment is set up correctly. Ensure that the computer to be used in the sessions is set up with the appropriate assistive technologies. Ask the participant ahead of time what screen reader or screen magnifier they use and what version. Also, find out if they use it in conjunction with a Braille display. If a blind user requires the use of a Braille display, request that they bring it with them. There are many different brands and models of Braille displays and they are very expensive, so it would be unusual to have a Braille device available at your work site. Allow time to connect and set up the Braille display to the computer. Some Braille displays require additional driver installation and the participant should be prepared with any drivers. In most cases, the screen reader provides drivers although some screen readers require an additional installation step which includes the selection of the Braille display. Ensure that the lab room is free from floor clutter so the participant won't trip on anything. For blind and visually impaired users, some basic orientation to the workstation area may be of value. There are no hard and fast rules as to the amount of orientation necessary so the best approach is to ask the participant what information might help them.
- Test the user interface with screen reader and ensure support of system font and size settings ahead of time. Testing will uncover any base code problems with interaction between the product and screen reader or base code support for support of operating system settings. This initial testing will allow the team to anticipate the accessibility issues. The test team needs to be familiar with both the user interface and the assistive technology. If possible, also run a pilot session with an in-house participant.
- Allot extra time at the beginning of the session for customization. The participant will need to customize their computer and screen reader configuration according to his or her preferences so that they can use it as they normally do. Many screen reader users have their speech synthesizers set to speak at a very fast rate. Consider asking the user to slow the speed of his screen reader some so that it is intelligible to the testers and observers attending the session.
- Note that people with low vision have a wide range of visual limitations. Some people need to have larger fonts and have no difficulty using color as part of the application user interface. Others use high contrast system settings which mean that the text and background can only be viewed in black and white or yellow. The use of color in the user interface may be difficult for such users to detect. A large number of low vision users use an assistive technology called a screen magnifier and must have extra time at the start of the evaluation session to configure the optimal settings for themselves. It is helpful to discuss this issue directly with your potential participants in order to get useful information from the exercises being performed. Explain where the session administrator will be and what they will be doing. Tell the participant If the session administrator will stay in the same room, where they will sit, that they'll be taking notes, etc. Also explain whether the session will be informal, e.g., participant and session administrator will discuss issues as they arise and participant can ask questions at any time, or more formal, e.g., in which the participant should try to do the tasks before asking for help.
- Provide an accessible task list for the participant to read. With short tasks, the session administrator can read them but generally it's best to have the task list available for the participant to go back and refer to while completing the task. Discuss this issue with the participant before they arrive to ensure the task list is accessible, e.g. in large print or Braille. Some blind users may request that a reader be available and for others it may be best to simply have an electronic version of the task list in .txt format available on the user's computer.
- Encourage the participant to think aloud at appropriate times. Realize that when the participant is using a screen reader they will not be thinking aloud at the same time as they're listening. Ask the participant questions when you observe that they may be confused, disoriented, or annoyed. Blind users do not always know when they are missing information or when information is insufficient or out of order, but they may become confused because of it. Also, they may become disoriented when the focus changes unexpectedly or the presentation of information is not logically ordered or meaningful. If too little, too much, or redundant information is presented, they may become annoyed.
- Be prepared for a guide dog to accompany the user. Some users may require that a guide dog accompany them to the session. You should plan to leave space under or next to the user's workspace for this reason. Always refrain from petting, addressing, or distracting a guide dog in any way, it is working too. Identifying an appropriate grassy area to the guide dog owner may be necessary.
Hearing impairments can arise from congenital conditions (for example, present from birth), disease, physical trauma (for example, repeated exposure to loud noises) and the ageing process. However, users who are deaf or hard of hearing can broadly be regarded as falling into one of two categories:
- late acquired deafness
- early deafness
The principle difference between the groups is that the late deaf had some hearing capability early in their life and became hearing impaired, whereas the early deaf have had no hearing from an early age. This difference is significant because the late deaf will have learned spoken language, whereas the early deaf will probably only understand sign language. Note that while sign language shares many of the same word concepts as spoken language, the grammar is significantly different, and so word-for-word translations may not necessarily make sense. The golden rule under such circumstances is to keep your use of language simple and clear. Also note that when considering international user trials or software, each country has its own particular type of sign language. So, for example, American Sign Language (ASL) and British Sign Language (BSL) are two distinctly different languages and someone who communicates in ASL would not be able to understand someone else using BSL. The following points should be considered when preparing a lab environment for users who are deaf or hard of hearing:
- Provide seating for an interpreter. Users who are deaf or hard of hearing may require an interpreter to accompany them, provide space for the interpreter to sit close to or across from the user.
- Communicating with the user. This is the principal difficulty that is likely to be encountered during the user trial. Assuming that the level of hearing impairment is sufficient that simply increasing volume is not sufficient for communication, then for late deaf users, print-based communication is most likely to be the most effective method of communicating. For early deaf users, sign language / lip-reading / use of print / trained translator are all possible methods of communication. Remember that each user will have developed their own method of communicating with non-signing people, and it is best to follow their lead for their preferred method of communicating with you. Realize that not all hearing impaired users will be able to speak clearly.
Motor impairments can arise from congenital conditions (i.e. present from birth, e.g. Down's syndrome, cerebral palsy), disease (e.g. Parkinson's, polio), physical trauma (e.g. strokes or accidents) and the ageing process (e.g. arthritis).
Common symptoms include:
- Tremor uncontrolled 'shaking' of the limbs or body. Tremor may have a dominant frequency, or may be completely random. The effects of tremor may become more pronounced under conditions of stress. Tremor is often associated with conditions such as Parkinson's disease,
- Spasm uncontrolled contraction of major muscle groups. Spasm often results in high velocity, high acceleration movements. There is anecdotal evidence to suggest that spasms increase in frequency with stress and concentration levels. It is particularly common for users susceptible to spasms to experience a spasm when the cursor is very close to a small or difficult to select target.
- Reduced range of motion some conditions such as arthritis make it painful for users to fully extend their joints, other conditions such as stroke may inhibit all gross movement.
- Reduced strength and stamina conditions such as multiple sclerosis can reduce both muscles strength and stamina meaning that users get tired quickly.
- Reduced co-ordination many conditions causing motor impairments involve damage to the brain or nerves. If such damage is present, it is also frequently connected to reduced hand-eye co-ordination.
- Limited use of arms and hands users may have use of only one hand, only one or two fingers, arms but not hands, no use of hands at all, etc.
The following points should be considered when preparing a lab environment for users with motor impairments:
- Leave space for mobility devices. These include wheelchairs, electric scooters, walkers, crutches, and canes. Room must be made to accommodate them and they should not be moved or touched without the users' permission.
- Communicate effectively with the user. Remember that many conditions causing motor impairments also affect the ability to speak. This may range from a slight slurring of speech (e.g. after a stroke) to being very softly spoken (for example, Parkinson's disease) to dysarthric speech (e.g. for cerebral palsy) which is very difficult for someone unfamiliar with that particular user to understand.
- Be a good host. Allow sufficient breaks for the participant and be flexible with when these are held. Provide refreshments — but note that some users will have a need for assistance with drinking beverages. Having straws handy is always useful.
- Consider influences due to medication. Bear in mind that many users with motor impairments will be taking medication for their condition and that their behavior may change noticeably depending on whether they have just taken their medication or are due for it.
- Reduced co-ordination many conditions causing motor impairments involve damage to the brain or nerves. If such damage is present, it is also frequently connected to reduced hand-eye co-ordination.
- Provide seating for a Personal Care Attendant (PCA). Some users may require the presence of an assistant depending on the severity of the disability. Be prepared to provide seating for this person directly next to the participant.
When laboratory sessions are carried out, users can be encouraged to bring assistive technologies they use at home (e.g. special keyboard and key guard) with them, but may not be able or willing to do this. A compromise approach that can work well is to hold sessions at a location specializing in computer access for people with disabilities, where specialized equipment is already available. In either situation, researchers should consider providing the following facilities, and plan to spend time at the start of each session making sure that the user is comfortable via:
- Table with easily adjustable height
- Moveable and adjustable chair
- Cordless keyboard and mouse that can be placed on a user's wheelchair tray
- Keyboard whose slope and orientation can be adjusted and then fixed in place on the table
- On-screen keyboard application
- Alternative pointing devices such as a trackball, touch pad, or switches
- Adjustable key repeat delay, key repeat rate, mouse gain, double click speed and any other software accessibility features to user's preferred settings
- Standard Operating System accessibility features are enabled, such as Sticky Keys etc., and are configured to user's preferred settings
The Section 508 Web site has a good illustration of the height and reach requirements for hardware so controls can be reached by someone in a wheelchair.
Users with cognitive and/or learning impairments exhibit a range of symptoms from simple memory loss to extreme behavioral traits. One of the most common sources of cognitive impairment is the ageing process and conditions associated with that. Some memory loss is typical in older adults, especially working memory (also known as short-term memory). Conditions such as Parkinson 's disease, strokes and Alzheimer's are often associated with much more severe memory loss.
Other causes of cognitive/learning impairments include conditions such as:
- Dyslexia this is an umbrella term for conditions that cause a disparity between a person's IQ and their measurable language processing ability, especially written language. The most important symptoms for user evaluations are a general difficulty in reading and writing and an anecdotally above average tendency to try to find unorthodox solutions to problems.
- Attention Deficit Disorder (properly Attention Deficit Hyperactivity Disorder) people with ADD (or ADHD) are often prone to easy distraction from the task in hand, need more frequent and often instantaneous gratification and also bouts of hyperactivity, where they find it difficult to 'sit still'.
- Autism Spectrum Disorders (also known as Pervasive Developmental Disorders) these are a range of conditions from the milder Asperger's Syndrome to the more severe Autistic Disorder. People with autism spectrum disorders exhibit difficulties in both verbal and nonverbal communication and social interaction. They may also demonstrate repetitive behavior patterns and may also respond in unexpected ways to a sensory stimulus.
Estimates of the prevalence of these conditions seem to be increasing, most likely as a result of better diagnosis methods.
The following points should be considered when preparing a lab environment for users with cognitive/learning impairments
- Keep it simple. Ensure that your evaluation session protocol is self-explanatory. Keep the use of language clear, simple and concise. Remember that the aim is to evaluate how understandable the system/application is and not your instructions.
- Do things in short bursts. Long, complicated procedures will almost certainly run into problems with this user community. Keep things in short, discrete chunks.
- Be prepared to explain things in different ways. This user group may not understand your first attempt at explaining what needs to be done. It is best to have thought of several different ways of explaining what you want them to do, e.g. an instruction such as "Click on the button," may need to become "Move the mouse. See how it moves the little white arrow? Now try to get the little white arrow over the grey box and press the left mouse button."
- Remove ALL distractions. Users with conditions such as attention deficit disorder, autism and the like are often easily distracted or intimidated by strange objects. Try to keep the area where the evaluations are being conducted free from any unnecessary objects.
- Avoid questionnaires. It is better to use interviews for briefing and de-briefing sessions.
Older adults will typically exhibit a range of impairment types. The aging process is typically associated with a degradation of user capabilities. This is most commonly encountered as reductions in hearing and visual capabilities (for example, cataracts). Conditions such as arthritis and Parkinson's disease are also frequently associated with ageing. Thus older adults will probably exhibit features of all of the impairment types discussed so far in this document, albeit most likely towards the milder end of the spectrum. Thus you need to go through all of the requirements for vision, hearing and motor impairments.
In addition, you should:
- Reassure the user. A stereotypical image, but one that you are likely to encounter. Older adults are likely to need additional reassurance that they are doing the correct thing and that there will not be any "clouds of smoke" or "loud explosions".
- Communicate effectively with the user. Remember to speak slowly and clearly and keep your use of jargon to a minimum. Be prepared to explain what the user has to do in entirely non-technical language if necessary ("Move the arrow to the grey box with words on it"). Remember that older adults may have a very different background to yours and may not understand your references and terminology.
- Keep to the point. Another stereotypical image, but one that you will most likely encounter is that while some older adults will be very reserved, especially the men, many of the others will be very talkative during the sessions. This can be very useful for the session administrators, but only if the conversation is kept relevant to the topic of interest.
It is difficult to completely recreate a user's working environment in a lab setting; people with disabilities often use a complicated or unique combination of assistive technology software and devices. This is why it is often beneficial to conduct user sessions in a user's natural surroundings. If you can not physically visit the user at his or her home or office, you can still conduct a session and obtain useful feedback. The following points should be considered when conducting remote sessions with users with disabilities.
- Send materials ahead of time. Send the user any paperwork or materials one week before the session as well as any software to install or Web links to bookmark for the evaluation session. Note that any materials sent ahead of time must also be in an accessible format or they will need to ask for assistance. If you need to evaluate the accessibility of the installation process itself of your software, tell them to wait to install the software during the evaluation session.
- Ensure the user has access to a speaker phone and Web meeting software. For evaluations with blind users using screen readers, good results have been obtained by simply listening to their screen readers as they work through the assigned tasks over the phone. Remember to ask the user to slow the speed of his screen reader some so that it is intelligible to the administrators listening at the other end. Indeed, session administrators will have to follow along on their side and remember to ask the user to speak aloud as he or she works. For users with low vision and other disabilities who may still use the mouse, it can be useful to use screen sharing meeting software so session administrators can see the user's screen as she performs the assigned tasks.
- Ensure that the user's system meets all your system's software and hardware requirements (operating system version, browser version, other required software versions like Adobe Reader or a Flash player, etc).
If you can be on-site at the user's location, and this is optimal, remember to do the following in addition to the points mentioned above.
- SPlan for travel appropriately. Leave enough time to travel to and from locations and ensure proper directions beforehand. A cell phone may be useful to call the participants from the road if you get lost or are going to be late.
- Determine how to handle interruptions. In the user's environment it is likely there will be many more interruptions and distractions during the session. For timed tasks you will want to stop timing when an interruption occurs and the user has to temporarily stop.
- Verify the user has a working monitor and Internet connection prior to the session. A blind user using a screen reader, while he may have a monitor, may not use it often so you should verify one is available and in working condition. Likewise, it is worth spending the time to verify there is a working Internet connection in case your evaluation session requires Web access or certain drivers need to be downloaded when installing software.
- CIf you move anything let the user know. Users with disabilities have unique set ups customized for them. Be sensitive to when you move something and always remember to tell them if you have moved something like a chair or lamp and be sure to move it back when the session is over, especially if the user is blind or visually impaired.
- When evaluating a Web site. send participants the URL and the tasks or questions to answer, which can be done at their own convenience.
If necessary, a third-party relay service (often using TTYs) can be used or captioning software for capturing your spoken instructions. There are two types of relay service, IP based and video relay:
Video Relay Services:
For people with physical impairments, lab-based evaluations are not always ideal. Individuals may have made many modifications to their home or work environments to allow them to work comfortably and accurately, and this will often be difficult to reproduce in a lab session. Furthermore, the user's impairment may make travel to sessions difficult and/or physically draining. If evaluation sessions can be carried out at the user's own location, then a more realistic usability evaluation can be performed. For users who employ specialized assistive technologies such as head pointing or eye gaze, it may be useful to schedule time for the user to explain these technologies to the researchers, as it may be difficult to understand what is happening if the operation of this device is unfamiliar.
- Communicating with the user (session administrator not present or remote). Verbal communication via telephone is better for some users than written communication, such as through instant messaging.
- Send out paperwork well beforehand. Legally it is necessary to get a user's signature on a consent form before they can participate in a session. However, such forms are often written in obtuse legal language that can be difficult to understand and often very intimidating. Allow the users plenty of time to understand those forms and any other paperwork that is needed. Make arrangements to explain any paperwork to them over the telephone, ideally before it arrives to explain what it is as well as while they are reading through it.
Installation of software (session administrator not present or remote). Some older adults may have difficulty installing any software that you send out. Ideally make your software self-installing if the users will permit that, so all they have to do is double-click on a file.
Users with disabilities are people first, like any other participant in this respect. Relax and act natural, there is no reason to be overly sensitive or avoid words like "see" or "look" for example, even if the user is blind. Use a normal tone of voice and always address the participant, not the interpreter or personal care attendant. You should be aware of personal space and never move or touch assistive devices such as canes, walkers, and crutches. Remember that all users are different and not simply a member of a particular user group, do not assume anything, always ask the user first. For example, always ask before giving assistance. Some users may look as if they are struggling to complete a task but may very well be happy to continue on their own, while others may appreciate the assistance. When in doubt always ask the user. Depending on the nature of the usability session, it may be a good idea to have one session administrator sitting near the participant in case any assistance is needed.
Remember, users with disabilities may fatigue more quickly than the researcher expects. The user's fatigue level should be carefully monitored and the researcher should be prepared to give frequent breaks, end a session early and split tasks over multiple sessions if necessary. Extra time should be allowed for computer based activities. Allow 2-3 times as long for someone with a moderate impairment and be prepared for some individuals to spend longer. For those whose disability has caused speech impairment, additional time for communication will be necessary, and the researcher may need to ask the user to repeat statements multiple times. Users are generally happy to do this in order to be understood. Researchers should also repeat responses back to the user to check that they have understood. In some cases, the user may choose to type responses into a document open on the computer. Some users may have difficulty signing a consent form. Some may sign an X or use a stamp to sign, others may wish to sign electronically. Be prepared for all of these. Users may prefer to respond to questionnaires verbally or electronically rather than use printed paper and pen.
Lastly, some disabilities have highly variable symptoms or may cause additional health problems. Be careful to confirm sessions near the time in case the participant is unable to attend or needs to reschedule.
Always send a thank you note to your participant after the session. Let them know their feedback was valuable and you look forward to working with them again. E-mail is fine for this and preferred by users of screen readers, screen magnifiers, and Braille devices. It's also recommended you tell the user the amount of the honorarium check that will be arriving in their mail and when they can expect it, assuming you did not pay them in person the day of the session.