Tablet test with blind users: A comparison of common tasks
4. February 2014
Summary of results
Before we describe how users managed to carry out the tasks, here is a summary for each tablet tested.
iPad mini (iOS 7)
Of the three devices tested, only the iPad mini with VoiceOver enabled allowed blind users to accomplish the tasks defined fully and mostly independently, though not without difficulties. The implementation of touch interaction, including meaningful hints, is better compared to the competition. Still, even on the iPad we found a number of issues, for example, not sufficiently descriptive buttons like "add" in the calendar, or the non-modal design of the pop-up window for entering a new calendar event - a design that more than once led to an accidental loss of data.
Nexus 7 (Android 4.4.2)
With the Nexus 7 tablet, some steps in our tasks (like specifying the date and time of events in the calendar app) were difficult or impossible to carry out for blind users. While the last Android and calendar update has corrected the date entry issue, the problem of specifying the event time with TalkBack enabled remains. This calls into question the overall usability of the calendar app for blind users.
In all, the labels, announced roles and user hints on the Nexus were a lot patchier than on the iPad and also the Windows 8 tablet. Some issues that occurred in our tests, for example, the non-modal behaviour of some pop-up widgets in the calendar, have been rectified by the latest update.
People in our tests used the standard stock Android calendar app. Versatile users may be able to identify and install more accessible Android apps. Regarding the browser, we chose Firefox instead of Chrome since it is generally deemed the most accessible for blind Android users.
ThinkPad Helix (Windows 8.1)
The interaction model of Lenovo's hybrid Windows 8 tablet ThinkPad Helix differs in many ways from the interaction model of the iPad and the Nexus tablet and requires some familiarisation. In our tests, the ThinkPad Helix was used without keyboard.
Our inexperienced users needed many hints by the test facilitators - a lot more than were needed on the iPad and the Nexus. The non-modal pop-up menus (charms bar, and the horizontal marginal menu bars) were often accidentally closed by swipe gestures, necessitating an at times laborious repetition of swipe navigation and input. The speech output was somewhat erratic and unpredictable - it was often difficult to reproduce the observed variations of output.
Design of the task test
Each test involved the test user, the test facilitator and a (silent) scribe. The average duration of the task tests was about 1.5 to 2 hours.
At the outset of each test, the test facilitator explained the basic touch-explore, swipe and tap gestures. The test user then had time to rehearse these gestures on the respective tablet.
In the consequent task test, the test facilitator described the respective task step by step which the test user then tried to accomplish. When users encountered difficulties, they could ask for help, which was provided by the test facilitator. Requests for help were recorded by the scribe.
When tasks required the use of additional gestures (e.g. use of the virtual keyboard or vertical swipe gestures) these were explained as and when they were needed.
We carried out the task tests on the iPad mini and the Nexus 7 with three users each. On the ThinkPad Helix (a Windows 8 hybrid, i.e., an ultrabook with a detachable screen that can double as tablet) tasks were carried out by just two users. One was already experienced in using the iPhone but had no prior experience with using a Windows 8 touch interface.
The noted differences in terms of number and expertise of users in the task tests with the Windows 8 tablet make a direct comparison with results of the other user tests difficult. But it should be clear anyway that our results are not statistically dependable due to the low number of users. Instead, the tests have a heuristic value, showing many of the issues that blind novice users encounter when using these touch devices. We have highlighted these generic issues in separate posts (Blind users using touchscreens: the issues).
The technical tests of the three different calendar apps that were used for the most complex task helped us validate and detail our results.
The overall task test was composed of five tasks:
- Unlock screen
- Find calendar and make new entry
- Find browser and open page (URL input)
- Find and read the previously made calendar entry in the agenda
- Change speaking rate in settings
1. Unlock screen
To unlock the Nexus 7 tablet in sleep mode users first have to press the physical button at the top of the right edge of the device. This device sits right next to the similar volume rocker buttons, they are not very easy to find and discern.
The device must then be unlocked with a slide gesture. Swiping right on the lock screen first renders a status widget and the current time and date, then the "slider lock". Our test showed that users first had to identify the right area for applying the slide gesture to unlock the screen since it only works in this area which is centred in the lower half of the screen and takes up perhaps a quarter of the overall screen area. Applying the gesture with the right speed also gave users some trouble. Often, sliding was carried out too fast or too slowly and was not recognised (see our post Blind users using touchscreens - the issues).
The instruction “slide right to unlock” is only provided if the finger happens to touch the small central lock icon in the slider lock area. It is actually incorrect because sliding from within the slider lock in any direction will unlock the device. Users familiar with the device will easily find the slider lock via touch-explore in the lower half of the screen.
Similar to the Nexus 7, the iPad mini first wants to be woken up by a physical button press. The on/off button is located at the top of the screen to the far right. The button itself is slightly more elevated than the Nexus button - it is also the only button on the top edge which makes it easier to discern.
Exploring the screen with swipe gestures leads to the output of time and date, like on the Nexus. Once the unlock button has been reached ("unlock - button") a simple double tap anywhere on the screen is enough to unlock the iPad - an advantage compared to the Nexus where the slide gesture has to be carried out in a defined area.
Users found unlocking the device relatively easy. In the tests, there was a problem that after a very short period of inactivity (about 5 seconds) the device went back into sleep mode, requiring a repetition of unlocking. This can be configured, however (and the short period could not be re-activated later in the technical test).
The start and volume buttons on the Thinkpad Helix are small and recessed and difficult to press and are therefore not especially easy to locate and use. In the default setting, users activating the device with a physical button press get the Windows lock screen. Unlocking first needs a vertical two-finger swipe to bring up the windows registration. There is no hint that this vertical two-finger swipe is needed. The windows registration then requires the input of the user password to unlock the device.
In the novice user test the password input with the virtual keyboard proved to be difficult and error-prone, especially when the password contains capital letters or numbers and the modifier key for those needs to be activated first. A check of input is not possible since the keyboard just provides "versteckt" (hidden) as echo for each character entered.
The windows registration can be turned off in the system settings, however, so that it is no longer displayed when waking up the device. It is still required after a reboot, though. In this case, several elements have to be traversed with swipe gestures until the button for unlocking receives focus. (Experienced users will instead locate the unlock button in the bottom-left corner.) After a simple double tap, the device is then unlocked and presents the metro or modern interface, with the first tile focused and spoken.
2. Find calendar and make new entry
The calendar app was easily found by all test users. However, Android first requires users to locate the apps icon at the bottom of the screen, which took novice users some time.
In the calendar app itself, the button to create a new event was found by all test users; only one dithered briefly whether this was the right choice. Here it shows that the descriptive labelling of the button ("new event") works much better than the terse and less descriptive options in the iPad ("add") and Windows calendar ("new").
The new event can then be entered in a new view. The text input for entering the title was found easily (even though it is not the first field focused). It was also easy to activate by double-tapping. Users did not notice, however, that this caused the display of the virtual keyboard. Some assumed that a keyboard was now present and confirmed this via touch-explore. (Note: When verifying test results under Android 4.4.2, we found that now the output “keyboard for text” is provided).
Entering the event name (title) via the virtual keyboard did not pose significant problems (apart from being slow). The 10-finger system, once explained, seemed to work well. Some users were puzzled that some keyboard keys changed depending on the context. Also, the labelling and pronunciation of the Entf. Key (for Entfernen = Delete) was considered difficult to understand.
The big problem in this task came when users tried to enter the date and time of the event. This simply did not work. A date could not be selected since days were not announced properly. Instead of rendering the full two-digit day figure, the screen reader TalkBack only read the first digit, which meant a meaningless "zero" for the day range of 01-09 , "one" for the range of 10-19, "two" for the range of 20-29 and "three" for 30 and 31. (This problem has been rectified in the latest update of Android 4.4.2, Calendar 201308023.)
In some cases users involuntarily closed the pop-up window for the date selection by tapping outside the pop-up. This problem has equally been rectified in the latest update.
In the Nexus calendar app, the event start time has to be put in via a complex rotation gesture. Even for test users who understood what was required, it was impossible to successfully enter the specified time. The gesture is in no way self-explanatory, it is unreliable and finicky even after training. Hours and minutes are provided as numbers (without label). This calls into question the overall utility for blind users of the stock Android calendar app (For more details, see the article Nexus 7 calendar technical test: Bugs and issues.)
The calendar app on the iPad mini was found and opened by all without problems. Then however came the first problem since the button to create an new event is called "add". This name was not easily understood and interpreted by novice users - they kept searching for another button. Another problem for users that kept searching was that horizontal swiping took them from the top menu elements to the date grid in the month view, while the inverse swiping gesture (still searching) lead to dates of the prior month. It needs a vertical swipe upwards to return to the top menu. The test facilitators had to offer a hint here. (With a more comprehensive training of VoiceOver gestures at the outset, this might not have been necessary.)
The display of the virtual keyboard when activating the first text input was not always recognised by users. As with the Nexus, users merely guessed (and confirmed) that a virtual keyboard should now be present. The display is actually indicated by a sound that might be interpreted reliably once users are attuned to it. Entering the event name caused no problems.
A more serious issue which occurred several times is the accidental closure of the event entry pop-up by touching the area outside of it. This led to the loss of entries already made, requiring the annoying re-entering of input. A dialogue for saving or discarding the event would be useful (as implemented in the Windows 8 calendar) to prevent accidental data loss. All three users were comfortable using the virtual keyboard on the iPad mini.
All users managed to specify the date although in some cases hints by the test facilitator were necessary. It wasn't obvious that with the selection of "Start", a fruit-machine-type slider menu would be presented which has to be operated with vertical swipe gestures. (While this instruction is given quite clearly by VoiceOver, not all users were patient enough to let VoiceOver finish). Users struggled here, since variations in applying the vertical gesture can easily move the focus from days to hours. With some practice this problem is bound to disappear. An additional irritation is the output of numbers without qualifying labels such as "hours" or "minutes".
When saving the new event, some users were slightly puzzled by the name of the button. "Fertig" (Done) was not immediately interpreted as equivalent of "Save". The system feedback that the event has been saved could be more explicit: it is just provided by a sound and the output of the event name. (For more details, see the article iPad mini calendar technical test: Bugs and issues.)
One user swiping through the Windows 8 tiles missed the Calendar app on the start screen. She only found it after swiping through all tiles till the end and then swiping back. When trying to open the calendar with a double tap, the user accidentally triple-clicked and thereby triggered the tile customize mode. This lead to the output "Von Start lösen – Menüelement" (detach from start – menu element). Still in customize mode, the renewed focusing of the calendar brought the instruction that the calendar can be activated with a double tap, trying that however just triggered the feedback "Der Befehl ist nicht verfügbar" (the command is not available). First, the customize mode had to be cancelled, which required help by the test facilitator. The second test user had no trouble opening the calendar.
As an aside, it is worth noting that the Narrator hint "double-tap to activate, triple tap to select" can easily be misunderstood. Both gestures are physically very similar, as are the terms activate and select. Hints like "double tap to open" (iPad) are less likely to confuse users.
Both users failed to find the button for creating a new event with standard swipe gestures: the menu bar containing it is inserted only after a vertical swipe from the bottom. Here, a hint by the test facilitator was necessary. The difficulty is down to the different interaction concept of Windows 8. Once users are familiar with it, they should be able to find the menu bars without problem. As with the vertical swipe needed to navigate calendar regions on the iPad, a more extensive training of WIndows 8 touch interaction would have reduced the number of facilitator hints.
The name of the button "Neu" (New) was not immediately interpreted correctly. "New event" (Nexus) is less ambiguous.
Both users managed to enter events, but only with difficulty. In the detail view of an event, several elements are badly labeled. For example, the start time input has not labels for day and hour. The order of inputs and buttons was also confusing. Some elements have several focus points (e.g., 1. label / 2. pop-up menu / and 3. a separate button) for opening a pop-up menu. The narrow pop-up menus for selecting hour and minute of an event were hard to operate since accidental touches outside the pop-up often led to focus loss.
Input via the virtual keyboard with the 2-finger system worked quite well. (The input mode can also be changed to the 10-finger system: write when lifting off the finger). What was irritating was the seemingly arbitrary variation in speech volume and the (mostly) missing keyboard echo when deleting input. (For more details, see the article Windows 8 calendar technical test: Bugs and issues.)
3. Find browser and open a page (URL entry)
None of the users had any problem finding and opening the browser "Mozilla". The input field for the web address (URL) was also found by all. The insertion of the virtual keyboard mostly went unnoticed. In some cases an accidental gesture led to the hiding of the keyboard. Writing a specified URL (bahn.de) into the address field worked fine for all test users. The name "Los" (let's go) for the Enter key was understood correctly.
Two test users quickly found the browser "Safari", the third took a long time to find it, due to accidental focus losses and inadvertent opening of other apps.
The address field was also found without trouble by two test users but no one noticed the insertion of the virtual keyboard. One test user was puzzled by the different name of the Enter Key: "Öffnen" (open).
The browser "Internet Explorer" (Modern version) was found and opened by both test users without trouble. In one case the address bar disappeared during the swipe exploration of its elements, possibly due to an accidental gesture. A hint by the test facilitator was needed to restore it.
The display of hints in a popup area above the address bar confused one user when she accidentally touched it. The name "Gehe zu" (go to) in place of Enter in the virtual keyboard was not considered self-explanatory. The second, more experienced user had expected the menu bar at the top of the screen (the usual place in desktop browsers) but could locate it at the bottom of the screen without much effort. Entering the URL and calling up the page was both successful.
4. Find and read the previously made calendar entry in the agenda
For the Nexus tablet we skipped this task since the creation of an event had not been successful due to the problem of specifying date and time (see task 2 above). One attempt to check events was successful in finding the agenda; however, an event in this agenda was rendered without date. (For more details, see the article Nexus 7 calendar technical test: Bugs and issues.)
It wasn't apparent to test users that the agenda can be found under "Suche" (Search). In several cases, a hint by the test facilitator was necessary to put users on the right path. In one case, the pop-up-window was accidentally closed (For more details, see the article iPad mini calendar technical test: Bugs and issues.)
Searching the Windows 8 calendar for recorded events in the month view created a cyclical output: first, the column headers for days (Mo, Di... So) were read, then the focus traversed all events entered, and finally it jumped back to the beginning of the month and traversed all days in the month grid. In the events cycle, the event name, date as well as start and end time were rendered completely, if somewhat verbosely: the output of date was repeated when speaking the end time. (For more details, see the article Windows 8 calendar technical test: Bugs and issues.)
5. Change speaking rate in settings
The element for the settings on the app screen was found by all users. None of the users, however, had expected the speaking rate setting in the rubric Text-in-Sprache-Ausgabe (text-in-speech output). One user thought the rate should be a setting of the screen reader itself, i.e., placed under TalkBack settings. The difficulty of identifying the location of the setting was compounded by the nearly incomprehensible pronunciation of "Text-in-Sprache-Ausgabe". Here, hints of the test facilitator were necessary. Changing the speaking rate was then accomplished without difficulties.
None of the test users had trouble finding the Element "Einstellungen" (Settings). They also managed to find the entry "Bedienungshilfen" (Accessibility). None of the users however understood the two-column structure or the settings screen.
Blind novices users did not recognise that a sub-menu opens in the right column; swiping runs through both columns consecutively, so that they appear to users as one long list without hierarchy. The focus travelled past the entry "Allgemein" to traverse the rest of the main menu items in the left column, then entering the submenu "Allgemein" in the right column where users finally found the entry "Bedienungshilfen" (Accessibility).
Within Bedienungshilfen, the entry "VoiceOver" was not recognised by test users. This is mainly due to the bad pronunciation of "VoiceOver" by the active German-language version of VoiceOver. The entry was only understood correctly after a hint by the test facilitator even though all test users had been told to look out for VoiceOver in the task description.
Changing the speaking rate was then no problem all for test users.
The setting for changing the speaking rate was only found after a hint by the test facilitator to look for Change PC settings in the charms menu. The more experienced user had expected but not found the settings on the app screen, in analogy to Android and iOS.
To reach the speaking rate settings, many menu levels and entries have to be traversed. The column-type arrangement of main menu and submenu resembles that of the iPad, which also means that the attribution of main menu entry and related submenu via swipe gestures is difficult since both menus are traversed consecutively. The less experienced test users needed a hint at the entry "Erleichterte Bedienung" (Ease of Access) to find the setting.
It is mainly the non-modal design of the Charms menu and its Change PC settings submenu which makes navigating to the speaking rate setting so error-prone. Both test users closed the menus several times while traversing it - navigation then had to start all over, which was frustrating.