Tablet test with blind users: Overview

5. February 2014

Tablets tested: Nexus 7, iPad mini,ThinkPad Helix

In November 2013 we tested three tablets with blind users: Google's Nexus 7, Apple's iPad mini, and Lenovo's ThinkPad Helix, a Windows 8 hybrid with detachable screen.

The tests with users were complemented with technical tests of the three devices. This article provides a summary of results and describes the background and approach followed.

Separate articles cover the individual results:

Two articles investigate general difficulties for non-sighted users using tablets:

Summary of results

Below, we list the pros and cons for each of the tablets (operating systems and screen readers) tested.

Note: List entries marked with * (asterisk) are often important determining factors for blind users choosing a device, but are not results of our tests.

iPad mini, iOS 7, VoiceOver

Pros of iPad mini

  • The best and most mature touch interaction concept
  • Most fluent and most understandable screen reader
  • Haptic home button to return easily to the home screen
  • Most comprehensive naming and labelling of controls
  • Most comprehensive output of the role and value of controls
  • Most comprehensive and intelligible interaction hints
  • Best implementation of switching on and off of the screen reader
  • Best start screen, good discoverability of apps
  • Good virtual keyboard
  • Technical implementation with very few bugs
  • Fluent set-up process with screen reader turned on (but no upfront tutorial)
  • Best implementation of unlock function
  • Best PDF manual (accessible)
  • Largest choice of accessible apps*

Cons of of iPad mini

  • Names of controls at times not sufficiently descriptive
  • Non-modal pop-ups for data entry can easily lead to data loss
  • Some issues with focus management (e.g., after closing pop-ups)
  • Some issues regarding user guidance (e.g., missing dialogue to save data entries)

Nexus 7, Android 4.4., TalkBack

Pros of Nexus 7

  • Mostly good implementation of functions and processes 
  • Good modal design of pop-ups
  • Names of elements at times clearer as on iPad
  • Usable OS interface, but separation of library (start screen) app screen more cumbersome than on iPad
  • Good virtual keyboard
  • Overall usable set-up process with screen reader turned on
  • Upfront tutorial of gestures during set-up
  • Workable unlock function (but harder than on the iPad)

Cons of Nexus 7

  • Less mature touch interaction concept compared to the iPad
  • Screen reader less responsive than on the iPad, occasional glitches requiring repeat actions
  • No haptic home button
  • Even standard apps (calendar) show serious gaps in the implementation of functions, calling into question the day-to-day utility of the device
  • Gaps in the naming and labelling of controls
  • Redundant focus positions of controls
  • Roles of controls not consistently provided
  • Error-prone gestural activation of screen reader for set-up process
  • Some obstacles during set-up process with screen reader on
  • Half-baked system updates cause changes (some improvements, some changes for the worse)
  • PDF manual only in English

ThinkPad Helix, Windows 8.1

Pros of ThinkPad Helix

  • Overall understandable start screen with tiles
  • Haptic Windows button to return easily to the Windows start screen (Modern Interface)
  • Usable screen reader output
  • Mostly descriptive names and labels of controls
  • Good but at times cumbersome interaction hints
  • At times good user guidance (e.g. dialogues for saving entries)
  • Overall good focus management, but weaknesses in the focus order
  • Usable virtual keyboard, but some weaknesses (diffcult interaction for hiding keyboard)
  • Possibility of running office applications (but this requires an external screen reader like JAWS, NVDA or Window-Eyes and usually also a keyboard)*

Cons of ThinkPad Helix

  • Less mature touch interaction concept which deviates strongly from that of market leaders iOS and Android
  • Frequent focus loss when using flyout menus (charms bar, apps menus)
  • Insufficient disambiguation of gestures, frequent touch input problems
  • Even standard apps (calendar) show serious gaps in the implementation of functions, calling into question the day-to-day utility of the device
  • Weaknesses regarding the focus order of elements / controls
  • Inexplicable volume fluctuations of speech output
  • Often missing keyboard echo when deleting erroneous user input
  • Redundant focus positions of controls
  • Implementation of screen reader appears flaky and immature, inexplicable variations of screen reader output of elements
  • The dual desktop (modern and classic) difficult to grasp: Narrator not sufficient for classic desktop *
  • Long and error-prone path to accessibility settings
  • Wake-up and unlocking of device more cumbersome than on iPad and Nexus
  • PDF manual incomplete, inaccessible (but information about Narrator on the Microsoft website)

Conclusion

The iPad is the clear winner, offering the best accessibility for blind people. The Nexus still shows significant accessibility issues and gaps which put a question mark on the day-to-day utility of Android-powered devices for blind users. Windows 8 comes in third place. The touch interaction model is still error-prone and the implementation of accessibility of the Modern interface and core Windows apps via the built-in screen reader Narrator still fees half-baked.

Interpretation of of results

The result is not really surprising and confirms experiences by blind users who tried to use Android on a day-to-day basis and gave up after a while - compare Marco Zehe's or Chris Hofstader's reports. Windows 8 tablets have a special role as they are usually sold as a hybrid / convertible with a keyboard, and are presumably mostly used in desktop mode with external screen readers.

Our tests have shown that blind users without prior experience with touch devices need some time before they can use a tablet confidently and independently. There are significant individual differences. Some users approached the tasks systematically and progressively understood the device and the interaction mode, others were more spontaneous and exploratory, which often led to discoveries but also caused frustration. Some users were quickly confident in applying the basic gestures (swiping, double-tapping), others struggled for a prolonged period, again and again running into problems such as focus loss through unintended touches. The interpretation of speech output (such as interpreting the meaning of a control) equally differed a lot between users. General results are provided in the articles Blind people using touchscreens: The issues and Interface issues when using tablets with screen readers.

With all users it was apparent that the use of touch devices became easier during testing. Still, hints by the test facilitator were often needed when users got stuck, involuntarily opened apps, or happened to call up states that were unrelated to the task at hand. Some of the user actions were easy to carry out, others proved very difficult or even impossible to accomplish with the screen reader. More about that in our post Tablet test with blind users: A comparison of common tasks.

Why test tablets with blind users?

Tablet devices are increasingly used alongside and in addition to traditional desktop workplaces – and this includes blind and low vision users. They are also taken to off-workplace locations (meetings in other rooms or at clients’ premises) for a variety of purposes: navigation and orientation, simultaneous communication, note-taking or browsing, or the capturing/scanning of flipcharts or documents.

This is why we wanted to find out to what extent tablets can be used by blind people without prior experience of touch devices. We have compared the most common tablet operating systems: iOS, Android and – currently a distant third – Windows 8. Further tests will focus on the accessibility of tablets for visually impared users.

A two-pronged approach: Initial set-up and common tasks

The user test was split in two parts: the Initial set-up test, and the common tasks test.

Initial set-up test

In the initial set-up test, we wanted to find out to what extent blind users without previous exposure to touch devices would be able to start using a tablet from scratch – from finding the button to turn on the device, to activating speech output, to registration and the use of tutorials and documentation. The test started with a minimal introduction and explanation of physical device layout, screen reader activation and the main touch input gestures. To separate the principal operability for blind users from the problems occurring in first-time use, the installation process itself was then investigated in more detail in an additional expert test.

The initial set-up of tablets was tested only for the iPad and the Nexus 7. The ThinkPad Helix comes with a keyboard which makes installation with touch a rather theoretical exercise. Also, Windows 8 is a full-blown desktop operating system and is in any case hard to compare to the mobile operating systems iOS and Android. Detailed results are provided in the article Tablet test with blind users: initial set-up comparison.

Common tasks test

For the common tasks test, we defined five generic tasks: Unlocking the device, making a new entry in the default calendar, calling up a page in a browser (URL entry), revisiting the calendar entry, and changing the default speaking rate setting. The tests showed to what extent blind users would be able to carry out these tasks without sighted help. We recorded the accessibility problems users encountered. When users got stuck, we provided help and took note of this fact. Detailed results are provided in the article Tablet test with blind users: A comparison of common tasks.

Additional technical tests

The common tasks tests revealed a number of deficiencies in the apps used. We took a closer look at the three calendar apps (iOS, Android, Windows 8) and listed the issues found there in our technical tests. Examples of issues are not sufficiently descriptive names of controls, missing output of element role or preset value, problems with focus management, or plain software bugs. Detailed results can be found here:

The test users

As test users for both parts, we selected blind users with experience in using desktop computers with screen readers in a professional context, but without prior knowledge of touch devices.

The basic tasks tests brought up a range of problems that were also partly related to the different dispositions and usage strategies of our test users. Due to the low number of test users, our results cannot claim statistical significance. Nevertheless, the tests revealed many accessibility problems and gaps, especially where input in screen reader mode was difficult or impossible, but also in terms of the naming/labelling of functions and the provision of user instructions and adequate feedback after user input.

Devices tested

Our choice of devices spanned the three most common operating systems. Our choice of actual devices / form factor was influenced by what we had at hand:

  • an iPad mini running iOS 7 (verified / amended in iOS 7.0.4)
  • a Google Nexus running Android 4.2 (verified / amended in Android 4.4.2),
  • a ThinkPad Helix running Windows 8.0 (verified / amended in Windows 8.1)

Transferability of results

We assume that most of our iOS and Android testing results can be transferred to its larger siblings, the 10 inch iPad / iPad Air and the Nexus 10 tablet (as far as these run under the same OS version). The results for the ThinkPad Helix are probably transferable to other tablets running Windows 8 / 8.1 (apart from the position of physical buttons on the device). We were unable to verify to what extent our Windows test results can be transferred to Windows RT. Here we would be grateful for hints and comments.

The Nexus test results, however, cannot simply be transferred to other tablets running Android since these often use a custom interface preinstalled by device makers such as Samsung, HTC, or Huawei. These interfaces, also called ‘skin’ – differ from Google’s default Android operating system and are often less well integrated with the built-in screen reader, TalkBack.

Since Google’s ‘Vanilla Android’ is usually most accessible compared to other skins, we therefore picked Google’s Nexus device for our tests. This choice also ensured frequent OS updates, meaning that the test was most likely to reflect recent accessibility improvements of the Android operating system, the Google apps used in tasks, and the built-in screen reader, TalkBack.

The problem of updates

The frequent updates of mobile operating systems indicate an important caveat. For a testing scheme like ours that involved the definition of a test instrument with tasks, the acquisition of test users, the scheduling of tests spread over several weeks and finally, a period of aggregating and analyzing results, mobile OS updates present a problem: they are just too frequent.

For our results to be most relevant for users considering the purchase of a mobile device, we have decided to check our tests results against the latest OS version that was available to us (which is the one buyers will get when purchasing a suitable new device – and with regard to Android, most suitable usually means a device running Vanilla Android). This meant that some earlier test results with users may needed to be qualified and partially revised.