Mobile devices user testing methodology
Last update: 28 February 2015
Our user testing methodology describes how we conduct tests with blind and low vision users in order to evaluiate the accessibility and usability of smartphones and tablets.
Note: This methodology is currently subject to revisions.
The utility of a smartphone or tablet for a user with some kind of visual impairment (from age-related long-sightedness to blindness) depends on a number of factors. While the technical review can establish values for a number of defined criteria such as text sitze, contrast or the avaiability of a particular function, user tests with blind or low vision people tell us more about the actual usability of a particular device, system or app in a realistic context.
User tests provide indications for accessibility problems as well as usability problems.
We conduct tests in a quiet room seated around a table. We usually do not use audio or video recording or covert observation. Our test setting usually comprises the test user, test facilitator and a scribe who observes the test and takes notes.
The test facilitator welcomes the test user. He or she will then explain the aim and context of the test, followed by an explanation of the tasks that will form the basis of the test. Where necessary the facilitator will invite the user to familiarise hiself or herself with the device, system, or apps to be used.
Our user tests usually take an hour or two, rarely longer. We usually finsih with a debriefing where the test user is invited to give his or her views on the tasks, the devices or apps, or the use of assisitve technologies involved.
Tasks and test instrument
For our user tests we define a number of practical tasks in a test instrument.
The tasks in the test aim to reflect common tasks in a workplace context. Examples are:
- Download and read mails, reply to a particular mail
- make a calendar entry for a meeting, setting date and time
- search online for a transpert connection, book an online ticket
- create a text document, mark up headings, copy and paste chunks of text
- save files or upload / synchronize documents to a cloud
The test instruments serves to lay dowen the task in a number of concrete steps, It defines certain measurable points (such as: does the user succeed in activating a particular function). In addition, the instrument has comment fields for all steps in the test instrument to register contextual information.
While the test user carries out the tasks we obserive the execution, looking out for aspects that cause difficulties for the user. Usually, we encourage users to 'talk aloud' about the task and any problem they encounter in using the device, system, app or any assitive technologies in play. If uncertain what causes the problems, we may also ask test users to describe the kind of problem they encounter.
On the test instrument, we record comments about the executon of a steps ("managed only after several attempts"), misunderstandings that we notice or are being told about by the test person (which often point to potential interface issue), or questions and calls for help by test users. By recording the points where the test facilitator gave a hint, we identify areas where the sytem or app might not be sufficiently clear or self-explanatory or cause issues in other ways.
Depending on the familiarity of test users with the device, system or the app under test, the execution of tasks may be preceded by a training phase where test users have time to familiarise themselves with the technical context.
We find out test users by informing about planned test on via mailing lists of organisations of blind and low vision people and people to come forwar if they match the user profile needed for a particular test. We ask test usders for the occupational background, their expertise regarding devices, systems and applications, and their particular kind of visual impairment. This way we have built up a small data base of test users that we can use to search for fitting users in planned tests.
For practical reasons the number of low vision or blind people available for testing is rather small. It is also usually not possible to achieve a homogenous group of users. Even when trying to focus on particular competences or kinds of visual impairments, there usually remain differences among them in several respects:
- Assisitve technology expertise: The dgree of familiarity with particular assisitve technologies, knowledge of expert functions such as particular gestures or keyboard shortcuts. Another aspect is knowledge about other assisitve technologies (for example, a user may just be familiar with JAWS or know several screen readers)
- mobile device competence: The degree of familiarity with mobile devices, touch input and integrated assisitve functions, knowledge about other deivces and systems with different interaction models
- Task-related knowledge: Familiarity with the subject domain of the respective tasks (such as the use of social media apps navigation apps, word processing, speech input, etc.)
Reliability and limits of the results of user tests
Due to the low number and high variability weith regard to assisitve technology expertise, mobile device competence and task related knowledge, the results of user tests usually cannot claim statistical relevance. The result however provide important pointers to accessibility and usability problems that can be followed up in technical reviews where the reasons can be tracked down via measurements or specific interpretations of the observied behaviour.
We invite all readers, especially low vision and blind people, to give us feedback on our approch so that we can improve and validate out test methodology and thereby improve the relevance and validity of our test results.