Audio GUIs: Interacting with Graphical Applications in an Auditory World

Keywords:

Introduction:

Imagine interacting with your computer desktop through a primarily auditory exchange. Perhaps you are driving to work and calling into your computer with your carphone. You might want to access your normal work environment from a small, mobile device such as a PDA, or you might be blind and need to work alongside your sighted colleague. These three scenarios require the interfaces to your standard graphical applications be transformed into efficient and intuitive auditory interfaces. The resulting auditory interface should leverage your knowledge of the graphical interface. Furthermore, for this strategy to be of general use, this transformation must be done without specific knowledge of the individual application.

The typical scenario to providing access to a graphical interface is as follows: While an unmodified graphical application is running, an outside agent collects information about the application interface by watching objects drawn to the screen and by monitoring the application behavior. This outside agent (or screen reader) then translates the graphical interface into an auditory and/or tactile interface. Not only does the screen reader translate the graphical presentation into an nonvisual presentation, but the screen reader often provides different user input mechanisms which are more appropriate with the new interface.

Our work in this area began with a simple question: How can we provide access to X Window applications for blind computer users. Historically, blind computer users have little trouble accessing standard ASCII terminals. The line-oriented textual output displayed on the screen is stored in the computer's framebuffer. An access program simply copies the contents of the framebuffer to a speech synthesizer, a Braille terminal or a Braille printer. Conversely, the contents of the framebuffer for a graphical interface are simple pixel values. To provide access to GUIs, it is necessary to intercept application output before it reaches the screen. This intercepted application output becomes the basis for an off-screen model of the application interface. The information in the off-screen model is then used to create alternative, accessible interfaces.

A primary goal of our work, called the Mercator Project, is to provide transparent access to X Windows applications for computer users who are blind or severely visually-impaired. In order to achieve this goal, we addressed two major problems. First, in order to provide transparent access to applications, we created a framework which could allow us to monitor, model and translate graphical interfaces of X Windows applications without modifying the applications. Second, given these application models, we developed a methodology for translating graphical interfaces into non-visual interfaces. This methodology mimics the advantages of GUIs in an nonvisual presentation.

GUI Models

The information gathered with the hooks forms the basis for an off-screen model of the application interface. A set of translation rules then processes the off-screen model, identifying higher-level objects in the graphical interface and then specifying the presentation of the objects in the auditory interface. The hypothesis behind this scheme is that some portions of the graphical interface directly contribute to the user's mental model of the application interface while other portions of the interface are simply artifacts of the visual presentation. An implicit question in this transformation is how is space used as an organizing medium in the graphical presentation. For example, objects such as menus are likely to be critical to the mental model of the application interface, but the spatial presentation of a group of menu buttons may be irrelevant. Does a column or row-based organization convey any information to the user? The spatial arrangement within the grouping may be important though. For example, are the buttons evenly spaced, or are some segregated from the others using distance as the only visual cue.

Audio GUIs

Mercator provides a separate navigation method based on the hierarchy of the interface to replace the visual, spatial-oriented mouse navigation used in GUIs. The relationships between objects in the interface are modeled as a tree structure. Users can simply navigate the user interface by walking through the tree structure [3].

Project Information

We are coordinating our current design efforts with the Disability Action Committee on X (DACX) which is directed by Trace Research and Development Center. This committee is made up of Unix workstation vendors (Sun, DEC, IBM), researchers, commercial access vendors, the X Consortium, and other interested parties. The goal of the committee is to design and implement standard access solutions to X Windows for people with various motor and sensory impairments.