CHI 97 Electronic Publications: Doctoral Consortium
The Multimodal GUI: Developing Auditory Cues as Tools for Performance and Usability
La Tondra A. Murray
Department of Industrial Engineering
North Carolina State University
Raleigh, North Carolina 27695
Tel: +1 919 515 7198
Email: lamurra1@eos.ncsu.edu
ABSTRACT
Designers who use sound in the computer interface must do so judiciously.
The inclusion of auditory cues within an interface should be a mechanism for
the improvement of task performance and the facilitation of usability.
Gaver [6] and Blattner [1] have demonstrated the utility of auditory cues in
communicating information to users. The usage of "spatially-enhanced" speech
and nonspeech elements could provide an additional source of data that might help
or hurt performance. The usefulness of an auditory cue could be linked to acoustical
parameters, spatialization, and task type. The proposed study will assess
the improvement of user performance for various types of auditory cues as
applied to spatial and verbal computer tasks. These results will be
important to multimedia developers who want to create software that
facilitates user acceptance or the quality of user performance.
KEYWORDS
Auditory I/O, human performance, multimedia, user acceptance, user interface design
© 1997 Copyright on this material is held by the authors.
INTRODUCTION
As computer products continue to improve in terms of their capabilities, the human
component of human-computer interactions must be given consideration. The advent of
multimedia and multimodal computer environments has generated a new set of
questions related to human information processing capabilities and task
performance. In critically evaluating these new multimedia elements, one
must focus on the quantification of their utility. Auditory cues may be
useful in splitting among sensory modalities the ever-increasing amount of
information produced by the computer.
Human factors specialists, designers and user interface professionals must
determine answers to relevant questions including the following:
1. Why use sound in the interface at all? 2. Why attempt to characterize task
performance in the context of the user interface? and 3. What salient research
goals can be pursued in an effort to utilize auditory cues such that task
performance is not degraded and users accept a given software product? The rise
of multimedia technologies and techniques has fostered the development of the
"multimodal" graphical user interface. The incorporation of icons, sounds, gestures,
video, and animation in the generic GUI (Graphical User Interface) has enabled user
interface designers to create a rich environment for the presentation of
information and the provision of feedback.
The use of sound elements within an interface generates interesting questions about
the role of auditory cues as tools for performance and usability. Since the auditory
channel is often viewed as underutilized, research is needed to determine the
utility of sound as more than an entertaining addition to an interface.
Sound should not be added to software products as an afterthought, but rather
added when it can be a powerful tool for the facilitation of task performance
and user acceptance.
SOUND AND TASK PERFORMANCE
The relation between task performance and auditory elements in computing systems is
not well understood. The relevance, utility and effectiveness of spatially-enhanced
sound cues outside of virtual reality environments (and specifically for the
context of the GUI) is not clearly defined. While research has assessed the
utility of auditory cues as effective purveyors of information [1,2,3,5,6,9]
or discussed detailed issues about their design [1,4,7,8], the impact of
sound on GUI performance for specific types of tasks remains unclear.
The issue of task performance has only been addressed for the sonification of
very specific interface elements such as buttons or scrollbars [2,3].
Further, there are few studies that compare the effectiveness of auditory
information to that of visual information for the facilitation of computer
task performance. Performance and usability are important aspects of the user
interface which must be applied to all elements of a multimedia
system.
RESEARCH GOALS
In conducting research to reflect the technical capabilities of many personal
computing systems, the following questions will be addressed:
- What types of nonspeech sounds should be used in a GUI as
informative auditory cues? Are abstract, musical, or "real world" tones
better or worse?
- How can simulation be employed to assess the utility of
auditory elements?
- What might spatialized sound elements add to an
interface?
- Are intuitive sounds more likely to facilitate task performance
than sounds that are strictly learned or created by the user?
- What parameters of an earcon can be enhanced to maintain or improve task
performance?
- Are specific sound types more optimal at conveying complex
information for certain tasks?
- Can a generalized model of auditory displays be successfully developed
from task performance data, and can this model then be generalized to assess
audio in other types of computer interactions?
For the purposes of this research, the term "auditory cues"
will be used to refer to all nonspeech auditory elements. Nonspeech auditory
cues will include earcons and auditory icons [1,8]. The focus of this work
will be primarily on the features of auditory cues that serve to improve task
performance from individually established benchmarks. Furthermore, the noise
of the real world can have a tiring effect and potentially impact
performance. Thus, there must be some notion of what interface designers and
human factors engineers view as a "useful" characterization of audio displays
which provides a metric for the appraisal of this utility.
PROCEDURE
The proposed study will be performed on a Macintosh platform using simulated user
interface tasks and pre-recorded auditory cues presented over headphones.
The data will be obtained in three experimental phases. Phase I will
determine which cues are more intuitive among nonspeech sounds. Phase II
will evaluate the effect of spatial enhancement on concept recognition and
retention both in the presence and absence of visual cues. Phase III will
determine how task performance varies as acoustical parameters (pitch,
duration, intensity), spatial enhancement (enhanced or not), and task type
(spatial or verbal) are manipulated. The results of these phases will be
used to develop a model for the validation of auditory cues.
SUMMARY
The goal of this research is to examine the relationship between auditory cues
and the enhancement of task performance. The results of this work (a) will
clarify the relationship between sound, performance and usability and (b)
will assist developers in utilizing multimedia sound technology optimally.
The formulation of a model for the assessment of auditory cues will
additionally provide a methodology for the evaluation of sound elements
within the interface.
REFERENCES
1. Blattner, M. M., Sumikawa, D. A. & Greenberg, R. M. (1989). Earcons
and icons: Their structure and common design principles. Human-Computer
Interaction, 4, 11-44.
2. Brewster, S. A., Wright, P. C. & Edwards, A. D. N. (1994). The design
and evaluation of an auditory-enhanced scrollbar. In Proceedings of CHI -94,
(pp. 173-179). Boston, Massachusetts: ACM Press, Addison-Wesley.
3. Brewster, S. A., Wright, P. C. & Edwards, A. D. N. (1995). The sonic
enhancement of graphical buttons. K. Nordby, P. Helmersen, D. J. Gilmore & S. A.
Arnesen (eds.), In Proceedings of Human-Computer Interaction INTERACT '95,
(pp. 43-48). Lillehammer, Norway: Chapman & Hall.
4. Darvishi, A., Munteanu, E., Guggiana, V. & Schauer, H. (1995). Designing
environmental sounds based on the results of interaction between objects in the
real world. K. Nordby, P. Helmersen, D. J. Gilmore & S. A. Arnesen (eds.),
In Proceedings of Human-Computer Interaction INTERACT '95. Lillehammer, Norway:
Chapman & Hall.
5. Edwards, A. D. N. (1989). Modeling blind users' interactions
with an auditory computer interface. International Journal of Man-Machine
Studies, 30, 575-589.
6. Gaver, W. W. (1989). Auditory icons: Using sound in computer interfaces.
Human-Computer Interaction, 2, 167-177.
7. Gaver, W. W. (1993). Synthesizing auditory icons. In Proceedings of INTERCHI '93,
(pp. 228-235). Amsterdam: ACM Press, Addison-Wesley.
8. Leimann, E. and Schulze, H.-H. (1995). Earcons and icons: An experimental
study. K. Nordby, P. Helmersen, D. J. Gilmore & S. A. Arnesen (eds.), In Proceedings
of Human-Computer Interaction INTERACT '95, (pp. 49-54). Lillehammer, Norway:
Chapman & Hall.
9. Mynatt, E. D. (1994). Auditory presentations of graphical user interfaces.
In G. Kramer (ed.), Auditory Display: Sonification, Audification, and Auditory
Interfaces (pp. 533-556). Reading, Massachusetts: Addison-Wesley.
CHI 97 Electronic Publications: Doctoral Consortium
