EVALUATING DISTRIBUTED ENVIRONMENTS BASED ON COMMUNICATIVE EFFICACY

One of the most rapidly expanding areas of HCI research is centered around supporting the collaborative endeavors of widely distributed participants. Whether the domain is advanced learning environments or desktop conferencing, the underlying goal of such efforts is to provide a maximally robust simulacrum of copresent interaction. The current trend in research is to characterize these computer- mediated communication environments -- and to argue their efficacy -- by focusing on the technical parameters of the environment. This approach places form above function, completely ignoring the communicative difficulties actually encountered by users. The work described here explores a new evaluative technique, based on methodologies originally developed by Conversation Analysts, which characterizes the communicative efficacy of a computer-mediated environment by documenting how well participants are able to maintain intersubjectivity throughout the interaction.

Keywords:

Collaborative work, distributed interaction, conversation analysis, evaluation, simulation, learning environments.

Introduction

As network connectivity and bandwidth improves, exploring ways to support the collaborative interaction of distributed participants has become an extremely popular topic of research and discussion. Increasingly powerful systems for desktop conferencing, group authoring, and distributed design [c.f. 1, 3, 5] promise to fundamentally change the way members of modern society interact with each other, both casually and in formal business contexts. Unfortunately, the aggressive pace of technical advancements has far outstripped development of metrics and techniques for characterizing and evaluating the novel communication environments being developed.

One reason for this is that the design of communication environments is fundamentally unlike the design of other engineered artifacts like space shuttles and compilers in that there is no deterministic formal model for communicative success. Where designers of a new rocket, for instance, can rely on the laws of physics, both to describe the system's behavior predictively and to rationalize it in retrospect, no such model has been developed for human communication in general. In this vacuum, the development of the current crop of computer-mediated environments has largely been driven by and oriented around the technical challenges posed by distributed interaction. By focusing on issues like bandwidth, frame rate, color depth, and sampling rate, these projects make the tacit assumption that "more is better"; that higher bandwidth and better resolution inevitably lead to a more robust and efficient interaction. Clearly, this approach places form before function, ignoring functional utility of the environment in favor of abstract parameters.

A more empirical evaluative technique is the task-centered approach [9] , developed for contexts in which participants are remotely collaborating on a specific task (e.g. shared drawing). The communicative efficacy of the environment is determined based on how quickly participants are able to accomplish the task at hand. Despite the empirical orientation of this technique, the primary focus is on the task rather than on the communication between participants as they work to accomplish the task.

By focusing on either the task space or the technical characteristics of the environment, both of these evaluative approaches fail to directly characterize or account for the communicative success or failure of the interaction, in terms of how well participants are able to maintain intersubjectivity throughout the interaction. What is needed is an approach that places the analytic focus squarely on the dialogue itself, as the central artifact of communication, documenting the ways in which participants construct and maintain shared interpretations of action in a given environment, with a particular emphasis on the communicative breakdowns experienced by participants.

The research described here investigates the utility of Conversation Analysis, an analytic methodology developed [6, 7] within the discipline of ethnomethodology, as an evaluative tool for characterizing the communicative efficacy of computer-mediated communication environments. Conversation Analysis is based on an epistemological foundation known as situated action [2, 8] , which stresses the knowledgeability of actors and how they use common-sense practices/procedures to produce, analyze and make sense of one another's actions and their local or situated circumstances. In practice, Conversation Analysis works to document the ways in which participants maintain the moment-by-moment coherence of conversation in response to the contingencies of local context.

The application of ethnomethodological techniques to computer-mediated environments raises several important questions and challenges which are addressed by this research:

1. How can the analytic techniques of Conversation Analysis be extended to distributed audio-visual environments? Existing work focuses largely on telephonic exchanges; extensions to document the effects of the visual channel (as mediated by the electronic medium) must be developed. Though some ideas in this direction have been explored [4] , it is not clear how they apply to distributed scenarios.
2. Can ethnomethodological techniques be applied to generate meaningful evaluative metrics for communication environments? Traditionally, Conversation Analysis has been used as a purely documentary tool, providing empirical evidence of how participants collaboratively regulate their interaction. Whether the episodes documented through this analysis can be summarized quantitatively to serve as a reliable characterization of communicative efficacy constitutes the central question addressed by this research.
3. Can the information gained using this technique usefully inform (re)design of distributed collaborative environments? While the proposed technique provides a way of characterizing the communicative efficacy of an environment, it is not clear whether this characterization can be applied prescriptively to somehow streamline the interaction.

EMPIRICAL STUDIES

To test the methodology of Conversation Analysis as an evaluative tool, the problem-solving interaction of two participants will be recorded in three communicative scenarios. In the first scenario, participants are seated at the same machine, and are able to communicate freely as they work. This establishes a baseline for evaluating the following scenarios. In the second and third scenarios, participants perform the same set of tasks, but are seated at machines in separate rooms communicating, respectively, by audio-only and audio/video link. The task domain for the experiment is basic cardiovascular physiology. Working together, participants are asked to use a graphic cardiovascular simulation tool designed to teach basic cardiovascular physiology to construct and interpret the behavior of several simple cardiovascular systems. Mutual access to the simulator in the distributed scenarios is maintained by running the simulator in a shared workspace displayed on each participant's machine.

The interaction of the two participants in each scenario is recorded on synchronized videotapes, which capture the shared workspace (i.e. the simulator) as well as frontal views of both participants. Each interaction is then painstakingly transcribed, preserving such features as timing between utterances and prosodic effects. Non-verbal behaviors (e.g. facial expressions, actions using the cursor within the shared workspace) and their relationship to the utterances of participants must also be faithfully represented. The transcripts for each interaction are analyzed both quantitatively (e.g. overall time, number of adjacency pairs) and qualitatively (e.g. the number, type, and severity of communicative breakdowns), yielding a characterization of communicative efficacy within the interaction. Results from individual interactions in each communication environment are combined to generate an overall characterization of the environment. Finally, these characterizations will be analyzed to assess the utility and accuracy of the evaluative technique itself.

CONCLUSIONS, STATUS AND FUTURE WORK

In sum, this work explores a technique for directly evaluating the efficacy of electronic communication environments. While existing approaches accept technical quality or the time to accomplish certain tasks as tacit evidence for the communicative efficacy of the environment, the technique proposed here examines the interaction itself, documenting the communicative troubles of participants.

The data for the three collaborative scenarios is currently being collected and analyzed. Results should be available by the Spring of 1995. In future work, I plan to further test the methodology developed here by using it to characterize interaction in a broad variety of electronic environments. Also, the utility of this methodology as a tool to inform the design of distributed communications environments must be explored.

1. 1. Abel, M.J., Experiences in an Exploratory Distributed Organization, in Groupware and Computer-Supported Cooperative Work, R.M. Baecker, Editor. 1993, Morgan Kaufman Publishers, Inc.: San Mateo, CA.
2. 2. Garfinkel, H., Studies in ethnomethodology. 1967, Englewood Cliffs, NJ: Prentice-Hall, Inc.
3. 3. Harrison, S. and S. Minneman. The Media Space: a research project into the use of video as a design medium. in Conference on Participatory Design. 1990. Seattle, WA:
4. 4. Jordan, B. and A. Henderson, Interaction Analysis: Foundations and Practice. 1993, Unpublished manuscript.
5. 5. Root, R.W. Design of a multi-media vehicle for social browsing. in CSCW'88. 1988. Portland, OR: ACM Press.
6. 6. Sacks, H., E.A. Schegloff, and G. Jefferson, A simplest systematics for organization of turn-taking in conversation. Language, 1974. 50(4): p. 696-735.
7. 7. Schegloff, E.A. and H. Sacks, Opening up Closings. Semiotica, 1973. 7: p. 289-327.
8. 8. Suchman, L., Plans and Situated Actions: The Problem of Human-Machine Communication. 1987, Cambridge, England: Cambridge University Press.
9. 9. Tatar, D., Using Video-Based Observation to Shape the Design of a New Technology. SIGCHI Bulletin, 1989. 21(2): p. 108-111.