Conversational Dialogue in Graphical User Interfaces:Interaction Technique Feedback and Dialogue Structure

Keywords:

Introduction

Graphical user interfaces, specifically those following direct manipulation principles, have made computers easier to use and accessible to more users. But graphical user interfaces are still difficult to design due in part to the lack of principles describing how users behave in an interactive human-computer dialogue. This work studies some aspects of human conversation in an effort to identify principles that can be used to build more natural human-computer dialogues. While some principles of human-human conversation have been studied in natural language systems, little research has been done on how those principles apply to graphical user interfaces. The principles studied here will in turn be used as requirements of the design of a dialogue controller in a UIMS.

Dialogue in human conversation is the exchange of ideas, information, commands, etc. between two or more participants. The dialogue flow follows certain patterns and rules that make it easy for participants to exchange information. In a conversation, there are usually several dialogues being interleaved; some parts are suspended and resumed later, others are suspended never to be resumed again. It is this flexibility in the flow of human dialogue that we would like to achieve in human-computer dialogues.

The theory of contributions to conversations, as presented in [2-4], has already been applied in the study of human-computer dialogues. In [5], Payne analyzes the interaction in MacDraw based on presentation/acceptance trees. Brennan and Hulteen have defined a model of adaptive speech feedback [6] based on the states of understanding (SOU) principles. The work described here uses the SOU principles as the basis for a feedback model for graphical interaction, such as click-and-drag operations.

SOU is one of many ways participants in conversations fulfill their goals in a collaborative process. The "listener" provides evidence of his/her SOU thus allowing the speaker to adapt according to the specific SOU. If a SOU is not clearly communicated by the "listener", the "speaker" will seek further evidence that the communication was received; that is the "speaker" seeks extra evidence to achieve grounding [4]. This process provides the starting point for the proposed model of feedback for graphical interaction techniques.

PROPOSED WORK

The work of this dissertation involves the definition of a behavioral model of feedback for graphical interaction techniques and the definition of a software runtime architecture for a UIMS dialogue controller based on conversational principles. The research will seek evidence to validate the behavioral principles presented in the feedback model. The runtime architecture incorporates the feedback model but also supports meta-dialogue turns in multi-threaded dialogues.

Feedback Model

The model of feedback for graphical interaction techniques defines the different types of feedback for interaction techniques that must be provided to the user, the purpose each feedback type fulfills in a collaborative dialogue, and the behavior each feedback type is intended to produce in the human dialogue partner. The model also accounts for a collaborative view of graphical interaction techniques.

The model contains five "simulated states of understanding" (SSOU): busy, buffering, ready, processing, and reporting. Each SSOU feedback state is intended to produce conversational behavior on the part of the user similar to that produced by the corresponding SOU [2]. The feedback model prescribes the type of feedback to be provided based on its conversational purpose but not the specific form in which it must be provided.

The principles defined in the feedback model will be evaluated with a formal evaluation. Participants will be asked to perform a problem solving task doing click-and-drag manipulations. The task involves pairing objects on the screen by dragging an object over to another object that matches in shape and color. The system at times will be "busy" and thus will ignore user actions. This state roughly corresponds to SOU 0: "Not Listening." At other times the system will be "delayed" and user actions will be accepted but will be processed after a "delay" to simulate the system being overloaded. This state doesn't correspond directly to any one of the SOU, but it is typical of human-computer dialogues and thus has been included in the model. Two treatments include providing feedback based on the proposed model and feedback based on traditional practices. Performance will be computed based on a task score, time to complete a task, and the number of repairs (incomplete or ignored user actions) in a task.

Software Architecture for Dialogue Controller

A limitation of dialogue controllers in UIMS is that they do not provide good facilities to define "global" operations, such as interruptions and cancellations. Most controllers define a global action or a global state to handle this requests. This makes it very difficult to handle these global requests in a generalized way based on the state of the dialogue in which they are invoked. The model proposed here defines interruptions and cancellations in a localized structure with the handling of them depending on the structure of the dialogue where they are invoked.

Many existing software notations and implementations require that the whole dialogue structure be defined at design time. Current software practices prohibit this design time definition. For example, software is designed in modular parts either through the combination of several applications to achieve one goal, such as Mosaic, or through the combination of runtime components to produce one application, such as OpenDoc or OLE. Due to this runtime combination, some parts of the dialogue structure have to be built or combined at runtime. The model proposed here allows for design time local structures to be organized at runtime to build the global dialogue structure.

This research defines a runtime architecture for a dialogue controller such that:
1) interruptions, cancellations, and suspensions are viewed as meta-dialogue turns in a human-computer dialogue,
2) handling of these turns are negotiated based on the dialogue context in which they were invoked,
3) handling is based on dialogue structure which is built at runtime by combining design time components.

The runtime architecture will be evaluated by building a prototype dialogue controller for a user interface management system. Several dialogues requiring negotiation over interruptions and cancellations will be constructed and usability studies will be performed on these to exercise the runtime architecture. The implementation of these dialogues will be compared with other notations and other models.

CONCLUSION

ACKNOWLEDGEMENTS

The Dialogue Research Group at the Naval Research Laboratory helped form many of these ideas. This work has been funded by the Economic Development Administration of Puerto Rico.