|
|
Christopher Fry
Scoutworks, Inc
cfry@shore.net
+1 617 863 9617
Diane Schiano
Interval Research
schiano@interval.com
+1 415 842 6099
This session brings together several leading structure visualization and browsing technologies for an entertaining yet informative "live" comparison. Users of each system will compete "head-to-head" in a series of races designed to simulate the stressful conditions under which real world browsing often takes place. Expert and novice operators will use four different visualization and browsing tools to complete a set of generic retrieval tasks as quickly and accurately as possible within the same information space. Attendees will be able to see for themselves which techniques work well or poorly as each system demonstrates its potential for a range of users.
visualization, browsing, navigation, interaction design, information retrieval, evaluation
© 1997 Copyright on this material is held by the authors.
Many innovative and effective structure visualization and browsing techniques have been demonstrated at CHI conferences over the years [1,2,3,4,5,6]. These systems range from dedicated, application-specific visualizations to general purpose schemes designed to work with any appropriately structured data set. Most have been presented as obvious improvements over more familiar browsing techniques, but these (sometimes implicit) claims have rarely been backed up with empirical data. Most systems have simply been demonstrated with a carefully chosen data set, with little or no systematic comparison against one another, against different data sets, or even against the basic browsing mechanisms provided by the common software platforms.
This session brings together several different visualization and browsing technologies for an entertaining yet informative "live" comparison. Users of each system will compete "head-to-head" in a series of races designed to simulate the stressful conditions under which "real world" browsing often takes place. Sixteen contestants (eight novices and eight experts) will use four different visualization and browsing tools to complete the same set of generic retrieval tasks within the same information space as quickly and accurately as possible. Contestants compete for fame and prizes, and have the added incentive of an audience that cheers (or jeers?) them onward while browsing along vicariously on video projection screens.
Entrants will compete two at a time, with the "winners" advancing to the next level in a short tournament format. The information structure and browsing tasks have been selected so as not to favor any particular browsing technology or approach to visualization. Contestants will be asked to locate specific items by name, location, or other criteria. Some criteria may require comparisons (e.g., "Find the leaf item with the longest name"). Contestants will know the general structure of the information space in advance, but the exact contents will not be disclosed until the race begins. The two entrants who correctly complete the browsing tasks in the shortest elapsed time advance to the final round.
The competing systems have been selected from a pool of respondents to a public call for participation For each system, attendees will see both an expert operator and a novice selected randomly from the audience to attempt the same browsing tasks. The experts are expected to demonstrate the potential of the system in the hands of a highly skilled operator (which we can assume most dedicated users would eventually become) while the novices provide some indication of the level of expertise needed to use the system effectively. Championships will be awarded in both Expert and Novice categories, after which the floor will be opened to questions from the audience. All contestants will be on hand to discuss the strengths and weaknesses of the different browsing techniques, the usefulness of the generic data set and browsing tasks, and the value of the "race" paradigm as an evaluation tool.
This session offers conference attendees a chance to experience and directly compare several state of the art browsing interfaces in an atmosphere that is far less scripted (and hence, more open to informative "surprises") than the typical conference demonstration. The race format can be both entertaining and informative. It offers an effective way to surface, in a greatly compressed time frame, the kinds of human performance issues that show up under extended, high intensity usage. Instead of being passively (perhaps even manipulatively) "demo-ed" at by a presenter on high, attendees can formulate their own opinions by directly observing performance on a set of tasks that have not been pre-selected to showcase any particular system.
The appropriateness of the race paradigm comes to mind every day as I rush to pack up my data for the train ride home. Will I forget something? Which was the latest version of that document? Do I have enough room for all these files? Was I really not linking to any external data? These are prototypical browsing problems that vary little from one system or application to another. As a designer, my hope is that this session will promote more cross fertilization between the different approaches to structure visualization and browsing by distinguishing ideas that work from those that need more work. Seeing several systems operating head-to-head on exactly the same data set should yield new insight into the mechanics of competing approaches and thus stimulate everyone's thinking in the direction of innovative, integrative new designs.
Human beings find it very difficult to conduct rational evaluations of very complex problems. We depend heavily on a subconscious "summarizing" capability that, while better than nothing, is necessarily subjective. Too often, what we think of as a rational evaluation is little more than rationalization. Thus the endless "religious" debates over which programming language, or development environment, or operating system, is "best." To this list we might now add, "browsing systems." As our access to more and more data accelerates accurate evaluation of the available browsing techniques will become increasingly important. This forum at CHI'97 is an attempt to:
Our aim is not to stifle creativity by imposing an overly constraining methodology, but to lay the foundation for an on-going activity that can stimulate discussion and innovation within the CHI community for years to come.
As an experimental psychologist interested in influencing UI design and practice, I'm very excited about the idea of a competitive "browse-off" at CHI'97. Neither the casual "demo or die" approach of all-too-many designers nor the precisely controlled "publish or perish" approach of all-too-many researchers provides what practitioners in the UI community really need, which is some useful benchmarks derived from direct performance comparisons using a variety of measures under reasonably challenging conditions. This session introduces some generic retrieval tasks that are useful for comparing browsing styles in a format designed to be fun but fair. Only robust effects are of interest here, but the results should be informative for everyone interested in creating or choosing browsing interfaces.
Extending the sporting metaphor a bit, it's clear that speed and accuracy measures (as in foot races and archery contests) can be supplemented by more subjective assessments (as in gymnastics events). The question of what the most appropriate forms of evaluation might be for generalizing into actual conditions of use is a topic of interest in itself. The session may also thus inform researchers in their efforts to pragmatically advance the state of the art in product design. What's essential is direct empirical comparisons of the human use of human interfaces, since it is inevitably the human factor that exerts the greatest influence on performance often in the most unexpected ways. My hope is that the browse-off will become a means by which the basic insights of experimentalism can help to inform UI design under conditions promoting the pursuit of excellence while also providing a compelling competitive spectacle for the players in us all.
The organizers gratefully acknowledge the inspiration and active encouragement of Dr. Ben Shneiderman as one of the prime motivators behind this proposal. We also thank Ben as well as his students past and present for their special interest and tireless effort in advancing the state of the art in the visualization and browsing of structured data. Finally, we thank Rodney Fuller for his insightful comments and constructive critique of our methodology and metaphysics in an earlier version of this proposal.
|
|