David Allport (1), Earl Rennison, Lisa Strausfeld
MIT Media Lab
20 Ames St.
Cambridge, MA 02139
E-mail: dea, rennison, straus@media.mit.edu
1. Visiting scientist from HP Laboratories, Bristol, England.
By a comparison of the values received at the four corners of the monitor we can derive position sensing for
up/down, left/right and forward/back, and thus implement a "virtual 3D joystick" where displacement from
a central position is mapped to the velocity of navigational movement. This virtual joystick has the
interesting property of being manipulable with one or two hands without any changes being made to
hardware or software.
Figure 1
a) Steering a camera to the left, objects move right on screen.
b) Indicating the direction to move, objects move left on screen.
The same hardware and software can support multiple different navigational models simultaneously, with
the crucial variable being the conceptual mapping the user makes, as in the classic "which direction should
the arrow point on a button?" problem. With our interface, the user can make different gestures supported
by the different models, yet given the physics of our input sensors, the distinction for the input device
between the gestures shown in (a) and shown in (b) is negligible.
With this same "flying fish" sensing hardware we can implement a variety of other virtual 3D input devices
in software. These could provide absolute position-to-position mapping (like a 3D tablet), or relative
displacement-to displacement mapping (like a 3D relative mouse). But for navigation through very large
spaces an absolute position-to-position mapping is impracticable, and a relative displacement-to-
displacement mapping would be inconvenient because of the constant "nulling problem"[1] when moving to
parts of the space not previously displayed.
Figure 2 Financial Viewpoints: A Euclidean information space.
Our experiences of navigating in this space have raised a host of issues concerning the underlying
conceptual models. There are many different ways to think of what is flying through the space; is it the
user's body, a steerable vehicle containing the user, a camera through which the user is looking, or is the
user (directly or indirectly) acting upon the information "objects" themselves? (see Figure 1a and Figure
1b).
We are experimenting with providing a number of different "virtual intermediary devices", where the
models which map gestures to navigational actions can change as the user moves into different spatial
contexts. User expectations based upon familiar real-world motor skills and concepts can be used to
reinforce the sense of immediacy of interaction.
A further set of issues arises when we are navigating through non-Euclidean spaces. We have developed an
abstract news and information space whose form is 2-1/2D, in that it uses an orthogonal projection of
multiple overlapping layers of news information [2], (see Figure 3). However, as the viewer moves closer
towards an object representing a particular topic, more details relating to that object appear, effectively
reconstructing the space. As a result, navigation and movement become the only language of
communication necessary for the dialogue between user and computer. The user not only browses "system
output" by navigation, but also expresses "input commands" in the same manner.
Figure 3 Galaxy of News: An abstract, non-Euclidean information space
Our bodies do not experience non-Euclidean spaces, but we can still use a natural gestural vocabulary that
can be dynamically mapped onto different navigational modes by system software and/or by the user, to
reflect whatever conceptual model is most appropriate for a given spatial context.
Abstract
Navigating large multidimensional information spaces presents a set of unique problems for user interface
design. The key challenge is not to provide fast and accurate object manipulation, but to prevent the user
from getting "lost", and to provide an intuitive way to move through the space. We have developed an
interface that uses electrostatic field sensing to interpret natural hand gestures as motion controllers. We are
investigating the conceptual models that provide intuitive mappings from hand gestures to movements in
multidimensional information space.
Keywords:
Gestural navigation, conceptual navigation models, input devices, abstract information
spaces
The "flying fish"
We are using a "fish" electrostatic field sensor in human transmitter mode [4]. A cushion on the user's chair
capacitively couples energy into the body, and four receivers mounted in "wings" around the bezel of a
workstation monitor sense the fields emitted from the user's hands. An input on the fish's user port also
allows two foot-switches to be connected, enabling us to select four different navigational modes.
Conceptual models of navigation
The "flying fish" provides a natural way to move through a 3D "virtual world", using gestures to point in
the direction that we wish our virtual bodies to fly. We have implemented a financial information space
whose form is analogous to a city with buildings, streets etc.[3]. From a distance, it can be thought of as an
object like a planet floating in space. From close up, we can think of it as if we are "inside" the virtual
buildings. However, it is unlike traditional virtual worlds in that the contents designated by the forms are
entirely conceptual (see Figure 2).
Summary
We do not think that there is a single "right answer" to the question of how to move through
multidimensional spaces. Rather we think that there are a range of natural gestures that users can employ to
indicate 3D navigation, a large number of different types of virtual space that can be constructed, and a
variety of different conceptual models that can be used to understand how gestures map to movements
within (different parts of) these spaces. We have developed hardware and software that allows us to explore
a large number of combinations of these different conceptual mappings.
