CHI '95 ProceedingsTopIndexes

InteractiveDESK: A Computer-augmented Desk Which Responds to Operations on Real Objects

Toshifumi Arai, Kimiyoshi Machii, Soshiro Kuzunuki, Hiroshi Shojima

Hitachi Research Laboratory, Hitachi Ltd.
7-1-1 Omika-cho, Hitachi-shi, Ibaraki-ken, 319-12 Japan



Office and engineering workers' workloads are reduced with a computer-augmented desk named InteractiveDESK. The desk has a large desktop display with a pen-input facility and an ordinary upright display with a keyboard, thus integrating features of conventional systems and pen-based systems. The desk detects the operations on real objects on its real desk top, and responds to the operations to reduce users' workloads. The prototype of the desk assists users in switching input methods and retrieving electronic files.


Augmented reality, pen-based system, computer-augmented desk


User-friendly interfaces are essential to make computers easy to use for everyone, and many pen-based systems which realize friendly interfaces are commercially available. However, most pen-based systems are designed as mobile computing tools, and have some drawbacks when used as computing tools in an office. For example, small displays and text-input methods based on handwriting recognition techniques are not desirable for certain office tasks. Because portability for mobile computing is not always compatible with usability in an office, it is necessary to study pen-based systems not only for mobile computing, but also for office or engineering environments.

We think that a pen-based system used in an office should be integrated with conventional computer systems, which have upright displays and ordinary input devices (keyboards and mice), and it should provide a large display to replace an entire desk top. Then we are trying to develop a computer-augmented desk which has an upright display embedded in a partition facing the user and a large desktop display with a pen-input facility on its surface. The desk enables the user to use the keyboard and mouse with the upright display, and to manipulate and make annotations on "electronic paper" on the large desktop display with the pen. Switching the input methods and displays ensures work is carried out effectively. We have named this desk the InteractiveDESK.

The InteractiveDESK, on the other hand, forces the user to manage multiple input methods and a large working space. It is accordingly necessary to provide certain assists to reduce the workloads. Then we try to utilize operations on real objects. There are many operations on real objects in the real world, and almost all are ignored by computer systems, even though the operations are unavoidable for users in most cases. Our system tries to detect such operations, and responds to them to reduce the number of the commands that the user is expected to perform.


Figure 1 shows a schematic of the InteractiveDESK. The desk features a camera to monitor the real desk top, and it processes the camera images to detect the user's operations on real objects. Figure 2 shows a photo of an InteractiveDESK prototype. A 26" CRT display is used in place of the partition display, and a projector display with an A1-sized transparent tablet is used to provide the desktop display with a pen-input facility. We are studying some new techniques for intuitive and efficient interaction on this prototype.

FIGURE 1: Concepts Implemented on the InteractiveDESK

FIGURE 2: Photo of the Prototype

Switching the Input Methods

The system provides a keyboard connected to the upright display for text entry, and a desktop display with a stylus pen for annotating. However, it is sometimes inconvenient to switch input methods, so the system should assist the user in switching them.

In order to make this possible, the positions of the keyboard are utilized to anticipate the current input method. If the keyboard is placed at the center of the desk, the system displays documents on the upright display. Otherwise, it displays documents on the desktop display. The prototype uses the desk monitoring camera to detect the position of the keyboard.

Operations on the keyboard to change its position are unavoidable for the user in many cases. But, ordinary systems are not concerned with these operations. Our system, on the other hand, tries to respond to them to reduces the users' workloads on switching the displays in accordance with the input methods.

Retrieving Files

In this section, we propose another feature which enables users to retrieve electronic files or directories maintained by computers easily. The InteractiveDESK provides a method for linking real objects to electronic files or directories. For example, it is possible to link a scrapbook holding some transparent sheets to an electronic file holding the original figures for the sheets. Once the link has been established, the user can retrieve the file by simply putting the scrapbook on the desk, without having to consider where the file is stored. Most users would probably prefer to retrieve files by just showing real objects rather than digging into the deep hierarchy of a file system.

In the prototype, when a scrapbook is placed on the desk, the system displays the list of the files linked to the scrapbook. The user can retrieve the appropriate files easily by choosing items from the list. If some documents have been displayed on the desktop when the scrapbook is placed on the desk, the system assumes the user is trying to link the scrapbook to those documents, and it shows a button on each document. The buttons enable the user to link the corresponding document to the scrapbook. In the current implementation, the prototype is able to detect only a few real objects with differently colored tags.


Weiser[1] has described a "tab," a "pad," and a "board" as essential information instruments, but we would like to add "desk" to the set. Desks have been the center of the space for intellectual human activities, and they will retain their importance. We want to bridge the real and electronic worlds via desks.

Wellner, working alone[2] and with Newman[3] have developed a system called DigitalDesk. The idea of realizing interactions between computers and users with real objects is proposed in the system. In the system, a method for monitoring the real desk top is used to allow users to input commands not only through electronic devices, such as a pen or a mouse, but with unencumbered hands and papers. In our system, however, the method is used to reduce the number of commands users are expected to input. But, the features of the DigitalDesk can also be applied to our system.

The idea of using real objects to retrieve electronic files is similar to the idea proposed by Johnson, et al.[4]. They developed a system called XAX, in which users can retrieve electronic files with a paper form on which computer-readable marks are printed. In the system, the form is sent to the document server by a FAX or a scanner, and the results are sent back to the user by the FAX or a printer. The XAX does not require any particular equipment. On the other hand, our system requires a camera to monitor the desk top, but it allows users to establish and trace links interactively.


We have described the features of the InteractiveDESK, which responds to users' operations on real objects on the desk top to reduce the users' workload. We are now investigating application fields to which our system is well suited. For example, it will work well in CAD systems that require a large working space and various input methods. Linking real objects to electronic files will improve the interfaces used by office workers.


1 Weiser, M. Some Computer Science Issues in Ubiquitous Computing, Communications of the ACM Vol.36, No.7, 1993, 74-84

2 Wellner, P. The DigitalDesk Calculator: Tangible Manipulation on a Desk Top Display, Proceedings of UIST'91, 1991, 27-33

3 Newman, W. and Wellner,P. A Desk Supporting Computer-based Interaction with Paper Documents, Proceedings of CHI'92 Conference on Human Factors in Computing Systems, 1992, 587-592

4 Johnson, W., Jellinek, H., Klotz, L. and Card, S. Bridging the Paper and Electronic Worlds: The Paper User Interface, Proceedings of INTERCHI'93, 1993, 507-512