|
|
Computer Science Institute
Freie Universität Berlin
Takustr. 9
D-14195 Berlin, GERMANY
+49 30 838-75131
kurze@acm.org
This paper describes a rendering method for generating tangible drawings of spatial real world objects based on a theory of haptic image perception and understanding. The method is based on an analysis of the process of drawing used by blind people and on cognitive considerations. A haptic rendering pipeline has been implemented which uses methods such as folding out or flattening to create 2D images from 3D models. The evaluation currently being carried out is described and the results are discussed in a broader application context.
haptic perception, tactile drawing, blind people, haptic rendering pipeline.
Ordinarily, we render images according to our perceptual habits, i.e. based on vision and optical effects. It is the common goal of many research groups in the computer-graphics field to render "photorealistic" images which do not imitate human perception but rather the photographic camera. Consequently, rendering is defined as the "procedure for applying a lighting model to obtain pixel intensities for all the projected surface positions in a scene" [5]. On the other hand, one of the best known books in computer graphics presents a more general definition: "The process of creating images from models is often called rendering" [4].
This raises the question of how to define an image. An image is a two dimensional representation of an n-dimensional referent; it is not visual by nature. Since vision is our primary sense of perception, the majority of images are created for visual perception. This excludes people who lack the sense of vision from images as a source of information.
On the other hand tactile images are available for blind people to give them access to two dimensionally represented information such as diagrams or maps [2].
Up to now no systematic approach has been developed to design tangible images of three-dimensional objects and scenes. There are only a few heuristics about how such images should not be designed [2]. In particular visual and optical effects like perspective distortion and shadows are useless in pictures for non-visual perception.
At present, there is no standard definition of the term haptic. Here haptic perception is used in the sense of a combination of tactile (skin) and kinesthetic (muscles and joints) perception. This definition has been chosen because perceiving tangible images (like perceiving real objects by touch) requires both tactile and kinesthetic stimuli.
The present paper addresses this problem by briefly analyzing haptic perception and then describing and evaluating a haptic rendering system and its evaluation. Finally our findings are discussed and some applications of the developed algorithms outside the pictures-for-the-blind field are mentioned.
Today there is no method or algorithm known for rendering haptic images of spatial objects and scenes. For this reason, we have used a user-centered approach: In an extensive study blind people were asked to draw such motifs as a chair, a table, and a toy-truck as well as mixed scenes which they had explored in the real world. Fig. 1 shows one of the drawings produced.
Fig. 1: A drawing by a blind person showing a table with a toy truck and a bottle
Using a specially designed tactile drawing tool [11], the process of drawing haptic images was studied. The methods discovered here and in the sparse literature (e.g. [10]) have been analyzed and generalized and constitute the core for a knowledge base implemented in the haptic renderer.
As a result of the experiments three general conclusions were reached:
In general, some sort of "fold-out"-method is very often used: Objects are dissected into their (more or less flat) faces and these faces are arranged on a flat surface in such a way as to preserve topology a far as possible and still show all parts of the object. Some simplification was applied beforehand. This method is not completely new in painting: In the arts, Picasso drew faces combining the front and side views; world maps show the earth either as a Mercator projection or as sliced orange shell as in Fig. 2. Here problems arise which are similar to those we will have to deal with, e.g. preservation of angles and/or distances during the transformation from 3D to 2D.
Fig. 2: A sphere, projected (or folded out) into a plane.
A number of problems must be solved in order to design useful (i.e. recognizable) haptic drawings:
The problem of drawings and blind people can be approached from three different directions:
Physical access: One prerequisite is the availability of tactile media or devices to make the drawings accessible. Currently, some pin-matrix-devices are being developed ([19], [21], [23]). These are very expensive and still do not offer the high resolution required for detailed spatial drawings. Haptic feedback devices like the Phantom [14] or the Pantograph ([16], [17]) offer high resolution affordable access to one point at a time on a virtual surface/line via a probe. Salisbury et. al. [18] used the term "haptic rendering" for the phantom style interaction with virtual objects. Two hand navigation (or even two point orientation with two fingers) is impossible. Hardcopies (swellpaper or deep draw foils [22]) have the advantage of high resolution 2D (or 2½D) media for a low price.
Semantic access: The combination of text and graphics is a crucial factor for successful information transmission in visual and in haptic images. Since written text (in Braille) clutters an image, a combination of image hardcopy, touch tablet, and a computer system with a speech synthesizer can solve this problem ([13], [15]). The blind user puts the tactile hardcopy onto the touch tablet and explores it; whenever he or she wants to know more about a particular object on the drawing, he or she pushes on the object, and the computer can give additional information via speech synthesis. The drawings themselves must be designed manually. In fact, nearly no spatial drawings for blind people exist because there is no general methodology for producing spatial drawings for haptic perception.
Perceptual psychology: The role of touch in perception was studied, for example, by Katz early in this century [9] and later Jansson [8] and Lederman [12] with the focus on blind people. Their findings show that touch is a very efficient modality for detailed exploration of objects. A small number of psychologists do research in the field of drawings for the blind. For example Kennedy [10] used an approach similar to ours: he let blind people draw spatial scenes and drew conclusions from the results. His subjects used several methods which were also observed by the author [11]. Kennedy likewise suggested a symbolic language for some spatial properties in haptic drawings. Since he did not employ the technical devices described above, he had to rely only on the graphic and therefore had to put more information into the picture itself which made them harder to read in those cases where the contents were more complex.
Non-photorealistic rendering: Apart from the limited area of graphics for blind people, some research on non-photorealistic rendering for visual perception has been performed in recent years. Burton developed a system called Rose [1] which imitates children's drawings based on spatial models. Basically Rose replaces all parts of an object by an elliptical form when it draws. Schumann et al. render architectural scenes from CAD-data as sketched line drawings [20]. Those authors used specific line-drawing algorithms to draw an image calculated with a regular camera-based rendering algorithm.
All these research efforts can contribute to the solution of the problem of representing haptic properties of spatial objects in a 2D tangible drawing.
A literature survey coupled with our own experiments formed the basis for a rendering system which takes into consideration the capabilities of a computer system in the context of blind people's haptic perception. This rendering system is described below.
The main input for the haptic renderer is a conventional scene description, the geometric model of the objects in the scene, their properties and positions. If the model contains text information about single objects (their names) these will be managed too. This model is then fed into the rendering pipeline. The sequence of operations performed by the renderer may lead to considerable modifications in the model and in the resulting drawing. Therefore each step can be tuned to the user's preferences. See below for details. The modifications are performed in the following order:
Fig. 3 The haptic rendering
pipeline. See the text for a detailed description. (Click
on image to magnify)
1 3D-simplification: The whole scene is examined object by object. The following manipulations are performed:
1.1 Very small objects are "hidden". Since they would not be tangible, relatively small objects (e.g. a key in a big room's door) are marked as "hidden and will not be considered in the current scale. However, if only a part of the scene (e.g. the door) is rendered, the small object may re-appear and will be treated like other objects.
1.2 Relatively flat/slender objects are approximated as their closest 2D equivalent: flat cubes are transformed into squares, slender ones into lines. The same function can be applied to cylinders and other extruded objects (see Fig. 3, (1)). Flatness and slenderness are defined as the ratio between the object's dimensions in its three main axes.
This approximation may lead to "dissolved" objects (e.g. a tabletop-polygon disconnected from the leg-lines). To maintain connections and topology, which is most important for haptic perception, the approximated objects must be moved or scaled accordingly. This in turn raises the problem of the point of orientation: Which object/polygon should be kept at its place and be the reference for the objects connected to it? We decided to use the model's hierarchy and the polygon-size as key properties to choose the reference point.
2 Polygonalization: To be able to fold out faces of objects, composed objects like cubes or cylinders are disassembled into a set of polygons (see Fig. 3, (2)). The information about the original shape is kept for later use.
3 Fold out faces: Each object's faces are now checked and optionally folded so that they are arranged in one plane. This plane is defined by one face of the object, the reference face. The reference face can be either the object's biggest one or the one closest to the observer. The faces connected to the reference face are folded out by rotating them around the connecting edge (see Fig. 3, (3)).
There are many possible fold-out strategies [3]. In our context the properties of 3D haptic perception should be considered: Only faces/edges that are tangible should be folded out; the bottom face of a cupboard is not accessible by touch and therefore needs not be drawn.
4 "Flattening" linear parts: While faces are rotated around one edge, the rotation axis of lines is not predetermined. The renderer puts the axis at the connected end of the line in an orientation which is determined with a simple energy-minimization algorithm: If the line is connected to the corner of a square, the axis lies perpendicular to the bisector of the rectangular corner (see Fig. 3, (4)). Otherwise the ratio between the two edges which connect at angle is considered so that a long table would have its legs flattened near to its longest primary direction (Fig. 4 ).
Fig. 4 Two equivalent solutions for flattening a long table. The upper one is chosen if the table is surrounded by other objects, the lower one if objects stand on the tabletop.
Both fold out and flattening may require rotation of more than just one object, continuing recursively at the end/edge of the rotated part. The rating step (see below) prevents the procedure from producing images that are too cluttered. Only those lines which are connected to other parts at just one end (like table legs) are flattened.
5 Image synthesis: While the first steps manipulate the object in 3D, now the mapping to 2D is carried out.
5.1 Select image plane: Basically there are three candidates for the image plane: the horizontal plane (like in maps), the observer's "viewing plane" ("film-plane" in photorealistic images), and the predominant face plane from the scene. We implemented the first two approaches and found that in many cases they are identical: Blind persons tend to observe a scene "from above", maybe because of the map-like style of presentation.
5.2 Rotate (flattened) objects so that they lie parallel to the image plane. This may lead to images showing a bottle parallel (lying) to the table top which in turn lies parallel to the image plane. Availability of context information means this is acceptable in most cases.
5.3 Render the line drawing using the "painter's-algorithm": start drawing the most distant object and progress line by line to the nearer objects. Remember: no perspective distortion must be calculated.
The result of this process is a draft version of the image containing all the lines of the final version but is possibly hard to read tactually.
6 Optimization in 2D: The draft image may contain lines which are very close to each other and therefore might not be distinguished by touch. This problem is solved by moving the objects (in 2D) away from each other. Also overlapping lines may cause confusion. Overlappings are either used as depth cues (see below) or must be avoided by moving one object.
7 Depth cues: Since all the depth information has been removed by the rendering pipeline, one can either rely on the context information and the reader's common world knowledge (which tells that table legs reach from the tabletop down to the floor and not straight away) or reintroduce depth cues as shown in Fig. 3 (7) and described in what follows:
7.1 Line width has been used by our test subjects and can be used here to show distance: thick lines indicate close edges, thin lines distant ones.
7.2 Derived from the line width cue, we use wedges for lines perpendicular to the image plane: the thick end points towards the observer, the thin end away from him or her. This convention is also used in chemistry for drawing molecules and bonds.
7.3 overlapping lines must either be avoided or used intentionally to distinguish between foreground and background. In the latter case it must be (tactually) clear which line runs in front of the other. This is achieved by interrupting the background line in the surrounding of the cross over (Fig. 3 (7)).
Up to this point many manipulations have been applied to the original 3D model. These manipulations have been designed to match the needs of haptic perception as closely as possible. In order to present as much information as possible, some of the original features have been changed to fit the result into a 2D drawing.
The subsequent question is: How much manipulation is enough and which should be combined?
We addressed this problem in two ways: First, we allow interactive fine-tuning of the rendering steps for each single object (see Fig. 5). This is comparable to Eisenberg's approach [3]. Because the production of haptic pictures in most cases can not be done in real-time, interaction may be inadequate for blind users. So we added a rating process to measure the amount of manipulation needed to transform the 3D scene into a 2D picture. We assume that this processing load is related (proportionally?) to the mental load of the 2D to 3D transformation by the (blind) reader.
Fig. 5 The user can fine tune the renderer to personal preferences. This is the dialog box for setting the preferred 3D to 2D mapping style.
The rating scheme is sketched here briefly: The more information (faces, edges, angles, and distances) presented the higher the rating. Many manipulations and many different methods of manipulation (fold outs, recursive fold outs, flattening, cross-sectional picture) result in lower ratings. In other words: If only a few rotations are needed to show many faces and lines, the rating is high. This is done with the perceiver mental load in mind: the more mental rotations/manipulations are needed the more difficult is the process of understanding the image.
Since even a moderately complex model offers a huge set of possible manipulations, the search tree for a trial and error strategy is too big to be traversed completely. For this reason we implemented some heuristics and limited the search depth. The heuristics are represented in a knowledge base which restricts the possible manipulations. This knowledge base contains rules about haptic accessibility. For example, a kitchen stove standing freely on a surface can be explored by touch from all sides. The same stove built in a modular kitchen can only be explored from above and the front. Consequently the sides of the stove will be folded out or not. Some simple rules from spatial reasoning are applied here to identify accessible/foldable parts.
In addition the ratings for each manipulation are normalized for each user. Previous research showed that blind people have various preferences of mapping-styles. Some prefer fold outs, others flattening or contour lines. Preferences are stored in a configuration file.
Fig. 6 The renderer with the simplification dialog and a chair already simplified, but not yet folded out. In the background: a visually rendered version.
Fig. 6 b: the same chair folded out
The haptic renderer has been implemented on a PC under MS-Windows® using C++. It accepts models in the Virtual Reality Modeling Language (VRML) format which is widely used in the World Wide Web (WWW), has a clear hierarchical structure of objects in the scene, and allows name-tags on each object. The renderer offers a graphical user interface (see Fig. 6) conforming to common standards and is still accessible using only non-visual interaction methods. It produces vector graphics in the Windows Metafile (WMF) format which can be printed as a raised line drawing and a corresponding description file for exploration by blind people, using either AudioTouch [13] or a custom-made exploration and guidance system developed by the author.
The design and implementation of the haptic renderer described above is based on a user centered approach. Although cognitive considerations lead to the conclusion that the drawings from the haptic renderer are easily recognizable, an evaluation is needed to verify the usefulness and usability of the approach.
Table 1 Characteristics of the blind subjects
|
sex |
age |
congen./late |
occupation |
|
M |
24 |
C |
student |
|
F |
26 |
C |
student |
|
M |
27 |
C |
student |
|
M |
26 |
C |
none |
|
M |
53 |
C |
teacher |
|
M |
51 |
L |
teacher |
|
M |
45 |
L |
secretary |
|
M |
55 |
L |
admin. |
|
M |
46 |
L |
civil servant |
|
M |
65 |
L |
retired |
The evaluation which we carried out with 10 blind test-subjects was structured as follows:
After a brief explanation of the system and the graphic symbol used to indicate lines "into" the drawing plane (wedges), four phases of evaluation were passed through:
First, drawings of single objects were presented without any contextual information and without text-support. Here we wanted to discover, whether the drawings are recognizable even without technical or personal assistance only by exploring the hardcopy.
After reading the set of drawings, the participants assigned the drawings to real objects presented to them. Some of these were similar to each other but only one object was the 3D-referent of each drawing (e.g. three different sorts of cups and Fig. 8).
Then drawings of composite scenes were presented with our presentation system, comparable to Nomad [15] or AudioTouch [13], which is based on a touch pad and a speech synthesizer. We used speech only to present object names, no spatial information was spoken. The participants then set up the scene using toy models and we compared the scene with the original 3D model.
Finally, a complex task had to be prepared and performed: A drawing of a kitchen unknown to the participants with closets, appliances and dishes is used to prepare for a coffee-brewing task (see Fig. 7). Immediately after exploring the image, the participants entered the real kitchen and prepared coffee with the material and the objects explored in the image.
An interview about the participants subjective problems and proposals
for enhancements concluded the evaluation session.
Fig. 7 A kitchen for a real world task, rendered visually (top) and haptically (bottom).
Generally we found that complex composed scenes with only a few manipulations (folded faces) were easier to recognize than relatively simple objects folded out completely (the cup in Fig. 8). Also the participants got used to the graphic language during the time the experiment took and felt quite familiar after the 3 hours of evaluation.
While some objects were difficult to recognize with no context or verbal cue given, spatial relations between objects were recognized correctly in almost all cases.
Since none of the participants ever used haptic drawings of spatial objects before, they had different favorite techniques: some preferred fold outs others flattened images. In spite of this, different from Heller [6], we observed no significant differences in performance between congenitally and late blind were found. 2 late blind people mentioned that they first build up a "visual" 2D-image in front of their inner eye and interpret this image afterwards, which increased the mental load for them.
One congenitally blind person expected the drawing to match his conceptual image of the objects drawn: In a drawing of an audio cassette he missed the connecting line between the reel and the accessible part of the tape.
These results show that the haptic rendered images can be recognized, if some context-information is provided, e.g. the place where you would find the displayed object or its size. Certain problems were encountered with the vessels or hollow objects (see Fig. 8).
Fig. 8 A pot. Note the difference between the two circles representing the opening and the bottom.
Concentrating on haptic features to render images requires analyzing the process of haptic perception and its relation to images. As one blind participant stated, haptic pictures are in a way "more objective" than visual ones. Many properties of the referent are maintained which are not in visual ones.
However, some learning is necessary even for haptic pictures. Most shapes can be recognized easily, but their orientation in space must either be known from contextual knowledge or extracted from a small "graphical language" which must be learned much as sighted people in western cultures learn to use and understand perspective distortion in visual images. In other cultures, other methods of drawing and reading images are used. Hudson [7] and Thouless [24] describe drawings of African and Indian origin which look much like those derived from the haptic renderer.
The haptic renderer seems to use rather unconventional mapping methods for the 3D to 2D transformation. Actually comparable techniques have been used in other areas before: cartography is an example (see above). An other field of application can be text-books for medicine where spatial bones and organs are mapped on a 2D drawing. Fig. 9 shows an example from a jaw surgeon's textbook.
Fig. 9 A jawbone, folded out to the plane.
The present work will be extended to further automate the design process. In addition, we will study textures or microstructures on faces to indicate spatial orientation. They will also be used to indicate spherical surfaces.
A very promising direction of future research is the combination of hardcopy images from the haptic renderer with a force feedback device like the Pantograph or the Phantom. Some properties of lines could be presented as forces applied to the exploring finger: A hand moving downwards along a table leg is pushed forward by gravity/Pantograph while an exploration upwards needs extra force to overcome gravity/Pantograph. This will be our next step in this research area.
In this paper a new technique has been introduced, haptic rendering, as a means of producing images for interactive haptic perception. We started by investigating real users' drawing methods and developed a model of haptic picture perception which also incorporates the cognitive load necessary to read these pictures.
Computer science can help to better understand the process of image perception and synthesis. But computer science can also give blind people concrete support in accessing information that is presented today with more and more graphical and spatial style.
Using existing technology like haptic devices, a touch tablet, speech recognition and speech synthesis, properly designed images are accessible and useful for blind people. The problem of properly designing images is addressed by the proposed haptic rendering pipeline. This pipeline incorporates user preferences and image context to end up with an image designed to be perceived haptically
Besides the immediate impact on blind people, the techniques and methods described here raise some interesting questions about haptic human computer interaction in general, and about alternative rendering techniques in computer graphics. We hope that this work will lead to a more detailed exploration of the technique and its theoretical base.
I would like to thank Immo Eitel, who helped to implement the haptic renderer. The blind participants in the evaluation suggested some important enhancements to the system. This work was supported by the HCM program of the European Union.
|
|
