The High-Tech Toolbelt: A Study of Designers in the Workplace

Tamara Sumner

Department of Computer Science and Institute of Cognitive Science
University of Colorado
Boulder, CO USA 80309-0430
Email: sumner@cs.colorado.edu

© ACM

Abstract

Keywords:

FIGURE 1:The Toolbelt Model

The designer assembles a collection of software tools and uses them to create different design representations. In the case shown here, the designer used a word processor (MS Word1) to create text documents, a flow charting tool (TopDown1) to create flow charts, and a database ( FoxPro1) to create tables.

Typical off-the-shelf software tools include word processors, spreadsheets, databases, flow charting tools, and CAD systems. Each tool is good at making a different type of representation. For instance, spreadsheets provide a lot of support for making tabular representations while flow charting packages make it relatively easy to construct node- link types of representations. Thus, one strength of the toolbelt model is that designers can flexibly pick and choose from a selection of relatively inexpensive, off-the- shelf tools and find ones that provide decent support for making the necessary representations.

However, designers often end up in a situation where different tools are needed for each different design representation. This situation is problematic because the different representations are usually interrelated; i.e., they are different views of the same information. Now, when the designer changes part of the design in one representation, she must remember to manually carry out the related modifications in all other representations. This makes it very difficult to modify the design because small changes in one representation can trigger a series of changes in other representations. Suddenly, that seemingly small task has blossomed into a large and tedious job. This "blossoming" effect hinders iterative design because designers are reluctant to make changes due to the effort required. Following an iterative design process is not only desirable; it is often necessary, because existing requirements change and new ones are uncovered as design proceeds [1, 6]. Thus, a weakness of the toolbelt model is that it hinders iterative design.

Empirical studies indicate that many design errors result from designers' cognitive limitations when managing such dependencies across representations [8]. Besides the danger of introducing errors when making the necessary changes across multiple representations using different tools, the designer must remember to make the changes in the first place. Thus, another weakness of the model is that the resulting toolbelt provides significantly suboptimal support for the design activity in that it does not help designers to deal with either the cognitive or manual burden of managing dependencies across design representations.

The following case study serves several purposes. First, its illustrates the toolbelt model and its strengths and weaknesses. Second, a detailed analysis of this case suggest how future software tools might be designed to overcome the weaknesses identified. And third, a look at how the designers' practices, tools, and representations have evolved over time allows us to compare the toolbelt model with alternative task- or domain-oriented design environment models in order to assess the benefits and limitations these different approaches.

CASE STUDY: VOICE DIALOG DESIGN

The domain studied here is the design of voice dialog applications; i.e., software applications with phone-based user interfaces. These phone-based interfaces consist of a series of voice-prompted menus requesting the user to perform certain actions; e.g., "to listen to your messages, press 1." The caller issues commands by pressing touch- tone buttons on the telephone keypad and the system responds with appropriate voice phrases and prompts. Typical applications are voice information systems and voice messaging systems. Voice dialog applications are an important technology for many businesses because they reduce the need for human phone operators and provide callers with direct access to information concerning business services. Designing in this domain means specifying the interface for an application at a detailed level.

The Larger Design Process

Depending on the size and complexity of the application being developed, this overall design process can take from six months to two years. In summary, the design process is long, complex, highly iterative, and involves a multidisciplinary group of stakeholders (marketers, designers, testers, simulation builders, vendors). Collaboration and communication between the stakeholders is made more difficult by the geographical distance separating them. The designer, marketers, and vendors typically reside in different states and much communication occurs via conference calls using standard telephones.

What Designers Do: Construct Multiple Representations

The flow chart is the core interface design representation (Figure 2). It consists of different kinds of nodes filled with text. The nodes represents common entities found in the domain, such as audio prompts, voice menus, decisions, and audio messages. These flow charts specify the control flow of the interface, possible user actions, and the text of all audio prompts and messages in the interface. Flow- charting tools such as TopDown or MacFlow1 are used to create this representation. Flow charts are primarily used to communicate with the marketing group and the simulation builder during design. However, these charts are also used by the vendor when implementing the design and by customer support representatives who must diagnose problems with the application after its release.

Figure 2 shows an excerpt from a flow chart depicting a voice messaging system interface. The chart illustrates that to change a security code, the caller must first navigate through two voice menus (Main Menu and Personal Options) and then listen to an audio prompt requesting the caller to enter 4 digits followed by the # sign. As this example illustrates, the different domain-specific entities have unique appearances and content. These appearances and content are built up by the designer using graphic primitives provided by the flow charting tool, such as text, boxes, and shading. Voice menus consist of a title (the shaded rectangle in Main Menu, Figure 2), an audio prompt (the text under the title), and possible actions associated with touch-tone buttons (the text and small boxes below the text). Each entity in the flow chart is assigned a unique identifier according to a naming scheme the design group has created (e.g. "MM.01" for the Main Menu in Figure 2). This naming scheme facilitates long distance communication involving the marketing group and the vendor. When talking over the phone, the stakeholders can ground their design discussions by referring to "prompt PO.02."

FIGURE 2: The flow chart and table design representations.

The table representation (Figure 2) is constructed primarily for the vendor who will implement the final application on a specialized hardware platform. These tables are constructed using database systems such as FoxPro. Figure 2 shows the table entry for the Personal Options menu. For every entity in the flow chart, there is a corresponding table. Correspondence between the two representations is established via the entity's unique identifier. The table representation is a more detailed elaboration of the entities in the flow chart. It contains both redundant information and additional information to that found in the flow chart. For the example shown in Figure 2, redundant information includes the unique identifier, menu title, and the menu prompt. The descriptions of inflows and outflows is another way of representing the links shown in the flow chart. To create the table, all this redundant information must be manually rekeyed in by the designer or explicitly copied and pasted using direct manipulation. The additional information includes more implementation-oriented details such as pseudo-code descriptions of conditionals and events not shown in the flow chart such as messages spoken in response to help buttons and illegal input conditions.

Test plans are semi-structured text documents created for each task the interface supports. These textual representations are constructed using word processors such as MS Word. The plans detail the actions that must be performed and the output that will be heard along a particular path through the interface. As such, these plans reorganize information already found in the flow chart and table representations. The flow chart shown in Figure 2 would yield several test plans including one for changing the security code and one for changing the recorded greeting. These representations are used by testers who are hired to exhaustively test all features of the final product for compliance with the product's specification.

Other representations containing additional design information not found in the flow chart, table, or test plan representations are also constructed. For the most part, these additional representations are primarily textual in nature and are constructed using word processors such as MS Word.

Iterating the Design: How the Tools (do not) Fit Together

As an example, imagine that usability testing reveals that most callers want to hear their existing security code before they change it. The designer decides to add a new confirmation message between the Personal Options menu (PO.01) and the change security code prompt (PO.02). This requires the designer to adjust all subsequent identifiers to make room for the new message. Additionally, each table corresponding to a changed entity in the flow chart must also be updated with the new identifier and all the inflow /outflow lists must be reworked. Besides being tedious, this change process is fraught with potential errors in that the designer must remember which particular entities in the flow chart changed (a potentially large number) in order that the right tables be updated with the correct information. If the test plans have already been constructed, the designer must also find and update every test plan traversing the path containing the new message.

This example illustrates how seemingly simple design changes can blossom into major jobs involving extensive editing of several design representations created with different software applications. The hardest part is that the burden of the change is placed on the designer - she must remember to make the change and then correctly implement the change without introducing errors or other inconsistencies across the representations. Unfortunately, with existing levels of application interoperability, manual rekeying and copy and paste using direct manipulation are the only methods for maintaining these consistencies! Thus, the cost of iteration (both cognitive and labor) is very high due to the complex dependencies across design representations and the lack of tool support for managing these dependencies.

The Co-evolution of Representations, Tools, and Practices Over Time

At the beginning of the period of study, designs were represented using primarily textual specification documents. These textual specifications distinguished between phrases, prompts, and menus. Important information such as dynamic conditions and help messages were buried in the middle of paragraphs. These textual specifications were augmented with simple tables and flow charts at the request of the vendor organization. The simple flow charts did not contain the full text of all audio prompts and phrases in the interface. The tables augmented the textual description of prompts and menus by depicting all possible actions and responses. MacDraw1 and sometimes Excel1 were used to construct the simple tables since early versions of MS Word did not support tables.

FIGURE 3:The co-evolution of tools and representations over a three year period.

In 1991, one designer was asked to create one of the most complex applications to date. He decided it was time for a whole new approach. The textual specifications were getting so large and complex that few people bothered to read them. Those that did had trouble understanding them. This designer established a personal convention of using flow charts and a new, complex table as the primary design representations. His flow charts contained four different domain distinctions: voice menus, prompts, messages, and decisions. Tables depicted dynamic conditions using pseudo-code, showed alternative messages (such as help) and provided new details on error handling. Excel did not support the complex layout needed for the new tables, so the designer switched to a database program called FoxPro. He used MacDraw for the first flow charts simply because that tool was readily available. When he later tried to switch to a flow charting tool, he found that those early applications could not handle large designs, so he continued to use MacDraw.

The two representational systems - textual versus flows and tables - coexisted for about one year. Other designers were reluctant to switch for two reasons. First, several had long- term relationships with their vendors and marketers and were reluctant to switch representations on them. Second, constructing the flow charts in MacDraw was tedious and time-consuming.

By mid-1992, several things happened that led to the emergence of flows and tables as the primary design representations. First, new versions of flow charting tools (e.g., TopDown) had greatly improved their capacity. Using these tools made it much easier to construct the flow chart representation. Second, new designers were hired to embark on a long series of enhancements to an existing product. These new designers readily adopted the flow and table representations.

The designers began to elaborate on the flow and table representations. New domain distinctions emerged such as inputs, error handlers, and digit collectors. A new release of TopDown supported patterned arrows and designers began to use these patterns to differentiate between different types of flow control. The vendor liked the formality of the table representation and suggested ways to formalize the pseudo- code parts even more.

Today, due to the dependencies between representations, it is difficult to maintain the table representation when iterating the design. One designer experienced this in a major way when the market group asked him to redo large parts of a design for which the tables had already been constructed. Most of the tables had to be significantly reconstructed. For this reason, designers are increasingly deferring constructing the table representation until later in the design process.

Not shown in Figure 3 are further signs of major change looming on the horizon. One designer has started working with a new vendor who (so far) has not required the table representation. In another new design, the marketers were dissatisfied with the increasing complexity of the flow charts; they think simplification might be in order.

Looking back over the course of this study, it is clear that several factors influenced the co-evolution of the designers' tools, representations, and practices:

1. The various stakeholder groups. When communication with a stakeholder group broke down, designers changed the representations to overcome the breakdown. Additionally, stakeholders often suggested new ways of representing things.
2. The designers. Over time, better approaches and techniques emerged through practice. As one designer noted, "part of my job is continually thinking of better ways to do things."
3. The tools. Affordances and hindrances of the tools affected both the content of the design representations and the practices of the designers. New features in tools enabled new domain distinctions to emerge and sometimes simply enabled the tool to be used. The flexibility of the tools allowed the representations to undergo quite a bit of modification before a new tool was required. Hindrances, or lack of support for important operations, caused both delayed tool adoption and changes in design practice.

Domain Enriching

FIGURE 4:Domain- enriching is a process whereby domain distinctions emerge and representations evolve to incorporate these emergent distinctions over time.

RECOMMENDATIONS FOR IMPROVING THE TOOLBELT

The cost of iteration would be lowered if tools could be extended to automatically manage the dependencies between design representations. The observed process of domain enriching suggests necessary extension mechanisms to allow designers to "tune" their toolbelts to better support their design practices:

• Within tools, designers need to be able to describe their own layer of named, domain distinctions. Tools need to be extended to be aware that there are named distinctions, such as prompts, voice menus, and links representing different types of control flow, which have a specific look and content. The contents may also consist of other named domain distinctions.
• Across tools, designers need to be able to establish automatic mechanisms to manage complex dependencies based on shared named domain distinctions. For instance, designers should be able to establish a dependency between the flow chart and table representations stating that all prompts with the same identifier should have identical text content. As a second example, designers should be able to establish a dependency that states that whenever a design entity is added to the flow chart, a corresponding table with the same identifier and type should be automatically created and information derivable from the flow chart should be automatically filled in.

The emergence of compound document architectures such as OLE [16] and OpenDoc [12] offer hope that creating tools that provide designers with such mechanisms will soon be possible. These architectures provide a standard way for applications to expose their internal objects for extensibility and programmatic control.

However, these architectures by themselves do not guarantee that the resulting extension mechanisms will be either useful or usable for end users such as the designers in this case study. The prevalent attitude in the software development community is that the key benefit of these architectures is how they enable assembling or bundling an initial collection of smaller components to form a cohesive toolbelt [17]. As this study shows, the benefit is not in the initial bundling; the benefit is in supporting end users to evolve the tools over time. Cohesiveness emerges as end users enrich the tools with knowledge about important domain distinctions and the relationships between domain distinctions. Thus, end users need to be provided with mechanisms supporting the types of extensibility or design-in-use [9] needs uncovered in this study.

RELATED WORK

Fischer [4] has long advocated the construction of knowledge-based domain-oriented design environments. He has noted that one of the key challenges when creating such environments is providing the necessary flexibility to support tasks not envisioned by the creators of the design environment [3]. Several approaches towards providing the necessary flexibility have been pursued, such as providing end user modification substrates [5] and programmable design environments [3]. Both of these techniques have emphasized enabling users to enrich their environment by describing new entities or operations. However, these approaches do not address the need for design representations to undergo significant evolutionary change, e.g., changing from flow chart to table representations. Thus, these domain-oriented design environments may lack the flexibility needed to keep up with changing practices in rapidly evolving domains. The research presented here suggests that domain-oriented design environments with the required flexibility can be constructed by building on off-the-shelf software components.

Until recently, Nardi had advocated the advantages of task- specific tools [10]. Similar to Fischer, her recent empirical work found that specially constructed task-specific tools are too inflexible to support the activities of professionals [11]. She also discovered that the definition of "task" was highly individual and depended on the fluctuating goals of the people involved. She found that professional slidemakers preferred collections of interoperable components to task- specific tools, and she suggested that such tools be tailored to capture regularities in their daily work [11]. The results of the voice dialog design case study suggest that the ability to describe domain distinctions and establish dependencies across tools based on these domain distinctions are two specific types of tailoring that professional designers require.

While the analysis here is based on a single case study, the basic finding that professionals need the ability to tune generic tools into their personal toolbelts is a general one. Preliminary interviews with designers of a multimedia title (i.e., an interactive book) have yielded similar evidence for the need to tune toolbelts via domain-enriching. The toolbelt used by this design team consisted of MacroMind Director1, MS Word, and a half dozen other applications for creating visual effects. During the course of several months, the design team evolved a set of domain distinctions and incorporated these distinctions into their representations. They had problems iterating their design due to the dependencies between design representations.

SUMMARY

References