Abstract

Keywords

1. Introduction

While this framework is claimed to be appropriate for the capture and subsequent use of DR, there is little empirical data on how useful DR documents constructed using this framework really are (but see [13] for a review related to this issue). By DR documents, we mean paper documents or electronic documents containing a semi-formal description of the DR. This paper presents a first experiment aimed at evaluating the usefulness of such documents.

The first requirement of this evaluation was to record the DR of a design project. This has been possible with the collaboration of a French aerospace company, Aerospatiale. We recorded the DR of a real industrial project conducted in a design office of this company. This design office specialises in mechanical engineering design. The goal of the design project was to evolve one of their products to meet new requirements (redesign). We examined the first phase of this project, which lasted for nine months. This study received the project manager's support, because he was convinced that a DR document could be a helpful information tool for those in charge of the next phases of the project.

To record the DR of this project, we used the QOC formalism [12]. Two reasons explain this choice: (1) to carry out a first experiment in an industrial setting, we needed a fairly simple method; IBIS (see [6]) and QOC meet this requirement; (2) one of the main differences between IBIS and QOC is that it is necessary to make design criteria explicit with QOC, while they may remain hidden in the arguments defending the positions with IBIS. The ability to be directly informed about the design criteria was judged by the project manager to be an indispensable requirement. Indeed, this makes it possible to immediately know the project objectives, and thus, to better assess the solution.

The record of the DR was ensured by two scribes, a company documentation specialist and the author. This was a condition required by the design office chief, who did not believe that the designers would take the time to document the DR themselves. Both scribes played a purely passive role in the design process. The DR record was based on the analysis of 4 audiotaped design meetings, bringing together mechanical engineers and technicians. Each design meeting lasted two hours.

The DR document was structured by reporting the new facts discussed at each new design meeting. After each formalization of a meeting using QOC, we asked the project team to review the new version of the DR document. The final document contains 31 questions, 46 options, 25 criteria, and 12 justifications of the option assessments.

This document was paper-based. Indeed, the company where the study was carried out preferred to first test the DR capturing method without any computer support, before investing in such a tool. One consequence of this choice is that the evaluation performed and presented in this paper does not cope with some aspects affecting the usefulness of DR documents. In particular, we do not evaluate how easy the access to the DR is, because this feature is completely dependant on the provided means for access.

Because the document was paper-based, we tried to respect the following rules intended to make it easier to read : (i) one problem per page; (ii) a recall of the origin of each question with a title at the top of each page, (iii) a graphic symbol for each type of node (questions were encircled, chosen options in a square frame, etc.), (iv) each node numbered to provide annexes with complete lists of each type of node (list of questions, list of criteria, etc.), with the number of each node and its wording.

In order to explore how useful DR documents are, three questions were addressed: (1) Do designers confronted with an unknown design need to know the design rationales? Depending on this answer, we will be able to say at least if DR documents should be useful; (2) Do designers use design rationale documents? Indeed, no matter what the designers' need for DR is, if they do not use DR documents, this would highlight serious limitations of our approach; (3) Do we succeed in capturing the rationales looked for by designers? Here too, if designers need DR, but if they do not find the facts that they are looking for in the DR documents, we will need to revise our approach. This paper presents the results of different analyses related to these three issues. We will finally conclude by recalling the lessons learned by this study.

The experiment presented in this paper was carried out in the field of mechanical engineering design. However, we believe that the lessons learned by this study should be valuable for other fields of design, because our conclusions are more concerned with the approach for capturing DR than with the application domain.

2. Method

During an evaluation meeting, a designer was provided with two kinds of information: (i) information describing the solution, and consisting mostly of blueprints; (ii) information describing the reasons for design decisions, and reported in a paper-document.

Designers were free to use blueprints and DR as they chose: this means that we did not force them to use the DR. This was necessary because we were interested in knowing how they would spontaneously use it. In addition, the documentation specialist and I were present to help find the information looked for by designers, or to answer their questions. Actually, our initial intention was to let the designers finding themselves their answers in the DR. However, because some of them systematically asked questions to us, we decided to give the appropriate answers, and thus, not to jam the natural process. One consequence of this decision is that we will not be able to draw conclusions about the usability of DR documents (but see [13], which reports some empirical data on usability of DR).

Each evaluation meeting proceeded as follows: we described the tasks assigned to the designer, and presented the documents available. Because the DR was a new type of document, we described its properties and how it should be interpreted. Then the designer began his/her tasks, without time constraints. At the end, we asked a few questions on his/her opinions concerning the usefulness of the DR. These evaluation meetings lasted about one hour each, and were all taped.

3. Do designers confronted with an unknown design need to know the design rationale?

One hundred and thirty eight questions were extracted from the 6 evaluation meetings. These questions were coded according to the following categories:

1. Blueprints questions: they aim at clarifying the objects represented (e.g., pointing at a blueprint "what is this element?"); this category also includes requests for other blueprints.

2. Structure questions: they concern the material used, the forms and the size of the objects composing the whole artifact, and the properties of the interfaces between these objects (e.g., "how is the plate that carries the equipment fixed?")

3. Behavior questions: they focus on the way the artifact carries out its functions (e.g., "What are the moves allowed by this arm?")

4. Design rationale questions: they ask about the initial motivations of the project, the reasons for the design decisions, the different possible choices and the comparison between these different choices (e.g., "Why 6 fuel tanks, and not just 4 bigger ones?").

5. History questions: typically, they include questions about what has changed between the previous design and the new one, mostly concentrating on the structural aspect of the artifact ("Did you change the way the plate is fixed?")

6. Document questions: they aim at clarifying the terms used in the DR documents, and to better understand the way these documents were constructed (e.g., "How did you decide to link one criteria to one option?").

Figure 1 gives the percentages of occurrence that we found for each category of questions.

Figure 1. Percentage of questions during evaluation meetings

This diagram shows that DR questions are by far the most frequent. In addition, this result remains true whatever the designers' competence is (63% of DR questions for engineers; 49% for technicians).

Discussion

However, one must note that these results contradict Herbsled and Kuwana's observations [8]: their analysis of questions in design meetings stated that design rationale questions (referred to as why questions) were rare. They then proposed different possible explanations of this result, among which: "the rationale behind decisions is simply relatively unimportant" (p.13). If this is true, how can we explain the high frequency of DR questions in our study ?

The discrepancy between our respective results might be due to the fact that question asking is a situated, context-dependent activity. In other words, questions observed in a specific situation, namely design meetings, cannot be considered as representative of questions asked in another situation, namely reuse situations. And indeed, there are two major differences between both situations:

1. Some previous studies [9] highlight the fact that explanations are most of the time given spontaneously in design dialogues, and not in response to a question. More precisely, in analyzing database design dialogues, it was found that 83% of the explanations were volunteered when one member of the group wanted to submit a proposal or to address a critic. As a consequence, one observes few explicit why questions in such design dialogues.

2. Furthermore, because participants in a project accumulate knowledge on this project (the requirements, constraints, problems with different alternatives, etc.), they are able to infer, in some cases, why such a proposal or such a criticism is relevant, without asking the corresponding why question.

Both aspects are present in the design meetings studied by Herbsleb and Kuwana, and missing from our evaluation meetings, where designers had to work with a previous design without having the historical knowledge of the project. Our conclusion is that knowing the design rationale is important, especially when the project team works on someone else's design.

This conclusion would appear as a strong argument for claiming the usefulness of DR documents if at least two conditions were satisfied: the first condition would be that designers want to use DR documents; the second condition would be that DR documents constructed with the QOC method contain the information looked for by designers. We test these conditions in the two following sections.

4. Do designers use design rationale documents?

The opportunistic use of DR

Indeed, very often when a designer raises a DR question, he/she first looks more precisely at the blueprints. In doing so, he/she tries to gain more information on the structural and functional aspects of the artifact. The new information found in this way (they also might ask a question) helps to infer the reasons for the design decisions. The following extract gives an example of how a better understanding of the artifact helps to discover the design rationale ("D" stands for designer, "DS" for documentation specialist, "* *" for personal comments):

D: (talking to himself) So, what lead to the choice of these tanks? (Now, he turns towards us) What is the principle of these tanks? Is the pressurization external or internal?

DS: Internal.

D: Ok, so one may assume that the cost was a criteria of choice in this case (*in fact, internal pressurization is cheaper than external pressurization*).

This extract highlights the fact that an improved understanding of the structure or of the behavior of the artifact makes it possible in numerous cases to answer DR questions. However, this is not always possible. The designer may then try to find the information looked for in the DR. This is what we call an opportunistic use of a document.

This kind of use could have been expected if we had related the use of DR with the use of manuals, guidelines, written instructions, etc. Indeed, several user studies (e.g., [4]) reported that users usually prefer to act before reading any documentation. This is what seems to happen with our designers.

No other study could have prepared us to observe the opposite way of using the DR, which we call an extensive use.

The extensive use of DR

D: (reading a DR question) "How can we reduce the vibrations at the DAAV level?"...first option "Design with DIAS"...ok, but it does not satisfy the weight criteria Right. "New design of the high level without DIAS" (*this is the second option*) ... ok ...could we look at the blueprints now (turning towards the blueprints) ...Ah ok, so here, the bottom of the ferrule comes under the DAAV, doesn't it ? (*this structural item explains how it is possible to reduce vibrations and weight at the same time by comparison with the previous design*).

DS : Yes, that's it.

In contrast to the opportunistic use, an extensive reading of the DR then mainly implies the following behavioral pattern: from DR to blueprints, or in other words, from design reasoning to the solution.

Note that for designers who extensively read the DR, a paper format of the DR might be more appropriate than a computerized format.

Discussion: An explanation of why designers use the DR differently

However, it is also important to stress the fact that the reasons found by these more experienced designers were not always right, even if they were satisfied by them[1]. This reveals that designers do not look for the real reasons for design decisions: finding out some good reasons seems sufficient. This also explains the fact that some designers rarely consult the DR, even if they are interested in the design rationales.

All these considerations lead us to revise our initial assumptions about DR documents: it is not because such documents (whether paper or electronic) are available that they will be consulted, even if they contain information needed by designers!

Our approach to circumvent this difficulty, at least partially, consists in including some facts which make up the design rationale such as comments on the blueprints (or any representation of the solution). Such an approach assumes that the documentation specialist will be able to select the important facts from the sets of facts composing the DR. It also assumes that designers will use an interface that offers them the possibility of hiding or showing the DR comments.

5. Do we succeed in capturing the rationales looked for by designers?

To address this issue, we accounted for the DR questions expressed by the designers that were answered by the DR. We considered that a designer's DR question was answered by the DR either if the designer found an answer in it, or if we provided him/her with a verbal answer which existed in the document. Seventy six DR questions were identified in this way.

We found that 41% of the designers' DR questions (31 out of 76) were answered by the DR. In addition, we noticed that this result did not significantly vary according to the designers' competence (43% for the technicians, 38% for the engineers).

These results alone are difficult to judge: for some, they may be viewed as a good result, because nothing existed before! For others, they may be considered as insufficient. To better assess these results, an analysis of the causes explaining why some DR questions were not answered by the DR was performed.

Analyzing the discrepancy between DR questions and the DR captured

                 Types of cause                     Frequency    
                                                        of       
                                                    occurrence   
1. Causes related to the analysts:			14            
	- Known justifications not reported			10       
	- Known issues not reported				2        
	- Known criteria not reported				2        
2. Causes related to the QOC method:			1        
	- Lack of weighted assessments				1        
3. Causes related to the readers:			19       
	- Misconceptions					12       
	- Inappropriate questions (Need for formal		7        
proofs)                                                          
4. Causes related to the project team: 			11       
	- Issue not handled					2        
	- Criteria not taken into account			2        
	- Option not considered					4        
	- Lack of global assessment				3        

Table 1. The causes and their frequencies explaining the discrepancy between designers' DR questions and the content of the DR document

1. Causes related to the DR analysts: 14 designers' questions were not answered because the analysts had not reported some facts, even though these facts were discussed during the design meetings recorded. Looking at these facts more closely, we found that some of them were not reported because the analysts had simply forgotten to report them. Others were not reported because the analysts might decide, not always consciously, that given information was not important, because given the current state of design elaboration, it was not essential. For instance, during the project, the designers had decided to elaborate one alternative among three, simply because they wanted to quickly test one idea, and not because this alternative was judged better than the other ones; however, at the end of the first phase of the project, only this alternative was elaborated, and during the evaluation meetings, the designers were provided only with the blueprints of this solution. Some of them asked why. We knew the answer, but it was not reported in the DR. In short, for different reasons related to the analyst, some known facts may not be included in the DR document. This observation may induce two opposite reactions: one may want to reduce this number, for instance by asking the analyst to be more "rigorous", or by introducing more controls in the DR elaboration. We believe that such an approach will be inadequate. Indeed, one has to note that, because it was the first time that we tried to capture the DR, we were as "rigorous" as possible. Introducing more controls does not seem to be the solution either: we did ask the project team to review the DR document after each new meeting; this team was also motivated by the fact that it was the first time that such an experiment was attempted within the company. However, these review meetings also were not sufficient to ensure the completeness of our document.

Considering these arguments, another reaction is possible: we need to introduce an iterative approach to DR capture. We believe that a DR document must be conceived as something unfinished at the end of a project. This document should be improved with the questions raised by its subsequent use. This improvement process should be centralized at the level of the DR analyst: he/she should receive the designers' questions, then answer them, and at the same time decide if some facts need to be added to the existing documents or not. If a computer tool is introduced to access the design rationale, this means that we must provide the user with the possibility of asking questions and receiving answers through the system (electronic mail for instance).

2. Causes related to the QOC method: the only problem noticed with QOC concerned its relative inability to represent weighted evaluation (X is better than Y according to criteria Z). Actually, it is possible with the use of link-thickness, but readers often fail to correctly interpret it. As a consequence, it was not possible to answer questions like "According to criteria Z, is X or Y better?" Some authors (e.g., [11]) claim that this kind of information is necessary, and include it as a requirement for DR capture support tools. However, because this problem was noticed only once in our study, we did not decide to modify our original notation.

3. Causes related to the readers: this category contains the highest number of questions unanswered (19 out of 45 in all). It includes two different types of cause:

- readers' misconceptions: most of the time, a DR question based on a misconception has no DR answer. For instance, a question such as "Why did they change the whole structure?" has no answer if only some parts of the whole structure were changed. The most frequent misconceptions encountered in our data were due either to a misundertanding of the artifact structure (4/12), a misunderstanding of the artifact functions (3/12), an incomplete or incorrect knowledge of the previous design (2/12), or facts learned during the meeting that were forgotten (2/12). It is important to note that misconceptions represent one factor that contributes to making wrong decisions in design, especially when a designer has to choose between different alternatives. These misconceptions were often not due to a lack of information, but to an incomplete analysis of the blueprints provided. We then believe that these misconceptions will always exist, because it is difficult to change the way people work in their daily settings. If we accept this statement, we must look for mechanisms of detection and correction of these misconceptions. One possible approach that we choose to explore further consists in taking advantage of the internal network of the company. We have said that designers should be able to ask questions directly to the DR analyst. The analyst will then be able to correct some designers' misconceptions.

- the problem of inappropriate questions is specific to our study. Because the design presented was only the result of the first phase of the project, several formal proofs (calculations) were not yet available. The designers may ask for these formal proofs if they are not convinced by the justifications provided in the DR. These questions are inappropriate, given the current state of design development. However, they are not inappropriate with respect to the designer's needs. As a consequence, we have learned from this observation that we will need to include in the DR the formal proofs that will be established later. These formal proofs will be represented as justifications of evaluations.

4. Causes related to the project team: this last category of causes is one of the most interesting, because it highlights that some claims concerning the benefits gained with a DR documentation rely on a false assumption (which we shared at the beginning of this study). It is common to think that a design has a rationale. This is what makes the idea of "capture" possible. Some authors (e.g., [12]) also promote the idea that the design space (i.e., the space of possibilities) can be analyzed by designers. All of these ideas have some limitations, because the space of possibilities is not fixed in time. Not only this space, which depends on the designers' background and experience, changes according to the designers involved in a project, but it also evolves with time (because technologies change, experience is growing, industrial strategies are different, etc.). This is what explains why some questions were not answered by the DR: some designers asked questions about issues, options or criteria not taken into account during the design. For instance, "why did they choose option X (reported in the DR) and not option Y (not handled during the project)?" It is then obvious that such questions have no answer in the DR, but it is also important to consider that it will never be possible to avoid such questions. Moreover, we can anticipate that the frequency of occurrence of this class of questions (currently 8/45) will increase, as the time goes by and designers' culture changes. We then speculate that long-term reuse of DR should be more problematic that short-term reuse.

In considering this problem, Terveen et al. [14] promoted the idea of a "living design memory". This approach consists of the creation of a design knowledge base and a design assistant program aimed at giving advice to the designer. The designer can accept or refuse the advice. The refused advice is discussed during a design review. These discussions may produce further knowledge which will be incorporated into the design knowledge base. This living approach to the design memory, while interesting, presents two limitations: firstly, Terveen et al.'s paper reports that many users still have problems with the current system; secondly, this approach radically changes the way designers are currently working. Designers at the design office where our study was conducted have rejected this approach for this reason.

Another possible approach exists, which acknowledges the intrinsic limitations of any memory system and develops a social approach to design memory [1]. According to this approach, the emphasis must be on dialogue in work settings, because it is there that people collectively (re)interpret past experience and control others' interpretations. Putting this approach into practice means giving the opportunity to designers to collaborate more easily and more often. CSCW applications may be a technical solution to this problem. However, they will not be sufficient. New work organizations must also be defined. We are currently attempting to put these ideas into practice at Aerospatiale.

6. Conclusion

This study highlights that traditional approaches for capturing DR exhibit some limitations. Possible adaptations of these approaches are then necessary. In this paper, we have considered (1) the integration of the DR with the information currently processed by designers (in our case, this means integrating DR with blueprints), (2) the introduction of an iterative approach to the DR capture, going beyond the design period, and (3) the need to promote collaboration between designers.

Acknowledgments

References

2. Boose, J.H., Bradshaw, J., and Shema D.B. Knowledge-based Design Rationale Capture: Automating Engineering Trade Studies. In: Green M. (Ed.) Knowledge Aided Design, NY: Academic Press, 1992.

3. Boy, G. Supportability-based design rationale. In Proceedings of the 6th IFAC Symposium on Analysis, Design and Evaluation of Man-Machine Systems, 1995.

4. Carroll, J.M. The Nurberg Funnel: Designing minimalist instruction for practical computer skill. Cambridge, MA: MIT Press, 1990.

5. Carroll J.M. and Moran T.P. (Eds.) Human-Computer Interaction, Special Issue on Design Rationale, 6(3&4), 1991.

6. Conklin, J.E. and Begeman, M.L. gIBIS: A Hypertext Tool for Exploratory Policy Discussion, ACM Transactions on Office Information Systems, 6,303-331, 1988.

7. Fischer, G., McCall, R. and Morch A.I. Design environments for constructive and argumentative design. In CHI'89, 1989.

8. Herbsleb, J.D., and Kuwana, E. Preserving Knowledge in Design Projects: What Designers Need to Know. INTERCHI'93, 1993.

9. Karsenty, L., and Brézillon, P. Cooperative problem solving and explanation. Expert Systems with Application, 8(4), 445-462, 1995.

10. Klein, M. Capturing Design Rationale in Concurrent Engineering Teams. IEEE Computer, 26(1), 39-47, 1993.

11. Lee J. and Lai K. What's in Design Rationale?. Human-Computer Interaction, 6(3&4), 251-280, 1991.

12. MacLean, A., Young, R.M., Bellotti, V. and Moran T.P., Questions, options and criteria: Elements of design space analysis, Human-Computer Interaction, 6(3&4), 201-250, 1991.

13. Shum S. & Hammond N., Argumentation-based design rationale: what use at what cost? International Journal of Human-Computer Studies, 40, 603-652, 1994.

14. Terveen, L.G., Selfridge, P.G. and Long, M.D. From "Folklore" to "Living Design Memory". In INTERCHI'93, 1993.