What Help Do Users Need?: Taxonomies for On-line Information Needs & Access Methods

A.W. Roesler, S.G. McLellan

Austin Systems Center
Schlumberger Well Services
8311 North FM 620 Road
P.O. Box 200015
Austin, Texas 78720-0015
roesler@austin.asc.slb.com

Abstract

The feasibility of using a general on-line help taxonomy scheme as the starting point for our interactive graphical applications' on-line help specifications was investigated. We assumed that using such a taxonomy would make it easier for users of the help system, regardless of the application used. The literature, software conferences, trade shows, and the like point to enormous differences of opinion about what help even IS, much less how it should be designed, accessed, displayed, stored, or maintained. While much research described sound design principles and access methods, very little was available on WHAT to organize or access. Our effort on defining a taxonomy for on-line help was based upon three tests:

Test1, a Wizard-of-Oz usability study of an application that identified what types of on-line help our interactive software users actually ask for;
Test2, a test that validated a general taxonomy for on-line help content for help providers, based on the results of Test1, and a general taxonomy of access methods derived from these content types; and
Test3, a repeat of Test1, substituting a prototype help system for Wizard-of-Oz help that successfully validated the usability of both on-line help content and access taxonomies for help users.

This paper summarizes the results of all three tests, highlighting the proposed taxonomies and key findings about them from Test2. Together, the results from all tests indicate that a general taxonomy of information needs and the taxonomy of access methods to particular information types make it easy both for help providers to understand what information they need to supply and for help users to find the help they need quickly.

Keywords

On-line help, taxonomy, user interface, usability, empirical evaluation, methodology.

THE PROBLEM

Early in 1993, as part of the requirements for a large commercial system of over 55 complex interactive graphics applications comprising some 16 million lines of code, an effort was begun to determine the feasibility of using a general on-line help taxonomy scheme as the starting point for all applications' on-line help specifications. The current help facility, utilizing a book mental model and related table of contents and indices as the primary navigational aids, had proven inadequate--for example, for the application chosen for the present study, only 32% of help requests from users could be answered or found in the current documentation. "One striking thing about a survey of the current literature on help systems," Mayhew [1] indicates, "is what is missing, rather than what is available. In particular, no paper or study surveyed here directly addresses the basic question of what the content or subject matter of a user assistance program should be. Few researchers seem to have posed and answered the question, ‘What kind of questions do novice and expert users actually ask when using a system?' All the fancy formatting, navigational ease and ease of access in the world will not be of much use if the information contained in the help system is not the information that users seek." Likewise, Elkerton and Palmiter [2] find that "knowledge about help content is underdeveloped, whereas research on help presentation and access is more widely available." In his 1993 book on help systems, Duffy [3] identifies the primary challenge in designing information for on-line help as "match[ing] the information provided to users with the different kinds of knowledge that they require." Clearly, what is required is a practical examination of the content of help that users need and if it is possible to categorize this content for reuse from application to application.

THE STRATEGY TOWARDS A SOLUTION

Our short-term efforts at defining a taxonomy for on-line help were based upon three basic tests:

Test1, a Wizard-of-Oz usability study of an application that identified what types of on-line help our interactive software users actually ask for,
Test2, a test that validated a general taxonomy for on-line help content for help providers, based on the results of Test1, and a general taxonomy of access methods derived from these content types, and
Test3, a repeat of Test1, substituting a prototype help system for Wizard-of-Oz help that successfully validated the usability of both on-line help content and access taxonomies for help users.

The longer-term goals of our work were to establish empirically-based guidelines that would:

make it simple for help developers to determine and place required help information so that help users can easily find the information where they expect it.
explicitly cover all questions indicated by Test1.
implicitly cover all similar specific questions that could arise, given a different set of tasks.
minimize user navigation for information.

Before presenting either taxonomy of information or access methods, let's examine what the first and third tests revealed about the kind of information users required and how the prototype help system satisfied these requirements.

Comparing Wizard-Of-Oz Help (Test1) & Prototype On-line Help (Test3)

A Wizard-of-Oz technique [4], whereby a person, the Wizard, received the test subjects requests for help and supplied that help as required, was chosen for Test1 to capture how users formulated their requests for help as they encountered obstacles or questions as they worked with an application. A specific interactive application, which allows users to edit graphical displays of complex geological data, was selected. A set of 15 specific tasks was formulated from a real-world use of the program in the field. Five clients with varying experiences using the application were asked to bring up the application and to think aloud as they performed these prescribed tasks. They were also told that they could rely on help at any time from an expert, the wizard, sitting alongside [5]. Results were videotaped to capture the entire session with user commentary, errors, questions, and the like.

Some months later, a usability test was run to validate if users could successfully use the prototype on-line help system that derived its specific taxonomy of help content and specific taxonomy of access methods from the questions asked and the errors made in Test1. Three users performed this Test3, the same as Test1 with one exception: the wizard used in Test1 was replaced by the prototype help system in Test3. At the beginning of each session, test subjects were shown how to use help ("Help on Help" entry from Help Menu in Figure 1). Again, user performance was videotaped.

Figure 1:Figure 1. Prototype Help System (Information & Access Methods).

Though we are still analyzing the data, our observations to date show the following:

All test subjects easily learned how to use the help system and referenced it as needed. No subject found it necessary to access help on help during the duration of the test.
Subjects in Test3 committed none of the three critical errors subjects in Test1 did. One reason was that all the critical need-to-know information was provided by the help system as a whole, whereas the wizard only provided information about the single problem a subject in Test1 was asking about. Typically, the subject in Test1 then committed the next critical error and asked the wizard for help.
There were 44% fewer overall errors than in the first test (44 for 3 users versus 131 for 5 users). In part, this resulted because repeated errors were less frequent in Test3. Videotapes reveal that test subjects repeatedly utilized prompt messages, reminding them about what components were (What Is) and what actions to perform and how to perform them (What Next) correctly.
No test subject failed to find the information he or she needed from the help system.
Even though our help documentation was terse, test subjects made comments that they would like for it to be even more terse. In particular, they are not interested in rationalizations for actions they have to perform being part of what they FIRST see in the help system. In short, they wanted some control of whether they saw any rationalizations or not.

Defining a Taxonomy of Help Information (Test2)

The taxonomy of help content was determined by starting with existing taxonomies in the literature [6,7,8,9,10,11], then tailoring them to a generalization scheme that incorporated our findings from Test1. In particular,

Users posed their requests for help as questions.
Example:
"Which button here controls the fill of the box?"
Based on the information requested and errors committed, required information included some unexpected types: for example, what an application CAN'T do and how to use Motif conventions (despite users indicating an expert level of Motif knowledge).
Examples:
"What about a double ended arrow?"
"How do I change the colors of a curve?"
A user lost text he typed into a text field dialog because the default button was changed to the Cancel button and the user pressed the Return key.
"Is there any difference between the Close and Exit buttons?"
Repeated and severe errors pointed to a need to provide users up front with cues to critical, need-to- know-first information to avoid early setbacks.
Examples:
Many users selected the wrong file to display after they saved data. All users did not initially set parameters to display all of the data they needed.
Since some information (for example, encountered bugs and workarounds) cannot always be anticipated ahead of time, alternative information about assistance (for example, Customer Support) must be available to users.
Example:
"Looks like the program is broken. What happened?"
Since all users, new or experienced to an application, experienced repeated impasses and needed reminders about HI components or next required actions, part of the taxonomy must include methods and components directly in the application interface that provide application-directed information when appropriate.
Examples:
"What are these arrow buttons"
"How do you close a polygon again?"
Any taxonomy of information needs includes information that is specific to the application and information that may span multiple applications.
Examples:
"What are the right color and pattern to indicate oil?"
"How do I get help on Motif?"

The taxonomy we derived from an examination of the questions is given in Table 1.

Table 1: Taxonomy for On-line Information Needs.

In Test2, a set of six test subjects, consisting of application developers and technical writers, were given

all of the questions from Test1 (a total of 209), along with the errors (a total of 131), in the context of the tasks that the original Test1 subjects were asked to perform, and
the taxonomy of proposed information types.

They were asked to place a checkmark under the help category (1-10) to indicate the location in help where they would go FIRST to find the answer to the question asked. Two additional categories were provided: "Don't Understand Question" if they did not understand the question at all and "Don't Know Category" if they understood the question but did not know what help category to go for help first. The test subjects were not shown the actual video recordings of interactions but were asked to infer the context for each question as best they could. Below is a summary of our results for Test2:

Test subjects took an average of 45 minutes to classify the 209 questions.
Test subjects were able to classify 80% of queries for help in the same way, even without the benefit of knowing the explicit context in which the earlier users actually asked the questions.
When test subjects were given additional context for the remaining 20% of queries, results climbed to 97%.
Example question: Is it beginning tail (to draw arrow)?
Revised question: I'm in arrow drawing mode. Is the head of the arrow at the first click or the tail?

At the end of the test, subjects were asked this question: "Based upon questions users asked and what errors users made, what information would you classify as critical information (to be included in the "What Must I Know First" help category)?" Responses confirmed how easy it is for help providers, with the aid of usability results, to point to the same list of critical information.

Defining a Taxonomy of Help Access methods (Test2)

The taxonomy of information needs was only half of the solution for help providers. What was still needed was to map this information uniformly to the access methods to them. Specifically, we speculated how we might minimize the number of times users had to leave an application and minimize the amount of navigation for help.

Implicit to our taxonomy of information were two general help categories: user-directed and application- directed. User-directed help means that a user determines the kind of help desired. Application-directed help means that an application determines the kind of help provided. Examining the percentage of answers placed in the "What Is" and "What Next" categories, we noticed that 37% of queries for help could be answered within the application with the help of a message box and two different access methods:

Button 1 down over an application's human interface primitive or composite component (e.g., button, menu item, menu bar) to display in the Message Box user-directed "What Is" information about the component
Button 1 up (activation) over a primitive to display in the Message Box application-directed "What Next". information about the next possible actions required or allowed.

This was confirmed in Test3, where 44% of the help that test subjects used was contained in the one line "What Is" and "What Next" messages.

A third access method, Button 1 down plus the "Help" key, would allow users to bring up the help application with additional context-sensitive information on the component without going to the Help menu in the menu bar and searching for help. This help would include more information about what the component is and how it relates to other components, what next actions are expected, including critical need-to-know-first information, and a link to the master lists of application components (What Is), possible states and actions (What Next), and tasks that users can and can't do (How Do I).

A fourth method, the help button in any HI window, would provide access to critical need-to-know-first information (What Must I Know First) about using this window. A fifth method, help in the Help Menu, would provide access to master lists of help specific to the application (What Must I Know First, What Is, What Next, and How Do I), as well as access to help common to a set of applications (Meaning of Terms, Help on Help, etc.).

Our access taxonomy was, then, derived from our taxonomy of information, not the other way around. This taxonomy of access methods is given in Table 2. We also implemented the same help menu and access methods in the help window for the application. This would allow users to obtain additional help in the same way as they did within the application itself.

Table 2: Taxonomy for On-line Access Methods.

Test3 demonstrated the usefulness of an access strategy that is simple and that utilizes messaging within an application. Specifically,

No test subject got lost navigating in the help system throughout the test.
44% of the help information that test subjects used was found in the one line "What Is" and "What Next" messages within the application. This was comparable to the 37% of questions whose answers were placed in either the "What Is" or "What Next" categories in Test2.
In almost all cases the test subjects used the minimum amount of navigation to the information they needed outside the application (only 35% of accesses required more than 2 actions, excluding help found in "What Is" and "What Next" messages).

WHAT WE LEARNED
Overall, the results from our tests indicate that a general taxonomy of information needs and the taxonomy of access methods to particular information types make it easy both for help providers to understand what information they need to supply and for help users to find the help they need quickly (See Figure 1). Our tests suggest that:
- Usability studies are needed to find and verify information in several help categories (e.g., critical information in the "What Must I Know First" category and tasks that can't be performed in the "How Do I" category).
- Subjects could not distinguish whether the information they needed was related to Motif or the specific application, despite their indication that they knew Motif behavior (e.g., how to delete the contents of a text field). Help was constructed so that subjects did not have to know whether their question was a Motif- related question or an application-specific question.
- The use of two types of prompts (e.g., for "What Is" and "What Next" information) directly in the application substantially reduces the occurrences of leaving an application for help.
- Constructing application-specific help around two levels (e.g., master lists of "What Next", "What Is", and "How Do I" versus specific "What Next" prompts, "What Is" prompts, and "How Do I" task descriptions) minimizes the amount of navigation of help outside an application.
ACKNOWLEDGEMENTS
The authors would like to thank Allison Elliott from Georgia Tech University and Ruven Brooks from Schlumberger Austin Research for their technical assistance in setting up the usability tests, compiling, and analyzing the results.
References

1. Mayhew, D. J. (1992). Principles and Guidelines in Software User Interface Design, Englewood Cliffs: PTR Prentice Hall.
2. Elkerton, J., & Palmiter, S. L. (1989). Designing Help Systems Using the GOMS Model: An Information Retrieval Evaluation, Proceedings of the Human Factors Society 33rd Annual Meeting, 1989, pp. 281-85.
3. Duffy T. & others (1993). On Line Help: Design and Evaluation. Norwood: Ablex Publishing Corp.
4. Hill, W. C. (1993). A Wizard of Oz Study of Advice Giving and Following, Human-Computer Interaction, 1993, Vol. 8, pp. 57-81.
5. Hill, W. C., & Miller, J. R., (1988). Justified Advice: A Semi-Naturalistic Study of Advisory Strategies, CHI ‘88 Proceedings, May 1988, pp. 185-90, ACM.
6. Sondheimer, N. K., Relles, N. (1982). Human Factors and User Assistance in Interactive Computing Systems: An Introduction, IEEE Transactions of Systems, Man and Cybernetics, Vol. SMC-12 (March-April 1982), pp. 102-107.
7. Hartley, J. R., & Smith, M. (1988). Question answering and explanation giving in on-line help systems. Artificial intelligence and human learning: Intelligent computer-aided instruction (pp.338-360). London: Chapman and Hall.
8. Miyake, N., & Norman, D. A. (1979). To ask a question one must know enough to know what is not known. Journal of Verbal Learning and Verbal Behavior, 18, 357-364.
9. Sellen, A. & Nicol, A. (1990). Building user-centered on-line help. In B. Laurel (Ed.), The art of human-computer interface design (pp.143-154). Reading, MA: Addisoon-Wesley.
10. Robertson, S., & Swartz, M. (1988). Why do we ask ourselves questions? Questioning Exchange, 2, (1), pp. 47-51.
11. Harris, R. A., & Hosier, W. J. (1991). A Taxonomy of Online Information. Technical Communication, 1991, 38(2), pp. 197-209.