Usability Inspection Methods

Jakob Nielsen

SunSoft
2550 Garcia Avenue
Mountain View, CA 9404301100
Email: jakob.nielsen@sun.com (primary), nielsen.chi@xerox.com (backup)

Abstract

Usability inspection is the generic name for a set of cost-effective ways of evaluating user interfaces to find usability problems. They are fairly informal methods and easy to use. Keywords: Usability engineering, heuristic evaluation, cognitive walkthroughs, pluralistic walkthroughs, feature inspection, consistency inspection, standards inspection.

Introduction

Software inspection [1][7] has long been used as a method for debugging and improving code. Similarly, usability inspection [25] has seen increasing use since about 1990 as a way to evaluate user interfaces. The four basic ways of evaluating user interfaces are automatically (usability measures computed by running a user interface specification through some program), empirically (usability assessed by testing the interface with real users), formally (using exact models and formulas to calculate usability measures), and informally (based on rules of thumb and the general skill and experience of the evaluators). Under the current state of the art, automatic methods do not work and formal methods are very difficult to apply and do not scale up well to handle larger user interfaces.

Empirical methods are the main way of evaluating user interfaces, with user testing probably being the most commonly used method. Often, real users can be difficult or expensive to recruit in sufficient numbers to test all aspects of all the versions of an evolving design, leading to the use of inspection as a way to "save users." Furthermore, project schedules or budgets sometimes impose restrictions that make informal methods like inspection desirable as a "discount usability engineering" solution [16][21] since they are highly cost-effective [8]. Several studies have shown that usability inspection methods are able to find many usability problems that are overlooked by user testing but that user testing also finds some problems that are overlooked by inspection, meaning that the best results can often be achieved by combining several methods [5][6][11].

INSPECTION METHODS

Usability inspection is the generic name for a set of methods that are all based on having evaluators inspect the interface. Typically, usability inspection is aimed at finding usability problems in a design [13], though some methods also address issues like the severity of the usability problems and the overall usability of an entire design [28]. Many inspection methods lend themselves to the inspection of user interface specifications [17] that have not necessarily been implemented yet, meaning that inspection can be performed early in the usability engineering lifecycle [19].

Heuristic evaluation [23][27] is the most informal method and involves having usability specialists judge whether each dialogue element follows established usability principles (the "heuristics" [15][18][21][22]).
Cognitive walkthroughs [12][29][30] use a more explicitly detailed procedure to simulate a user's problem solving process at each step through the dialogue, checking if the simulated user's goals and memory content can be assumed to lead to the next correct action.
Formal usability inspections [9] use a six-step procedure with strictly defined roles to combine heuristic evaluation and a simplified form of cognitive walkthroughs.
Pluralistic walkthroughs [3][4] are meetings where users, developers, and human factors people step through a scenario, discussing each dialogue element.
Feature inspection [2] lists sequence of features used to accomplish typical tasks, checks for long sequences, cumbersome steps, steps that would not be natural for users to try, and steps that require extensive knowledge/experience in order to assess a proposed feature set.
Consistency inspection [31] has designers representing multiple projects inspect an interface to see whether it does things in the same way as their own designs.
Standards inspection [31] has an expert on some interface standard inspect the interface for compliance.

Heuristic evaluation, cognitive walkthroughs, feature inspection, and standards inspection normally have the interface inspected by a single evaluator at a time (though heuristic evaluation is based on combining inspection reports from a set of independent evaluators to form the list of usability problems). In contrast, pluralistic walkthroughs and consistency inspections are group inspection methods. Finally, formal usability inspections combine individual and group inspections. Many usability inspection methods are so easy to apply that it is possible to have regular developers serve as evaluators, though better results are normally achieved when using usability specialists [20].

Acknowledgments

This summary is based on the author's own work, presentations at the CHI'92 workshop on usability inspection methods [14], and chapters in the recent Usability Inspection Methods book [25]. Workshop and book contributors include Brigham Bell, Randolph Bias, Louis A. Blatt, Janice Bradford, Patricia Brooks, George Casaday, Heather Desurvire, Robin Jeffries, Sandra Jones, Michael J. Kahn, Clare-Marie Karat, James A. Knutson, Thomas Lanzetta, Clayton Lewis, Robert L. Mack, Allan MacLean, Frank Montaniz, Peter Polson, Amanda Prail, John Rieman, Linda Tse, Cathleen Wharton, and Dennis Wixon. The present discussion of usability inspection is the responsibility of the author and does not necessarily represent the positions of these other people.

References

1. Ackerman, A. F., Buchwald, L. S., and Lewski, F. H. (1989). Software inspections: An effective verification process. IEEE Software 6, 3 (May), 31-36.
2. Bell, B. (1992). Using programming walkthroughs to design a visual language. Technical Report CU-CS-581-92 (Ph.D. Thesis), University of Colorado, Boulder, CO.
3. Bias, R. (1991). Walkthroughs: Efficient collaborative testing. IEEE Software 8, 5 (September), 94-95.
4. Bias, R. G. (1994). The pluralistic usability walkthrough: Coordinated empathies. In Nielsen, J., and Mack, R. L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 65-78.
5. Desurvire, H. W. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing? In Nielsen, J., and Mack, R. L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 173-202.
6. Desurvire, H. W., Kondziela, J. M., and Atwood, M. E. (1992). What is gained and lost when using evaluation methods other than empirical testing. In Monk, A., Diaper, D., and Harrison, M.D. (Eds.), People and Computers VII, Cambridge University Press, Cambridge, U.K. 89-102.
7. Fagan, M.E. (1986). Advances in software inspection. IEEE Transactions on Software Engineering 12, 7 (July), 744-751.
8. Jeffries, R., Miller, J.R., Wharton, C., and Uyeda, K.M. (1991). User interface evaluation in the real world: A comparison of four techniques. Proc. ACM CHI'91 Conf. (New Orleans, LA, April 28 - May 2), 119-124.
9. Kahn, M. J., and Prail, A. (1994). Formal usability inspections. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 141-172.
10. Karat, C. (1994). Comparison of user interface evaluation methods. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 203-232.
11. Karat, C., Campbell, R. and Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings of CHI'92 (Monterey, California, May 3- 7, 1992), ACM, New York, 397-404.
12. Lewis, C., Polson, P., Wharton, C., and Rieman, J. (1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. Proc. ACM CHI'90 Conf. (Seattle, WA, April 1-5), 235-242.
13. Mack, R. L., and Montaniz, F. (1994). Observing, predicting and analyzing usability problems. In Nielsen, J., and Mack, R. L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 293-336.
14. Mack, R. L., and Nielsen, J. (1993). Usability inspection methods. ACM SIGCHI Bulletin 25, 1 (January), 28-33.
15. Molich, R., and Nielsen, J. (1990). Improving a human-computer dialogue. Comm. ACM 33, 3 (March), 338-348.
16. Nielsen, J. (1989). Usability engineering at a discount. In Salvendy, G., and Smith, M.J. (Eds.), Designing and Using Human-Computer Interfaces and Knowledge Based Systems. Elsevier Science Publishers, Amsterdam, the Netherlands. 394-401.
17. Nielsen, J. (1990). Paper versus computer implementations as mockup scenarios for heuristic evaluation. Proc. IFIP INTERACT'90 Third Intl. Conf. Human-Computer Interaction (Cambridge, U.K., August 27-31), 315-320.
18. Nielsen, J. (1990). Traditional dialogue design applied to modern user interfaces. Communications of the ACM 33, 10 (October), 109-118.
19. Nielsen, J. (1992). The usability engineering life cycle. IEEE Computer 25, 3 (March), 12-22.
20. Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Proc. CHI'92 Conf. (Monterey, CA, May 3-7), ACM, New York, 373-380.
21. Nielsen, J. (1993). Usability Engineering, Academic Press, Boston, MA.
22. Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Proc. ACM CHI'94 Conf. (Boston, MA, April 24-28), 152-158.
23. Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R. L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 25-64.
24. Nielsen, J. (1994). Scenarios in discount usability engineering. In Carroll, J. M. (Ed.), Scenario Based Design: Envisioning Work and Technology. Book under preparation.
25. Nielsen, J., and Mack, R. L. (Eds.) (1994). Usability Inspection Methods, John Wiley & Sons, New York.
26. Nielsen, J., and Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proc. ACM INTERCHI'93 Conf. (Amsterdam, the Netherlands, 24-29 April), 206-213.
27. Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. Proc. ACM CHI'90 Conf. (Seattle, WA, April 1- 5), 249 - 256.
28. Nielsen, J., and Phillips, V. L. (1993). Estimating the relative usability of two interfaces: Heuristic, formal, and empirical methods compared. Proc. ACM INTERCHI'93 Conf. (Amsterdam, The Netherlands, April 24-29), 214-221.
29. Polson, P., Lewis, C., Rieman, J., and Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 5, 741-773.
30. Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994). The cognitive walkthrough method: A practitioner's guide. In Nielsen, J., and Mack, R. L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 105-140.
31. Wixon, D., Jones, S., Tse, L., and Casaday, G. (1994). Inspections and design reviews: Framework, history, and reflection. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods, John Wiley & Sons, New York, 79-104.