February 7, 1997
Contact:
Daniel Engelberg Computer Research Institute of Montreal 1801 McGill College Ave., Suite 800 Montreal, Quebec, Canada H3A 2N4 Tel. (514) 840-1235 Fax (514) 840-1244 e-mail: engelber@crim.caWe would like to send one participant to the workshop: Mr. Alain Robillard from CRIM.
CRIM is a non-profit R&D institute in computer science. The HCI group, composed of approximately 20 people, performs both consulting and precompetitive research. Our approach emphasizes multidisciplinary teams, participation of our own software engineering experts on usability projects, and sensitivity to our clients' business goals and culture. Research focuses on methodologies for designing and evaluating interfaces. For more information on the HCI group, please consult our organizational overview paper presented at CHI '96 [VER96].
Mr. Alain Robillard has an M.Sc. in communications and two years of work experience at CRIM. Mr. Robillard was responsible for all of our usability tests of Web sites, and for most of our heuristic evaluations of Web sites. He has also participated in a number of usability inspections of non-Web applications. Mr. Robillard's main research interests are multimedia applications (especially training applications), development of Web sites and applications, and new communication technologies.
Dr. Christian Bastien is a former employee of the HCI group at CRIM, and now works at INRIA (National Institute for Research in Computer Science and Automation) in France. His research specialty is in the development and use of guidelines for evaluating human-computer interfaces. He has published several articles on this topic (e.g. [BAS96], [SCA96]) with Dr. Scapin, also of INRIA.
Our group has extensive experience with a variety of inspection methods including heuristic inspection, industrial usability testing and scientific research approaches. Most of our testing experience is based on the inspection of non-Web applications, however we have tested two Web sites formally: a self-learning site (Self-Learning Guide) and an experimental Web navigator site (Francoroute/CHEOPS). We applied standard testing methods to these sites, i.e. observing users performing a pre-established scenario or task while using a think-aloud protocol. The observations were followed by a standard set of interview questions concerning the user's satisfaction, interest level and comments.
In results from these two tests, we have seen some differences in relative frequencies of problem types compared to non-Web applications. The main distinction seems to be in problems of navigation.
We have also performed four heuristic inspections of Web sites, which seem to confirm the pattern of problem types that we have seen in the usability tests. These inspections used the criteria developed by Bastien & Scapin [BAS96], [SCA96].
Bastien & Scapin's ergonomic criteria are not an inspection method as such, but constitute a set of reference points for the development of comprehensive evaluation approaches. These criteria can be applied at the preliminary design stage, for heuristic inspection, or in usability testing.
1. Basic thesis: The null hypothesis: For the most part, differences in Web site design constraints and in Web site problem profiles do not imply a difference in testing methods. This may seem unintuitive at first, but the whole point of standard usability tests is that they are not task-specific. Our experience testing Web sites confirms this thesis.
1.1 Possible exception (a): Web sites tend to have an objective of providing information and publicity rather than transforming information. This is not a necessary property of Web sites, but it is a tendency which might benefit from adaptations to the standard "vanilla" methods.
1.2 Possible exception (b): The content (and format) of Web sites can change frequently.
2. For known types of usability problems about Web sites, testing should only be a last resort: To the extent that any type of usability problem is already known to be especially common in Web sites, rather than testing the problem we should proactively avoid the problem. We can do this by incorporating the appropriate guidelines in our designs in the first place, and by paying special attention to the appropriate evaluation criteria in heuristic inspections before testing.
For example, Web sites often have longer and more irregular wait times than other applications, but Mayhew [MAY92] and Nielsen [NIE95] have already covered response time guidelines fairly extensively, the latter publication addressing response times specifically in the context of the Web. We don't need to re-test these guidelines on new applications. (However if the guidelines are outdated, we may need more basic research to bring them up to date.)
3. Defining the terms of the discussion: "Usability problems in Web sites": It is important to recognize that the usability problems which seem to pertain especially to Web sites are only at the level of a tendency. We are primarily discussing differences in problem frequencies as compared to non-Web applications.
Conclusion: For any usability problem that we find in a Web site, there exists a corresponding problem in non-Web applications. Therefore we don't have to invent new methods: we can consult the existing literature.
4. Navigation issues: Web sites exhibit navigation problems more often than other applications, and as such navigation is an especially important usability criterion in Web sites. This needs to be addressed at more than just the level of testing.
4.1 We need to minimize unnecessary navigation before testing it. This can be done by a thorough analysis of the task, of users' objectives and of the frequencies of these objectives.
4.2 After the Web site design has been put into place, we can monitor patterns of navigation in order to optimize the design. This could be considered a type of testing method, to the extent that it is empirical. (This approach is relatively light in resources, since neither users nor experimenters need to be conscripted for the data collection -- data collection can be automated and applied to regular site visitors without interfering with their task.)
4.3 Basic research on navigation complexity and representation formats: We believe that in order to advance analysis and design methodologies for structuring information and navigation, we need basic research in the following topics:
[BAS96] J. M. C. Bastien, D. L. Scapin and C. Leulier. Looking for usability problems with the Ergonomic Criteria and with the ISO 9241-10 dialogue principles. In M. J. Tauber (ed.). Conference on Human Factors in Computing Systems. CHI'96 Conference Companion. New York, NY : ACM. P. 77-78. (Vancouver, Canada, April 13-18 1996.).
[MAY92] Mayhew, Deborah J. Principles and Guidelines in Software User Interface Design (Englewood Cliffs: Prentice Hall, 1992), pp. 507-519.
[NIE95] Nielsen, J. Jakob Nielsen's Alertbox for December 1995: Guidelines for Multimedia on the Web. Consulted Feb 3, 1997. http://www.useit.com/alertbox/9512.html.
[SCA96] D. L. Scapin and J. M. C. Bastien. Inspection ergonomique d'interfaces et critères ergonomiques. In conference proceedings, Les sciences cognitives et la conception des systèmes informatiques. (Florianopolis, Brazil, February 26-28, 1996.).
[VER96] F. de Verteuil and D. Engelberg. HCI Group at Computer Research Institute of Montreal. In M. J. Tauber (ed.). Conference on Human Factors in Computing Systems. CHI'96 Conference Companion. New York, NY : ACM. P. 147-148. (Vancouver, Canada, April 13-18 1996.).
