As a staff usability consultant at Tec-Ed, I have used two methods for usability study of World Wide Web sites: usability testing and heuristic evaluation. This position paper describes the effectiveness of these methods in those web-site usability studies.
By the time CHI97 convenes in Atlanta, I will have conducted four usability studies of web sites. I will also be able to cite web-site studies conducted by other Tec-Ed usability consultants. Through my experience with these studies (for clients who include Netscape, Sun Microsystems, Thomas Publishing, and Knight-Ridder), I believe I will bring practical strategies to the workshop.
Overview of Experience with Usability Study of Web Sites
I have completed two usability studies of web sites, and I will have completed two more by CHI97. The two I have completed are:
• A comparative usability test of downloading software from two different computer company web sites (one extremely large and one smaller).
• An heuristic evaluation of a preliminary web site, to be published later this year, that will be used to search for product information within a proprietary database.
The studies I am currently designing and preparing to implement are:
• A follow-on usability test of the same product information web site for which I performed an heuristic evaluation.
• Another study of downloading software at a different company's widely used web site
This position paper first introduces the purpose of each study. Then it describes the strengths and weaknesses of heuristic evaluation and usability testing in investigating the usability of web sites.
Usability Testing of Downloading Software from Web Sites
A client with a large and well-established web site wanted to learn why, although thousands of users were visiting the site, extremely few were downloading trial software from the site, and even fewer were buying software from the site. The client also wanted to know how its downloading process compared to another company's web site.
An heuristic evaluation would have uncovered many of the problems we found during usability testing. However, many large players "owned" the site; the organization would have considered the feedback "someone else's opinion" and would not have been committed to using it. Only primary user data would drive design changes and, in fact, the week after the test sessions were completed, the site already reflected some major improvements recommended from the study.
Tec-Ed is planning to employ the same general approach used for this study in the upcoming web-site downloading study being performed for another client. Thus, I will have a second, comparable experience to contribute to the workshop discussions.
Tec-Ed's methodology for collecting data during usability tests of web sites is to work from a script that provides specific prompts for note-taking about user activities, and also to have a printout of the web pages themselves on which to jot down where users visited and in what page order. The browser history list does not adequately record the order of pages visited, nor does it indicate which pages received the most time from the user. Our method captures both of these types of information, which we believe are critical to understanding the scope of usability problems at a web site. Of course, we also videotape the test sessions, but our clients usually want the results more quickly than we can deliver if we need to watch all the videos.
Heuristic Evaluation of Web Sites Used for Proprietary Database Searching
A client developing a web site that provides search access to a proprietary database of ultimately hundreds of thousands of product descriptions wants to build usability into the design from the beginning. Tec-Ed has nearly completed the first of three phases for this study: a first-iteration heuristic evaluation of the preliminary web pages. That feedback has been well-received and incorporated into a version of the web site that will undergo usability testing and second-iteration evaluation. Following those activities, the web pages will receive a final round of usability testing. The client has accepted this approach to achieve its goal of publishing a web site that must successfully convert users from existing print-based resources they are comfortable with to an online resource they perceive as easy to use and useful.
Another client with a web site providing search access to a database (already published to a limited audience) asked Tec-Ed to perform an heuristic evaluation of the web pages to confirm feedback received about usability problems. This experience was less successful, perhaps because the client had also requested an heuristic evaluation from the firm that had been involved in the UI design, thus receiving two sets of results and having to pick and choose which to implement. Tec-Ed's recommendations, some of which would have required extensive rework of user interface text, were largely ignored.
Tec-Ed's method in performing heuristic evaluations is to assign at least two usability specialists, who perform independent evaluations of the user interface and take notes on their findings. The evaluators then discuss their separate findings and find common ground for communicating the findings to the developers. We generally organize our findings into four categories: user task support, UI behavior, presentation, and terminology. Although there tends to be overlap in findings among these categories, using the categories ensures that we give full attention to each aspect of a usability problem.
Strengths and Weaknesses of Usability Testing vs. Heuristic Evaluation of Web Sites
Tec-Ed's overall experience with heuristic evaluation and usability testing of web sites agrees in general with the findings in the literature (and anecdotes from UTEST) about these methods:
• Heuristic evaluation is useful for identifying usability problems according to established guidelines, providing a basis for “cleaning up” a user interface prior to usability testing. It does not replace collecting user data.
• Usability testing, in addition to collecting primary user data, is especially effective for finding whether the UI corresponds to users' work processes and mental models. It also identifies user perceptions of problem severity.
This experience is also true for our web site usability studies. However, web site usability studies bring a few differences to the table:
• Early versions of web sites are likely to become published before they undergo usability assessment, and thus they elicit actual user (visitor) feedback that may be misinterpreted. While this data can be useful for defining the issues to evaluate during either a usability test or an heuristic evaluation, the uncertainty of the user profile and the user's motivation for visiting the site limits effective generalization of this feedback. It is Tec-Ed's experience that users are likely to enter bogus data into a web-based form, whereas one-on-one recruiting and participant observation will yield more reliable data.
• Web sites are revised more quickly and more often than software that will reside on a user's computer, so while usability feedback can be more immediately implemented, there may be less motivation to conduct formal usability testing because the version that was evaluated no longer exists! Especially when navigation from the home page is an issue, a changing web site can degrade the protocol developed to explore the issues identified for a study. Cooperation is needed from web-site developers to "drive a stake" in a particular web-page version while usability evaluation takes place, and from usability evaluators to be willing to adjust the protocol right up to the day before the test, if needed.
• Availability of visitor feedback and the turf wars of web-site ownership may make organizations less willing to act on formal heuristic evaluation findings. Conducting an heuristic evaluation before the site is published should result in a better site design that makes a more favorable first impression on visitors. In addition, if the evaluation team consists of several objective, independent evaluators who consolidate their findings, this level of feedback may outweigh individual ownership issues.
• The vast number of user path alternatives at a web site, especially a large web site, makes performing a thorough heuristic evaluation more difficult, and makes usability test task scenarios trickier to scope. Rather than directing users to paths, our approach has been to allow users to go wherever they please; we track where they go and their stated reasons. It is advisable to recruit a greater number of users than the possible pathways they can follow, to assess which pathways are more frequently traveled and why.
Bio and Contact Information
Laurie Kantner has designed and administered many usability studies of computer software, hardware, and documentation for the Windows, Macintosh, and UNIX platforms, employing a variety of methods. In her 19 years in technical communication, Laurie has also planned, written, edited, and managed user training, documentation, and marketing communication projects for print and online delivery.
Laurie serves on the Editorial Board of Common Ground, the publication of the Usability Professionals' Association. Her article on “Techniques for Managing a Usability Test” appeared in the September 1994 issue of the IEEE Transactions on Professional Communication. Laurie co-presented “Uses and Misuses of Heuristic Evaluation” at the 1995 Usability Professionals' Association Conference, organized a session on usability test participant recruiting for the 1996 UPA conference, and has made presentations on documentation quality control and editing at the Society for Technical Communication.
For information about this position paper, please contact:
Laurie Kantner
Tec-Ed, Inc.
P.O. Box 1905
Ann Arbor, Michigan 48106
(313) 995-1010 fax (313) 995-1025
email laurie@teced.com
