Gregory A. Wilt

Contact Information

Biographical Sketch

Gregory A. Wilt founded and leads the Usability Engineering program at Bell Atlantic. Concentrating on software developed for Bell Atlantic employees, this group espouses a user-centered design approach, conducts formal usability tests, and utilizes a variety of experimental, heuristic, and observational techniques. A Ph.D. candidate at George Mason University, Wilt is currently completing his dissertation on empirical comparison of knowledge elicitation techniques. His background in applied cognitive psychology supports current research interests in the effective use of World Wide Web and virtual reality environments for training.

Issue

Internet and intranet technology are becoming increasingly used to deliver and access important material on an as-needed basis. Uses such as this are logical, as hyper-based technology allows for fast and easy access to large data sets, exploration, and learning by discovery. Following from the latter, such technology is now being used for training and educational purposes. As the goals of training are somewhat different from those of information exploration or data-mining, it is important to assess the degree to which the same technology is as user-friendly for one purpose as it is for the other.

Methods

Bell Atlantic has conducted a number of formal usability tests toward this end. These tests have been both empirical and observational, objective and subjective, quantitative and qualitative. The content of these web projects is instructional material intended for Bell Atlantic employees, and as such, representative user samples were tested.

The most successful of these tests was a series of formally structured tests, with pre-defined search tasks based on the material in the domain. Subjects were given no training on the browser (Netscape), although most indicated they had some prior experience. Data collected for these tests include objective data (time to complete tasks, number and pattern of links to reach target), observational data, self-report (QUIS) data, and mental model construction data.

For this application and this sample, time to complete search tasks emerged as the most valuable data, clearly differentiating experienced from novice users. Further, mental model drawings were indicative of performance. Those who accurately drew the contents of the pages as a hierarchical/tree structure or as a network of nodes and arcs performed well; subjects with less well-defined drawings performed poorly. The self-report data was also valuable, in that the pattern of positive and negative responses tended to mirror observations; subjects responded positively to items dealing with the look and feel of the interface and negatively to items dealing with performance aspects of the system, or with the efficiency of the tool as an information source.

Less valuable were data for the pattern and number of links accessed to accomplish the search tasks. With additional research effort, it is possible that these data will yield a wealth of information regarding the mental models of subjects. However, for our pragmatic purposes - making the application more easily navigable as quickly as possible - these data were too esoteric. An interesting research program would follow from comparing patterns of link usage for training versus non-training web applications with similar interfaces and navigation controls.

Usability Problems Detected

Focusing on time to complete search tasks as the key measure of usability of these web pages, the most severe usability problem uncovered in this test, and subsequently in others, is the failure of designers to promote a clear visualization of the information space. While one subject in the first study did accurately represent the space, utilizing nodes and arcs, most were less precise. The representation of the space by others as a hierarchical structure, while not fundamentally wrong, suggests that these subjects viewed the search tasks as digging rather than browsing. Their mental models, and the observed behavior, consisted of going `down' into the material and coming back `up' again, rather than moving easily `across' or `through' the space.

This problem was addressed in redesign by developing as the home page to this material a matrix which clearly illustrated both the depth and the breadth of the space. This implementation resulted not only in an average improvement of thirty minutes in search time across all tasks, but perhaps more importantly, dramatically closed the gap in performance between novice and expert users. This suggests that a clearly portrayed depiction of the information space is essential to efficient navigation.

This content organization problem also emerged as a key issue in a test of another project, the latter more training-oriented than the former. The usability test for the latter project was less quantitative in nature, as superimposing a series of artificial search tasks over the inherent tasks of completing units of instruction, taking quizzes, and so on, would be less than optimal. Rather, subject comments and a comparison of observed navigation to an expert model of the space were used as data.

While the usability problem identified, inaccurate conceptualization of the space, is the same as in the former tests, the ramifications are very different, and manifest on two levels. The first is that users are attempting to complete different tasks at a web training site (learning, completing units) than at a purely informational site (browsing). Thus, inability to conceptualize the entire space and navigate efficiently through it is potentially much more frustrating for the trainee than for the surfer.

The second points to a potentially much more critical relationship between the web as a delivery mechanism and the nature of instruction. Specifically, instructional content is typically organized in a very structured format, with higher level learning objectives building on lower level ones. Web-based presentation, in contrast, fosters a looser mental organization on the part of users (i.e., browsing, surfing). Given this, instructional systems designers must carefully consider the goals of training projects in conjunction with the advantages and disadvantages of the web as a delivery mechanism to ensure usability.

If the overriding goal is to present instructional material via the web in a structured, instructionally-sound sequence, some features of browsing software should be de-emphasized. Multiple links to the same material allowing multiple paths through the course clearly confused subjects in Bell Atlantic usability tests. Under this model, usability can be enhanced if users are provided fewer opportunities to stray from the intended, and expected, organization of material.

If, however, the more important goal is to exploit the flexibility of the current technology, usability is improved when the structure is relaxed and users are encouraged to browse the material in any order. A global view of the hyperspace should be predominant and easily accessed. Behavior becomes less constrained and frustration with the interface decreases. As a caveat, the model chosen should be clearly communicated to users. Such communication goes a long way toward managing expectations.

Thus, the definition of usability for web-based applications is not a constant. The example presented of delivering training material on web sites introduces additional variables into the equation. While learning by discovery is a powerful tool, real-world constraints, competing theory, and user expectations sometimes mandate a different approach. The challenge for usability engineers is to develop models of the ways in which usability standards, and hence testing methodology, relate to the purpose of the application. The Bell Atlantic data are at least a starting point.

Organizational Issues

Usability testing of web sites presents an interesting organizational problem. Ideally, usability testing first occurs at a functional prototype stage. Here, an interface has been specified and mocked up so that base functionality is in place, although content or data may be missing. This allows testing at a stage where we know enough about what the application is supposed to do to be able to make meaningful design recommendations but where we have not invested too many resources, in case major interface redesign is required.

Web sites are different however. The usability of a web interface is closely tied to the underlying information. In order to adequately assess the amount of frustration and confusion which will result when a user has to tunnel through numerous links and pages to get one desired piece of information, all those links and pages and all that information must be in place. Thus, as usability professionals, we are in some ways undermining our hard-fought battle for early involvement by telling designers it is best to wait a bit. We must effectively communicate this rationale to ensure that we are not sending mixed signals.

Final Thoughts

Clearly, significant issues exist regarding usability testing of web sites. Of the millions of sites currently in place, there are surely as many ways of organizing the underlying information and relating this organization to the intended purpose of the site, sometimes communicated to the end user, sometimes not. Facilitating mental models of web sites to be shared by designers and users is our challenge. Measuring this shared model in the usability laboratory follows. As businesses, educators, governments and individuals strive to utilize this incredibly powerful technology in new and innovative ways, we must keep pace. I welcome the opportunity to participate in a dialogue of these and other issues.