Department of Mathematics and Computer Science
Georgia State University
Atlanta, Georgia USA 30303
e-mail: ckilpatrick@gsu.edu
http://www.cs.gsu.edu/~matcek
Overview
This paper includes a discussion of some of the usability evaluation and testing methods that have been applied to websites in a university environment during the past year. The methods were applied in several projects I have directed and were used in web projects in three of my human-computer interaction (HCI) classes at Georgia State University. Five sample projects are discussed. For each, a project description is given. This is followed by a discussion with emphasis on one of the usability evaluation methods that was applied in the project. Rationale is given for use of the method in the particular project setting. An overview of the general effectiveness of the method within the context of each project is also included.
In each of the five projects discussed, emphasis was placed on incorporating usability engineering methods in the development process of the project website. For example, participatory design, paper prototype development, design-level evaluation and user testing were typically incorporated in the process. Evaluation and testing methods used were all based on methods that have been used successfully within general software development.
Discussion
Several evaluation and/or user testing methods were used for each of the five projects discussed. One method in particular will be emphasized for each.
Evaluation of paper prototype.
University educational application.
The first project involves the development of a web application for use in university-level ESL (English as a Second Language) classes. Key features of the application include the following: students submitting class writing assignments; professor grading submitted assignments; students viewing graded assignments; and students and professor assessing student progress. The prototype design team included an ESL professor, three HCI students (computer science majors) and myself. During preliminary discussions about the site, the team identified key tasks (for both students and professors) and developed several high-level scenarios. The three students on the team then engaged in parallel design of the application. Each constructed a paper prototype.
Three design-level evaluations were done on each paper prototype. The key tasks and high-level scenarios were read to each evaluator. Evaluators then used the ten general principles termed “usability heuristics” by Jakob Nielson in Usability Engineering [Nielson93]. The principles provided a framework for focusing the evaluation activity as well as a way to characterize potential problems identified. The evaluations were given orally with half the sessions videotaped and half hand recorded.
Combined results of the evaluators showed, for example, potential problems related to language, navigation inconsistencies, layout inconsistencies (that might present usability problems concerning search time for important information) and feedback that could be handled at the site. Examples of violation of the usability heuristics included the following: Simple and Natural Dialog: link names leading to unexpected information and improper grouping of information; Feedback: no feedback when information had been successfully submitted to the database containing student work; and Speak the User's Language: the variety of navigation methods provided, offered no clear model of how to navigate the site.
The primary rationale for development of paper prototypes was to explore their use in website development. This appears to have been successful in this situation with novice, computer science student designers. It provided a way to promote the importance of non-implementation oriented design activity. The 10 usability heuristics were general enough to be useful in helping an evaluator identify and classify even those usability problems that are quite specific to the web.
Method to discuss: Think-aloud protocol on exploratory prototype.
Children's educational application.
The second project concerns the development of a site for children with asthma. The prototype development team included three HCI students (computer science majors), an adult domain expert professional and myself.
A think-aloud protocol with four children was used to get initial feedback about the site. The student designers for the site were also the site testers. Each think-aloud protocol session was videotaped. Though site specific tasks had been generated in preliminary site discussions, children were not asked to do specific tasks during testing. Instead, they were given the general task of exploring the site and commenting about information they read, paths they followed and services they used that were provided at the site. As is often the case, it was difficult to keep the children verbalizing as they explored the site. The testers each made a second pass through the site with the children as they went.
Several key usability issues came out during the testing. Several icons were not interpreted properly by most of the children even though each had an associated caption. The children were not as excited or interested in some of the features of the application as the development team had expected. In addition, no one on the development team actually had asthma and, for example, the children pointed to language they felt was not used properly.
As a follow-up to initial work on the site, a group of two other HCI students (computer science majors), two education students and an adult domain expert continued work on the site. The videotapes from the initial think-aloud protocol with the children were viewed and analyzed by this second group at the start of their project activities.
Think-aloud protocol on existing sites.
Document search.
The third project involved testing using a think-aloud protocol with four users on two existing websites. Testing was done by two HCI students (computer science majors). One student recorded observations and the other interacted with users during testing. These two sites each offered search of an online database of government documents. Each site was tailored for use in different parts of the country. The testing was preliminary work prior to design of a similar site to be tailored for use in the Atlanta area.
The testing provided excellent insights about the user populations for the Atlanta area. Users were given specific search tasks to complete. The tasks were key to making a successful search at each site. During testing, none of the users successfully accomplished any of the given tasks. As a consequence, only high level information was collected. However, this testing was used to support recommendations for education on use of the web for those who would be using the site. It also gave the designers insight into how their site could be effectively tested. In addition, it provided experience with web access and use in an environment where the site would be accessed often.
Method to discuss: Informal evaluation by non-team members.
Student class projects.
The fourth project discussed is an overview of the projects done in two different HCI classes. Students did initial development of a website with an Atlanta-area community organization. They followed a software development process, incorporating traditional methods and techniques associated with usability engineering in the development of their project website. Project teams included two or three students and one or two domain experts from participating organizations.
The development process for each project involved the following: identify users, identify key tasks, organize information at the site, develop a navigation strategy, construct a paper prototype, develop several scenarios, implement an online prototype and do informal testing/evaluation with community participants from other class project teams.
Informal testing/evaluation by community participants was done quite early in website development. It served to point out conceptual big-picture problems. These included, for example, a lack of clarity concerning content available at the site. It also served to identify more specific types of problems, for example, consistency, language choice and in some cases, abundance of detail of textual information provided.
Method to discuss: Comparison of results from several usability methods.
University computing services information.
The fifth project concerns current work involving evaluation and user testing of the redesign of a university organization's website. No usability evaluation or testing was done on the original site. One redesign has been done and a second is under development. A heuristic evaluation was done on the first redesign. In this evaluation, the 10 general usability heuristics outlined by Nielson [Nielson93] served as a framework for the evaluation activity.
A number of potential usability problems were identified in this first evaluation. The major problem involved site focus. The extent of information and services available were not clear especially as one browsed the site. Entry from other university web pages external to the site did not consistently pose as great a problem.
The results of this evaluation will be incorporated where applicable into the second redesign. The following lists the next two steps in evaluation and testing of the second redesign. These steps will be completed within the next month:
Problems identified in each of the steps will then be compared to provide insight into the types of usability problems found for each method.
Conclusions
Usability engineering methods that can be applied to software design in general can be applied to the web. This is in no way to say that new methods specifically oriented to the web are not needed. However, education and use of existing methods can be as key to effective website design as they are key to effective software design in general. Experimentation with use of these methods within website development points out ways the methods can be applied. This experimentation can also point to ways they can be modified and extended for the web.
References
[Nielson93] Nielson, J. (1993). Usability Engineering. London, Academic Press, Inc.
Biographical sketch
Dr. Kilpatrick has been an Assistant Professor of Computer Science since 1990 at Georgia State University in Atlanta, Georgia, USA. From 1991 to 1993, she was on academic leave of absence at IBM, Bergen Environmental Sciences and Solutions Centre (a multidisciplinary, multicultural environmental research site in Norway). Dr. Kilpatrick was in charge of usability activities related to the various projects at the center.
Dr. Kilpatrick received a Ph.D. in Information and Computer Science from Georgia Institute of Technology in 1991. Her work addressed the collection and display of performance information for parallel and distributed applications. The focus on the work involved systems issues along with consideration of issues of user specification and user interpretation of information related to the collection and display of performance information.
