Richard Whitehand trained as an ergonomist at Loughborough University of Technology (UK), completing a four year Honours degree in Information Technology and Human Factors in 1995. Since then he has been working as a consultant ergonomist and researcher for Nomos Management AB, a human factors consultancy in Stockholm, Sweden. In the past year, the majority of his consultancy work has been concerned with World Wide Web usability evaluation and design support activities for a number of large Swedish and international organisations. He has written several articles about WWW usability and ergonomics, and recently co-authored an article entitled “World Wide Web: Quality, usability and ergonomics”, published in Arbete Människa Miljö (The Scandinavian human factors journal).
At Nomos we provide design support and usability evaluation services to a wide range of clients. Specifically, we have worked (or are working) on large Web projects for a Swedish bank, a large Swedish government agency, several international investment companies, and a large international software corporation, amongst others. The majority of work has been with Internet Web sites, but with a recent growth in Intranet work. We are also working together with research organisations in the development of several usability evaluation support tools.
We have experience in the application of a broad range of usability testing methods in Web environments. Most of our methods are adapted and extended versions of those we have applied for some time in `traditional' software interface testing. Many of these tools and methods have been developed around International Standards concerning software ergonomics (for example, ISO 9241 Parts 10-17).
We have experience in applying the following usability testing methods and tools to World Wide Web sites:
· Expert assessments/inspections, with reference to principles adapted and extended from those set out in ISO 9241 Part 10.
· Participatory evaluations (experts with users).
· Small and large scale usability lab tests with representative users carrying out representative tasks on web sites.
· A specialised Web logging tool which captures a very rich set of information about how users interact with, and navigate through, a site.
· An online subjective usability assessment questionnaire specifically developed for the Web, based on SUMI (Software Usability Measurement Inventory).
The methods and tools we employ for a given web site naturally depend on it's state in the development process and individual client requirements. In addition to testing methods, we employ context and requirements analyses as necessary, and a variety of prototyping techniques, as part of a user-centred Web design and evaluation process we use when working on development projects with clients.
Below is a brief summary of the circumstances in which we employ each of the usability testing tools and methods listed above, together some of their relative advantages and disadvantages.
Expert assessments/inspections
These are the most frequent types of evaluation we carry out for clients and are based on expert knowledge and principles adapted from ISO 9241 Part 10. They are used at intervals throughout the development process when we work with clients developing a new site. When used as a quick and inexpensive way of evaluating an existing site, an expert evaluation is often combined with a small context study, the combined results of which give a good picture as to the usability of the site. This can then form the basis for planning future evaluation and design activity.
Advantages:
· Relatively quick and inexpensive - the extent of the evaluation and the number of experts used (typically 2 or 3) can vary depending upon the nature of the site and client requirements.
· Can be carried out on paper or incomplete prototypes.
· Captures usability problems which might not be found during lab tests using a specific environment (e.g. issues concerning differences between browsers, accessibility for disabled users).
· Gives quick feedback into the development process - crucial in the short development time scales for Web projects.
Disadvantages:
· Requires a working technological knowledge of certain aspects of the web (for instance, unlike `normal' software interfaces, the expert needs to be aware of differences in the way interfaces may appear on different browsers). This is especially important if useful recommendations are to be made as a result of the inspection.
· Whilst usability issues can be identified, in some cases it is difficult to judge how much trouble certain problems will cause users, such as aspects of navigation support.
Participatory evaluations (experts with users)
In these evaluations we evaluate a web site together with representative users. Several usability experts will `pair up' with representative users who will work together with the web site. This method is often employed with Intranet sites, or sites with specific applications, where representative users can easily be found, and a set of representative tasks can be pre-defined for users to work through in the evaluation.
Advantages:
· Involving representative users enriches the experts' feel and understanding of the way in which users use the Web site and the interaction issues concerned with it.
· Can be used as a method to work with users when only partially-working prototypes are available.
· Gives quick feedback into the development process - crucial in the short development time scales for Web projects.
Disadvantages:
· By having experts working with users, the users' way of working may well be affected.
· Users may become overly-critical of interface issues.
Small and large scale usability lab tests
We carry out usability `lab' tests with users in a variety of different ways. Small scale studies can be carried out in particular environments (e.g. user's home or office) with portable video recording and logging equipment. Larger scale studies are carried out in the Nomos usability lab - a fully equipped usability lab designed originally for a variety of software and hardware testing.
Typically, users have a period of `free interaction' with the web site being tested, followed by a series of pre-defined representative user tasks which may include searching for particular information, working with particular applications, etc. Usability tests are often combined with user interviews and questionnaires, and automatic user interaction logging (logging tools and online questionnaires are detailed later).
Advantages:
· User interaction with a web site can be observed in detail.
· Interaction is recorded and can be analysed later.
· Performance, efficiency and effectiveness measures can be made for particular tasks.
· “Lab tests” have a perception of high validity on the part of clients.
· Video recordings of usability problems can be very persuasive in convincing programmers and designers of the need to make changes to poor interfaces.
Disadvantages:
· Relatively time consuming (and thus costly).
· Requires recording equipment.
· Difficult to use with prototype sites and therefore usually used only towards the end of the development process, or on existing sites.
· User interaction may not be completely natural.
· May not capture usability problems experienced by users using a different Web browser or computer platform to that used in the test (less of a problem for Intranet sites, where such factors are known and the test environment constructed accordingly).
Specialised web interaction logging tool
In partnership with a large computer science research institute in Stockholm, Nomos has developed a logging tool which enables detailed logging of user interaction with a web site which is much richer and more detailed than standard server log files. The logging tool and analysis software has been specifically developed as a means of moving the software usability test laboratory on to the Web itself.
Advantages:
· Particularly useful in analysing the way in which users navigate around a web site.
· Only requires the user to use the web site as normal, in his or her natural environment.
· Data collection is automatic once specified and thus data from a large number of users can be obtained relatively inexpensively.
· Can be combined with results from online subjective assessment questionnaires to give a rich picture of web site usability in a highly automated way.
Disadvantages:
· In some situations the logging may have some affect on performance of the Web interface (although minor compared with the large fluctuations in performance often experienced on the Web).
· Can produce vast quantities of data and thus requires care and some expert understanding in the specification of logging parameters and subsequent analysis.
· Requires a finished prototype or existing site.
· There are limits to the logging abilities when using Java applications.
Online subjective usability assessment questionnaires
In partnership with the Human Factors Research Group in Cork, Ireland, (the developers of the SUMI - Software Usability Measurement Inventory), Nomos is developing a generic web-administered user satisfaction questionnaire for web sites. SUMI is a paper-based questionnaire used by a variety of companies in Europe, and gives statistical measures of software usability on five scales - “User efficiency”, “Affect”, “Helpfulness”, “Control” and “Learnability”, as well as a global measure. The Web version is intended to give similar measures for Web interfaces, and has already been trialed on a number of Web sites in Sweden and the UK.
