Richard C. Omanson & Amy L. Schwartz
Ameritech
Introduction
The Human Factors department at Ameritech has played an active role
in
defining and evaluating a number of Web sites including our external
corporate site (www.ameritech.com), Corporate Communications
internal
site, our Internet access service site (www.ameritech.net), and a site
we created for Chicago96, the hosting committee for the Democratic
National Convention (www.chicago96.org). Our philosophy is that
good Web
sites are best designed with a specific purpose and population in mind.
If a site, like Ameritech's corporate site, is accessed by multiple
groups, multiple sections should be created within the site that cater
to each group. A commitment to designing sites for target populations
and goals makes it important to assess both the site and the user (e.g.,
their purpose in coming to the site, what they expect to see there and
how they expect it to be organized). Our research has both of these
questions.
We have used a number of methodologies to assess both the usability
of
sites and to understand what our users want and expect. Specifically,
we used:
1. Observing free-form exploration
2. Observing simulated tasks
3. Beta user questionnaires
4. Focus groups
5. Card Sorts
6. Feedback Analysis
7. Search Term Analysis
Observing Free-form Exploration
Part of our dial-up Internet access service involved a Web site that
contains a directory of 50,000 reviews of other Web sites. One
method
we used to test this site was to observe users exploring it. Subjects
were paid to come to a testing facility where they accessed and
explored a beta-version of the site. These explorations were video
taped and observed through one-way glass from an observation room.
Subjects were interviewed after their exploration.
This testing was done during the creation of the site with rapid
turn-around (time from experiment design to report delivery was three
days). Since we did this testing a number of times, we reduced costs
by
having subjects come into an office rather than a testing facility.
Observation was done in an adjacent office via a TV monitor rather
than
through one-way glass. Thus, our only expense was the rental of the
camcorder, tripod, and TV monitor and subject payments. The results
were just as effective.
This methodology gave us insight both into goals of our users (what did
they choose to do?) and weaknesses of the site (which parts confused
most users). Of the two, we probably learned more about our site than
we did about the users. The strength of this method was its ability to
uncover problems arising from using the site in ways that were not
anticipated. The weakness was that coverage of the site was not
systematic or consistent across subjects. Moreover, it was not truly
free-form. Rather than randomly or systematically clicking on links,
users tended to generate their own goals (guess I'll look for a my
favorite rock group). Thus, "explore this site" is a directed task just
as much as is "find a good vacation site."
Observing Simulated Tasks
To get a more systematic coverage of the site, we also observed these
same subjects as they performed specific tasks we gave them (e.g.,
find
a good place to vacation, create a Web page). This methodology gave
us
additional insight into weaknesses of our site. The advantage of this
approach was that coverage of the site could be controlled. The
limitation was that our Web site was so large, only a small portion of
it could actually be accessed. So while coverage was controlled and
consistent across subjects, it was not complete. Moreover, the tasks
were artificial and may not, in fact, correspond to how users actually
use the site.
Beta Test Questionnaires
Once our Internet access site was fully designed, we launched a Beta
test. Participants were asked to fill out questionnaires after each
milestone was completed (e.g., system installed, browsed on the
Web).
The questionnaires asked about their reactions to and problems with
the
site and service. The advantage of this approach was that it identified
problems that users actually encountered. However, since it was
basically a voluntary recall measure, descriptions were often
incomplete
and selective. The beta test was a big project that required a
full-time staff for several months. It was effective in identifying
customer support and other operations problems, but its impact on site
design and understanding users was minimal.
Focus Groups
We have used focus groups both before the launch of a new section
(e.g.,
putting jobs on our external corporate site and our employee
newspaper
on our internal site) and after site launch. Pre-launch focus groups
were conducted with intended users and were useful in learning about
what users expected to find. Post-launch focus groups were useful in
discovering navigation problems, and learning what content and
capabilities users wanted but weren't finding. The post-launch groups
were also helpful in identifying the tasks for which the users actually
used the site. The limitations of focus groups are well known. For
example, group interaction can influence individual reports.
Search Term Analysis
The terms users enter in a Web site's search tool can give insight into
their goals for coming and into what content they would like to see. We
keep a log of all the search terms entered on our external corporate
site and have classified according to the goals we infer they represent
and according to the content for which people searched. The
advantage of
this method is that it is inexpensive and it based on how users actually
use the site in the field. The disadvantage is that it can be a very
biased sample. Users that prefer to browse via menus, or that are
looking for content that has a pointer on the front page will never
enter a search term. Finally, it is always the case the reason why a
person came to a site must be inferred from the search terms used and
often what the reason is, is not evident.
Feedback Analysis
Our external corporate site has feedback forms on virtually every page.
The questions asked on these forms are logged. As with the search
terms, we have classified these forms according to the goals we infer
they represent and according to the about which people ask. This
methodology has similar strengths and weaknesses to the search term
analysis. Like search terms there may be considerable bias in who use
these forms if users fill them out only if they are having difficulty.
Moreover, the difficulty in inferring user goals that exist for search
terms, to a lesser degree, also exist here.
Card Sorts
To assess how users expect our Web site content to be organized, we
had
subjects do a card sort of our 50 top-level sections of our external
Web
site. We discovered that some of the distinctions we were making in
our
menus never showed up in the card sorts and were probably lost on
users.
We found some differences between experienced and novice Web
users, but
suspect now that even greater differences would be found between
groups
representing investors, shoppers, and the media. Even more than
different populations, we suspect that the specific goal with which
users arrive at a site will affect the organization they expect to see.
This methodology is inexpensive and can be done either before or after
a
site has been launched.
Contact Information
(only Rick Omanson will be able to attend)
2C-43D
2000 W. Ameritech Center Drive
Hoffman Estates, IL 60196-1025
Office: (847) 248-5555
Email: richard.omanson@ameritech.com
URL: www.ameritech.com/people/omanson/
Rick Omanson is a Member of Technical Staff in the Human Factors
and
Media Prototyping department at Ameritech. He is part of the core
Web
team that runs Ameritech's internal and external (www.ameritech.com)
Web
sites and is responsible for their overall design and evaluation. He is
co-author of Ameritech's Web page standards and design guidelines.
Prior to coming to Ameritech in 1992, Rick was a Member of
Technical
Staff at AT&T Bell Laboratories and a research fellow at the Learning
Research and Development Center at the University of Pittsburgh.
Rick
received his B.S. in psychology from the University of Illinois and his
Ph.D. in developmental psychology from the University of Minnesota.
Amy L. Schwartz is a Technical Manager in the Ameritech Human
Factors
and Media Prototyping group. She is co-author of the Ameritech
Phone-Based Interface Standards and Design Guidelines. Her current
interests are in the design of phone-based interfaces, usability testing
and usability metrics, methods for quality assurance and assessment,
and
Internet research applications. She was formerly President of Applied
Cognitive Research, a human factors consulting firm specializing in user
interface design. She received her B.A. in Psychology from Columbia
University in 1980 and M.S., M.Phil., and Ph.D. degrees in Cognitive
Psychology from Yale University. She is the former President of the
Chicago Metropolitan Chapter of the Human Factors and Ergonomics
Society.
