Rick's Notes

Intro

Will focus on testing and assume that analysis & design has been done.
Nigel: why exclude it if it occurs throughout.

Intros
Michael Levi- Bureau of Labor Statistics
Fred Conrad  - Bureau of Labor Statistics 
Laura Downey NIST CS usability
Ginny Redish - Linguist via documentation Bethesda independent consultant
Sharon Laskowski - NIST CS visualization expert system - visualization on Web
Laurie Kantner - TechEd via documentation
Mary Utt - Open market startup secure financial transactions documentation
Kathy Frederickson-Mele - BLS economist software development
Richard Whitehead - Numos UK human factors 
Greg Wilt - Bell Atlantic usability
Nigel Bevan - NPL UK methods for usability working on standards 
Carol Kilpatrick - Georgia State CS teaching HCI
Gary Marchionini - U of Maryland library services via education documentation
Alain Robillard - Montreal HCI group
Irv Katz - ETS Cognitive Psychology, does CS visual disabilities

---------------------------

One notion is the difference between Web design versus Web testing.
1.	what are usability problems on the web
	how do we find them?  bottom up

2.	certain facets that are unique to web?

3.	Organizational context - how to implement it.

Cost is an issue.
	Communicating results to funders and developers
	Convincing the developers that the changes are needed.

Summary of Position Papers
Usability Problems Exposed
   - unclear labeling/vocabulary
   - user needs to remember too much
   - graphics are cluttered and overused
   - inaccurate understanding of site design
   - poor match between site design and user¹s needs
   - navigation problems
   - designed without clear target user populations in mind
   - design is not guided by user¹s goals
   - inadequate ³fan out²
   - insufficient privacy and security
   - inadequate buy-in from stakeholders
Methods
   - Heuristic evaluation and other inspection methods
   - paper prototypes, storyboarding
   - end user sessions;Exploratory assigned task, think aloud
   - sorting and mapping tasks
   - analysis of usage logs, feedback and search terms
   - satisfaction judgments
   - interviews with support staff/site contacts
   - focus groups
   - checklists
THEMES POSITIONS, OBSERVATIONS
methodological challenges:
   - Many sites too large to test exhaustively with any method
   - native data collection tools inadequate:
      + browser history lists do not capture order or time
      + server logs provide too much data (mostly irrelevant)
      + cannot be confidently segmented into sessions
      + do not capture info about cashed pages
   - search terms and feedback are indirect measures of goals and not typical
   - need better tools for remote testing  
   - Target users and actual users may differ
   - platforms browsers, speed of connection are not under developers¹ control
   - organizations may be reluctant to support usability testing 
     even if they support it for conventional software

Dimensions of Web Sites
   Size:  large institutional sites have many designers and no one 
          repository of design information
      -- large sites are harder to change because consequences are 
         harder to anticipate
      -- many loyal users may be disrupted
   Retrieval vs. transactions:
      -- sites for doing business are interface to web applications
      -- Web site as document versus software
      -- Web sites as trees vs. webs
         - some user tasks lead to depth first search strategies
         - some sites are designed to be explored depth first
         - these do not always correspond
    Web sites vs. Software
      -- Web sites are software and have the same usability problems 
         as software but in different proportions
      -- web sites have disproportionately more navigation problems
         same methods should be used in both cases as in HCI methods 
         literature, no position papers advocated exclusive use of 
         heuristic evaluation
      -- good design involves identifying the users, but web sites have 
         unique characteristics
      -- published instead of released
         + except they can be interfaces to applications
      -- have an informating purpose, 
         so we need to test what knowledge users actually acquire
         + but so does educational software
      -- sites can change often, so we need to rapid evaluation methods
         + web users might expect less predictability
      -- cannot design confidently for target users

-------------------
Brain Storming Session
Features Unique to Web

navigation	3,4,7
disorientation	2,3,4,5
distraction	1,4
Frames	1,3,5
broken links	5
different types of links	2,4
Browser differences	1	
slow graphics	1,3,4
inconsistent terms 	4,2,1
printing	1,3
scrolling	3,1, 
screen differences
color differences
not user¹s terms 
inconsistency
lack of structure
structure not conveyed to use
timeliness - obsolete
poor use of feedback
lack state
confusion of browser & page navigation
proliferation of links
click here
spawning multiple browsers
back button problems
back-forward loop
helper aps and plug-ins
no text access
lack of sibling navigation
depth to breadth 
fireball restrictions
mailtos
hard to use forms
search 
poor use of color
site identity
growing beyond initial design
multi-purpose agendas
suitability of icons
irrelevant information
content organization  
poor chucking
long pages 
within vs. Between document navigation
significance of codes
lack of starting points
internationalization 
suitability for task
too many clicks
personalization
security
disabled users
poor html techniques
user differences computer skills
user differences content experts
inconsistent layout
overuse of consistency
context of use - 
designing for browsing vs searching
over relaince on helper apps
consistency with user expectations
rapid evolutio
moving targe 
error tolerance
interacting with databse
display of results
misuse of animation
lack of tools
info not carried
URLs and page titles

---------------------------------------
Categories used to classify problems
Provided by Richard Whitehall 
from ISO 9241 part 10.  Dialogue principles
----------------------------------------------

1.	Suitability For The Task
2.	Self Descriptiveness
3.	Controlability
4.	Conformative To User Expectations
5.	Error Tolerance
6.	Suitability For Individualization
7.	Suitability For Learning

-------------------------------
Breakout Session Notes
Group 3 Classification Notes - 
Types Of Problems
--------------------------------

We started by trying to classify the problem along several dimensions:

1st Pass:
what can you do:
user goals: information vs. transactions
user populations:
technology used: 
  html version
  helper aps & plugins
  browser level
  interaction with other systems

2nd Pass:
Things that constain the HTML
Things that augment the HTML (e.g., interacting with other systems)

3rd Pass: Types of Sites
info: reference
info: reading
task: transactions
entertainment

4th Pass: We agreed to define a type of site along several dimensions and focus on the a specific type of site within that space.  The 4 dimensions are:
Purpose, Target Population, Target Technology, and Site complexity.  
For our readout, we selected the following instance:

Purpose:
   goal: get information
   user action: get refence information

User population: 
   computer knowledge: moderate
   content knowledge: naive    
   age: adults
   handicaps: none
   Language: English 

Technology
	HTML: 3.2
	scripting: CGI & not Java
	helper aps: none
	browser level: Mozilla 2.0 and up
	platforms: Mac & Windows 95

SITE Complexity
  	interaction with other systems: none
  	site complexity: simple (no sub-sites)
  	search engine: yes
----------------------------
Group 1 Readout
----------------------------

Approach: 
   - Define a context
   - indentify a key issues
   - Organize error list
   - Different context means different key issues

Context = Dimensions
   - User Goal (entertainment, browing, get reference info)
   - User Action (looking for specific info)
   - User Profile (computer sophisticated, content naive)
   - Site Complexity (size, subsites, search engine)
   - Technology (target browser, html level, scripting types, target platform)

Categorization
   -  Indentity (6%)
      growing beyond design
      multi-purpose agenda

   -  Navigation (40%)
      disorientation
      broken links
      poor feedback
      structure not conveyed
      dead ends
      lack of fan-out (sibling navigation)
      know where link goes (e.g., off site)
   
   - page layout (20%)
      slow graphics
      overuse of color
      extraneous formatting
      hard to use forms

   - Content (10%)
      inconsistent terms 
      obsolete data

   -  Other (20%)
      firewall problems
      helper aps
      consistency

User¹s Tasks/Issues
   - Find the right site 
      + Indentify

   - Find the right place 
      + Navigation within documents
      + Navigation between documents

   - Identify the right information 
      + Reject information that¹s not relevant
      + Page loyout/graphics
      + Content
   - Understand inforamtion
      + Cotnent
      + Page layout/graphics
 
   - Act on information

-------------------------
Readout: Group 2
--------------------------------
Types of usability issue
   1. management/maintenance
	  e.g., broken links
	  obsolete data

   2. Context of use (interviews, etc
	  2.1 user characteristics
	     - frequency of use
	  2.2 user goals and tasks (task analysis)
	  2.3 technology constraints
	     - e.g., frames color, plugins, cache
         - does it work on the intended technology
         - has the most appropriate technology been
           used to support the design?
         - tools, expert/heuristc, technology testing

   3. Navigation
	  - e.g., scroll, back . dead ends
	  - can you find it?
	  - how long does it take?
	  - how long do experts take?
      - logging tools

   4. Structure
      - e.g., content organization, chunking
	  - does the structure match the user¹s mental model?
	  - card sort, graphical representation of mental model

   5. Content
	  - e.g., terminology, focus, international 
	  - do the intended users find the right info?
	  - user interviews, paper prototyping

   6. Design principles (page & site)
	  - does the page enable the users to achieve their task?
	  - expert/heuristic evaluation, checklists, standards, guidelines
      
   *  the BIG PICTURE 
      - can the user acheive their goal
	     + effectiveness
	     + efficientc
	     + satisfaction
      - user testing
      - questionnaires

-------------------------------
Readout Group 3
------------------------------
Focus:
   -  Problems with large site - structure
   -  Objective:  What can developed to enhance usability 
      testing for thos types of problems, in particular what 
      types of tools exist for automation

Caveat
   we understand that we can¹t replace user testing,
   but we can:
      - automate easy characteristics 
      - find/indentify where/what to test


PROBLEM ==> TESTING SUPPORT TOOL
timeliness ==> freshness stamp & email to page owner

specifying design ==> automated card sorter

consistency ==> style guide checker

structure ==>  - broken link checker
               - count in/out link ratios
               - complexity metrics 
                 (pages/link)
                 (using hypertext metrics from Neilsen & Shneiderman)
               - page hits
               - time on page
                 (orgnization goals determines if staying long is good)
               - page performance (e.g., simulation of time delays)
               - client-side monitors to collect better info on user actions
               - tracking IP addresses over several visits 
                 to identify goals and novice vs. expert
               - spelling including jargon check ³hetero-scadisity², income
we¹d like to know when its time to restructure
We¹d like the tools to evolve quickly

-------------------------------------
Brainstorming session
How is web different?
-------------------------------------

1.	Being used by a variety of users
2.	being used by a variety of platforms
3.	Rapid evolution
4.	boundaries are fuzzy
5.	professionals and amateurs share same space.
6.	types of sites
7.	fluid boundaries of size and time
8.	intangible
9.	multiple servers
10.	decentralization of development and maintenance
11.	unknown entry points
12.	variety of skills required does not match developer skills 
13.	lack of  gatekeeping
14.	amaturs and experts share same space
15.	relationships info a
16.	artificial relationships (don¹t make sense)
17.	way people find out about site 
18.	online advertizing
19.	competition for space with other istes
20.	importance off first impressions
21.	time to load 
22.	closed vs. open community
23.	page as unit
24.	browser as intermediary
25.	catering to minority interests
26.	multipurpose sites
27.	stage of development less clear
28.	no taxonomy 
29.	early in development
30.	automatic usage logging
31.	less privte tha alone software
32.	can police web use

Need methods....
--------------------------------
Break!
--------------------------------
There was discussion of what phase we were going to focus on

Life Cycle 
  Analysis
      --> prototype:
  System
      --> beta test
  Release

Propose that we focus on informational sites

------------------------------------
Continuation of break out session
-------------------------------------
Gary:
Had a site that changed based on how people used it.
Come up with dependent measures that assess the effect of the change - 

How do you address variety of users, platforms and browsers?

Remote testing
Distinguish what can be handled via guideline and checklist versus that needs actual testing.  Speed can be handled by filesize guidelines  , and graphics composition (put them on the bottom).

Testing 
Richard Whitehall described a nice tool that adds java wrappers to pages so that keystrokes and anchor clicks are recorded and sent to the server.  Working on analysis tools that accompany its.

----------------------------------
Group 2 Methods read out
----------------------------------

ISSUES              ==>   METHODS

   variety of users|         |remote usability test.
   platforms &     |    ==>  |online questionnaires
   browsers        |         |logs
                   |         |global heuristic evaluations

   Note: to test overall structure, do least platform   
   dependent as possible for prototype

   fluid size and time  ==> ongoing testing

   time to load         ==> function test checklist 
                         (file size, graphics placement)

   decentralization     ==> checklist standards, training classes
   of development 

   maintenance          ==> note: different levels of techniques

   unknown entry points ==> collect log 
                            include scenarios that don¹t start at home page

Web specific tool from Richard:
   -  Enables remote testing by users and experts
      + rapid feedback on tototypes
	  + reduced cost
	  + realistic¹ conditions

   - Collecting data about pages accesses, search terms, etc (server side)
	  + traditional server logging
	  + use f cooke for accuracy

   - Detailed user interaction logging
     (client and server tools)
	  + capture browser actions (bak/forward)
	  + within pace interaction
	  + accurate timing
	  +  additional ³pop-up¹ user questions, etc

   - Online questionnaires
	  + feedback forms
	  + satisfaction measurement (wammi/sumi/etc)

   - HTML checkers extended with the style conformance checking
     Standard user measurement inventory

------------------------------------------
Group 3 readout matching method & problems
-------------------------------------------

Overarching Context
   - Rapid
   - Remote
   - Automated

Problems Addressed
   - Diversity of development (amateurs & experts share same space)
   - Global, diverse user base
   - Multiplicity of goals
   - Rapid evolution

Diversity of Development
	key: consistency
   
   - Desgin: testplan, framework, template, tools
     Evaluation: Expert inspection (pages from across developers)

   - evalutaion via case studies
     [amateurs & experts]
	  + examples
	  + design rationale	
	  + evaluation examples with executive sumary
   
   Still need an approach to use the case studies

Global User Base
   - Remote testing
	  + recruiting --> advertising, guest book
	  + schedule & specific tasks
	  + data: videotape mirror of user interaction, verbal via network

   -  Cooperation among organizations
	  + reciprocal user testing 
	  + provides alternative viewpoints

Two More Problems
   - Multiplicity of goals
	  + individual, group, instititions 
	  + Approach: scenarios
	      o traditionally --> well defined goals
	      o Web --> scenarios with all goals
	                scenarios with undev goals,  
                    but used in parallel 
	                usability tests
   - Rapid evolution
	  + approach: sampling of pages from large sites

------------------------
Group 1 readout
-------------------------
Variety of users, platforms, browers
   - define
      + users
	  + usage context
		 o focus groups
		 o questionnaires
	  + minorit needs
	  + image
	  + usability goals
		 o perceived quality
		 o # hits on important pages logs
		 o # completeing transactions vs. attempted logs

Page as unit, relationship - rich
   - Prioritize home page design
	  + Image
	  + indicate scope of site
	  + motivate user to click
		 o user testing
		 o prototypes
   - Every page can be an entry point
	  + Title should be standalone
	  + Link to home page and newxt level up
		 o check list /heuristics
	  + meaningful, cimple URLS
      (Discussion of subsites and how high up to jump)

Decentralized development
   - Analogy with traditional publishig
   - metrics to make business case
   - hard sell - perceived as high effort
      + Criteria for publishing could be to 
        test or validate against checklists

New Browser features
   - usability community needs to define usability 
     benefits and provide guidance for use

Special needs
	- if it usable for the seeing-imparied, 
      it¹s more usable for the signted.
	- Use special needs as a baseline.

-----------------------------------------
Organizational Issues to get things Implemented
Brainstorming the Issues:
----------------------------------------

Multiple owners
Lack of knowlege of web
Lack of corporate knowledge
Not clear goals
institutional barriers to getting access to users
Lack of standards
No time for testing 
No money for testing
Developers are defensive
Duplication of efforts
Inertia - resistence to change
Media seen only for advertising
Web seen as threat
Evaluator & designer are the same person
it appears to be too easy
Non-publishing culture
No professional standards
lack of infrastructure
too many standards
bandwidth
tight deadlines
perception that evaluation is fast and easy
called in too late
fear of change
loss of control
understaffed, overtasked
corporate instead of user view of purpose
easy to convert
site doens¹t have mission
how do you measure success
multiple developers

-------------------------------------
Discussion of solutions with whole group.
---------------------------------------

how do get through the distrust of developers
   - Say what is good
   - work with them early on and build relationship
   - Even when pages are outsourced, 
     you have a similar acceptance dynamic, 
     only not with the developer but rather with the owner.
     In either case, you have to train the person 
 
Need to change corporate culture.  
   - Shoot down the developer cowboys.
   - often the sponsor has mltiple agendas,   
   - Usability can support some of these agendas.
     But it may not be possible to change the 
     corporate culture very queickly.  
   - The buy in comes by tying it to the bottom line.

Different issues of saving money versus usability.  
   - A sponsor has limited resources 
     and the tradeoff has to be presented to them.
   - usability testing gets justified by 
     if they don¹t find it or go there, 
     they don¹t see it or buy it.

What type of benefits are there for informational sites?  Reducing costs?

The fear that the Web will change the organization is valid.
   - Lots of banks in Sweden are going online. 
     
Being able to do a desired task is an aspect of usability.  
So that some types of summative evaluation are valid for usability

Can we come up with a checklist of issues that we can use 
to say why usability testing is important.

usability is a key component, but not only component to outcomes such as traffic, saving money, making money.

What is the role of the usability test in the broader goals for the site?

Need to push people to develop goals for why they want to put things on the web.  
The web is a form of publishing.

Getting the goals 
Getting the buyin

Part of getting goals is agreeing on what the metaphor is.

-------------------------------------------
Last Exercise: Matching Methods & Problems
--------------------------------------------
Problem areas
   a. magnt/maint
   b. tech constraints
   c. navigation
   d. structure
   e. content
   f. Missing goals
   g. page Layout

Methods
   1. Expert instpection
   2. paper protyping/storygoarding
   3. Exploratory End User
   4. Scenario-based End User
   5. Sorting and mapping
   6. opprtunistic log analysis
   7. intentiaonl log analyssi
   8. search term analysis
   9. satisfaction questioaries
  10. focus groups
  11. user interviews
  12. check lists
  13. automated checking tools
 

Stage = Mid: Prototype

Stage = Early in Design

Stage = Late: Beta Testing

Wind up, down, around:

Consolidation:
Process - writing a book on this topic takes too long 
- obsolete in 2 years.
perhaps article in Interactions
perhaps an article in a commercial web magazines
perhaps a web site

usability is an important aspect of the current dialog going on on measurement and site management

Web site would be for others doing usability of web sites
NIST will do the email exploder