Michael J. Muller* , Rebecca Carr**, Catherine Ashworth*, Barbara Diekmann*, Cathleen Wharton*, Cherie Eickstaedt**, and Joan Clonts**
* U S WEST Technologies, 4001 Discovery Drive, Boulder CO 80303 USA, +1-303-541-8182 (fax)
** U S WEST Communications, 1801 California, Denver CO 80202 USA, +1-303-896-2563 (fax)
Muller: michael@advtech.uswest.com, +1-303-541-
6564.
Carr: rcarr@future.mnet.uswest.com, +1-303-896-
8278.
Diekmann: diekmann@advtech.uswest.com, +1-303-
541-6769.
Eickstaedt: ceickst@future.mnet.uswest.com,
+1-303-896-5963.
Clonts: jclonts@future.mnet.uswest.
com, +1-303-778-4023.
Ashworth: ashworth@clipr.
colorado.edu, +1-303-541-6613.
Wharton: cwharton@
advtech.uswest.com, +1-303-541-6292.
We begin with a brief overview of the work of Directory
Assistance (DA) operators, followed by a review of HCI
research on telephone operators. We then present data
from a detailed task analysis conducted with DA operators
at a U S WEST Communications DA office. We describe
a variety of types of expertise-based work that operators
perform on more than 50% of the calls that they handle.
We supplement this major analysis with a second,
confirmatory task analysis conducted with DA operators at
a second U S WEST Communications site. For
comparison purposes, we briefly report a case study of the
work of one toll and assistance (TA) operator.
Previous reports characterized the DA operators' role as
intermediary between customers and a system or database
[10]. Our results support a more human-to-human view of
the job, with the important knowledge and expertise vested
in the operators, rather than in their computer support tools.
Our interpretation is in some ways similar to Kidd's
analysis of knowledge work [9]. Our view is aligned with
Floyd's process-oriented paradigm (the human work
process is the starting point), rather than with the more
system-focused, product-oriented paradigm [4].
In our view, DA operators serve as expert consultants.
They help customers articulate their needs, engaging in
collaborative query refinement at one or more stages of a
DA call. Operators serve as experts in a variety of domains
of relevance to their customers' lives, helping them to
navigate through government agencies, complex business
hierarchies, partially remembered geographies, and
dynamic changes in their customers' worlds. In a small
case study, we find similar themes in the work of TA
operators. These interpretations - that is, that operators
perform significant knowledge work - have powerful
implications for work practices, training, support
technologies, and partnerships between management and
the labor force.
The hallmark of DA work is fast accuracy. In U S WEST,
the average call to a DA operator takes less than half a
minute. During this brief time, the operator listens to the
customer's request, and often engages in collaborative
refinement of that query with the customer. Based on her
or his analysis of the customer's needs - and perhaps
supplemented by new information that may have been
communicated from the business office at the beginning of
the work shift - the operator executes one or more
searches in a complex set of specialized databases, most of
which are internally partitioned. The operator then reports
only the relevant subset of the search results to the
customer - this is a second opportunity for collaborative
refinement. To save work time, the operator often invokes
a computerized audio report support tool that delivers the
telephone number to the customer. In some localities, this
tool may optionally dial the call for the customer.
Campbell and Velius [2] and Stuart and Gabrys [15] used a
modeling approach to predict the impact of experimental
technologies on DA calls. Unlike previous successes in the
modeling and prediction of Toll and Assistance (TA)
operators' work [6,8], both of these papers found large
deviations from their models' predictions. These may
signal trouble for proposals to automate DA calls [11,12].
Some of the subtle complexities of operators' work,
explored in this paper, may help to explain these outcomes.
Conceptions of operators' work are important to U S WEST
Communications' Operator and Information Services (OIS)
organization and to the operators' union, the
Communication Workers of America (CWA), which are
together developing knowledge worker descriptions of
other job titles [3,7]. It has been important to OIS and
CWA to understand operators' work in detail, so that the
company and the union can support their work and the
quality of service that they provide to U S WEST's
customers.
We applied CARD and PICTIVE in a "bifocal" analytical
approach [14]. CARD (which uses card images to represent
work tasks and events [16,17]) was used to analyze higher-
level task flow issues, and PICTIVE (which uses paper-
and-pencil representations of interaction media [13]) was
used to analyze lower-level interactions with specific
artifacts, including workstation screens. These techniques
have been described extensively, and will not be detailed
here. However, we note that one aspect of the CARD
technique was particularly valuable: This was the ability to
include explicit representations of the operator's goals,
strategies, and other mental operations as part of the
workflow.
We verified these results with representatives of CWA. We
then turned to a quantitative task analysis to determine, in
part, how often operators engaged in knowledge-work
activities, and what resources supported those activities.
Figure 1. An example of a Directory Assistance call, focusing on Operators' knowledge work.
Some of the credentials of the operator participants are
summarized in Table 1. Five operators were women, and
five were men.
Table 1. Backgrounds of Operator Participants
Fifty calls from each operator's videotape were
subsequently analyzed in detail. Of these calls, 73 percent
asked about businesses; 18 percent were for residences; 8
percent were for government agencies; and 1 percent were
other miscellaneous requests (area code, time of day, etc.).
In this brief paper, we cannot present all of our results. The
following two subsections focus on operators' knowledge
work, and on conditions enabling that work.
Figure 2. Occurrence of knowledge work in DA calls.
Figure 3 details the types of knowledge work that we
analyzed, based on the qualitative task analysis and on
operators' narratives during the participatory analysis.
Figure 3. Types of knowledge work in DA calls.
These included:
Figure 4. Occurrence of customer-volunteered information
(in addition to locality and listing) in DA calls.
Figure 5. Types of customer-volunteered information
(in addition to locality and listing) in DA calls.
Lawrence, Atwood, and Dews also have described some
aspects of customer-provided information. They argued
that the lack of specificity in the customers' information
could be a problem: "They might say that the name is
'something like,' or that 'it might be in Cambridge,' or 'it's
near Queens Boulevard'" [10, p. 402]. Our experience has
been that operators are adept at making sense and meaning
from these approximations, although this process may
require an extra conversational turn (see Figure 1, and the
knowledge work categories of "Schools," "Events," and
"Neighborhoods," above). A customer's approximate
query for a government agency provides a similar
opportunity for collaborative query refinement, as the
operator probes the customer's needs. We have observed
operators asking questions such as "Vialsystics? Do you
mean birth records?" [vital statistics] and "Well, was it a
moving violation?" [to look up the appropriate court] as
they helped customers to articulate their needs.
Did operators use the customer-volunteered information for
their knowledge work? As Figure 6 shows, knowledge
work was significantly associated with customer-
volunteered information (X2=5.87, p<.02).
Figure 6. Correlation of knowledge work and customer-
volunteered information in DA calls.
Eight operators participated. Approximately 50 calls were
scored for each operator, using the taxonomy of knowledge
work presented above, for a total of 410 calls. DA
operators performed knowledge work on 41 percent of the
calls, and customers volunteered information on 37 percent
of the calls. However, the association between knowledge
work and customer-volunteered information in this sample
did not achieve significance (X2=3.40, p>.05).
In terms of overall percentages, these results are broadly
consistent with the 1993 task analysis described above,
showing over 40% of the calls involving knowledge work
(over 50% in the 1993 task analysis). This similarity
occurred despite the fact that the two DA offices were
separated by 700 miles, different local work practices, and
different vendors' workstations.
DA operators maintain extensive long-term
knowledge of the structure and details of information in
their databases. Using this knowledge, they can trim
seconds off their calls while providing accurate
information. Examples include:
We note that our conceptions of skill and expertise are
based on the mundane, every-day skills of an experienced
worker (e.g., [9]), and not on the extraordinary skills of an
exceptional performer. For a discussion of mundane skills,
see [1].
Could it be that, while DA operators engage in knowledge
work on over half of their calls, TA operators do none?
One of us worked with one TA operator to explore this
question. This section briefly introduces the work of TA
operators, and then describes our tentative findings.
Customers' requests for assistance are sometimes the
telephone company's first notice of trouble: TA operators
therefore serve as first-line trouble diagnosticians,
analyzing and reporting system or service problems to
other offices in the telephone company. They are aided in
their diagnostic work by electronic mail announcements of
problems that may affect service. For example, during our
observations, the operator received notice of a major forest
fire in Montana which had the potential to affect telephone
service. The operator used this information to help one
customer plan alternate strategies for contacting someone
who was on the opposite side of the affected area.
One of us observed 128 TA calls that were handled by a
U S WEST TA operator during one weekday morning in
summer 1994. Calls were scored in real-time, without any
recording medium. As noted above, this may have led to
an undercount of subtle types of knowledge work.
A total of 38 percent of the calls involved significant
knowledge work by the TA operator, in areas such as
customer contact and negotiation skills, collaborative
refinement of the customer's needs and request, dialing
instructions suited to the customer's ability to understand
(especially in the case of international dialing), remote
operation of telephone equipment (coin telephones, line
tests), and analytic skills (diagnosing errors by other
operators, errors by network elements, database or network
errors requiring repair). Quantitative details are in Figure
7.
Figure 7. Knowledge work by one TA operator.
Task Analysis of One TA Operator's Work
These results must be interpreted tentatively. They will
have to be repeated with a larger sample of operators and
of calls. Nonetheless, this case study suggests that the
earlier research [6,8] may have been based on only a subset
of TA operator work. In all probability, the subset was
restricted to call-completion services, in which the TA
operator handles alternate billing arrangements, such as
credit card, collect, third-party, and so on. In fact, 46
percent of the calls in our case study involved call
completion, and only 17 percent of the call-completion
subset (6 percent of the 128 calls in the study) required
operator knowledge work.
While it is certainly true that operators mediate access
between customers and databases [10], this formulation
appears to omit important aspects of their work, such as
extensive knowledge that is not contained
within the database, collaborative refinement of customer's
queries, and expert diagnostic and problem-solving skills.
Similarly, while it is also true that many tasks performed by
operators are routine and repeatable [6,8], this formulation,
too, appears to omit critical aspects of the application of
expert knowledge and analysis, as well as human-to-human
skills. Operators are thus a hybrid case between Kidd's
clerical workers (who rely primarily on
external data and resources) and knowledge workers
(who rely primarily on their own internal
representations of information) [9].
We do not know, at present, how these more sophisticated
skills contribute to customer's perceptions of the quality of
service that they receive, nor to the operators' experience of
work that is both pleasant and satisfying. We suspect that
these components of knowledge, skill, and expertise are
crucial for both of these important considerations.
Abstract:
We present two large studies and one case study
that make a strong case for considering telephone operators
as knowledge workers. We describe a quantitative analysis
of the diversity of operators' knowledge work, and of how
their knowledge work coordinates with the subtle resources
contained within customers' requests. Operators engage in
collaborative query refinement with customers, exhibiting a
rich set of skilled performances. Earlier reports
characterized the operators' role as an intermediary
between customer and database. In contrast, we focus on
operator's consultative work in which they use computer
systems as one type of support for their primarily cognitive
activities. Our results suggest that knowledge work may be
a subtle feature of many jobs, not only those that are
labeled as such. Our methodology may be useful for the
analysis of other domains involving skilled workers.
Keywords:
Telephone operators, knowledge work,
expertise, skilled performance, participatory design,
participatory analysis
Introduction
This paper develops and documents U S WEST's emerging
view of telephone operators as knowledge workers. We
describe qualitative and quantitative task analyses that
show the diversity and depth of operators' skilled
performances.
The Work of Directory Assistance Operators
DA operators are the women and men who look up
telephone numbers in response to customers' requests.
Different countries have different practices in this area. In
the US and Canada, all DA calls go to operators who have
access to databases for residential, business, and
government telephone numbers (by contrast, in Poland,
different operators are responsible for each type of
listings). Because of the volume of work and the number
of operators who perform it, telephone companies are often
concerned to minimize the time required to handle each
DA call. Savings of even a tenth of a second per call are
multiplied into significant corporate economies.
Previous Research on Telephone Operators
In prior work, Lawrence et al. analyzed the work of DA
operators as a special form of mediation between customer
and database. In their terms, operators are a particular case
of the general class of work involving two humans and a
computer: "In these interactions, one person, typically a
customer, wants to accomplish some goal with a system but
does so by interacting with a human intermediary. The
computer operator is, in effect, a 'surrogate user'." [10, p.
399]. They provided examples of several types of
translations and inferences that DA operators perform
while working in this mode, and continued earlier
quantitative research into operators' abilities to time-share
their human-to-human and human-to-computer
interactions. In this paper, we provide a more rigorous
taxonomy of DA operators' cognitive and social work, and
quantitative estimates of the occurrence of knowledge
work.
TASK ANALYSIS WITH DIRECTORY ASSISTANCE
OPERATORS
Our major task analysis took place in two stages. The first
stage was a qualitative study, conducted by subject matter
experts (SMEs) from the OIS training organization, in col-
laboration with a human factors worker. The results of this
stage guided the second, quantitative stage. The
quantitative task analysis was conducted by one SME and
one human factors worker, with ten operator participants
who were selected and recruited by CWA. Analyses of the
videotaped data were conducted by a team of HCI
specialists, in close collaboration with the SME. Interim
and final results were validated with representatives of
CWA.
Collaborative Qualitative Analysis
Method
The qualitative task analysis used the CARD and PICTIVE
techniques from participatory design [13,17]. We
emphasize that we were not, in this stage, engaged in a
participatory activity, because no operators (users) were
involved in this preliminary analysis. However, we used
participatory techniques in our qualitative analysis.
Results
Through the qualitative task analysis, we produced an
initial description of the work of DA operators, including
representative task flows, which were shown in [16]. It
quickly became obvious that there were many opportunities
for operators to perform a variety of types of knowledge
work during these task flows. Figure 1 shows one example
of a variety of types of knowledge work (for convenience,
two different calls have been combined in the one figure).
Specific descriptions of types of knowledge work will be
provided in the next section.
Quantitative Analyses
Method
The qualitative task analysis used the method of direct,
videotaped observation of one hour of live traffic handled
by each of ten operator participants. Calls were recorded
during three weekdays during autumn 1993, between the
hours of 8:00AM and 8:00PM. Video recordings of the DA
calls were supplemented by each operator viewing her or
his videotape and providing explanatory comments; these
participatory analysis sessions were also videotaped. Results: Operators' Knowledge Work
Figure 2 provides an overview of the results relating to
operators' knowledge work. Fifty-three percent of the calls
involved at least one type of knowledge work. Thirty-one
percent involved more than one type.
Results:
Customer-Volunteered Information
What resources do DA operators use to perform this
knowledge work? One often-overlooked resource is the
information that customers volunteer - that is, information
that is over and above the standard locality and
name of the listing information that operators
are trained to request, and that technologists design toward
[11,12]. When we analyzed customer-volunteered
information, we found that it occurred during more than
half the calls (Figure 4). Details of the types of customer-
volunteered information are provided in Figure 5.
CONFIRMATORY TASK ANALYSIS AT A SECOND
DIRECTORY ASSISTANCE OFFICE
During the summer of 1994, one of us had the opportunity
to conduct a quantitative task analysis at a second DA
office in U S WEST. Unlike the first analysis, the second
exercise could not use videotaped recordings. Calls were
observed and characterized in real-time. Therefore, we
suspect that subtle types of knowledge work and customer-
volunteered information may have gone undetected.
OPERATORS' WORK AS MUNDANE EXPERTISE
We compared our view of DA operators' knowledge work
with published conceptual analyses of expert performance.
DA operators' work fulfilled all seven of the criteria listed
by Glaser and Chi [5]:
DA
operators have detailed domain-specific knowledge that
includes the computer systems that they use, the business
and geographical domains of their customers, and skills
in conducting brief but effective conversations.
DA operators routinely extract meaning from
large numbers of potential answers to customers' queries.
Some of these are quite complex - e.g., departmental
listings of government agencies, whose naming
conventions are often more responsive to the agency's
internal logic than to citizens' needs for access.
As noted above, DA
operators work quickly, and with high
accuracy.
DA operators routinely maintain many
details of the customer's request in short-term memory.
The best keying strategy is often to search on the main
name information, and to use address or department
information as a cognitive (but not keyed) aid in sorting
through the listings returned from the database.
DA operators possess a detailed
structural knowledge of the database, in terms both of its
abstract structure (different files, different partitions
within files) and in terms of the formal organization of
government or business department hierarchies. They
are also skilled at diagnosing database entries that have
become inaccurate or even subtly misleading, as well as
telephone trunks that, while still usable, are in need of
maintenance. Their trouble reports and proposed
solutions often serve as first-line indications of problems
that other offices in the company then attend to.
While DA operators handle most calls very quickly, they
are able to identify customer queries that fall outside of
their usual domain of speeded expertise. On this subset
of calls, they spend extra time to analyze the problem in
detail. In extreme cases, to economize on individual
work time, operators transfer the caller directly to a
Customer Service Consultant (sometimes called a
Service Assistant), whis is assigned to spend the needed
time on difficult calls.
DA
operators are expected to maintain a certain performance
level, usually expressed as average call duration. Failure
to achieve this criterion impacts the entire office, forcing
other operators to work harder, and causing longer
waiting time for customers. Operators maintain a sense
of whether they are meeting this criterion. If they are,
they can spend more time on a difficult call. If they are
not, then they attempt to speed their work, so as regain
their targeted performance.
CASE STUDY WITH
ONE TOLL AND ASSISTANCE OPERATOR
We were curious to know if our results regarding
knowledge work were applicable to another operator job -
that of Toll and Assistance (TA) operators. The work of
TA operators has been extensively studied and modeled,
with spectacular successes for cognitive modeling in real-
world settings [6,8]. However, the requirements of these
modeling activities appeared to contrast with our
knowledge work findings in DA operators. For the
modeling techniques to be effective, the modeled behavior
must be routine and repeatable. Variations in the behavior
should be quantitative, rather than qualitative. For the
modeled calls, John wrote: "There is typically no problem
solving involved; the [operator] simply recognizes the call
situation and executes routine procedures associated with
that situation" [8, p. 107]. Similarly, according to Gray et
al., "TAOs [toll and assistance operators] recognize each
call situation and execute well-practiced methods, rather
than engage in problem solving." [6, p. 241].
The Work of Toll and Assistance (TA) Operators
Operators who work in TA in the US have different
responsibilities from those who work in DA. TA operators
are primarily concerned with call completion
- that is, with helping a customer who already knows the
number s/he wants to call, but needs operator assistance to
place the call. Assistance may take the form of special
dialing, but is most often concerned with alternate billing
arrangements (collect calls, billing to credit cards or calling
cards, or billing to the caller's home telephone number if
the caller is traveling). TA operators may also receive
nearly all other requests for assistance. These include a
great miscellany of problems, ranging from a customer
who needs to interrupt someone else's on-going telephone
call with an emergency message, to someone who is
confused with advanced calling features, to preliminary
inquiries about billing, repair, and other telephone
company operations that are subsequently handled by other
staff at other offices. DISCUSSION
Customers often call operators not for mediated access to a
database, but rather for expert assistance in finding
information that the customers need in order to live and
work in their worlds. As we have shown, operators
respond to these needs through a mixture of their
knowledge of the customers' worlds, of the changing
circumstances of those worlds, and of the
structure and content of their database systems. In this
way, we have framed our analysis in terms of the
human processes in the work, rather than in
terms of the software product that plays a
supporting role for one of those humans (see [4]).
