Thomas W. Mastaglio, Jeanine Williamson
Loral Federal Systems
12461 Research Parkway, Suite 400
Orlando, FL 32817
(407) 823-7345
tmast@greatwall.cctt.com
Dynamics Research Corporation
12461 Research Parkway, Suite 400
Orlando, FL 32817
(407) 823-7345
willt@greatwall.cctt.com
work together to accomplish team objectives. This paper
discusses the development approach being used in CCTT; it
specifically focuses on the usability design issues and how
they are being addressed.
The purpose of CCTT is to train Army units to operate as a
team [4]. The training context for CCTT is a virtual
environment -- a realistic model of the terrain that is shared
by the training audience. Over 50 different interfaces to that
virtual enviroment are being designed, implemented and
tested as part of the CCTT development effort. Some of
these interfaces are simulators with visual ports into the
virtual environment, others are workstations used to control
the interaction of intelligent agents which operate in that
virutal environmnet. Developing these user interfaces
presents many unique challenges that require us to apply
and adapt HCI methodologies developed over the past
decade.
In CCTT the training audience operate as a team interacting
within a virtual environment. Designing that environment in
itself presents numerous challenges. But designing and
testing the interfaces to that environment requires the
application of proven HCI techniques and technology. The
approach we are taking in CCTT and the lessons learned
from our experiences will assist others who will have to
design interconnected virtual reality systems in the future.
Interfacing even a single user to a synthetic world using
currently available technology requires a user-centered
approach to design, continuous user testing, and an iterative
approach. This challenge is compounded in CCTT because
we are also developing a large scale system with a complex
HCI environment comprised on many numerous and varied
user-computer interfaces [7].
The CCTT system will be installed at 32 sites throughout
the world. More than 500 simulators and over 300
workstations will be built and fielded. These sites will be
located at Army posts in fixed configurations or placed into
trailers to provide a mobile capability to train National
Guard units in their home towns. The configuration of a
single fixed site will depend upon the units which train
there, but in general they will have approximately 40
simulators and upwards of 20 workstations available. The
system requirement calls for seven different types of
simulators and 12 different types of workstations. Each type
of simulator is being designed to meet the specific training
requirements of a prototypical crew. Four types of
workstations are run by contractor personnel and their
purpose is to support the training, the other eight types will
be used by soldiers to interact with the virtual environment
that is shared with and among the simulators.
IEEE has established a standard for this type of interactive
virtual environment, the IEEE Standard 1278 for
Distributed Interactive Simulations (DIS) [3]. The key to
DIS is that each node on the network maintains its own copy
of the virtual environment and the state of each object
within its area of concern within that environment. A set of
protocols, defined by the standard, pass entity state
information to other entities so they can update local state
information. This allows any simulator or workstation on
the network to continuously display a correct view of the
synthetic world. Each node is in itself a view port into that
synthetic world.
Designing the interface for each node on the network is our
main challenge. Prior to contract award we conducted an
analysis of the CCTT system concept to identify all
potential user-computer interaction (UCI) components,
characterize their target users, and to determine what type of
task analysis was required to support design of each
interface. That pre-design analysis determined that within
CCTT there would be 57 different interfaces. We
considered an interface as any medium used to communicate
with the system or operate within the virtual environment.
Our notion is somewhat unique because, in addition to
traditional screen and input devices, our notion of interfaces
includes those UCI where users must cognitively (and
manually) respond as if they were operating actual
equipment.
Four different types of interfaces were identified:
Three types of user categories were identified:
Tasks to be performed at each interface are based upon
either an assessment of what the operator has to accomplish
or what tasks are targeted for training in the system. Both
types of analsyis are necessary to the design of the interfaces
in CCTT.
Another challenge for this program is managing the HCI
design effort within a large engineering organization. The
CCTT development team has approximately 300 engineers.
Most of them are collocated in an Integrated Development
Facility in Orlando, but some hardware engineers work in
the manufacturing facility, also in Orlando. The visual
system engineering and development is taking place in Salt
Lake City. The effort is organized using a concurrent
engineering (CE) approach. There are four CE teams, each
responsible for one type of system component:
Teams are further broken down to build specific products
(e.g., there is a module CE team building the M1 Abrams
Tank simulator). Human Factors engineers are an integral
part of each CE team. Although not formal members of sub
teams they must interact with them to support design
analysis and evaluations. To assist in this process a three
man detachment of Army experts (soldiers) work in our
integrated development facility full time. Figure 1 is a
conceptual overview of that organization.
FIGURE 1
The CCTT Development Organization is
organized into Concurrent Engineering Teams by
system component
An Integrated Development Team (IDT) was formed to
build and test CCTT. That team is a mosaic of skills and
knowledge; it is comprised of system and software
engineers and users organized around a concurrent
engineering approach. There are four concurrent
engineering (CE) teams developing each of the major types
of components. Each team is compsied of members from all
engineering disciplines needed for a project this complex:
systems, software, hardware, human factors, and test.
Figure 2 shows the interrelationship of engineering
disciplines and CE teams.
FIGURE 2
Engineering and support disciplines are active participants
in the CCTT development effort as members of each of the CE Teams
We extended the CE concept by including members of the
user community on each team [6]. A User Optimization
Team comprised of three subject matter experts is part of
our overall integrated development appraoch. These three
experts are active members of the CE teams. They are
involved on a daily basis with system requirements analysis,
high level design, implementation, and evaluation.
The system is being developed using a spiral methodology
to produce seven incremental system builds. The builds
integrate those components of CCTT developed to date.
Figure 3 graphically depicts the general content of each
spiral build and completion dates. An exit condition for
each build is a user evaluation. These evaluations are
conducted as scenario-based training exercises prototypical
of situations that will be used on the objective system. The
results of the user exercise are analyzed by human factors
engineers for technical shortcomings in the design or
implementation. Those deficienices are reported back to
engineering to be addressed prior to government acceptance
of the system for independent formal testing.
The CCTT development effort will take three years from
contract award until delivery of a complete system for user
acceptance testing. A number of issues were identified early
in the program which would impact the success of that test
and appropriate HCI approaches selected to address those
issues.
Early User Involvement
It was imperative to have early user involvement in the
design process [9]. To achieve that we implemented several
intiatives.
FIGURE 3
CCTT is being developed as a series of seven incremental spiral builds spread over a
two year
period. Dates shown in the figure represent the completion of each build to include a user
evaluation.
Simulator module fidelity is a major HCI issue for CCTT.
At first glance it appears to be a fairly simple problem, just
make a simulator that looks and operates like a real vehicle.
The challenge is to keep the cost of that simulator
reasonable. Flight simulators have been developed
following a full fidelity approach for last decade, but they
cost upward of tens of millions of dollars per copy. CCTT
cannot afford such costly modules in the required quantities.
A selective fidelity approach is used instead.
An up front task analysis determined the task steps which
could and should be performed to accomplished effective
training in CCTT. Based on that analysis, a determination is
made for each control, display or indicator in the actual
vehicle to determine if it has to be functional, pictorially
depicted (e.g., using a decal), or can be left out all together.
The results of that analysis were reviewed by the user
community and then used to build prototype engineering
development models which are subject to further user
review.
The virtual environment in which soldiers train using CCTT
must be a high fidelity replication of real world terrain. The
sparse and often artificial worlds used in most laboratory
systems and entertainment-based virtual reality systems are
inadequate. There is a danger that training in an unrealistic
synthetic world will result in soldiers acquiring incorrect
skills, ones which may cause them to make incorrect and
lethal decisions in actual combat. On the other hand, we
cannot afford the computational cost for a visual system that
provides a completely realistic model of the world.
We had to analytically determine which simplifications are
acceptable, such as displaying a building as a simple two
dimensional polygon rather than a full fidelity three
dimensional model that can be entered and occupied. In a
process similar to the module fidelity analysis discussed
above, an up front analysis has been conducted of the terrain
database design. The implemented terrain database is
subjected to user review to identify any possible oversights
or implementation anamolies.
Another major design issues associated with this complex
system is the control of intelligent agents. In CCTT, they
are called semi-autonomous forces or SAF. To add to the
realism of a large virtual battlefield, semi-automated
friendly and enemy forces operate in the same scenario as
do trainees in the simulator modules and at operations
center workstations. The SAF can be either friendly or
enemy units. When observed in the virtual environment,
manned forces and SAF must appear and behave similarly.
The design issues here is a classic automation problem,
where to interject the human into the loop.
The operators of the SAF will be highly trained tactical
experts who use the system daily. Ideally, the design will
allow operators to control a large number of individual
vehicles. These forces cannot operate autonomously, some
human intervention is required. The IDT analyzed the real
world execution of military tactics and determined which
decision points can be automated and which must be
controlled by a human operator [2]. The problem is
complicated by the design issue of how the UCI can present
all of the required information, while providing easy control
and minimizing the operator workload. The SAF
workstation is a MOTIF interface running on a high fidelity
X-Station 19 inch monitor. The interface includes a graphic
depiction of unit organizations, task execution matrices, and
report windows. The screen also shows a topographical
disply of the virtual battlefield, the units in the exercise,
follows their movement, their status and their firing
actitivies.
The users of this class of interfaces will be infrequent users,
available for only a short orientation on the system. These
users are members of the command and control and support
elements of the training audience, e.g., fire support, combat
engineering, resupply, and repair specialists. Normally,
they do their planning using acetate map overlays, dispense
orders orally and track status by monitoring radio
communications. The UCI for their workstations must
allow them to perform as they would normally, but while
actually commanding synthetic units in the virtual
environment. They must talk to their units via a computer
interface which is unnatural. Additionally, these users will
be evaluated on how well they directed their SAF elements.
To achieve user acceptance we have established close and
continuous communications with the user community, some
techniques used for doing that were discussed earlier. We
also established a series of user evaluations of the system.
These evaluations are conducted in the form of exercises in
which users are placed into training scenarios and subjected
to data collection methods designed to assess the
effectiveness of the system. To review, CCTT components
are being developed and integrated into an operational
system using seven incremental builds each subject to a user
evaluation. Each incremental build delivers a partially
functional system, and, of course, the overall functionality
increases with each build.
User subjects who are not familiar with the system are
provided by the Army. The members of the User
Optimzation Team members are not used because of their
familiarity with the CCTT design. The subjects are brought
to the Integrated Development Facility where, after
receiving an orientation and completing a biographical
questionnaire, they are put on the system and required to
perform prototypical training scenarios designed to teach
specific collective or team tasks. During these scenarios
data is collected by human factors engineers who observe
the training. After each scenario is completed,
questionnaires are administered to solicit specific target
information about the system. At the conclusion of all
scenarios a structured interview at the team level is
conducted to establish a consensus opinion when necessary.
As with many typical large-scale system development
programs, most CCTT hardware and software engineers
have very little understanding of the military training
environment for which they are building the system.
Conversely, the typical Army user has very little
understanding of the complex development environment in
which his system is being developed. To help bridge this
gap, CE teams conduct Display and Control Conferences
early in the design process. These conferences require athe
participation of a prototypical user for a particular
component of the CCTT system. For instance, an M1A1
Tank Commander attended the M1A1 Conference and
experienced SAF operators supported the SAF workstation
Conference.
At these conferences, the operator tasks are compared to the
CCTT preliminary design using early mockups and
prototypes. The notable aspect of these conferences is the
user involvement at an early phase in the design process.
With very simple prototyping tools and materials, designers
are able to convey the hardware and software design
concepts to users and conduct beneficial discussions. The
results of these conferences are then incorporated into the
preliminary hardware and software design. Both users and
designers came away with a better understanding of how the
other group does business
Our data collection approach during the user exercises has
been to analyze all feedback to determine specific user
comments. User comments are then subjected to a second
level analysis to determine which of them address specific
technical issues. The set of technical issues are further
categorized as to those which are software or hardware
deficiencies in the design or implementation versus those
which either identify system capabilities scheduled for a
later delivery or user requirements that are outside of the
scope of the current design. User evaluations for the first
three builds of CCTT resulted in 244 user comments, 156
technical issues, and 96 development problems that need to
be addressed prior to delivery for formal testing.
Developing any complex computational system is a
challenge, one which is based on a virtual environment and
has the complexity CCTT is especially challenging. We
have instituted an integrated approach that uses the best HCI
practices. In making that effort, the following observatons
are relevant to both CCTT and similar virtual training
environments as well as any large-scale complex system
with a significant HCI requirement.
CCTT is halfway though a 5 year development effort. It is at
an appropriate juncture to report on our efforts to date to
apply the appropriate usability engineering [8] and user-
centered development techniques. We have learned many
lessons of value to the HCI community. A User
Optimization Team approach is important for a complex
system development. It is crucial that users be brought into
the project early, their value is just as critical during systems
analysis as during prototype evaluations or testing.
The specific reviews of UCI analysis, design documents and
prototypes need to be clearly defined in terms of their
objectives, what will be provided, and who needs to review
it. It is important to get the user community to support the
efforts so they understand what it is they will see and do,
and to insure errors and ommisions are caught early.
Using operational scenarios is key to successful user
evaluations of an incrementally developed system.
Scheduled user evaluations of not just the interfaces, but the
entire system to insure it works as advertised are critical.
The requirement on developers to be prepared for these
evaluations motivates them to complete promised
functionality on time. It is important that a perspective be
maintained that user exercises are for the engineers doing
the development; their primary purpose is not for
management to audit progress nor for customers to
prematurely evaluate the system. Lastly, with a large
complex system, the number of user subjects can grow
quickly. It is important to determine if feedback from a user
is guidance an engineer might consider or rather a user
approved position that must be accommodated.
CCTT is the first of an entire class of networked systems in
which users interact within a virtual environment. The
military envisions using these environments to train their
organizations and perform analysis of future concepts and
systems. The attraction of virtual reality is not just the
realistic immersion of individual users but the promise of
interconnected users sharing computer-generated worlds for
a host of functions ranging from education to entertainment.
This class of system will challenge the HCI community with
new and unique design issues. This paper has presented one
approach to addressing some of those issues, hopefully it
can serve as a starting point for others confronting these
challenges in the future.
Abstract
An integrated development team comprised of inductry
engineers, government engineers, and user community
representatives is developing a large-scale complex
networked virtual environmnet for the United States Army.
The effort is organized into concurrent engineering teams
responsible for each system component. Prototypical users
who are formally called a User Optimization Team are an
integral part of the development effort. The system under
development is the Close Combat Tactical Trainer (CCTT).
It is comprised of a network of simulators and workstations
which interface with a virtual environment representing real
world terrain. The nature of these systems requires user
involvement in all phases of systems engineering, software
development, and testing. The development organization
and the usability engineering approaches used are mosaics
of engineering skills, knowlege and HCI techniques.
Keywords:
User-Centered Development, User
Evaluations, User Optimization Team, Concurrent
Engineering, Integrated Development, Spiral System
Development
Introduction
Virtual environments or virtual reality-based systems are
starting to migrate from research laboratories into real world
applications [5]. The ultimate goal for virtual reality is to
provide a common synthetic world in which two or more
users can interact. The Close Combat Tactical Trainer
(CCTT) is a full scale development of such a system. It is a
precursor to an entire family of applications based on
similar technology [1] that will have to deal with a complex,
but common, set of human-computer interaction (HCI)
issues. Such systems could be a primary communications
media of the future, will provide much of our entertainment,
and will be useful for training people to
THE CHALLENGE
DEVELOPMENT APPROACH
MAJOR DESIGN ISSUES
Simulator Fidelity
Virtual Environment Design
Controlling Autonomous Software Agents
Training Effectiveness for Workstation Operators
The challenge of insuring effective transfer of learning from
the virtual to the real environment is related to the above
issue of controlling intelligent agents. Some workstations,
those located in the Operations Center component of CCTT,
must control these semi-automated forces while at the same
time allowing the operator to perform tasks required of him
in a battlefield environment. In other words, we need an
interface that allows control of agents by the operator, yet is
not so highly automated that it impedes transfer of training.
UCI To Support Group Interactions
The users of the workstations may be organized to operate
in situations as a group of individuals assigned to
accomplish one function or as one individual assigned to
perform that function. Therefore, we need interfaces that
support single as well as group interaction with the
computer system. In a group situation, several individuals
are performing tasks manually, they are located near the
workstation. In other situations, one individual could be
operating alone. The issue we are addressing is how to
design a single interface to accommodate both situations.
Balancing Commonality and Usability
A UCI style guide was written early in the project to
influence consistency and commonality among the different
concurrent engineering teams. Trading off commonality
and usability becomes an issue in a system this complex
because there is a range of users -- expert to novice.
Different users will use the same information for different
reasons. Should the same information be presented to
different users in the manner most useful to them? For
example, a contractor operating the system for a training
exercise is interested in summary information related to fuel
levels for vehicles. A logistics officer, however, is
interested in fuel levels in terms of precise gallons. For
consistency sake, we have to consider how this type of
information be should presented.
User Acceptablity
Systems based on synthetic environments whose purpose is
to train specific skills or knowledge must be demonstrated
to have user acceptability for their intended purpose. Their
success or failure cannot simply be determined by the
quality of the virtual environment, the usability of
interfaces, or the efficiency of the software. These types of
systems must, as a whole, achieve acceptability with the
target user community and they must effectively train or
teach their users the target skills. In the latter case transfer
of learning to situations calling for the skills to be
performed in a real environment is the only true measure of
success.
PROCESS FOLLOWED AND RESULTS
The CCTT development effort has delivered and evaluated
four of the seven incremental builds. A series of Design
and Control Conferences allow users to review the design
approach earlier in the development process. User
Exercises at the conclusion of each build are a development
team controlled approach to evaluating progress toward
achieving the porject goals.
CONCLUSIONS
