Abstract
Most studies in multimedia have not been controlled and
have focused on student satisfaction rather than examining
what students have learned. This poster will describe
results from current research designed to evaluate the
efficacy of multimedia-based training in producing
increased learning and retention of factual knowledge and
skills.
Keywords:
education, multimedia, evaluation
Introduction
Computers can be used to provide students with an
opportunity to actively learn material and perform tasks in
the same way as professionals in the field. Currently at
NMSU, only one method class is available for upper-
division students and there are no labs connected with
upper-division classes. Some courses have a special need
for labs (e.g. teaching students operant conditioning using
live rats). Availability of inexpensive personal computers
has led to the possibility of developing computer software
dedicated to training and tutoring students [2]. Although
the initial cost may be high, it is more cost efficient for
long-term laboratory use to develop a computer lab. For
example, the estimated cost for purchasing and maintaining
laboratory rats is $2,000 per student [1]; however, using a
computer virtual rat has an initial cost of less than $2000
for a single computer and software and more than one
student can use the virtual rat software in a computer lab
over the course of a semester. In addition, computer-
generated animation and multimedia capabilities of
computers provide an opportunity to expand the ways in
which students can learn [4]. Thus, a full computer
laboratory to augment psychology courses could be
developed.
Most studies in multimedia laboratories have not been
controlled and have focused on student satisfaction rather
than on examining what students have learned (for an
exception [4], a multimedia evaluation in the laboratory but
not in the classroom). Thus, the goal of the current
experiment was to evaluate the efficacy of a multimedia
laboratory in producing increased learning and retention of
both factual knowledge and skills in experimental methods.
Since the multimedia laboratory was supplemental to
students' regular coursework, we also predicted that
students would have an increased motivation for learning.
METHOD
Design. A within-subjects design was used. For the first
half of the semester, half the subjects used the operant
conditioning software and half listened to a recitation-style
lecture on the same topic. In the second half of the
semester, the students who heard the recitation in the first
half used software on eyewitness memory and those who
used operant conditioning software listened to a recitation
on eyewitness memory.
Subjects. Sixty four undergraduate students at NMSU
taking an upper-division course on learning were used.
Five students' results did not complete the experiment.
Materials. Two software packages were used in this study.
The first software package was Sniffy the rat, a simulated
laboratory module designed to train subjects on the
principles of operant conditioning [1]. We developed our
own accompanying background material and exercises for
students to do. The second software package was designed
to train students on experimental methodology using an
experiment on eyewitness memory as its basis [3]. For the
recitation-style lectures, materials (lecture notes and
overheads) covered the same concepts as those in the
software programs.
Procedure. The same basic procedure was used for the
operant conditioning and eyewitness memory training.
Before the labs were started, subjects were given a pre-test.
Then, over a four week period, subjects were brought into
the lab in groups for training (2 students per computer; 8
students listening to the recitation). After all students had
completed the lab, a post-test on operant conditioning was
given. In addition, subjects were given a questionnaire and
cognitive ability tests.
RESULTS
Operant conditioning results. Subjects in both conditions
improved from pre-test to post-test (t(57)=6.4, p<01). A
mixed model multiple regression was performed with test
(pre-test and post-test) as the within-subjects factor and
condition (computer or recitation) as the between-subjects
factor. A trend in the data indicates that for the recitation,
the lecture helped subjects scoring low on the pre-test more
than those scoring high on the pre-test; but, for those using
the computer, multimedia helped all subjects equally (F(1,
57)= 3.22, p<.08) This result may be due to the large
number of conceptual questions on the pre- and post-tests.
From the additional post-test questions, when subjects
listed behavior displayed by two babies on a video, there
was a trend in the data indicating that subjects who used the
computer listed more behaviors than those who listened to
the recitation (mean computer = 15.1, mean recitation =
13.7, t(27)=2.0, p<.06).
Eyewitness memory results. Subjects in both conditions
improved from pre-test to post-test (t(58)=7.8, p<.01). A
mixed model regression analysis was performed with test
(pre-test and post-test) as the within-subjects factor and
condition (computer or recitation) as the between-subjects
factor. For those using the computer, subjects who scored
lower on the pre-test improved more than those who scored
high on the pre-test; but, for those in recitation, all subjects
were helped equally well by the lecture (pre-test by
condition interaction F(1, 58)= 4.08, p<.05).
DISCUSSION
Measuring the effects of change in knowledge rather than
just motivation or preference is better for evaluating
educational efficacy. This study evaluated two multimedia-
base training systems using a pre-test, training, post-test
procedure. The operant conditioning and eyewitness
memory training produced very different results. In the
operant training, the recitation provided substantial benefit
to students who scored low on the pre-test. In contrast, in
the eyewitness memory training, the multimedia software
provided the most benefit to students who scored low on
the pre-test. In other words, both conditions suggest that
replacing lectures with multimedia lab may benefit certain
subjects but not others. The differences in results between
the operant conditioning and eyewitness memory training
may be caused by: 1) Software differences such as amount
of feedback for student actions, degree of interactively, and
goal-directedness of instruction, 2) Test (pre-test/post-test)
differences such as number of procedural versus number of
conceptual questions, and 3) Individual differences, such as
intelligence or spatial abilities. Preliminary analyses of the
cognitive ability tests and intelligence scores suggests that
these individual difference factors may not contribute to the
present findings. Thus, future research will focus on
differences in software and test construction.
The present evaluation procedure, though complex, has
several advantages over more typical evaluations of
multimedia. That is, comparing multimedia to a lecture
provided a more appropriate baseline than comparison to a
no training condition. Further, the within-subject design
allowed us to observe the effect of different types of
multimedia.
CONCLUSION
The effects of multimedia may be subtle. Therefore,
evaluations of multimedia must be carefully performed in
order to prevent prematurely accepting or rejecting a piece
of software.
References
1. Graham, J., Alloway, T., & Krames, L. Sniffy, the
virtual rat: Simulated operant conditioning. Behavior
Research Methods, Instruments, & Computers, 26(2),
(1994), 134-141.
2. Lee, A.Y. Using tutoring systems to study learning: An
application of HyperCard. Behavior Research Methods,
Instruments, & Computers, 24(2), (1992), 205-212.
3. Loftus, E.F., Miller, D.G., & Burns, H.J. Semantic
integration of verbal information into a visual memory.
Journal of Experimental Psychology: Human Learning
and Memory, 4(1), (1978), 19-31.
4. Mayer, R.E., & Sims, V.K. For whom is a picture worth
a thousand words? Extensions of a dual-coding theory
of multimedia learning. Journal of Educational
Psychology, 86(3), (1994), 389-401.
