"It's the Computer's Fault" -- Reasoning About Computers as Moral Agents

Keywords:

Introduction

METHODS

Note:
A considerable effort was made to interview equal numbers of females and males; however, a low enrollment of female computer science majors made this goal unfeasible.

The interview contained questions in three general areas: (1) Students' views of computer agency (the capability to make decisions and the capability to have intentions). (2) Students' assessments of computer system characteristics and limitations. And (3) students' judgments of moral responsibility for two scenarios that involved delegation of decision-making to a complex computer system. One scenario involved a computer system that administers medical radiation treatment, and due to a computer error over-radiates a cancer patient. The second scenario involved a computer system that evaluates the employability of job seekers, and due to a computer error rejects a qualified worker. For each scenario, three conditions were investigated: a fully automated computer system that entails no human intervention; a token human intervention in which a person with little authority and status in the organizational hierarchy and little content area expertise operates the computer system (e.g., a hospital orderly in the radiation treatment scenario); and a non-token human intervention in which a person with authority and status in the organizational hierarchy and content area expertise oversees the use of the computer system (e.g., the attending physician in the radiation treatment scenario).

A coding manual was developed from half of the interviews and then applied to the remaining half of the data. To insure reliability of the coding scheme, an independent scorer trained in the coding manual recoded 28% of the data. Intercoder reliability for evaluations was 96%, for content responses 97%, and for justifications 74%.

Non-parametric statistics were used to analyze the categorical data. The McNemar statistic was used to determine a change in students' evaluations across measures (e.g., evaluation of blame across conditions). The amount of blame students' assigned to each potential agent was treated as score data. Then matched-pair t-tests were used to determine differences in students' assignments of blame across agents and conditions.

RESULTS

Computers as Agents

Students' reasons for their assessments were also obtained. In justifying their positive or negative assessment of computer decision-making, virtually all students (95%) appealed to computers as deterministic systems that make use of rule-based or algorithmic processes, or lack free will. For example, in support of computer decision-making one student said, "[the computer is] deciding based on a clear strict algorithm...it's a decision but not an open-ended one." In contrast but also drawing on the idea of computers as deterministic systems, to buttress a negative assessment another student said, "the decisions that the computer makes are decisions that somebody else has made before and programmed into the computer....it can analyze its input and take various actions depending on what the nature of the input is, but somebody has already told it how to proceed in the case of various inputs." Thus, students shared a view of computers as deterministic systems, but differed in their assessments as to whether or not deterministic activity constitutes genuine decision-making.

Students drew on a largely different set of reasons to support their assessments of computer intentions. Of the students who judged computers to lack intentions, 36% appealed to deterministic systems, 14% to emotions, 7% to consciousness, 7% to the soul, and 36% provided unelaborated responses. In many of these cases students referred to the absence of qualities in computer systems such as a lack of consciousness (e.g., "The program is not actually knowing... it's like a level of consciousness...it's just a computer that executes these lines of code...so there's no intention on the part of the program."). In contrast, students who judged computers to have intentions encountered difficulty explicating their reasons. Although probed to the same degree as students who did not attribute intentions to computers, all of these students (100%) provided vague, unelaborated justifications that often did little more than reassert their assessment.

While the above findings overall provide a positive portrayal of computers as agents, students also judged computers to be different than humans along similar dimensions. Of the students who judged computers to have decision-making capabilities, 100% judged computer decision-making to be different from human decision- making. Similarly, of the students who judged computers to have intentions, 77% judged computer intentions to be different from human intentions (Z=2.30, p<.05).

Responsibility for Computer-Error

Roughly one-fifth of the students (on average 21%) consistently blamed the computer system itself for the computer-based error. No significant differences were found across the three conditions and two scenarios for students' evaluations of blame and the amount of blame. However, the amount of blame finding should be understood with caution as only those students (n<=6) who blamed the computer were assessed for the amount of blame.

A central concern of this study is how students understand computers to be accountable, if at all, for computer error. Thus, it is useful to examine students' reasons for blaming or not blaming computers in relation to their reasons for blaming or not blaming people (the computer system designer, the computer system's human operator, and the organization's administrators). Averaging across conditions and scenarios, virtually all of students' justifications for blaming the computer (96%) referred to the computer's participation in the sequence of events that led to harm. In contrast, the large majority of students' justifications for blaming people (80%) referred to failing to meet some commonly expected reasonable level of performance (e.g., negligence). When students did not assign blame, differences were also found among the justifications students used for computers and for people. Again, averaging across conditions and scenarios, virtually all of students' justifications for not blaming the computer (97%) referred to qualities of computers that diminish its agency and thus undermine computers as being the sort of thing that can be blamed. Notably, the appeal to diminished agency was used exclusively in reference to computers. In contrast, students' justifications for not blaming people primarily referred to adequately meeting commonly expected levels of performance (55%) and deferring to an authority perhaps due to habit, lack of autonomy, or the authority's greater power or knowledge (41%).

DISCUSSION

Acknowledgments

2. Walker, J. H., Sproull, L. and Subramani, R. Using a Human Face in an Interface, in Proc. CHI '94 Human Factors in Computing Systems (Boston, April 24-28, 1994), ACM Press, 85-91.