The Out-of-the-Loop Performance Problem¶

Status: established
Last updated: 2026-06-15
Sources: Endsley Kiris 1995 Out Of The Loop Level Of Control
Tags: [out-of-the-loop, situational-awareness, supervisory-control, levels-of-automation, automation, passive-monitoring, vigilance, complacency, skill-decay, human-automation-teaming, remote-operations]

Summary¶

The out-of-the-loop (OOTL) performance problem is the diminished ability of operators of automated systems to detect failures and resume manual control, relative to operators who perform the same task by hand. Endsley and Kiris (1995) argue that a loss of situation awareness (SA) underlies much of this decrement, produced by three mechanisms: vigilance and complacency under a monitoring role, a shift from active to passive information processing, and reduced or altered feedback. A between-subjects experiment automating a navigation decision task across five levels of control showed that decision time after an automation breakdown rose with the level of automation experienced beforehand, that Level 2 (comprehension) SA was lowest under full automation while Level 1 (perception) SA was unaffected, and that workload was not reduced by the automation. The authors identify the active-to-passive shift as the most likely driver of the SA loss and recommend intermediate levels of automation, which keep the operator in the decision loop, as the means to ameliorate the problem. The concept is the conceptual anchor for the loop-removal theme that recurs across this knowledge base.

Body¶

Context¶

Endsley and Kiris (1995) combine a literature review with a controlled experiment to test why operators struggle to take over when automation fails, and to test whether the level of operator control moderates that struggle. The lens is situation awareness — defined as the perception of elements in the environment (Level 1), the comprehension of their meaning (Level 2), and the projection of their future status (Level 3) (PDF pp. 2–3, orig. pp. 382–383). Within this knowledge base the article is the foundational statement of the out-of-the-loop problem that later articles invoke: it is the parent concept behind the ATC monitoring study in Active Control Vs Passive Monitoring Atc, the maritime transparency work in Human In The Loop Automation Transparency, and the complacency mechanisms detailed in Automation Complacency And Bias. Its five-level control scheme is the direct precursor to the ten-level taxonomy in Levels Of Automation Taxonomy.

Key Points¶

The problem and its two roots. Operators working with automation have a reduced ability both to detect system errors and to perform manually after a failure, compared with operators who do the task by hand. Endsley and Kiris link this to two issues: loss of manual skills, and loss of awareness of the state and processes of the system (PDF p. 1, orig. p. 381). They make the loss of SA the central thesis — skill loss alone cannot explain the data, because operators in their study retained decision quality yet were still slower and less aware after the breakdown.

Three mechanisms of SA loss. The review groups the evidence into three causes (PDF pp. 2–3, orig. pp. 382–383). First, vigilance and complacency: humans are poor passive monitors and over-trust highly reliable automation, so they are slow to notice failures — complacency effects appear chiefly in multitask settings (Parasuraman, Molloy & Singh, 1993). Second, a move from active processor to passive recipient of information, since passive processing of information is inferior to active processing and weakens the dynamic updating of system state in working memory. Third, a loss of or change in feedback: when automation removes raw cues or processed displays mask the true state, "people are indeed out of the loop" (Norman, 1989) (PDF pp. 4–5, orig. pp. 384–385). The authors note automation does not always hurt SA — it can raise it by integrating information or relieving excessive workload (Billings, 1991) — but caution that workload reduction does not reliably follow, echoing Bainbridge's (1983) observation that automation helps least when workload is highest.

Five levels of control. Automation is not all-or-none. The study operationalises five levels of operator control over a cognitive decision task (PDF p. 5, orig. p. 385):

The experiment. Eighty undergraduates (sixteen per condition) performed a simulated automobile-navigation decision task on a Macintosh, choosing among three routes across six scenarios; a simulated expert system supplied option probabilities at the four assisted levels (PDF pp. 5–7, orig. pp. 385–387). The expert system "broke down" after the fourth scenario, placing every subject in a purely manual condition for scenarios five and six, so manual performance after the failure could be read as a function of the automation level experienced before it (PDF p. 7, orig. p. 387).

Findings. Decision time after the breakdown rose with prior automation level, though a Tukey test found only the manual and full-automation conditions significantly different (p < 0.05) (PDF p. 8, orig. p. 388).

Level 2 SA (comprehension) differed significantly across conditions, F(4,75) = 2.54, p < 0.05, and was lowest under full automation; Level 1 SA (perception) did not differ — subjects were monitoring the system but had not built the higher-level understanding needed to act (PDF pp. 9–10, orig. pp. 389–390).

Two results sharpen the interpretation. Workload (NASA-TLX) was not reduced by the automation, even under full automation (PDF p. 10, orig. p. 390). And, contrary to hypothesis, confidence was higher under the automated conditions, with no objective loss of decision quality — subjects were equally likely to pick the optimal route before and after the breakdown (PDF pp. 9–10, orig. pp. 389–390). Because skill and decision quality were preserved while SA fell, the authors conclude the decrement cannot be attributed to skill loss alone.

Conclusion¶

Endsley and Kiris (1995) conclude that full automation produced more out-of-the-loop problems than partial automation, which in turn produced more than manual control, and that the loss of SA — specifically of Level 2 comprehension — is the mechanism, driven most plausibly by the shift from active to passive processing rather than by vigilance lapses or feedback loss in this single-task design (PDF pp. 11–12, orig. pp. 391–392). The practical recommendation is to keep the operator in the decision loop: implementing automation while maintaining a high level of operator control gives "definite benefits in minimizing the out-of-the-loop performance problem" relative to full automation (PDF p. 13, orig. p. 393). This is the empirical seed of the intermediate-automation argument developed for dynamic control in Levels Of Automation Taxonomy, and it frames the recurring design question for any remote or supervisory control centre — how much authority to leave with the human so that the human can still step back in.

Related¶

• Levels Of Automation Taxonomy — Kaber & Endsley's ten-level LOA taxonomy and the dynamic-control evidence for intermediate automation; extends this paper's five-level scheme
• Active Control Vs Passive Monitoring Atc — ATC counterpart: passive monitoring reproduces the OOTL decrement in conflict detection (Metzger & Parasuraman)
• Automation Complacency And Bias — the vigilance/complacency mechanism named here, developed into an attentional model (Parasuraman & Manzey)
• Use Misuse Disuse Abuse Of Automation — over-trust and misuse as the behavioural face of complacency
• Human In The Loop Automation Transparency — the maritime out-of-the-loop problem and interface responses to it
• Multi Ship Remote Operations Workload Sa — cross-domain workload, attention, and SA in one-to-many supervision

References¶

Bainbridge, L. (1983) 'Ironies of automation', Automatica, 19(6), pp. 775–779. To be validated.

Billings, C. E. (1991) Human-centered aircraft automation: a concept and guidelines (NASA Tech. Memorandum 103885). Moffett Field, CA: NASA Ames Research Center. To be validated.

Endsley, M. R. and Kiris, E. O. (1995) 'The out-of-the-loop performance problem and level of control in automation', Human Factors, 37(2), pp. 381–394. doi: 10.1518/001872095779064555. endsley1995loop

Norman, D. A. (1989) The problem of automation: inappropriate feedback and interaction, not overautomation (ICS Report 8904). La Jolla, CA: University of California, San Diego, Institute for Cognitive Science. To be validated.

Parasuraman, R. and Manzey, D. H. (2010) 'Complacency and bias in human use of automation: an attentional integration', Human Factors, 52(3), pp. 381–410. doi: 10.1177/0018720810376055. parasuraman2010complacency

Parasuraman, R., Molloy, R. and Singh, I. L. (1993) 'Performance consequences of automation-induced complacency', International Journal of Aviation Psychology, 3(1), pp. 1–23. To be validated.

Open Questions¶

• The active-to-passive shift is identified as the primary mechanism by elimination (feedback was held constant, vigilance was unlikely in a 10-minute single task), not by direct manipulation. The authors note this factor "has not received much explicit consideration" and needs dedicated study.
• The task was a discrete, high-level cognitive decision. How the findings transfer to continuous psychomotor control, where proprioceptive cues may carry SA-relevant information, is left open.
• Confidence rose under automation while SA fell — a miscalibration between felt and actual competence. Whether this over-confidence compounds the OOTL deficit in the field is not tested here.