Trust in Automation

developing Updated: 2026-06-02
trust automation autonomy human-automation-interaction appropriate-reliance calibration reliability safety maritime

Trust in Automation

Status: developing
Last updated: 2026-06-02
Sources: Hfes.46.1.50_30392.Pdf, S13437 025 00401 9.Pdf
Tags: [trust, automation, autonomy, human-automation-interaction, appropriate-reliance, calibration, reliability, safety, maritime]

Summary

Trust in automation is the attitude that an agent will help achieve an individual's goals in a situation characterised by uncertainty and vulnerability. An adequate level of trust is necessary for the safe and appropriate use of automated systems: operators should avoid disusing safe systems and avoid misusing fallible ones. Lee & See (2004) frame the design goal as appropriate reliance — trust calibrated to the automation's true capabilities — and identify performance, process, and purpose as the bases on which operators form that trust. Trust depends not only on the system's trustworthiness but also on individual, organisational, cultural, and environmental context. Empirical research with maritime bridge officers (Aalberg & Kongsvik, 2026) identifies twelve factors that practitioners consider necessary for safe automation, spanning technological reliability, human control, competence, and governance.

Body

Context

This article rests on two primary sources. Lee & See (2004) provide the theoretical foundation: an integrative review that links organizational, sociological, interpersonal, psychological, and neurological perspectives on trust to human–automation interaction, and defines the design goal of appropriate reliance. Aalberg & Kongsvik (2026) supply empirical grounding, investigating what factors bridge officers consider necessary for the safe introduction of maritime automation and autonomy through a mixed-method design that combines qualitative coding and structural topic modelling on 1,009 free-text responses from a Norwegian Maritime Authority survey. The article sits at the intersection of Supervisory Control Of Automation, Automation Autonomy And Ai, and Human Robot Interaction, formalising the trust-calibration problem raised in those articles. It connects to Human Error And Reliability through concerns about system failure and redundancy, to Situation Awareness through concerns about monitoring and deskilling, and to the Ironies Of Automation critique of over-trust and complacency.

Key Points

Trust, reliance, misuse and disuse (Lee & See, 2004). Lee & See define trust as "the attitude that an agent will help achieve an individual's goals in a situation characterized by uncertainty and vulnerability," where the agent may be automation or another person (PDF p. 2, orig. p. 51). Because people respond to technology socially, trust guides reliance — particularly when complexity and unanticipated situations make a complete understanding of the automation impractical (PDF p. 1, orig. p. 50). Flawed partnerships between people and automation appear as misuse (over-reliance, when people fail critical assumptions and rely on automation inappropriately) and disuse (under-reliance, when people reject capable automation), following Parasuraman & Riley (1997) (PDF p. 1, orig. p. 50). Trust guides but does not fully determine reliance.

Calibration, resolution, and specificity (Lee & See, 2004). Appropriate reliance depends on how well trust matches the automation's true capabilities. Calibration is the correspondence between a person's trust and the automation's actual capabilities: overtrust is poor calibration in which trust exceeds capability (leading to misuse), and distrust is trust that falls short of capability (leading to disuse). Resolution is how precisely trust differentiates between levels of automation capability, and specificity is the degree to which trust is associated with a particular component or time. The design goal is well-calibrated, high-resolution, appropriately specific trust (PDF p. 6, orig. p. 55).

Bases of trust: performance, process, purpose (Lee & See, 2004). Operators form trust on three grounds. Performance refers to what the automation does — its current and historical reliability, predictability, and ability to achieve the operator's goals. Process refers to how the automation operates — whether its algorithms are appropriate and understandable for the situation. Purpose refers to why the automation was developed — the degree to which it is used within the designer's intent (PDF p. 10, orig. p. 59). These map onto the broader ability/integrity/benevolence dimensions of interpersonal trust (Mayer et al., 1995, cited in Lee & See, 2004), and trust in automation can be conveyed through analytic, analogical, and affective processes (PDF p. 1, orig. p. 50).

Definition and model of trust (Aalberg & Kongsvik, 2026). Drawing on this foundation, Aalberg & Kongsvik adopt the same definition of trust (PDF p. 4, orig. p. 4, citing Lee & See, 2004) and treat calibrated trust as the correlation between subjective trust and the system's actual performance, so that operators neither over-trust fallible systems nor under-trust capable ones. According to McKnight et al. (2011, cited in Aalberg & Kongsvik, 2026), a system's trustworthiness comprises three dimensions: reliability (working consistently), functionality (capability), and helpfulness (adequacy and responsiveness) (PDF p. 4, orig. p. 4).

Trust develops through a feedback loop: information assimilation and belief formation lead to trust evolution and intention formation, which may lead to reliance actions where automation is used. Based on information about the automation and its display, trust is further negotiated. This process occurs within an individual, organisational, cultural, and environmental context (PDF p. 5, orig. p. 5).

Empirical findings: twelve integrated topics. The study identifies twelve topics that bridge officers consider necessary for safe automation, organised into factors they are worried about and factors they believe are needed (PDF pp. 20–23, orig. pp. 20–23):

Technological trustworthiness factors:
- Automation reliability — the overarching main topic; systems must be reliable, and fallibility necessitates human monitoring
- Redundancy — adequate backup systems and reactions when something goes wrong
- Automation capabilities — ability to handle heterogeneous external conditions such as weather and traffic

Functionality and helpfulness factors:
- Human override — opportunities for human control and manual takeover when needed
- Emergency handling — worry about ability to handle crises such as fires, collisions, and blackouts
- Key tasks — concern about consequences for maintenance, evacuation, and other functions
- System training and user-friendliness — need for user-friendly interfaces and operator training

Contextual factors:
- The human factor — strong call for continued human presence on board
- Competence — worry about deskilling and loss of navigational knowledge
- Crew on board — demand that safety staffing not be reduced
- Governance of implementation — need for controlled, gradual implementation with external oversight

Mapping topics to trust dimensions. The topics map onto the trust model as follows. Reliability, redundancy, and automation capabilities relate to the trustworthiness dimension of reliability. Human override, emergency handling, and key tasks (regarding capabilities) relate to functionality. System training and user-friendliness relates to helpfulness. The remaining topics — human factor, competence, crew on board, and governance — form the individual, organisational, and processual context in which trust develops (PDF p. 24, orig. p. 24).

Implications for practice. The study offers several recommendations. First, communicate the sociotechnical design of the autonomous system to end-users, including the division of tasks, roles, and plans for maintaining competence. Second, shift rhetoric from "removing people" to empowering human control and oversight, consistent with supervised autonomy approaches. Third, promote a climate for sharing performance data of autonomous systems, so that reliability data and failure reports reach seafarers and help adjust preconceptions (PDF pp. 25–26, orig. pp. 25–26).

Conclusion

The two sources converge. Lee & See (2004) conclude that trust influences but does not determine reliance, and matters most under uncertainty and complexity where an exhaustive evaluation of options is impractical; the design objective is therefore to support appropriate, well-calibrated trust through the automation's performance, process, and purpose rather than to maximise trust. Aalberg & Kongsvik (2026) reach a compatible practitioner-grounded conclusion: trust depends not only on technological aspects but on a functioning sociotechnical system, with their twelve topics forming a layered structure — automation reliability and human control at the core, surrounded by competence, organisation, and governance of implementation. Together they frame trust calibration as a problem spanning the technology, the operator, and the organisation, and point toward design, evaluation, and training criteria for trustworthy and trusted autonomous systems.

References

Aalberg, A.L. & Kongsvik, T. (2026) 'How can maritime automation and autonomy be safely implemented? A mixed-method topic model', WMU Journal of Maritime Affairs. doi: 10.1007/s13437-025-00401-9. aalberg2026trust

Lee, J.D. & See, K.A. (2004) 'Trust in Automation: Designing for Appropriate Reliance', Human Factors, 46(1), pp. 50–80. doi: 10.1518/hfes.46.1.50.30392. lee2004trust

Mayer, R.C., Davis, J.H. & Schoorman, F.D. (1995) 'An Integrative Model of Organizational Trust', Academy of Management Review, 20(3), pp. 709–734. doi: 10.5465/amr.1995.9508080335. Verified via RAW + web (cited in Lee & See, 2004; not held in RAW).

McKnight, D.H., Carter, M., Thatcher, J.B. & Clay, P.F. (2011) 'Trust in a specific technology: An investigation of its components and measures', ACM Transactions on Management Information Systems, 2(2), pp. 1–25. To be validated.

Parasuraman, R. & Riley, V. (1997) 'Humans and Automation: Use, Misuse, Disuse, Abuse', Human Factors, 39(2), pp. 230–253. doi: 10.1518/001872097778543886. Verified via RAW + web (cited in Lee & See, 2004; not held in RAW).

Open Questions

  • How should autonomous systems communicate their reliability and limitations to calibrate operator trust appropriately?
  • What training methods are effective for maintaining manual skills alongside increased automation (countering deskilling)?
  • How do the trust factors identified in maritime contexts translate to other safety-critical domains such as aviation, healthcare, or road transport?
  • What governance structures best support gradual, controlled implementation of autonomous systems?