Human-Centered Design of Artificial Intelligence

emerging Updated: 2026-05-31
human-centered-ai explainable-ai human-in-the-loop machine-learning human-computer-interaction ai-ethics federated-learning

Human-Centered Design of Artificial Intelligence

Status: emerging
Last updated: 2026-05-31
Sources: 9781119636113.Ch42.Pdf
Tags: [human-centered-ai, explainable-ai, human-in-the-loop, machine-learning, human-computer-interaction, ai-ethics, federated-learning]

Summary

Human-centered design of artificial intelligence (HCD for AI) applies the established human-centered design process from human-computer interaction (HCI) to the development of AI-empowered systems. Margetis et al. (2021) argue that, because human-level autonomous intelligence remains unattainable for the foreseeable future, humans must be kept in the loop of AI systems, and that the ISO human-centered design process supplies a tested framework for doing so. The chapter examines six concepts that operationalise this goal — explainable AI and human-in-the-loop, semantic/cognitive/perceptual computing, visual predictive analytics, interactive machine learning, federated learning, and UX design for AI — and proposes a framework that layers these around the ISO design cycle. It treats ethics as a distinct concern, surveying conceptual frameworks and AI approaches for embedding ethical behaviour while retaining human control. The work positions itself as a first methodological approach to designing AI in a systematically human-centered way.

Body

Context

Margetis et al. (2021) examine how the human-centered design (HCD) process from human-computer interaction can be applied to the development of AI-empowered systems. The chapter is a methodological proposal rather than an empirical study: it argues that the ISO human-centered design cycle supplies a tested framework for keeping humans in the loop of AI, reviews six concepts that operationalise that goal, layers them into a single framework, and treats ethics and human control as a distinct concern. It positions itself as a first systematic methodology for designing AI in a human-centered way. As the first article in this knowledge base, it sets the reference point for the AI/HCI strand here — the human-in-the-loop, explainable-AI, and design-methodology questions that later articles on machine learning, AI ethics, and interaction design will build on.

Key Points

The case for human-centered AI rests on the limits of autonomy. Drawing on McCarthy's account of seven unsolved problems — including representing common-sense knowledge, supporting non-monotonic reasoning, and formalising action — the authors argue that fully autonomous human-level intelligent systems cannot exist in the near future, so the human must be placed in the loop to address these barriers. Human involvement is therefore a structural requirement of current AI, not a usability courtesy. The intellectual lineage runs through cooperative, rather than replacement, models of computing: Licklider's (1960) "man-computer symbiosis" assigned goal-setting and evaluation to humans and routinizable work to machines, and Engelbart's (1962) Intelligent Amplification aimed at augmenting human intellect rather than building autonomous machines. AI and HCI historically competed for resources, with one field prospering as the other declined (Grudin, 2009), AI being the "rationalistic" approach to problem-solving and HCI the "design" approach (Winograd, 2006); the chapter holds that the two must now work together so that pervasive AI genuinely serves human needs (PDF pp. 2–3, orig. pp. 1086–1087).

Human-centered design supplies the organising framework because it is both standardised and flexible. The approach descends from User-Centered System Design, introduced by Norman and Draper (1986) at the intersection of psychology and AI, and was standardised by ISO as ISO 13407:1999 and later ISO 9241-210, latest revised as ISO 9241-210:2019; "human-centered" was preferred over "user-centered" to signal that stakeholders beyond direct users are involved. The current standard sets out six principles — explicit understanding of users, tasks and environments; user involvement throughout; user-centered evaluation; iteration; addressing the whole user experience; and multidisciplinary teams — realised through four iterative activities: understand and specify the context of use, specify user requirements, produce design solutions, and evaluate the design. The authors hold that HCD is not only relevant but imperative for AI and ML systems, including those with limited direct user interaction (PDF pp. 3–4, orig. pp. 1087–1088).

The chapter distinguishes guidelines from frameworks and argues the field needs the latter. It reviews corporate guideline sets — Microsoft's 18 guidelines across four UX phases (Amershi et al., 2019), IBM's Design for AI covering accountability, value alignment, explainability, fairness and user data rights (IBM, 2019), and Google's People + AI Guidebook (Google, 2019) — alongside academic proposals such as Xu's (2019) extended HAI framework and Shneiderman's (2020) two-dimensional model of automation and human control. Using a travel metaphor, the authors state that these efforts mark the destination and the hazards but leave the route undefined, which is the gap their framework targets (PDF pp. 4–5, orig. pp. 1088–1089).

Explainable AI (XAI) is presented as the precondition for trust and the first concept. Drawing on the DARPA XAI programme (Gunning & Aha, 2019), the chapter describes three strategies — deep explanation, interpretable models, and model induction — and a three-part classification of explanatory ML into processing, representation, and explanation production (Gilpin et al., 2018). Specific techniques include proxy models, automatic rule extraction, salience mapping, attention models, disentangled representations, and self-explanatory models, while "informed machine learning" (von Rueden et al., 2019) integrates prior knowledge — including human feedback — into the learning pipeline. An explanation interface is treated as a fundamental component through which systems and humans communicate (PDF pp. 6–7, orig. pp. 1090–1091).

The remaining five concepts each provide a route for human participation. Semantic, cognitive, and perceptual computing pursue human-centric rather than machine-centric computation through iterative cycles of representation, interpretation, and the search for new data, with cognitive computing linking neurobiology, cognitive psychology and AI (Valiant, 1995). Visual predictive analytics combine automated analysis with interactive visualisation to mitigate information overload and make ML pipelines transparent, with knowledge-generation models (Sacha et al., 2014) and ML-pipeline frameworks (Sacha et al., 2016; Lu et al., 2017) inserting the analyst at each step. Interactive machine learning, attributed to Fails and Olsen (2003), directly engages end-users in a model's training loop and is treated together with active learning (Settles, 2009), with a six-activity workflow from feature selection through transfer (Dudley & Kristensson, 2018). Federated learning, introduced by Google (Konečný et al., 2016; McMahan et al., 2016), enables decentralised, privacy-preserving training across everyday devices and is classified into horizontal, vertical, and federated transfer learning (Yang et al., 2019). UX design for AI is identified as the hardest, because AI violates the deterministic, closed assumption of conventional UX; Yang et al. (2020) attribute this difficulty to capability uncertainty and output complexity (PDF pp. 7–13, orig. pp. 1091–1097).

The framework layers these AI concepts around the ISO HCD cycle through three objectives: explainable AI, the active involvement of humans for improving algorithms through training and feedback, and UX design of AI. Visually, HCD sits at the centre, surrounded by expanding circles — explainable AI, which should always be pursued; processes involving ML; and knowledge-reasoning-and-planning processes. AI practitioners need not change their existing practices but must involve UX experts and end-users at the phases each circle identifies, mapping AI activities such as data preparation, feature selection, model training, and model validation onto the parallel HCD activities of user requirements, solution design, and evaluation (PDF pp. 13–15, orig. pp. 1097–1099).

The framework also surfaces methodological consequences for evaluation. Because an AI system is dynamic and continuously evolving, established rules on the number of users required for usability testing (Sauro & Lewis, 2016) become invalid, motivating crowdsourced evaluation. Design must contend with "design for uncertainty" (Ries, 2011), since the interface and interactions of a non-deterministic system cannot be fully specified in advance, and the authors suggest AI itself might assist the design process by deciding how best to present information (PDF pp. 17–18, orig. pp. 1101–1102).

Ethics is treated as extending well beyond transparency. The chapter reviews conceptual frameworks including the IEEE Ethically Aligned Design framework (2019), with its three pillars of universal human values, political self-determination over data, and technical dependability, and the European Commission AI HLEG principles of beneficence, non-maleficence, autonomy, justice, and explicability (European Commission, 2019). It classifies AI-based ethics approaches into ethical-dilemma exploration, individual and collective ethical decision frameworks, and frameworks for ethics in human-AI interaction (Yu et al., 2018), noting reinforcement-learning approaches to ethical learning (Abel et al., 2016; Noothigattu et al., 2019). The authors warn that human values may not transfer cleanly to machines, which lack guilt and empathy, and that values depend on context and on sensor data calibrated by humans and therefore potentially biased (Rossi & Mattei, 2019; Bonnemains et al., 2018). They caution that when humans vote on ethical dilemmas, care is needed to represent the whole population, since participation skews toward upper and middle social classes (Ames et al., 2014) (PDF pp. 16–17, orig. pp. 1100–1101).

Conclusion

Margetis et al. (2021) conclude that, while autonomous human-level intelligence remains unattainable, AI can still be developed in a systematically human-centered way by building on the established ISO design process. Their framework binds explainable AI, human involvement in algorithm improvement, and UX design around that cycle, requiring practitioners to engage UX experts and end-users without abandoning their existing methods. On ethics they hold that a gap persists between abstract values and technical implementation (Hagendorff, 2020), and that closing it depends on multidisciplinary collaboration with human control retained throughout.

References

Abel, D., MacGlashan, J. and Littman, M. L. (2016) 'Reinforcement learning as a framework for ethical decision making', in Workshops at the Thirtieth AAAI Conference on Artificial Intelligence. To be validated.

Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R. and Horvitz, E. (2019) 'Guidelines for human-AI interaction', in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-13. To be validated.

Ames, M. G., Bardzell, J., Bardzell, S., Lindtner, S., Mellis, D. A. and Rosner, D. K. (2014) 'Making cultures: empowerment, participation, and democracy - or not?', CHI '14 Extended Abstracts on Human Factors in Computing Systems. To be validated.

Bonnemains, V., Saurel, C. and Tessier, C. (2018) 'Embedded ethics: some technical and ethical challenges', Ethics and Information Technology, 20(1), pp. 41-58. To be validated.

Dudley, J. J. and Kristensson, P. O. (2018) 'A review of user interface design for interactive machine learning', ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2), pp. 1-37. To be validated.

Engelbart, D. C. (1962) Augmenting human intellect: a conceptual framework. Menlo Park, CA. To be validated.

European Commission (2019) Draft ethics guidelines for trustworthy AI. Available at: https://ec.europa.eu/digital-single-market/en/news/draft-ethics-guidelines-trustworthy-ai. To be validated.

Fails, J. A. and Olsen Jr, D. R. (2003) 'Interactive machine learning', in Proceedings of the 8th International Conference on Intelligent User Interfaces, pp. 39-45. To be validated.

Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M. and Kagal, L. (2018) 'Explaining explanations: an overview of interpretability of machine learning', in 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80-89. IEEE. To be validated.

Google (2019) People and AI Guidebook. Available at: https://pair.withgoogle.com/guidebook/. To be validated.

Grudin, J. (2009) 'AI and HCI: two fields divided by a common focus', AI Magazine, 30(4), p. 48. To be validated.

Gunning, D. and Aha, D. W. (2019) 'DARPA's Explainable Artificial Intelligence program', AI Magazine, 40(2), pp. 44-58. To be validated.

Hagendorff, T. (2020) 'The ethics of AI ethics: an evaluation of guidelines', Minds and Machines, pp. 1-22. To be validated.

IBM (2019) Design for AI. Available at: https://www.ibm.com/design/ai/. To be validated.

Konečný, J., McMahan, H. B., Yu, F. X., Richtárik, P., Suresh, A. T. and Bacon, D. (2016) 'Federated learning: strategies for improving communication efficiency', arXiv preprint arXiv:1610.05492. To be validated.

Licklider, J. C. (1960) 'Man-computer symbiosis', IRE Transactions on Human Factors in Electronics, 1, pp. 4-11. To be validated.

Lu, J., Chen, W., Ma, Y., Ke, J., Li, Z., Zhang, F. and Maciejewski, R. (2017) 'Recent progress and trends in predictive visual analytics', Frontiers of Computer Science, 11(2), pp. 192-207. To be validated.

Margetis, G., Ntoa, S., Antona, M. and Stephanidis, C. (2021) 'Human-Centered Design of Artificial Intelligence', in Salvendy, G. and Karwowski, W. (eds.) Handbook of Human Factors and Ergonomics. 5th edn. Hoboken, NJ: John Wiley & Sons, pp. 1085-1102. doi: 10.1002/9781119636113.ch42. margetis2021humancentered

McMahan, H. B., Moore, E., Ramage, D. and Hampson, S. (2016) 'Communication-efficient learning of deep networks from decentralized data', arXiv preprint arXiv:1602.05629. To be validated.

Norman, D. A. and Draper, S. W. (eds.) (1986) User-centered system design: new perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates. To be validated.

Noothigattu, R., Bouneffouf, D., Mattei, N., Chandra, R., Madan, P., Varshney, K. R., Campbell, M., Singh, M. and Rossi, F. (2019) 'Teaching AI agents ethical values using reinforcement learning and policy orchestration', IBM Journal of Research and Development, 63(4/5), pp. 2:1-2:9. To be validated.

Ries, E. (2011) The lean startup. London: Penguin. To be validated.

Rossi, F. and Mattei, N. (2019) 'Building ethically bounded AI', in Proceedings of the AAAI Conference on Artificial Intelligence, 33, pp. 9785-9789. To be validated.

Sacha, D., Sedlmair, M., Zhang, L., Lee, J. A., Weiskopf, D., North, S. and Keim, D. (2016) 'Human-centered machine learning through interactive visualization', ESANN. To be validated.

Sacha, D., Stoffel, A., Stoffel, F., Kwon, B. C., Ellis, G. and Keim, D. A. (2014) 'Knowledge generation model for visual analytics', IEEE Transactions on Visualization and Computer Graphics, 20(12), pp. 1604-1613. To be validated.

Sauro, J. and Lewis, J. R. (2016) Quantifying the user experience: practical statistics for user research. San Francisco: Morgan Kaufmann. To be validated.

Settles, B. (2009) Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences. To be validated.

Shneiderman, B. (2020) 'Human-centered artificial intelligence: reliable, safe and trustworthy', International Journal of Human-Computer Interaction, pp. 1-10. To be validated.

The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems (2019) Ethically aligned design: a vision for prioritizing human well-being with autonomous and intelligent systems. First edn. IEEE. Available at: https://standards.ieee.org/industry-connections/ec/autonomous-systems.html. To be validated.

Valiant, L. G. (1995) 'Cognitive computation', in Proceedings of IEEE 36th Annual Foundations of Computer Science, pp. 2-3. IEEE. To be validated.

von Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Pfrommer, J., Pick, A., Ramamurthy, R., Walczak, M., Garcke, J., Bauckhage, C. and Schuecker, J. (2019) 'Informed machine learning - a taxonomy and survey of integrating knowledge into learning systems', arXiv preprint arXiv:1903.12394. To be validated.

Winograd, T. (2006) 'Shifting viewpoints: artificial intelligence and human-computer interaction', Artificial Intelligence, 170(18), pp. 1256-1258. To be validated.

Xu, W. (2019) 'Toward human-centered AI: a perspective from human-computer interaction', Interactions, 26(4), pp. 42-46. To be validated.

Yang, Q., Liu, Y., Chen, T. and Tong, Y. (2019) 'Federated machine learning: concept and applications', ACM Transactions on Intelligent Systems and Technology (TIST), 10(2), pp. 1-19. To be validated.

Yang, Q., Steinfeld, A., Rosé, C. and Zimmerman, J. (2020) 'Re-examining whether, why, and how human-AI interaction is uniquely difficult to design', in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-13. To be validated.

Yu, H., Shen, Z., Miao, C., Leung, C., Lesser, V. R. and Yang, Q. (2018) 'Building ethics into artificial intelligence', in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pp. 5527-5533. To be validated.

Open Questions

  • The chapter proposes a framework but reports no empirical validation of it. How does HCD for AI perform when applied to a real system, and against what outcome measures?
  • The authors note that crowdsourced evaluation is needed for continuously evolving AI systems but do not specify how many participants or what protocol suffices. What evaluation standard replaces the fixed user-count heuristics of classical usability testing?
  • Embedding ethics in AI faces the gap between abstract values (fairness, autonomy) and technical implementation. What concrete methods close this gap while keeping humans in the loop?
  • The framework assumes UX experts and AI practitioners can bridge fundamentally different mental models. The chapter treats this as an open challenge rather than a solved problem.