title: Gaze-Based HCI and Usability Research
status: established
last_updated: 2026-05-31
sources: Jacob Kam 2003 Eye Tracking Hci Commentary, Novak 2024 Eye Tracking Usability Ux Review, Plopski 2022 Eye Tracking Xr
tags: [eye-tracking, HCI, usability, user-experience, gaze-interaction, Midas-touch, XR, VR, systematic-review]

Gaze-Based HCI and Usability Research¶

Status: established
Last updated: 2026-05-31
Sources: Jacob Kam 2003 Eye Tracking Hci Commentary, Novak 2024 Eye Tracking Usability Ux Review, Plopski 2022 Eye Tracking Xr
Tags: [eye-tracking, HCI, usability, user-experience, gaze-interaction, Midas-touch, XR, VR, systematic-review]

Summary¶

Eye tracking is used in HCI research in two distinct modes: retrospectively, to analyse user attention during interaction; and in real time, as a direct input medium. Jacob and Karn (2003) identified that both modes share an analogous core challenge — using gaze data appropriately without over-responding to involuntary movements. Recent systematic reviews confirm a trend toward automated, machine learning-assisted analysis, and the extension of gaze-based interaction into extended reality (XR) headsets introduces new interaction paradigms and privacy considerations.

Body¶

Context¶

This article draws on three sources spanning two decades of gaze-based HCI: Jacob and Karn's (2003) commentary framing eye tracking's two roles and its core challenges, Novák et al.'s (2024) systematic review of 90 included usability and user-experience (UX) papers (from 1,988 screened, search to 2022), and Plopski et al.'s (2022) survey of gaze interaction in head-worn extended reality (XR). Together they cover retrospective analysis, real-time interaction, and the move into immersive devices. Within this knowledge base the article is the human-computer-interaction strand: it draws the foundational involuntary-movement problem from Fixational Eye Movements, depends for real-time interaction on the kind of camera-based estimation examined in Appearance Based Gaze Estimation, and runs parallel to the other applied-domain cases, Eye Tracking In Surgery and Visual Occlusion Attentional Demand. The Plopski et al. (2022) survey is the deferred large source for this KB (see Open Questions), so its statements below carry no page locator.

Key Points¶

Jacob and Karn (2003) framed eye tracking in HCI as serving two areas, usually reported separately: retrospective usability analysis, where recordings are analysed after a session and the eye movements do not affect the interface in real time, and real-time gaze interaction, where eye movements act as a direct input medium (PDF p. 1, orig. p. 573). For real-time use the central difficulty is the Midas touch — because the eyes are always looking somewhere, an interface that acts on every fixation issues commands the user never intended, and the "look at it and have it happen" case is in general impossible to distinguish from ordinary looking (PDF pp. 17–18, orig. pp. 589–590). They trace the technique to Dodge and Cline's (1901) corneal-reflection method, the first precise non-invasive eye tracking (PDF p. 2, orig. p. 574), and note that the technology was only gradually becoming robust and inexpensive enough for real interfaces, with adoption rising mainly in usability labs (PDF p. 16, orig. p. 588).

Novák et al. (2024) found a significant shift toward more technologically advanced UX and usability evaluation, with an emphasis on automatic data processing and machine learning used to recognise various kinds of emotions linked to users' interactions (PDF p. 2, orig. p. 4485). They position eye tracking as an implicit measurement modality alongside self-report and performance measures (PDF p. 2, orig. p. 4485), and identify ML-based emotion recognition via gaze as an emerging area, proposing several opportunities for future research and gaps connecting UX, eye tracking, and machine learning (PDF pp. 1–2, orig. pp. 4484–4485).

Plopski et al. (2022) surveyed gaze interaction for head-worn XR across devices such as Meta Quest Pro, HTC Vive Pro Eye, Varjo XR-4, and Apple Vision Pro. Techniques include dwell-time selection, gaze-gesture commands, gaze-contingent foveated rendering, and social gaze signalling in multi-user settings. Foveated rendering reduces peripheral resolution to match lower peripheral acuity, cutting GPU load without perceptual degradation, and depends entirely on accurate real-time gaze estimation. They identify privacy as a central open challenge: always-on gaze data encodes cognitive state, identity, and health information, raising consent and governance questions existing frameworks do not address.

On the Midas touch specifically, Jacob and Karn (2003) argue that the way forward is interaction techniques that address the problem case by case, and they point to combining real-time eye-movement data with other, more conventional input modes — that is, multimodal interaction — rather than relying on gaze alone (PDF p. 18, orig. p. 590). Plopski et al. (2022) note that in XR, where switching modality is awkward, multimodal confirmation is a common pattern.

Conclusion¶

The three sources are complementary rather than competing, tracking the same field across time. Jacob and Karn (2003) set the conceptual frame — two roles, the Midas touch, and a technology only gradually becoming practical — that the later work still operates within. Novák et al. (2024) show the retrospective-analysis branch moving toward automated, ML-assisted UX pipelines, while Plopski et al. (2022) show the real-time branch moving into XR with new techniques such as foveated rendering. They agree that gaze must be used without over-responding to involuntary movement; the unresolved issues they add are contemporary — the open research gaps around ML-based emotion recognition (Novák et al., 2024) and privacy governance for always-on gaze data (Plopski et al., 2022).

References¶

Dodge, R. & Cline, T. S. (1901) 'The angle velocity of eye movements', Psychological Review, 8, pp. 145–157. To be validated. jacob2003hci

Jacob, R. J. K. & Karn, K. S. (2003) 'Eye tracking in human-computer interaction and usability research: Ready to deliver the promises', in J. Hyönä, R. Radach & H. Deubel (eds.) The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research. Amsterdam: Elsevier Science BV. doi: 10.1016/B978-044451020-4/50031-1. jacob2003hci

Novák, J. Š., Masner, J., Benda, P., Šimek, P. & Merunka, V. (2024) 'Eye tracking, usability, and user experience: A systematic review', International Journal of Human–Computer Interaction, 40(17), pp. 4484–4500. doi: 10.1080/10447318.2023.2221600. novak2024usability

Plopski, A., Hirzle, T., Norouzi, N., Qian, L., Bruder, G. & Langlotz, T. (2022) 'The eye in extended reality: A survey on gaze interaction and eye tracking in head-worn extended reality', ACM Computing Surveys, 55(3), pp. 1–39. doi: 10.1145/3491207. plopski2022xr