Cognitive Modeling Tools for Design Evaluation (HEE / CogTool)

emerging Updated: 2026-06-08
usability-evaluation cognitive-workload human-performance-modeling cogtool klm cockpit-design aviation-human-factors design-prototyping task-analysis

Cognitive Modeling Tools for Design Evaluation (HEE / CogTool)

Status: emerging
Last updated: 2026-06-08
Sources: Feuerstack 2015 Human Efficiency Evaluator.Pdf
Tags: [usability-evaluation, cognitive-workload, human-performance-modeling, cogtool, klm, cockpit-design, aviation-human-factors, design-prototyping, task-analysis]

Summary

Cognitive modeling tools let designers predict how a proposed interface will perform before building it, by simulating an operator rather than recruiting one. Feuerstack, Lüdtke and Osterloh (2015) present the Human Efficiency Evaluator (HEE), an extension of CogTool aimed at making this kind of analysis usable by engineers without cognitive-modeling expertise. The tool takes annotated photos or sketches of cockpit instruments, records a task as a demonstration, and then simulates a pilot model in the CASCaS cognitive architecture to predict task-execution time and workload-over-time. Applied to three generations of slats/flaps instrument designs for an aircraft approach, the analysis showed task performance and workload falling with each generation. The work connects this knowledge base's usability-evaluation and performance-modelling strands: it operationalises cognitive modelling as a practical design-evaluation step.

Body

Context

Feuerstack, Lüdtke and Osterloh (2015), at the European Conference on Cognitive Ergonomics, address why cognitive workload analysis — though established — is rarely used in industrial development: it depends on complex architectures and proprietary cognitive-model notations accessible only to experts (PDF p. 1). Their response is a tool, the HEE, that hides this machinery behind a demonstration-based interface for engineers and analysts. Within this knowledge base the article sits at the meeting point of Usability And User Experience (evaluating design prototypes before build), Mental Workload and Nasa Tlx (the workload it predicts), and Human Performance Modeling (the predict-before-build enterprise). Its CogTool basis descends from the ACT-R lineage in Cognitive Architectures, and it applies to the Aviation Human Factors domain.

Key Points

The motivation is the cost and limited reach of human testing. Feuerstack et al. note that user testing of design prototypes scales poorly: only a few design variants can be tested, often only with a functional prototype, and only against a small set of situations — yet in commercial aviation an instrument change can affect hundreds of standardised procedures, including rare ones (instrument damage, specific weather) that cannot all be covered in user tests (PDF pp. 1–2). Problems found late, on a working prototype, carry high re-engineering cost. A cognitive system analysis that predicts task execution and workload early is the proposed alternative.

The HEE extends CogTool with domain-specific instruments and a pilot simulation. The tool re-uses CogTool's interface, in which the user generates task scripts by demonstrating a task on a design sketch (PDF p. 2). Its contribution is twofold: support for adding new, domain-specific cockpit instruments modelled as state charts whose elements carry cognitive operators with associated workload values, and the automated creation of a virtual environment and pilot model that is then simulated in the CASCaS cognitive architecture to predict the operator's workload over time (PDF pp. 2–3). The workflow has three activities — defining designs (annotating photos or sketches), demonstrating procedures, and running operator simulations of those demonstrations (PDF p. 3).

Workload is computed from a channel-based metric. The HEE assigns workload through the McCracken–Aldrich–Bierbaum (VACP) approach, which rates demand on visual, auditory, cognitive, and psychomotor channels on a scale to 7, producing a workload-over-time profile rather than a single end-of-task score (PDF pp. 5–6). The paper situates this against subjective instruments such as the NASA Task Load Index (see Nasa Tlx) and notes the channel metric's origins in Subject-Matter-Expert data (PDF p. 6).

The case study compares three instrument generations. Using a slats/flaps setting task during an aircraft approach — a scenario drawn from Hutchins — the authors evaluate three generations of cockpit instrument design, from minimal support to the support in modern aircraft (PDF pp. 6–7). The HEE analysis showed that task performance and workload were significantly reduced with each new instrument setup, demonstrating the benefit of the successive pilot-assistance systems and showing how workload-over-time predictions can feed design decisions (PDF p. 7).

Conclusion

Feuerstack, Lüdtke and Osterloh (2015) conclude that the HEE makes cognitive task-performance and workload analysis available to non-expert engineers and can demonstrate, before any prototype is flown, that successive cockpit designs reduce pilot workload (PDF p. 7). They are explicit about limitations: the tool currently predicts single tasks, whereas a real cockpit runs several procedures in parallel — and attention shifts between concurrent tasks can substantially change the performance of any one of them — and the channel workload metric is a subjective measure first validated on military helicopter pilots, only recently extended to commercial pilots (PDF p. 7). The contribution is methodological: a route from a design sketch to a quantitative, model-based workload prediction that an engineer can run, positioned as a complement to, not a replacement for, human testing.

  • Usability And User Experience — the HEE is a formative, model-based evaluation method for design prototypes, an alternative to empirical usability testing when testing does not scale
  • Mental Workload — the quantity the tool predicts; the channel-based metric is one operationalisation of operator load
  • Nasa Tlx — the subjective workload instrument the paper contrasts with its model-based, over-time prediction
  • Human Performance Modeling — the predict-before-build modelling enterprise this tool applies
  • Cognitive Architectures — CogTool (the HEE's basis) descends from the ACT-R lineage; CASCaS is the simulating architecture
  • Aviation Human Factors — the application domain (cockpit instrument design for an approach)

References

Feuerstack, S., Lüdtke, A. and Osterloh, J.-P. (2015) 'A Tool for Easing the Cognitive Analysis of Design Prototypes of Aircraft Cockpit Instruments: The Human Efficiency Evaluator', in Proceedings of the European Conference on Cognitive Ergonomics 2015 (ECCE '15). New York: ACM, pp. 1–8. doi: 10.1145/2788412.2788434. feuerstack2015hee

John, B.E., Prevas, K., Salvucci, D.D. and Koedinger, K. (2004) 'Predictive human performance modeling made easy', in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). New York: ACM. To be validated.

McCracken, J.H. and Aldrich, T.B. (1984) Analyses of Selected LHX Mission Functions: Implications for Operator Workload and System Automation Goals. Technical Note ASI479-024-84. Fort Rucker, AL: Anacapa Sciences. To be validated.

Open Questions

  • The HEE predicts single tasks only, yet its own conclusion identifies parallel procedures and attention shifts as decisive for cockpit workload. The attention-allocation modelling needed to lift this limitation is exactly the AIE/CASCaS line developed for driving in the eye-tracking corpus — a cross-KB link candidate.
  • The channel-based workload metric is subjective and SME-derived, validated mainly on military helicopter pilots. How well do its predictions agree with measured workload indices (e.g. pupillometry, NASA-TLX) for commercial cockpit tasks?
  • CogTool/KLM and CASCaS make different modelling commitments. How much does the HEE's workload-over-time prediction depend on the choice of underlying architecture?