Human-in-the-loop digital twins integrate live biosensor data into virtual models of operators and their environments. This article examines the research foundations, technical architecture, and practical implementation of physiological digital twins in driving simulators and beyond.

Bringing Real-Time Human State Into Simulation Systems

Digital twin architectures have advanced rapidly across transportation, aerospace, healthcare, manufacturing, and industrial automation. In simulation-heavy domains such as automotive development and aviation training, virtual models can now replicate physical systems with remarkable fidelity, incorporating real-time telemetry, environmental variables, traffic conditions, and increasingly complex predictive models.

Yet despite these advances, many digital twin implementations still treat the human operator as an external variable rather than an actively modeled component of the system itself. A driving simulator may accurately reproduce vehicle dynamics, road conditions, and traffic behavior while remaining largely insensitive to the driver’s evolving cognitive state. Likewise, a flight simulator can capture aerodynamic behavior with high precision while offering limited visibility into pilot workload, fatigue, stress, attentional allocation, or decision-making processes as they unfold in real time.

Human Performance as a System Variable

For researchers working in human factors, neuroergonomics, cognitive engineering, and simulation science, this limitation is becoming increasingly significant. In many operational environments, human performance is not simply another output variable, it is actually one of the primary determinants of system behavior and safety. The challenge is no longer only to simulate the environment around the operator, but to continuously model the operator within it.

Human-in-the-loop digital twins attempt to address this gap by integrating real-time physiological and behavioral data directly into simulation architectures. Rather than treating the human as a static or generalized entity, these systems continuously estimate operator state using multimodal biosensor data streams such as eye tracking, EEG, electrodermal activity (EDA/GSR), ECG, respiration, and facial expression analysis.

From Passive Observation to Adaptive Simulation

This fundamentally changes the role of the simulator. Instead of functioning purely as a controlled task environment, the simulator becomes part of a dynamic feedback system in which the operator’s internal state is continuously measured, interpreted, and potentially used to influence the behavior of the system itself.

The result is a synchronized experimental environment capable of capturing both external system behavior and the evolving physiological and cognitive state of the human interacting with it. Researchers can investigate not only whether an operator succeeds or fails at a task, but how attention, workload, fatigue, stress, and emotional regulation fluctuate throughout the interaction, often at sub-second temporal resolution.

Why Adaptive Systems Require Human Modeling

This capability is becoming increasingly important in domains where adaptive systems and human-machine collaboration are central research concerns. Automated driving, air traffic control, surgical robotics, industrial control rooms, military training, and teleoperation systems all depend on understanding how humans behave under varying cognitive and physiological demands. As automation grows more sophisticated, accurately modeling operator state becomes critical for designing systems that can adapt intelligently to human limitations and capabilities.

This article examines the research foundations of human-in-the-loop digital twins, the technical infrastructure required to build them, and the role multimodal biosensor platforms such as iMotions play in operationalizing them for academic and applied research.

Human-in-the-loop digital twins

Human-in-the-loop digital twins extend conventional simulation architectures by incorporating continuous models of the operator’s physiological and cognitive state alongside the surrounding system environment. While the exact implementation varies by application domain, these systems generally share several defining characteristics.

Continuous physiological state estimation

The defining feature of a human-in-the-loop digital twin is that the operator is continuously instrumented and modeled during task execution. Rather than relying on generalized assumptions about driver or pilot behavior, the system ingests live physiological and behavioral signals and uses them to estimate latent cognitive and affective state variables in near real time.

Typical biosensor modalities include:

Eye tracking for gaze behavior, fixation patterns, blink behavior, and pupillometry
EEG for workload, vigilance, and attentional dynamics
EDA/GSR for autonomic arousal and stress responses
ECG and heart rate variability for cardiovascular workload and fatigue indicators
Facial expression analysis for observable emotional and behavioral reactions
Respiration and motion tracking for additional behavioral and physiological context

These signals are not typically interpreted in isolation. Instead, they are combined into multimodal models capable of estimating higher-level state variables such as cognitive workload, fatigue, situational awareness, distraction, stress, or emotional regulation.

This distinction is important because the goal of a human-in-the-loop twin is generally not to measure physiology for its own sake, but to construct a continuously updated representation of operator state that can be integrated into the broader system model.

Synchronized task environments

A second defining characteristic is the tight temporal synchronization between physiological monitoring and the active task environment. The operator performs tasks inside a driving simulator, flight simulator, VR environment, surgical trainer, or operational workstation while physiological signals and environmental telemetry evolve simultaneously on a shared timeline.

This synchronization requirement is non-trivial. Biosensor streams often operate at dramatically different sampling rates and originate from heterogeneous hardware systems with independent clocks and communication protocols. EEG may be sampled at hundreds of Hertz, eye tracking at 120 Hz, simulator telemetry at 60 Hz, and physiological sensors such as GSR or ECG at entirely different rates again.

For the digital twin to function coherently, these streams must remain aligned with sufficient temporal precision to support meaningful causal analysis and real-time state estimation. This is one reason why synchronization frameworks such as Lab Streaming Layer (LSL) have become foundational infrastructure within multimodal simulation research.

The importance of synchronization becomes especially apparent when studying transient cognitive events. A spike in workload following an automated takeover request, a delayed gaze response to a hazard, or a physiological stress reaction during a high-demand maneuver may unfold over only a few hundred milliseconds. Without precise temporal alignment, these interactions become difficult to interpret reliably.

Closed-loop adaptive systems

The third defining property is the ability to close the loop between operator state estimation and system behavior.

In conventional simulation studies, physiological data is often recorded passively for later analysis. Human-in-the-loop digital twins instead allow state estimates to influence the simulator or operational environment while the task is still unfolding.

For example, a workload estimation model might dynamically adjust simulator difficulty if cognitive demand exceeds a threshold. An automated driving system could modify handover timing based on detected driver vigilance. A training simulator might introduce interventions when fatigue signatures emerge or adapt task complexity according to real-time performance and physiological indicators.

This transforms the digital twin from a passive observational model into an active adaptive system.

Importantly, the goal is not necessarily full automation. In many research settings, the objective is to better understand how adaptive systems should respond to fluctuating human state under varying operational conditions. Human-in-the-loop twins therefore provide a controlled experimental framework for studying adaptive automation, operator assistance systems, workload management strategies, and human-machine collaboration more broadly.

The combination of continuous physiological modeling, synchronized multimodal acquisition, and closed-loop adaptation enables research paradigms that are difficult or impossible to investigate using conventional offline analysis alone. Researchers can examine how workload evolves during autonomous driving transitions, whether physiological fatigue markers emerge before behavioral degradation, or how adaptive training protocols influence skill acquisition and retention over time.

While many of the individual technologies involved are well established, their convergence into coherent real-time architectures represents a significant methodological shift for simulation science and human factors research.