Abstract:
Virtual reality motion sickness (VRMS) seriously affects user comfort and Quality of Experience (QoE) in immersive video systems, making objective and continuous assessment essential. Existing questionnaire-based methods are subjective and discontinuous, while physiological-signal-based approaches often rely on unimodal sensing or shallow fusion. To address these limitations, this paper proposes a multimodal physiological representation and cross-modal fusion framework for VRMS assessment using EEG, ECG, GSR, and eye-tracking signals. Modality-specific encoders are designed to extract VRMS-related neural, autonomic, and ocular patterns, and learnable semantic anchors, modality identity embeddings, and Transformer-based interactions are used to align heterogeneous modalities. In addition, a sample-level router and auxiliary unimodal supervision are introduced to adaptively weight modality reliability and enhance fused representation learning. To improve robustness under incomplete sensing, modality dropout is adopted during training, and pseudo-modality tokens are generated from visible streams during inference. Experiments on a self-collected dataset and a public VRMS dataset show that the proposed method outperforms representative baselines and achieves robust VRMS recognition under noisy or missing physiological signals.
