Abstract:
Safe intent recognition for assistive robots remains challenging when integrating neural, muscular, and gaze signals, particularly under sensor degradation or failure. We propose TriSaFe-Trans, a lightweight tri-modal transformer that fuses electroencephalography (EEG), electromyography (EMG), and eye-tracking (ET) data via modality-specific temporal convolutional networks (TCNs) and bidirectional gated recurrent units (BiGRUs), combined with reliability-gated cross-modal attention. The model is trained using multi-objective selfsupervised pretraining, followed by supervised fine-tuning, to simultaneously decode action (REST vs. ACTION) and task (rest plus five assistive tasks) from fixed-length windows. We collected a synchronized EEG, EMG, and eye-tracking dataset from N=20 healthy adults, and evaluated the model using a balancedwindow leave-one-subject-out (LOSO) protocol. Under nominal sensing (S0), TriSaFe-Trans achieves an action accuracy of 0.903±0.079( balanced accuracy 0.882±0.079) and a task accuracy of 0.727±0.096 (balanced accuracy 0.680±0.100), with positive multimodal synergy gain. Across LOSO folds, TriSaFe-Trans improves task decoding and shows statistically significant gains over the baselines on key metrics. We assess single-modality failures via test-time sensor-dropout scenarios S1 (drop EEG), S2 (drop EMG), and S3 (drop ET) masking the dropped modality, and introduce safety policies (P0-P2) with safety metrics to quantify safety performance trade-offs. Under SO, a safety-first policy reduces unsafe actions and improves rest safety at the cost of increased missed intent. Code and supplementary video are available at: https://github.com/madibabaiasl/SafeIntentDetectionPaper/wiki and https://youtu.be/pdMp_myF-1I, respectively.
