Mood-TBI: Rehabilitation of Traumatic Brain Injured Patients — Patient Mood Analysis from Multimodal Video
Published in 25th IEEE International Conference on Image Processing (ICIP 2018) Workshop, 2018
Abstract
Monitoring the emotional and affective state of Traumatic Brain Injured (TBI) patients during rehabilitation is critical for personalising therapeutic interventions and tracking recovery progress. However, TBI patients exhibit highly atypical expressive behaviour — including facial paralysis, limited voluntary muscle control, and reduced body mobility — making standard unimodal emotion recognition systems unreliable in this clinical context.
This paper presents a multimodal video-based mood analysis system that fuses information from both facial expressions and upper-body movements to produce robust affective state estimates for TBI patients. By combining complementary cues from face and body channels, the system achieves more stable performance across the diverse symptom presentations seen in TBI populations.
The approach is validated on a dataset collected from a neurological rehabilitation centre, covering multiple rehabilitation activity types. Results demonstrate that multimodal fusion substantially outperforms unimodal facial or body-only analysis for this challenging population.
Key Contributions
- Multimodal fusion of facial expression and upper-body movement cues for TBI mood analysis
- Evaluation in ecologically valid clinical rehabilitation settings
- Demonstrated benefit of cross-modal fusion over unimodal analysis for impaired populations
- Contribution to the evidence base for affect-aware assistive and robotic rehabilitation systems
Methodology
The system pipeline includes:
- Facial Region Processing — detection, alignment, and spatial feature extraction under constrained expressivity
- Upper Body Pose Estimation — skeletal joint tracking to infer body posture and gesture states
- Feature-Level Fusion — late fusion of face and body representations using learned combination weights
- Temporal Smoothing — sequence-level inference for stable mood labels across rehabilitation sessions
Venue
Presented at the IEEE Signal Processing Society workshop in conjunction with the 25th IEEE International Conference on Image Processing (ICIP 2018), Athens, Greece.