AI-Powered Learning Dashboard with Multimodal Metrics for Embodied Performance in XR Training Simulation

Abstract

The growing adoption of XR-based simulations in educational settings has highlighted the need for intelligent systems to monitor, predict, and visualize learners' experiences in real-time. This paper presents the design and implementation of a Real-Time Intelligent Monitoring System (RTIMS) for XR Learning that predicts learners' Quality of Experience (QoE) by analyzing cognitive, affective, and behavior data collected during XR-based simulation learning. The system leverages 14 sets of multi-dimensional data from HoloLens 2, including head and hand positions and pupil size measurements captured via NEON eye-tracking glasses. Using the RTIMS, cognitive (e.g., attention, task load), affective (e.g., self-efficacy, learning satisfaction), and behavioral (e.g., task accuracy, completion time) domain data were gathered through controlled experiments and used to train deep learning models. The RTIMS predicted QoE scores and generated a Learning Effectiveness Index (LEI), visualized in a real-time dashboard, enabling continuous monitoring and predictive insights into learners' learning quality progress. This research lays the groundwork for advancing XR-based learning by enhancing learner monitoring through QoE prediction, improving instructional adaptability, and fostering high quality learning experiences.