Gábor Baranyi, Zsolt Csibi, Kristian Fenech, Áron Fóthi, Zsófia Gaál, Joul Skaf, András Lőrincz

Adaptive Assistance Framework for Ambient Intelligence Rehabilitation

Gábor Baranyi, Zsolt Csibi, Kristian Fenech Áron Fóthi, Zsófia Gaál, Joul Skaf, András Lőrincz
Dept. of Artificial Intelligence, Eötvös Loránd University, Budapest, H-1117, Hungary.
Paper arXiv Code Supplementary
pipeline
(a) Patient with smartphone, (b) 3D patient model reconstruction (Choutas and et al., 2020), (c) 3D NeRF or Gaussian Splat of the home environment (Savva and et al., 2019) reconstruction with NeRFStudio (Tancik and et al., 2023), (d) colored semantic map for path planning to the exercise area, (e) optimal position of mobile tripod and patient, (f) optimal pose of camera and (g) patient, (h) navigation module guides the human partner to set up the tripod and to go to the start position and take the starting pose, (i) Visual and verbal instructions for the exercise delivered via a mobile phone, video recording, background elimination, conversion recording into a skeleton and an avatar representation for privacy reasons, (j) videos are shown about the correct and the time warped execution of exercises for visual comparison, (both (i) and (J) are shown in the clinical environment) (k) VLM verbal feedback to the therapist provides information about potential errors. The design has resources from Flaticon.com.

Abstract

This paper introduces the Ambient Intelligence Rehabilitation Support (AIRS) framework, an advanced artificial intelligence-based solution tailored for home rehabilitation environments. AIRS integrates cutting-edge technologies, including Real-Time 3D Reconstruction (RT-3DR), intelligent navigation, and large Vision-Language Models (VLMs), to create a comprehensive system for machine- guided physical rehabilitation. The framework is demonstrated in rehabilitation scenarios following total knee replacement (TKR), utilizing a database of 263 video recordings for evaluation. A smart- phone is employed within AIRS to perform RT-3DR of living spaces and has a body-matched avatar to provide visual feedback about the excercise. This avatar is necessary in (a) optimizing exercise configurations, including camera placement, patient positioning, and initial poses, and (b) address- ing privacy concerns and promoting compliance with the AI Act.

The system guides users through the recording process to ensure the collection of properly recorded videos. AIRS employs two feed- back mechanisms: (i) visual 3D feedback, enabling direct comparisons between prerecorded clinical exercises and patient home recordings and (ii) VLM-generated feedback, providing detailed explana- tions and corrections for exercise errors. The framework also supports people with visual and hearing impairments. It also features a modular design that can be adapted to broader rehabilitation contexts. AIRS software components are available for further use and customization.

BibTeX

@article{,
  author    = {},
  title     = {},
  journal   = {},
  year      = {},
}