Multi-Modal Fall Detection -- From Vision to Personalized Wearables

A research project exploring robust fall detection, first using viewpoint-independent depth cameras, and later with personalized, deep-learning models on smartwatches.

Falls are a leading cause of injury and death among older adults, making automated, real-time fall detection a critical area of research for assistive technology. Our research tackles this problem through two distinct but complementary modalities. We first developed a robust, vision-based system for in-home monitoring using depth cameras. Subsequently, to create a solution that is mobile and user-centric, we explored the potential of ubiquitous personal devices, leading to our work on smartwatch-based detection which culminates in a fully personalized, adaptive system.

Viewpoint-Independent Vision-Based Detection

Our early work in this area focused on creating a reliable, less intrusive system for in-home environments using Kinect depth cameras. We introduced a statistical method that, unlike other vision-based systems of the time, is viewpoint-independent and does not rely on hardcoded thresholds (Zhang et al., 2012).

The system works by first performing background subtraction on the depth stream to isolate the person. It then tracks the person’s head and extracts five novel features related to its movement (e.g., total head drop, maximum speed, duration). These features are combined in a Bayesian framework to probabilistically determine if a fall has occurred. A key contribution was our evaluation protocol, where the system was trained on data from one camera viewpoint and successfully tested on a completely different one, demonstrating true robustness to camera placement.

The simulated apartment environment used to collect depth-video for training and testing our viewpoint-independent fall detection method.

Wearable-Based Detection with Deep Learning

To create a solution that is mobile and works outside a fixed environment, we shifted our focus to wearable devices. This led to the development of the SmartFall system, an Android application that uses accelerometer data from a commodity smartwatch to detect falls in real-time (Mauldin et al., 2018).

The system is based on a three-layer IoT architecture where the watch acts as the edge sensor, a paired smartphone performs the computation, and a cloud server provides back-end services. We demonstrated that a Deep Learning model, specifically a Recurrent Neural Network (RNN), significantly outperforms traditional machine learning approaches because it can automatically learn subtle, temporal features directly from the raw sensor data.

Left: The three-layer IoT architecture for the SmartFall system. Right: The SmartFall user interface for user profile setup and fall alerts.

A key challenge in this domain is the scarcity of large, annotated datasets. To address this, we explored Ensemble Deep Learning techniques (Mauldin et al., 2019), (Mauldin et al., 2020), showing that an ensemble of smaller, diverse RNN models consistently outperforms a single, larger model.

Personalization via Edge-Cloud Collaboration

Finally, recognizing that a “one-size-fits-all” model is not optimal, our latest work introduces a collaborative edge-cloud framework for personalized fall detection (Ngu et al., 2020), (Ngu et al., 2021).

In this advanced architecture, the fall detection logic runs directly on the smartwatch. When a false alarm occurs, the user provides feedback through a simple on-watch UI. This labeled data is securely synchronized with a cloud backend, which automatically triggers a personalized re-training pipeline. The new model, now tailored to the user’s specific movement patterns, is pushed back to the watch. This automated feedback loop allows the system to continuously adapt and improve its precision, dramatically reducing false positives while maintaining high recall. This work is grounded in our broader research into robust IoT middleware architectures (Ngu et al., 2016).

Left: The automated data flow between the watch (edge) and the cloud for personalized model training.
Right: The simple user interface on the smartwatch for providing feedback after a potential fall is detected.

References

2021

  1. Collaborative Edge-Cloud Computing for Personalized Fall Detection
    Anne H Ngu, Shaun Coyne, Priyanka Srinivas, and Vangelis Metsis
    In IFIP International Conference on Artificial Intelligence Applications and Innovations, 2021

2020

  1. Ensemble Deep Learning on Wearables Using Small Datasets
    Taylor Mauldin, Anne H Ngu, Vangelis Metsis, and Marc E Canby
    ACM Transactions on Computing for Healthcare, 2020
  2. Personalized Fall Detection System
    Anne H Ngu, Vangelis Metsis, Shaun Coyne, Brian Chung, Rachel Pai, and Joshua Chang
    In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), 2020

2019

  1. Experimentation and analysis of ensemble deep learning in iot applications
    Taylor Mauldin, Anne H Ngu, Vangelis Metsis, Marc E Canby, and Jelena Tesic
    Open Journal of Internet Of Things (OJIOT), 2019

2018

  1. SmartFall: A smartwatch-based fall detection system using deep learning
    Taylor R Mauldin, Marc E Canby, Vangelis Metsis, Anne HH Ngu, and Coralys Cubero Rivera
    Sensors, 2018

2016

  1. IoT middleware: A survey on issues and enabling technologies
    Anne H Ngu, Mario Gutierrez, Vangelis Metsis, Surya Nepal, and Quan Z Sheng
    IEEE Internet of Things Journal, 2016

2012

  1. A viewpoint-independent statistical method for fall detection
    Zhong Zhang, Weihua Liu, Vangelis Metsis, and Vassilis Athitsos
    In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), 2012