View-agnostic Human Exercise Cataloging with Single MmWave Radar

Advances in mmWave-based sensing have enabled a privacy-friendly approach to pose and gesture recognition. Yet, providing robustness with the sparsity of reflected signals has been a long-standing challenge towards its practical deployment, constraining subjects to often face the radar. We present RF-HAC- a first-of-its-kind system that brings robust, automated and real-time human activity cataloging to practice by not only classifying exercises performed by subjects in their natural environments and poses, but also tracking the corresponding number of exercise repetitions. RF-HAC's unique approach (i) brings the diversity of multiple radars to scalably train a novel, self-supervised, pose-agnostic transformer-based exercise classifier directly on 3D RF point clouds with minimal manual effort and be deployed on a single radar; and (ii) leverages the underlying doppler behavior of exercises to design a robust self-similarity based segmentation algorithm for counting the repetitions in unstructured RF point clouds. Evaluations on a comprehensive set of challenging exercises in both seen and unseen environments/subjects highlight RF-HAC's robustness with high accuracy (over 90%) and readiness for real-time, practical deployments over prior art.