Answers are grounded in this paper and cite their sources.
Larger, sophisticated sequential models excel in Human Activity Recognition (HAR) using multivariate time-series data but may not suit compute-constrained smartphones due to latency issues. Knowledge distillation offers a solution by training smaller models based on larger teachers, but a single teacher often struggles to perform uniformly well across diverse activity classes. To address this limitation, we propose the Distillation, Augmentation, and Filtering (DAF) framework, leveraging Multiple-Architecture based Multi-Teacher Distillation (MAMTD). This approach identifies the best-performing teacher model for each activity class and uses Contrastive loss-based Distillation to align a smaller student model with the most effective teachers while distancing it from less effective ones. For challenging categories, a peer student model is employed with data augmentation to focus on areas where the first student struggles. Finally, a novel checkpoint ensemble via probability filtering combines the strengths of both student models, achieving a 21.4-24.6% increase in accuracy for certain confusing categories compared to typical distilled networks, while maintaining low latency.
No takes yet. Share an insight, caveat, or question.
Dutta et al. (2025) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: