Optimization of Swim Pose Estimation and Recognition with Data Augmentation

Swim pose estimation and recognition is a challenging problem in machine learning and artificial intelligence as the body of the swimmer is continuously submerged under the water. The objective of this paper is to enhance existing ML models for estimating swim poses and for recognizing strokes to aid swimmers in pursuing a more perfect technique. We developed a novel methodology augmenting raw video data and adjusting a YOLOv7 base model to enhance swim pose estimation. We found the standard multi-class classification using a CNN to be insufficient for stroke recognition due to the similarity between strokes, so we designed a hierarchical binary classification tree using multiple ensembles of multilayer perceptron (MLP), CNN, and residual network (ResNet) models. Through these optimizations, the confidence level of pose estimation has increased by over $30 %$, and the ensembles of our recognition model have achieved approximately $80 %$ accuracy. Fine-tuning of our recognition models and research combining joint keypoint coordinates with angle measurements as inputs could further increase the accuracy of our models.