A CNN–LSTM Framework for Player-Specific Baseball Pitch Type Prediction from Video Sequences

The performance of the pitcher is the cornerstone of baseball, often determining the flow and ultimate outcome of a game. Given this centrality, understanding the mechanics of an elite pitcher and decoding their strategies are paramount for both internal optimization and competitive scouting. This study proposes an end-to-end deep learning pipeline for automatically classifying five distinct pitch types from raw broadcast footage of MLB pitcher Max Scherzer between 2015 and 2020. By formulating pitch delivery as a time-series classification problem tailored to the unique biomechanics of an elite athlete, the proposed CNN–LSTM framework integrates per-frame spatial feature extraction using an advanced CNN backbone (YOLOv8s-cls) with a two-layer long short-term memory (LSTM) network to capture subtle biomechanical cues across a standardized 20-frame delivery sequence. While skeletal pose estimation primarily focuses on tracking major joints to analyze standard pitching mechanics, the proposed pixel-based method preserves fine-grained visual cues—such as finger grip and wrist rotation—that are critical for distinguishing pitch variations. The proposed framework achieved an accuracy of 91.8% under a standard Random Split and, importantly, 84.5% under a strict Chronological Split across different seasons, validating the feasibility of automated pitch “tell” detection from broadcast video. The resulting system provides coaches and analysts with an objective, data-driven tool for generating personalized scouting reports, identifying mechanical inconsistencies, and refining pitching strategies.