Development of a Robotic Manipulator for Piano Performance via Numbered Musical Notation Recognition

This paper presents a piano-playing robotic system that integrates numbered musical notation recognition with automated manipulator control. The system captures the notation using a camera, applies four-point detection for perspective correction, and performs measure segmentation through an orthogonal projection method. A pixel-scanning technique is then used to locate the positions of numerical notes, pitch dots, and rhythmic markers. Digit recognition is achieved using a CNN model trained on both the MNIST handwritten digit dataset and a custom computer-font digit dataset (CFDD), enabling robust identification of numerical symbols under varying font styles. The hardware platform consists of a 3D-printed robotic hand mounted on a linear rail and driven by an ESP32-based embedded controller with custom driver circuits. According to the recognized musical notes, the manipulator executes lateral positioning and vertical key-press motions to reproduce piano melodies. Experimental results demonstrate reliable notation recognition and accurate performance execution, confirming the feasibility of combining computer vision and robotic manipulation for low-cost, automated musical performance.