Robust Upper Limb Motion Learning and Generalization Using a Hierarchical Neuromechanical Control Model with Deep Reinforcement Learning and Muscle Synergy

Bio-inspired musculoskeletal robots offer inherent advantages in flexibility, robustness, and compliance. However, achieving efficient and generalizable motion learning while producing natural, human-like movements remains challenging. Neuromechanical mechanisms provide a promising framework to emulate biological motor control and skill acquisition. In this study, we present a biomimetic hierarchical neuromechanical control model (HNCM) that integrates deep reinforcement learning with two distinct lower-level controllers. The model is designed to manage high-dimensional muscle control, enhance robustness to motor and sensory perturbations, generalize to previously untrained tasks, and generate human-like reaching movements. A high-level controller based on the proximal policy optimization algorithm models the supraspinal neural system and generates motor commands. The dual-path adaptive lower-level controller emulates both synergy-based and non-synergy-based motor pathways, preserving the benefits of structured exploration while allowing flexible, task-specific behavioral refinement. Comprehensive evaluations on center-out and random reaching tasks demonstrate that HNCM significantly outperforms baseline methods in control accuracy, learning efficiency, and generalization under noise perturbations, while producing more natural and biologically plausible movement trajectories. This work advances the understanding of neuromechanical control and provides a solid foundation for future applications in musculoskeletal robotics.