Multi-Modal Audio-Visual Stimuli based Brain-Machine Interface with Safe Model Predictive Control for Robotic Arm Trajectory Tracking in Human-Robot Collaboration

Achieving effective Brain–Machine Interface (BMI) control of robotic manipulators remains challenging, particularly when integrating low-bandwidth neural commands with safety-constrained robot execution. This paper presents a pilot proof-of-concept framework for BMI-driven control of a UR5 robotic arm using a switchable audio–visual EEG paradigm combined with nonlinear model predictive control (NMPC). Steady-State Visual Evoked Potentials (SSVEP) and Auditory Steady-State Responses (ASSR) are employed as mutually exclusive BMI channels for discrete task selection, while Error-Related Potentials (ErrP) provide supervisory feedback for safety intervention. Three stimulation frequencies are mapped to predefined manipulation tasks executed along offline, pre-computed trajectories. Robot execution is evaluated under open-loop control, a nonlinear PID controller as a reference baseline, and a safe NMPC scheme. In the proposed implementation, ErrP detection triggers suppression of robot velocity within the NMPC framework, acting as a real-time safety stop. Pilot human–robot interaction experiments demonstrate the feasibility of decoding-to-execution integration and consistent trajectory execution under NMPC. The NMPC scheme achieved tracking errors in the range of 0.043–0.137 m and task completion times of 7.3 s –7.5 s while enforcing a velocity limit of 0.25 m/s. The FBCCA method yielded average classification accuracies of 93.5% (SSVEP) and 87.5% (ASSR), averaged across two participants. These findings provide preliminary evidence of system-level feasibility, rather than population-level statistical validation, and motivate future studies with larger cohorts.