A neuromorphic electronic artist for robotic painting

Abstract Recent advances in computer vision and deep learning have led to a surge of interest in the field of AI-generated art, including digital image creation and robot-assisted painting. Traditional painting machines rely on static images and offline processing to incorporate visual feedback into their painting process. However, this approach does not consider the dynamic nature of painting and fails to decompose complex overlapping patterns into individual strokes. As an alternative to frame-based RGB cameras, neuromorphic cameras capture changes in light intensity within a scene via asynchronous event streams, promising to overcome some of the inherent limitations of traditional computer vision techniques. In this project, a robotic system for physical painting is presented which utilizes event-based visual input from a Dynamic Vision Sensor (DVS) camera. To take advantage of the camera's ultra-low latency and sparse encoding, the proposed system also employs event-based information processing, implemented with spiking neural networks on the neuromorphic DynapSE-1 processor. The robotic system receives DVS sensory data which represents the trajectory of a brush stroke and computes the required joint velocities to recreate the stroke with a 6-DOF robotic arm in a closed-loop manner. The controller additionally integrates tactile feedback from a force-torque sensor to dynamically adjust the end-effector’s distance towards the canvas depending on the brush’s deformation. Within the scope of the project, it was further demonstrated how speed information about a perceived brush stroke can be extracted from DVS data. The system was tested in a real-world setting and successfully generated a collection of physical brush strokes. The proposed network is a first step towards a fully spiking robotic controller with the ability to seamlessly incorporate event-based sensory feedback, providing ultra-low latency responsiveness. Beyond its utility in robot-assisted painting, the developed network is applicable to any robotic task requiring real-time adaptive control.