STDP-based Visual Sensing and Timing-based Learning using Vision Sensors

This study presents a groundbreaking event-driven visual-tactile perception system, incorporating a novel tactile sensor inspired by biological mechanisms and leveraging multi-modal spike-based learning. Our tactile sensor, named NeuTouch, boasts scalability by virtue of its event-based design, enabling efficient handling of an expanding array of taxels. Complementing this, our Visual-Tactile Spiking Neural Network (VT-SNN) facilitates swift perception, particularly when interfaced with event sensors. To comprehensively evaluate our system's capabilities, we conduct rigorous testing on two pivotal robot tasks: container classification and rotational slip detection. Encouragingly, our approach yields commendable accuracies relative to conventional deep learning methods across both tasks, showcasing its efficacy in real-world scenarios. Furthermore, recognizing the importance of fostering research in multi-modal event-driven robot perception, we openly share our visual-tactile datasets to facilitate future advancements in this domain. This study underscores the promising potential of our approach in advancing the development of intelligent, energy-efficient robot systems, thereby paving the way for transformative advancements in robotics research and applications.