Spiking-Transformer Optimization on FPGA for Image Classification and Captioning

This paper presents an FPGA (Field Programmable Gate Arrays) architecture specifically crafted to optimize spiking transformers on FPGA. The initial phase involved designing transformer networks and conducting training for image classification and captioning tasks using the MNIST and COCO (Microsoft Common Objects In Context) datasets, respectively. Following this, an architecture was developed to replace artificial neurons in the transformer network with spiking neurons. Mixed precision was employed in the design, and the implementation was carried out on both CPU and GPU platforms. Profiling of the program was conducted to identify resource-intensive areas, leading to the proposal of an FPGA model designed to accelerate both training and inference processes. The results showcase significant performance improvements in throughput and a modest enhancement in prediction accuracy compared to certain existing benchmarks. Specifically, the image captioning task achieved approximately 80% accuracy, while the image classification task reached around 93% accuracy.