Edge-optimized threat detection with Florence v2 for autonomous surveillance in resource-constrained environments

This research presents a portable, edge-optimized system designed to overcome the limitations of traditional surveillance in detecting modern threats such as disguised individuals, hidden weapons, and flying drones, particularly in environments with limited resources. The system leverages a modified Florence v2 Vision-Language Transformer for military-grade object detection, with its core deployed on a Raspberry Pi 5. A separate ESP32-S2 DevKit handles wireless alerts using the ESP-NOW protocol for fast, local communication. To achieve high accuracy with low computational overhead, we adapted the Florence v2 model using Parameter Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA). The model was trained on a dataset of 12,000 images, augmented with manual annotations and synthetic images generated by Generative Adversarial Networks (GANs). The final model achieved an overall accuracy of 88.01%, outperforming other popular models such as You Only Look Once version 8 (YOLOv8) and Fast Region-based Convolutional Neural Network (FRCNN). Our model achieved a Mean Average Precision (mAP@50) score of 64.83, compared to YOLOv8 (64) and FRCNN (56). After training, the model was optimized through quantization and pruning to ensure efficient execution on edge hardware. It processes a frame in 16.6 s, utilizing only 38.4% of the Raspberry Pi’s RAM and 50.9% of its CPU. The integrated ESP32-S2 module enables the system to transmit critical alerts over long distances without the need for cloud connectivity or a central server. This system demonstrates the capability of autonomous, real-time surveillance across critical domains including military borders, urban airspaces, and civilian infrastructure.