Edge AI is redefining the deployment of computer vision systems by enabling real-time inference directly on resource-constrained edge devices. This shift offers significant advantages in terms of reduced latency, data privacy, and operational autonomy in bandwidth-limited computing environments. In this paper, we present a systematic performance benchmarking of multiple variants of YOLOv8 on the NVIDIA Jetson Orin NX platform, focusing on object detection tasks. We evaluate inference latency, frame throughput, and computational resource usage across varying input sizes and model complexities. Furthermore, we validate the deployment effectiveness through practical use cases, such as vehicle and package detection. Our findings show that the TensorRT model outperforms PyTorch by 17.7% at a batch size of 2, although PyTorch presents greater stability at larger batch sizes (e.g., 8), where TensorRT encounters resource constraints. In terms of memory usage, it increases linearly with batch size: 69% from batch 1 to 4, with TensorRT requiring 429.20 MB at batch size 2 compared to PyTorch’s 451.24 MB. Furthermore, the processing time per image decreases by 42% when scaling from batch size 1 to 4, highlighting a critical saturation point for edge resources. In summary, the results provide insight into the trade-offs between model size and speed, offering guidance for selecting detection architectures tailored to real-time edge applications.
Aljami et al. (Sun,) studied this question.