Hybrid TensorRT–PyTorch Inference Acceleration for Attention-Based CNNs with Dynamic ROI Sampling: A Case Study on Jetson Nano

Edge AI deployment requires low-latency inference on devices with tight constraints in size, weight, power, and cost. NVIDIA TensorRT accelerates inference through layer fusion, precision reduction, and kernel auto-tuning, but a model containing even one unsupported operator cannot be compiled into a single engine. This paper presents a hybrid TensorRT–PyTorch deployment in which the supported subnetworks run as TensorRT engines while the unsupported operations fall back to native PyTorch, and applies it to an attention-based convolutional neural network (ACNN) whose region-of-interest sampling operators are unsupported. On an NVIDIA Jetson Nano, the hybrid approach reduced the end-to-end latency of ACNN-dsc, the fastest native PyTorch model, from 38.37 ms to 20.40 ms (1.88×) under FP32. Under FP16, the hybrid ACNN-k3 achieved the lowest latency of 17.78 ms, a 2.23× speedup over its PyTorch FP32 counterpart and 2.16× over the fastest PyTorch model. Stepwise profiling of the image preprocessing pipeline showed that transferring raw images and performing most conversions on the GPU more than halves preprocessing latency. Further analyses across devices, precisions, and frameworks show that practical latency on edge AI accelerators depends on hardware and runtime factors beyond FLOPs and model size.