Adaptive inference can reduce the nominal cost of edge vision systems, but its deployment cost depends on runtime effects that are not visible in operation counts. This article characterizes these effects on a single commercial edge-NPU platform: an embedded Arm evaluation board running TensorFlow Lite with the vendor's NPU delegate. We measure four variables relevant to adaptive vision inference: first-invocation cost versus steady-state latency, co-resident model switching, INT8 quantization format, and CPU–NPU output consistency. The results show that the NPU path has low steady-state latency once warm, but first invocation can be second-scale and must be amortized. Co-resident cascades are physically feasible and switching between warm models is inexpensive, but the break-even estimate is only a simplified steady-state, latency-only approximation. Quantization format can dominate the deployment trade-off: for MobileNetV2, the accuracy-safe per-channel model is slower than the per-tensor model in this setting. We distill these measurements into deployment decision boundaries—amortization counts, switch-overhead scales, and a quantization-format gate—for adaptive inference on the studied edge-NPU platform, and release the measurement harness and data as a public artifact.
Jiadong Liu (Fri,) studied this question.