This preprint presents a reproducible measurement protocol --- not a set of measured results --- for benchmarking the inference efficiency of three sub-3B open-weight instruction-tuned large language models on a free NVIDIA T4 GPU provided by Google Colab. The three models targeted are meta-llama/Llama-3. 2-1B-Instruct (1. 24B parameters), Qwen/Qwen2. 5-1. 5B-Instruct (1. 54B parameters), and google/gemma-2-2b-it (2. 61B parameters), all loaded in BF16 precision with the default Hugging Face Transformers backend. The protocol collects three metrics per model: (1) peak VRAM, measured via torch. cuda. maxₘemoryₐllocated () ; (2) time-to-first-token (TTFT), measured as the wall-clock latency from input submission to the first generated token; and (3) tokens per second (TPS), measured as the steady-state decode-phase throughput with the prefill phase excluded. To ensure reproducibility, the protocol uses three warm-up runs and three measured runs per model, reports the median, and uses greedy decoding for bit-identical token sequences across sessions. To make the protocol concrete, the preprint additionally provides a table of reference estimates for the three target models. These estimates are derived transparently from each model's published parameter count, the BF16 weight size (2 bytes per parameter), the T4 GPU's published memory bandwidth (320 GB/s), and the attention-head configurations described in each model's technical report. They are clearly marked with a dagger in the preprint and are NOT measured values. The companion Colab script (benchmarkcolab. py, included as a supplementary file) produces measured values on a real T4 GPU, which should replace the reference estimates before the preprint is cited in downstream work. The preprint is deliberately framed as a protocol paper with reference estimates rather than as a benchmark paper with measured results, because the latter would require execution of the script on real hardware. The analysis framework maps the architectural differences between the three model families (grouped-query attention head counts, sliding-window attention, parameter count) onto testable hypotheses about each metric, so that the numbers produced by the protocol can be interpreted in terms of the underlying design choices rather than treated as opaque. Limitations are stated explicitly: the numerical values are reference estimates not direct measurements; only the default Transformers backend is benchmarked (no vLLM or TensorRT-LLM) ; only a single prompt length is tested; and absolute TPS values are not directly transferable to HBM-equipped data-center GPUs, although the relative ranking of the three models is expected to be preserved.
Anamay Tripathy (Fri,) studied this question.