Batch Scaling and Goodput of a Tuned Attention Kernel on TPU v6e: Throughput, Latency, Energy, and Cost Measurements of vLLM Serving Gemma 4 31B

This is the third study in a measurement series on large language model inference efficiency on Google TPUs. The first study (DOI: 10.5281/zenodo.21212010) recovered a factor of 12.7 in throughput on a single TPU v5e chip serving Gemma 2B, partly by retuning the decode block size of the Ragged Paged Attention v3 kernel. The second study (DOI: 10.5281/zenodo.21404069) showed that the shipped block size heuristic costs 27.7 to 68.7 percent of achievable throughput on a sharded TPU v6e slice serving Gemma 2 27B and Gemma 4 31B. Both studies left specific questions open, stated in their own limitation sections, and this paper closes them. On a Cloud TPU v6e slice of four chips (tensor parallelism TP=4) serving Gemma 4 31B through vLLM, with the kernel override held at its measured optimum, the paper reports the following. Throughput saturates at 256 concurrent sequences (7,375 tokens per second); beyond that point tokens per second stay roughly flat while time per output token doubles with every doubling of the batch, so goodput under a 50 ms per token service objective falls to zero exactly where raw throughput is still near its maximum, making batch 256 the optimum on both axes. The kernel override outperforms the shipped heuristic by 56.6 to 59.6 percent at the largest batches, a 36 to 37 percent reduction in cost per generated token at identical output, since the token streams are identical by construction. A long context probe reverses the result: with 1,500 token histories the shipped heuristic is 28 percent faster than the small block, locating the regime boundary that both prior studies could only predict. An experimental SparseCore collective offload, found to be inactive by default for decode scale collectives, changed throughput by less than 0.2 percent when enabled. The measurements are converted into cost per million tokens, energy bounds per token, and capacity in concurrent streams within a latency objective, with all pricing and power assumptions stated and separable from the measured ratios. All results derive from timestamped logs and per run JSON exports at a run to run coefficient of variation at or below 0.18 percent. The upload includes the compiled paper and the complete LaTeX source with the figure generation script. The measurement harness (provisioning scripts, kernel substitution and patch, campaign drivers, and the parser that reconstructs every table from the JSON records) is described in Section 10.