Answers are grounded in this paper and cite their sources.
The software stack for training transformers has moved on from NVIDIA Volta. FlashAttention 2 and 3 do not support sm_70, Volta has no bf16, and Triton's matrix-multiply primitive does not use Volta's tensor cores. As a result, PyTorch on a V100 falls back to slower attention and to unfused elementwise kernels, even though V100s are now among the cheapest datacenter GPUs available. We present hand-written CUDA and Triton kernels for training language models on a single V100: a variable-length causal attention kernel built directly on the mma.sync.m8n8k4 tensor-core instruction with fp32 accumulation, a fused Canon-convolution and rotary-embedding kernel, and a fused cross-entropy head that never materializes the logits. Our attention is 1.9x (forward) and 2.5x (backward) faster than PyTorch's memory-efficient attention. Every kernel we measure for precision (attention, the Canon+RoPE forward and the LM head) matches or beats the stock path's error against an fp64 reference. End to end, pretraining a 143.7M-parameter model runs 1.52x faster than a torch.compile baseline at equal micro-batch. Because the kernels also need less memory, they fit the production micro-batch the baseline cannot, which raises the gain to 1.59x: 59.2k tokens/s, 50% of the card's real tensor-core peak. We also report the optimizations that did not work, with measurements, and an fp16 overflow that ends training runs and that Volta, lacking bf16, cannot sidestep.
No takes yet. Share an insight, caveat, or question.
Joseph Daniel Jones (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: