Answers are grounded in this paper and cite their sources.
Most people use AI by renting it from a cloud service, paying per token or per month and sending their data off the device. We ask whether low-cost local hardware can do the job instead, using only free data and open tools. Study A analyses 243 de-duplicated MLPerf Tiny results (v1.0 to v1.3) measured on microcontrollers, neural processors and FPGAs. Int8 made Cortex-M inference 1.9x to 3.8x faster than float32 with no loss in classification accuracy, a dedicated NPU spotted keywords 25x faster than the median Cortex M4, and energy per inference tracked latency with a log-log slope of 0.97 (R² = 0.72). Study B trains three transformer language models (0.8M to 6.4M parameters) on a 2-core CPU with a dependency-free NumPy implementation and evaluates post-training quantization from 16 to 1.58 bits. Four-bit weights in blocks of 32 raised validation loss by only 0.09% to 0.55% while shrinking the models 6.4x to 6.9x; below 4 bits the loss depended on block size, model scale and layer type, and clipping-based scale search reduced weight error yet increased model error. Portable C kernels decoded 0.4B- and 1.1B-scale weight geometries 2.7x faster at 4 bits than at FP32. Roofline, energy and cost models translate these results into bandwidth requirements, energy per token and the volume at which owning hardware beats renting a service.
No takes yet. Share an insight, caveat, or question.
Waghmare et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: