Ember: Freeze-Survivable Pretraining of a 124M-Parameter Language Model on a Single Unified-Memory Accelerator

We report Ember, a from-scratch pretraining run of a 124-million-parameter GPT-2-class transformer on a single NVIDIA GB10 "DGX Spark" desktop accelerator (128 GB unified LPDDR5x) that simultaneously serves a co-resident inference workload. On this shared box, over-committing the single physical memory pool does not raise a catchable error — it livelocks the whole machine — and it had already forced a hard power-cycle twice under earlier workloads. That recurring failure, not foresight, motivated a freeze-survivable protocol (atomic checkpointing every 250 steps plus transparent auto-resume); when a routine co-tenancy event drove memory to zero again shortly after step 6,000, the protocol absorbed the resulting mid-flight kill with no discontinuity in the loss curve. Over 10,000 steps the model consumed 1.31 billion tokens of FineWeb-Edu (~half the Chinchilla-optimal budget), and its training loss fell along a clean power law to ≈5.1 nats. It would have been easy to stop there. Instead we ran a full post-hoc evaluation on the checkpoint — held-out loss, LAMBADA, HellaSwag, WikiText, calibration, per-position and frequency-stratified loss, embedding geometry, and degeneration metrics — and it uncovered the paper's central finding: a train/eval objective mismatch. A double target shift (the data loader shifts targets by one, and the framework's loss shifts them again) meant Ember was trained to predict the token two ahead, not the next token. The loss curve was blind to this for the entire run, because it consistently measured that shifted objective; only held-out benchmarks exposed it — as a next-token language model Ember scores at chance on LAMBADA (0.5%) and HellaSwag (26.5%) and worse-than-bigram perplexity (986), while under its as-trained objective it is coherent (perplexity 152) and remarkably well calibrated (ECE 0.014). We present this as a case study in why a healthy training curve is necessary but not sufficient, alongside the systems and scaling results it does not invalidate: a backtested single-run scaling fit with a pre-registered prediction, verified 15–19% MFU accounting, well-structured learned embeddings, and a near-zero generalization gap that confirms the single-pass no-memorization regime. We situate all of it in the training-run-report literature (Pythia, OPT, Cerebras-GPT, Llama-3's failure statistics) and specify the corrected compute-optimal follow-up.