One person. One borrowed TPU. 47 starts.

The grant ends

Google's TPU Research Cloud grant formally expires and a renewal is declined for capacity. The TPU keeps running because nobody reclaims it. From here on, every day of training is borrowed time.

Stage 1 begins

A 1.6B-parameter model, 243,000 steps planned, about 2.1 million tokens a step, 13.3 seconds a step on a TPU v4-32 with four hosts. The first log file was not kept; the record starts at step 28,650 on August 23.

The first scar

Five restarts in sixteen hours from the step 38,000 checkpoint. Checkpoints are written every 1,000 steps, so each crash throws away up to four hours of work.

Thirteen deaths at one step

A failed read leaves a page of zeros in the data mount's cache. One worker reads it and quits, the other three hang at the barrier, systemd restarts everything from the last checkpoint, and it dies at the same page again. Thirteen times. The log tail looks healthy the whole time; the only real signal is that the newest checkpoint folder stopped advancing. Thirty hours pass before the cause is found by comparing bytes at the failing offset against the cloud copy.

The restart guard

The same bug, the same day, three more deaths and nine hours. After this a guard is written: a state file counts starts without a new checkpoint, and on the third one the script remounts the data and restarts itself.

The billing lapse

The billing account goes delinquent and cloud storage starts returning 403. The run dies at step 122,733 looking exactly like the mount bug, so the first fix is the wrong fix. Paid, the account reopens 24 minutes after the run died, and the next automatic restart picks up from step 122,000.

Three more rounds

The zero-page bug comes back on September 10 and 21, and single crashes land on the 15th and 18th. The guard catches the last one on its third start, as designed. Four incidents and roughly 50 hours lost over the run. Every one of them is on the curve below.

Stage 1 ends, and the plan is found broken

Step 198,999 at 14:53 UTC. The notes say PRETRAIN FINISHED. That afternoon it turns out the launcher for stages 2 and 3 was lost in a laptop migration and only stage 1 ever ran, so the learning rate never came down. The stage 1 checkpoint is eleven hours from automatic deletion; it gets copied out by hand, and stage 2 starts at 22:29 the same night with far more math and code.

Stage 3 and the truncation default

Stage 2 ends on the 26th. Before stage 3, a framework default is found cutting every document at 4,096 tokens, which had been throwing away the answers on the long reasoning data. Fixing it breaks the saved data-iterator state; the state gets patched by hand after a test on a toy pipeline. Stage 3 runs the cool-down and finishes on September 30 at 02:48 UTC with a clean exit.

The bug that hid the scores

While chasing why the model answered a counting question with nothing at all, the export script is found shipping a repetition penalty in the generation config. The evaluation harness kept it; none of the baselines had one. Every generation score had been understated for weeks. Removed, counting accuracy goes from 4.0 to 41.3. The same day the 32K context phase ends and the 64K phase begins.

The last checkpoint

Phase C, 64K context with synthetic long-document tasks, writes its final checkpoint at 15:24 UTC. 524 billion tokens seen. Chat fine-tuning starts the same day: nine passes by October 6.

Ten benchmarks, one harness

Every model in the tables runs through the same evaluation harness on the same machine. The home 4090 goes offline overnight and the last runs finish on a second box. Five wins each against Meta's Llama 3.2 1B, which trained on about 18 times more data.

Out to 128K

The long-context runs, 4K to 128K, against the same baseline. Ahead at four lengths, behind at one, level at the last.

Release

Weights, intermediate checkpoints, the chat preview, GGUF builds, training and evaluation code, raw evaluation outputs and the full report, all Apache 2.0. Out of pocket for the whole run: about $320. The TPU was a grant. The rest was one person and time.