4 posts
I lost 810 training steps because a 2 GB checkpoint was copied straight onto its final name and the machine rebooted mid-copy. I fixed the write on the primary volume. Two days later the same bug was living in the second-drive mirror — and it had gotten worse: the truncated file passed the resume check.
I resumed a run from the last checkpoint and it looked healthy — steps advancing, gradient in range, zero errors. Except the learning rate had quietly shrunk by three orders of magnitude. The cause: the LR scheduler restores more than it should when loading old state, and the fix was two lines.
The Phase 4 run dropped mid-flight with no traceback, no OOM, no error log: the process simply stopped existing. Chasing the cause turned into a lesson about watching the wrong metrics — and about the invisible cost of letting your GPU driver sweep up your garbage.
The codec post ended with a dangling issue: the lm_head diverging in validation. Two weeks later, a methodical audit rescaled the model's head, generated a clean checkpoint, and became the foundation of Phase 4 — the Resurrection.