pytorch

3 posts

Sep 06, 2026

The training that woke up slow — the inheritance hidden inside a checkpoint

I resumed a run from the last checkpoint and it looked healthy — steps advancing, gradient in range, zero errors. Except the learning rate had quietly shrunk by three orders of magnitude. The cause: the LR scheduler restores more than it should when loading old state, and the fix was two lines.

Continue reading →

tatuengine fase4 pytorch

Step 322 — when training dies in silence and VRAM is left orphaned

TatuEngine

Sep 03, 2026

Step 322 — when training dies in silence and VRAM is left orphaned

The Phase 4 run dropped mid-flight with no traceback, no OOM, no error log: the process simply stopped existing. Chasing the cause turned into a lesson about watching the wrong metrics — and about the invisible cost of letting your GPU driver sweep up your garbage.

Continue reading →

tatuengine fase4 pytorch

TatuEngine

Aug 30, 2026

TatuEngine: the lm_head Returns — the Audit That Unlocked Phase 4

The codec post ended with a dangling issue: the lm_head diverging in validation. Two weeks later, a methodical audit rescaled the model's head, generated a clean checkpoint, and became the foundation of Phase 4 — the Resurrection.

Continue reading →

tatuengine lm-head checkpoint

Browse by tags

The training that woke up slow — the inheritance hidden inside a checkpoint

Step 322 — when training dies in silence and VRAM is left orphaned

TatuEngine: the lm_head Returns — the Audit That Unlocked Phase 4