training

3 posts

Sep 14, 2026

Two gigabytes that weren't two gigabytes

I lost 810 training steps because a 2 GB checkpoint was copied straight onto its final name and the machine rebooted mid-copy. I fixed the write on the primary volume. Two days later the same bug was living in the second-drive mirror — and it had gotten worse: the truncated file passed the resume check.

Continue reading →

tatuengine checkpoint training

TatuEngine

Sep 08, 2026

The model that came back from the coma

A catatonic BitMamba 1B — broken tokenizer, 10^18 gradients, a midnight ENOMEM and a CUDA wedge that faked a dead GPU — closed 500 steps and the validation reproduced the eval loss to the fourth decimal. The diary of Phase 4B on a 12 GB RTX 3060.

BitMamba 1B — training the first SSM model

The journey (and the craters) of training a Mamba-2 with 1.4B parameters from scratch on an RTX 3060 — warmup, SFT, catastrophic eval, and Phase 4 of the resurrection.

Continue reading →

tatuengine bitmamba training

Browse by tags

Two gigabytes that weren't two gigabytes

The model that came back from the coma

BitMamba 1B — training the first SSM model