
TatuEngine: the codec that made the model fit — and the lm_head that wouldn't cooperate
The model that needed to fit
BitMamba-2 1B is a 1.58-bit SSM — weights are ternary: {-1, 0, +1}. On disk, the packed .bin is already small. But inside TatuEngine, the inference representation was a 1:1 BVH: one lens per weight. Each weight became a triangle with position, phase and refraction — and a 1B-parameter model became a structure hundreds of times larger than the file it came from.
It didn’t fit in VRAM. It barely fit in RAM. And it was absurd to load that volume to represent information the disk already stores compactly.
The tools I use on this project live in the skills store — the first one is free.
Context
TatuEngine’s field theory treats inference as optical traversal: a PhaseWave travels through the lens BVH, and each weight contributes a phase shift. More lenses = more expensive traversal — in both memory and time.
The 1:1 path was simple but suicidal:
BVH 1:1: bursts the GPU's VRAM
BVH block: fits in L2 cache
The idea: instead of one lens per weight, one compact lens per block of ternary weights — 2 bits per weight packing, plus scale and mean η. A small number of bytes in place of 1024 individual lenses.
The struggle
The conversion itself was mechanical: unpack the ternary tensor, group into blocks, store the position in the original tensor. The conversion code pre-computes the ternary values for the whole tensor, then creates the blocks — avoiding re-unpacking per block.
Traversal became a loop with register accumulation — the heart of the lens.
Then came the part that hurt. The test compared the traversal output against the reference bridge_matmul, with memcmp + MSE per tensor:
in_proj: MSE in the ULP range (rounding differences)
out_proj: MSE in the tenths range (suspected accumulation)
lm_head: divergence debug in progress
in_proj came out perfect — rounding differences at the last-place-unit level. out_proj reported MSE in the tenths range, already suspicious: it smelled like accumulation order, not lost information. But lm_head genuinely diverged — and it was the most important tensor, the one mapping to the vocabulary.
Resolution
Even partial, the codec delivered what it promised: the structure shrank by orders of magnitude, and the save/load roundtrip with checksum proved the representation was stable.
Validation didn’t stop at the codec. The same day, Sprint 17 turned block traversal into a CUDA kernel with parallel reduction and optimized BVH loading. The Golden Test — CPU↔GPU coherence — closed with full structural coherence.
And the batch kernel showed why it was worth it: orders of magnitude of speedup on GPU traversal against CPU.
The out_proj lesson confirmed itself in practice: the traversal sums blocks in BlockBVH order, while bridge_matmul sums in original tensor order — different accumulation order, slightly different result. Not lost information, just floating-point algebra. The lm_head stayed as a registered TODO — the GPU golden test ended up validating the structural coherence the codec had introduced.
Metrics
| Metric | Value |
|---|---|
| BVH 1:1 (one lens per weight) | unworkable — bursts VRAM and RAM |
| Block BVH (ternary codec) | orders of magnitude smaller |
| in_proj MSE vs bridge_matmul | ULP range (rounding) |
| out_proj MSE | tenths range (accumulation order) |
| lm_head | diverged at initial commit (open TODO) |
| CPU↔GPU coherence (Golden Test) | full structural coherence |
Takeaways
- Model compression doesn’t need to be lossy to be radical: ternary weights are already 2 bits — the waste was structural (one lens per weight), not in the bits.
- Per-tensor validation separates the problems: ULP-perfect
in_proj, accumulation-errorout_projand divergentlm_headare three different diagnoses — merging them into one number would hide the story. - Accumulation order is the silent villain of any numeric refactor: a tenths-range MSE with block-identical
memcmpisn’t a bug — it’s floating point being floating point. - Honest commit > perfect commit: the codec’s first commit documented “partial validation” with lm_head open. That became the trail for the next stage (GPU), instead of a buried lie.
- The
lm_headis still on the backlog — the vocabulary blocks deserve their own chapter.
Next chapter is the kernel that makes this traversal fly — and the batch speedup. Spoiler: the lm_head shows up again.
Related entries
Same project, same ground — the stories connect.
Next step
If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.
Done — I will send the next article when it is out.
No spam, one-click unsubscribe. I am also here: Skills store · Portfolio