TatuEngine: the codec that made the model fit — and the lm_head that wouldn't cooperate
TatuEngine·

TatuEngine: the codec that made the model fit — and the lm_head that wouldn't cooperate

The model that needed to fit

BitMamba-2 1B is a 1.58-bit SSM — weights are ternary: {-1, 0, +1}. On disk, the packed .bin is already small. But inside TatuEngine, the inference representation was a 1:1 BVH: one lens per weight. Each weight became a triangle with position, phase and refraction — and a 1B-parameter model became a structure hundreds of times larger than the file it came from.

It didn’t fit in VRAM. It barely fit in RAM. And it was absurd to load that volume to represent information the disk already stores compactly.

The tools I use on this project live in the skills store — the first one is free.

Context

TatuEngine’s field theory treats inference as optical traversal: a PhaseWave travels through the lens BVH, and each weight contributes a phase shift. More lenses = more expensive traversal — in both memory and time.

The 1:1 path was simple but suicidal:

BVH 1:1:  bursts the GPU's VRAM
BVH block: fits in L2 cache

The idea: instead of one lens per weight, one compact lens per block of ternary weights — 2 bits per weight packing, plus scale and mean η. A small number of bytes in place of 1024 individual lenses.

The struggle

The conversion itself was mechanical: unpack the ternary tensor, group into blocks, store the position in the original tensor. The conversion code pre-computes the ternary values for the whole tensor, then creates the blocks — avoiding re-unpacking per block.

Traversal became a loop with register accumulation — the heart of the lens.

Then came the part that hurt. The test compared the traversal output against the reference bridge_matmul, with memcmp + MSE per tensor:

in_proj:  MSE in the ULP range (rounding differences)
out_proj: MSE in the tenths range (suspected accumulation)
lm_head:  divergence  debug in progress

in_proj came out perfect — rounding differences at the last-place-unit level. out_proj reported MSE in the tenths range, already suspicious: it smelled like accumulation order, not lost information. But lm_head genuinely diverged — and it was the most important tensor, the one mapping to the vocabulary.

Resolution

Even partial, the codec delivered what it promised: the structure shrank by orders of magnitude, and the save/load roundtrip with checksum proved the representation was stable.

Validation didn’t stop at the codec. The same day, Sprint 17 turned block traversal into a CUDA kernel with parallel reduction and optimized BVH loading. The Golden Test — CPU↔GPU coherence — closed with full structural coherence.

And the batch kernel showed why it was worth it: orders of magnitude of speedup on GPU traversal against CPU.

The out_proj lesson confirmed itself in practice: the traversal sums blocks in BlockBVH order, while bridge_matmul sums in original tensor order — different accumulation order, slightly different result. Not lost information, just floating-point algebra. The lm_head stayed as a registered TODO — the GPU golden test ended up validating the structural coherence the codec had introduced.

Metrics

MetricValue
BVH 1:1 (one lens per weight)unworkable — bursts VRAM and RAM
Block BVH (ternary codec)orders of magnitude smaller
in_proj MSE vs bridge_matmulULP range (rounding)
out_proj MSEtenths range (accumulation order)
lm_headdiverged at initial commit (open TODO)
CPU↔GPU coherence (Golden Test)full structural coherence

Takeaways

  • Model compression doesn’t need to be lossy to be radical: ternary weights are already 2 bits — the waste was structural (one lens per weight), not in the bits.
  • Per-tensor validation separates the problems: ULP-perfect in_proj, accumulation-error out_proj and divergent lm_head are three different diagnoses — merging them into one number would hide the story.
  • Accumulation order is the silent villain of any numeric refactor: a tenths-range MSE with block-identical memcmp isn’t a bug — it’s floating point being floating point.
  • Honest commit > perfect commit: the codec’s first commit documented “partial validation” with lm_head open. That became the trail for the next stage (GPU), instead of a buried lie.
  • The lm_head is still on the backlog — the vocabulary blocks deserve their own chapter.

Next chapter is the kernel that makes this traversal fly — and the batch speedup. Spoiler: the lm_head shows up again.

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio