
TatuEngine — first experiments with wave field theory
Context
TatuEngine was born from a silly question: “Do RT Cores process neural networks?” It sounds like a joke, but the idea behind it had real foundation — a neural network is, in the end, a function that transforms an input vector into an output vector. The right question was: “if I treat inference as wave propagation in a phase field, what happens?”
That question started the experiments.
The tools I use on this project live in the skills store — the first one is free.
The original dream: weights as geometry
The initial metaphor was too elegant to abandon:
Ternary weight w ∈ {-1, 0, +1} → lens with η = w
Lens position → f(layer, row, col) in 3D space
Input token → ray with initial phase φ₀
Output → interference of all waves at the detector
Each network weight became a triangle in a BVH. An inference was literally a ray-trace: fire rays, compute phase deviation, sum coherently.
The problem appeared quickly: 1 billion parameters in float32 = tens of GB of lenses. No chance of fitting in consumer GPU VRAM.
The first pivot: hybrid approach
Instead of converting EVERYTHING to geometry (impossible), I split the pipeline: heavy matmuls on GPU (99% of FLOPs), the rest on CPU. This division wasn’t obvious at first — the Mamba SSM is inherently sequential, it doesn’t parallelize over time like attention.
The discovery that changed everything: Block Codec
The hybrid approach worked, but it was still expensive. Then came the insight: ternary weights follow local patterns. Consecutive blocks of weights share zero, sign, and similar magnitude.
Instead of 1 lens per weight in a 1:1 scheme, a compact block fit in a lean block — a fraction of a byte per weight, well under the 1:1 scheme. The traversal kernel processes entire blocks in parallel.
What I learned from these experiments
1. Post-training ternarization destroys coherence
The model I tested (BitMamba-1B) was already natively ternary. I tried ternarizing a float32 model post-training and the logits turned into noise. Ternary models need to be trained that way from the start.
2. Softplus with float32 silently overflows
expf(89.0f) = +inf. This broke my SSM step tests for 2 days until I realized the softplus of A_log was generating inf in the last batches. The fix is the numerically stable formulation.
3. SSM doesn’t carry history in the hidden state
I expected the hidden state h(t) to carry context, like an RNN’s hidden state. But Δh ≈ 0 between consecutive tokens. Mamba’s real memory is in the kv_cache (conv buffer + SSM state). The hidden is almost a pass-through.
What’s next
The hybrid pipeline is functional, but the original vision hasn’t been fully realized: the weights aren’t real geometry during inference — Block Codec is a compressed representation, not an actual optical traversal.
The next step is connecting the MotorGPU with the agent pipeline we built afterward — Immunosystem, associative LTM, GoalStack, ToolUseBuffer. The agent should be able to call tools, remember past sessions, and self-correct the phase field based on feedback.
TL;DR: Started wanting to do ray tracing for neural networks, discovered hybrid SSM is the real path, compressed the model with Block Codec and achieved orders-of-magnitude GPU speedup. TatuEngine is no longer a crazy idea — it’s a functional inference engine that treats network weights as a phase field.
Related entries
Same project, same ground — the stories connect.
Next step
If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.
Done — I will send the next article when it is out.
No spam, one-click unsubscribe. I am also here: Skills store · Portfolio