TatuEngine — first experiments with wave field theory
TatuEngine·

TatuEngine — first experiments with wave field theory

Context

TatuEngine was born from a silly question: “Do RT Cores process neural networks?” It sounds like a joke, but the idea behind it had real foundation — a neural network is, in the end, a function that transforms an input vector into an output vector. The right question was: “if I treat inference as wave propagation in a phase field, what happens?”

That question started the experiments.

The tools I use on this project live in the skills store — the first one is free.

The original dream: weights as geometry

The initial metaphor was too elegant to abandon:

Ternary weight w ∈ {-1, 0, +1} → lens with η = w
Lens position → f(layer, row, col) in 3D space
Input token → ray with initial phase φ₀
Output → interference of all waves at the detector

Each network weight became a triangle in a BVH. An inference was literally a ray-trace: fire rays, compute phase deviation, sum coherently.

The problem appeared quickly: 1 billion parameters in float32 = tens of GB of lenses. No chance of fitting in consumer GPU VRAM.

The first pivot: hybrid approach

Instead of converting EVERYTHING to geometry (impossible), I split the pipeline: heavy matmuls on GPU (99% of FLOPs), the rest on CPU. This division wasn’t obvious at first — the Mamba SSM is inherently sequential, it doesn’t parallelize over time like attention.

The discovery that changed everything: Block Codec

The hybrid approach worked, but it was still expensive. Then came the insight: ternary weights follow local patterns. Consecutive blocks of weights share zero, sign, and similar magnitude.

Instead of 1 lens per weight in a 1:1 scheme, a compact block fit in a lean block — a fraction of a byte per weight, well under the 1:1 scheme. The traversal kernel processes entire blocks in parallel.

What I learned from these experiments

1. Post-training ternarization destroys coherence

The model I tested (BitMamba-1B) was already natively ternary. I tried ternarizing a float32 model post-training and the logits turned into noise. Ternary models need to be trained that way from the start.

2. Softplus with float32 silently overflows

expf(89.0f) = +inf. This broke my SSM step tests for 2 days until I realized the softplus of A_log was generating inf in the last batches. The fix is the numerically stable formulation.

3. SSM doesn’t carry history in the hidden state

I expected the hidden state h(t) to carry context, like an RNN’s hidden state. But Δh ≈ 0 between consecutive tokens. Mamba’s real memory is in the kv_cache (conv buffer + SSM state). The hidden is almost a pass-through.

What’s next

The hybrid pipeline is functional, but the original vision hasn’t been fully realized: the weights aren’t real geometry during inference — Block Codec is a compressed representation, not an actual optical traversal.

The next step is connecting the MotorGPU with the agent pipeline we built afterward — Immunosystem, associative LTM, GoalStack, ToolUseBuffer. The agent should be able to call tools, remember past sessions, and self-correct the phase field based on feedback.

TL;DR: Started wanting to do ray tracing for neural networks, discovered hybrid SSM is the real path, compressed the model with Block Codec and achieved orders-of-magnitude GPU speedup. TatuEngine is no longer a crazy idea — it’s a functional inference engine that treats network weights as a phase field.

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio