29 posts
Arachne had a whole feature written, tested and green — and that was never wired up. When I fixed the mounting, it stayed broken with a 422 on every call. Two truths sailed over the same code without touching it: a test that calls the route function directly never sees the assembled app.
A pip install fixed it. Then a measured bisect showed the line between working and breaking sat in a dependency, not in my code. And along the way, twelve copies of a five-line function became a single helper.
A single config line with a placeholder where the key should be killed production login and signup for almost nine hours — and the real key existed nowhere at all. The story of the best-intentioned lockout I ever shipped, and of the doctor that diagnoses a pair without ever seeing the secret.
A week after fixing the Dogwalk endpoints that refused to connect, I learned a repeated handshake is worse than none: the second accept() killed the connection right after it opened. And the first fix still let a dead socket into the room.
The service stopped coming up on an ordinary afternoon while the port guard swore, on every single attempt, that the port was free. 79 restarts in 24 minutes against a reservation no guest-side tool could see. The fix was to change the oracle.
Two machines, the same code, the same key. One got 200, the other took a 403 with error code: 1010. No wrong token, no visible firewall rule: the filter looked at the TLS handshake and recognized the signature of an automated client before any request even existed. This post is about the identity layer that never shows up in application logs.
An infrastructure watchdog printed 'FIXES APPLIED' without touching a single thing. Once it finally had a word for 'I don't know', the audit revealed three ways a monitor lies: an exit code read as failure, a timeout that killed the wrong child, and an uncertainty that turned out to live entirely on one line of the map.
Capivara's chat started answering without sources: HTTP 200, a polished sentence, zero memory. Restarting the embedding service looked like the obvious fix — and was precisely what kept it from coming back. Underneath, two defects from the same family: a process that only opens its port after waking up, and a fallback that doesn't know how to undo itself.
The embedder was not broken, it was loading. A one-way latch turned a cold load of minutes into a system that stayed broken until the next restart.
An LLM call ends in 200 OK with empty content and the pipeline moves on like nothing happened. Diagnosis: reasoning models burn the token budget thinking and nothing is left for the answer. The fixes: a token ceiling with real margin, and a parser that tolerates a data: [DONE] glued to the JSON.
79 tests green one day, 4 failing the next, zero production changes. Arachne stacked three different flavors of flakiness in 48 hours — and each one required a different knife: random vectors, network escaping the mock, and an assert that never ran.
A test was deleting RAG tables on teardown. One run, the environment variable pointing at the isolated database was ignored by the engine — and the DELETE landed on the production arachne.db, with the live server writing to it. What saved the data was an error message.
A catatonic BitMamba 1B — broken tokenizer, 10^18 gradients, a midnight ENOMEM and a CUDA wedge that faked a dead GPU — closed 500 steps and the validation reproduced the eval loss to the fourth decimal. The diary of Phase 4B on a 12 GB RTX 3060.
The Arachne landing page loaded without CSS, the watchdog restarted the service in an invisible loop, and a zombie socat relay kept stealing the port from its rightful owner. Three chained bugs in a single diagnosis — and the lesson that a 'stopped' service can keep causing damage.
300 training steps, a loss stuck at 10.8125, and a model that insisted on coming out blank. The cause wasn't a low learning rate — it was exploded weights in the starting checkpoint. The story of how a stalled run became a full diagnosis.
The LifeLog E2E suite started failing with 'Expected "" Received "estudos"' on 24 assertions. It wasn't a visual bug — it was the test helper drifting from production logic: it didn't filter hidden:true posts, and the frontmatter regex didn't match the CRLF line endings of the .mdx files. A story of how a test becomes the source of truth when it stops mirroring production.
Local services hanging intermittently: Qdrant on port 6333, helper APIs — everything pointing to 'localhost'. On Windows, 'localhost' resolves to IPv6 (::1) first — and the connection hangs. '127.0.0.1' forces IPv4 and works. Confirmed 88 times until it became a rule: always use 127.0.0.1.
After 5 days making the circle start from the click on desktop, mobile still used a crossfade — by documented decision: clip-path paints per frame and janks weak GPUs. Until Samuel asked for the opposite. What made the reversal possible? The anti-stutter fixes that were already in global.css.
TatuEngine had 5.7GB of AI models committed to git since day one. GitHub rejects objects above 2GB with a 422 error — that push would never succeed. The fix wasn't 'pushing through': it was an orphan branch that preserved code, tests, and docs and rewrote the repository's history from scratch.
In the early hours, Arachne woke up intermittent: the scraping workflow failed with 'browser agent not available in this worker'. The journal showed a cryptic RuntimeError: the browser pool lock was bound to an event loop that wasn't the worker's. The cause: asyncio.Lock() created at import time. The fix: lazy init — born inside the right loop.
Arachne responded 200, but authenticated routes rendered empty. A Bug Hunter found 1,239 QueuePool errors in the journal since 02:57. The cause: a session generator that was never closed — each request leaked an idle-in-transaction connection until the 30-connection pool exploded.
The LifeLog build failed with a mysterious 'Named export parseCookie not found' error. The code was right, the CI passed, but local builds broke. The cause: a 253MB node_modules in ~/ with [email protected] hijacking module resolution for EVERY Node project on the WSL.
Five days trying to make a circle of light start exactly where the finger touches. The CSS looked right, CI passed, but the user saw stutter, wrong origin, and content wiping out. The root cause wasn't the code — it was the interaction between the View Transitions API, Chromium blend modes, and a ghost node_modules.
All Dogwalk WebSockets returned HTTP 500 on handshake. The app ran, the database responded, but real-time login, chat and walk tracking simply wouldn't connect. The cause: a receive_text() called without the accept() Starlette requires.
Broken Pix BR Code, focus trap stealing keyboard input, scroll jumping sections — a deep audit fixed everything at once.
Hermes kept entering a continuation loop 4 times before giving up. The cause: Ollama, by default, only allocates 4096 context tokens — regardless of the model supporting 262K. The hard way: num_ctx of the model ≠ num_ctx of the inference slot.
I added Pix to my portfolio with BR Code generation. Only the QR Code didn't work in any banking app. The culprit: String.fromCharCode generating control characters instead of decimal digits in EMV 2022 length fields.
How a public endpoint that returned 401, a bundle that crashed, and 42 console errors became a single commit
YAML indentation, missing dependencies, tests that need a running server — the classic pipeline debugging spiral.