8 posts
Three surfaces that said OK without running: the backup that skipped a 300MB database, the retention that watched half the files, and eight assertions that never executed.
A scraping bridge asked the database for a column it never had. Nine days of runs, exit code zero, nothing collected. The uncomfortable part: the test suite passed because the fixture carried the exact same mistake as the query. Fixing the bug was the easy half — the lesson was about what a green test actually proves.
For months I looked at the pipeline's green dot as if it were proof. The moment I removed the two lines that swallowed the test suite's exit code, the first honest run in the repo's history returned 8 failures out of 417 tests. The fix wasn't to hide them again: it was to give the debt an address, an owner, and an expiry date.
I built an ecosystem status aggregator and, eighteen minutes after it was born, it couldn't prove its own existence. The fix wasn't code: it was deciding what counts as evidence. A timeout isn't an outage, 429 means alive, a 401 on a gate means the door is shut to people who shouldn't be there, and an open port doesn't mean a running process.
Arachne's response cache returned 'Brasília' to anyone asking about France — cosine 0.8531 against a 0.85 threshold. The story of a bug born in a CI investigation, the math that hid it, and the entity guard that fixed it without sacrificing recall.
On Saturday night I edited a script watched by the file integrity monitor and updated its baseline in the same block. Six hours later the alarm fired again, pointing at the exact change I had just approved. The monitor didn't fail: it did the only job it can do. What was missing was a process that knew how to tell a legitimate edit from a suspicious change.
On August 11 I put a secret scanner on this blog's pipeline. The next day it flagged a build cache file as if it were a key. I silenced it with an entry pinned to that finding's exact fingerprint. Sixteen days later I had to come back to the same spot and write the exception again, this time as policy. That gap is the difference between suppressing an event and defining what should never alarm in the first place.
The Security Agent cross-references findings from two hunters — the Dogwalk Bug Hunter and the Security Hunter watchdog — and outputs a single deduplicated alarm. A health gate to filter infrastructure false positives, discarded network error patterns, local-state dedup, and the no_agent contract: silence when everything is healthy.