
The auditor that gave a perfect score without looking at the screen
The dashboard's visual audit gate printed a full score without capturing a single image: stale PNG, wrong interpreter and a prompt that invited false positives.
67 posts

The dashboard's visual audit gate printed a full score without capturing a single image: stale PNG, wrong interpreter and a prompt that invited false positives.

Seven tests failed and pytest aborted the whole run with a WindowsPath error. The cause was a monkeypatch on os.name leaking into the process, and the error surfaced where no test was running.

An idle process wiped 9 destillations from the history. The max() fix created the next defect: a new process increment vanished.

A security patch that swapped the HTTP client of twenty routes passed review, lint and build. The full suite reported 22 failures, 20 of them mine. The worst group was not the one failing loudly, it was the one still green while testing nothing.

Three scripts measured the same checkpoint and silently read a twin frozen two days back. The path looked right, which is why nobody doubted it.

The portfolio suite had 599 green tests and four files that never exercised the code they claimed to cover. One called GET on a route that only exports POST. Another mocked the entire React module, making unreachable the exact case it was supposed to prove.

The Capivara test routine created real invites, and the invite endpoint really sent email. In five days, 410 messages went out to addresses that do not exist, 295 of them bounced. The sending domain is the same one that serves the store, the dog-walking app and the portfolio.

A silent NameError inside the subscription webhook, an idempotency key written too early and a test that required the password reset token in the HTML.

A visual regression test that depended on seeing the login error stopped being trustworthy because the preview server answered 502 on the very route the mock was supposed to intercept. The config now picks the server according to the test, and the Turnstile message stopped describing the system to say what the user must do.

An audit guard demanded four provenance fields on every work plan. I ran it against the archive and found that 1 of 500 plans complied — 0.2%. The cause was not laziness from whoever wrote the plans: the rule existed only on a branch that was never merged. What changes when the rule lands in the contract with a ceiling proportional to plan size, and why adherence went from 0.2% to 15% in three days.

The terminal game was still announcing the old domain months after the migration. I wrote a guard for that class of bug and it started lying in three different ways: it ignored PDF, it could not read PDF text, and in CI it did not have the tool to open the file.

Sixteen coherence checks passed, the gap between CPU and GPU sat in the billionths, and the engine still produced nonsense. The defect only surfaced when a second engine, written by somebody else, read the same model file. A test that compares a system to itself always agrees.

Recent evolution of the blog: accessibility WCAG AA improvements, tag corrections, and the 2-posts-per-day cycle. A journey of quality engineering and consistency.

The E2E suite broke CI twice with the same symptom: the preview server never came up. The first culprit had a name. The second was hiding behind it, with an identical signature. Two defects, one exit code — and a lesson about uptime as evidence.

Two commits on the same day in Dogwalk. The first normalized the booking date to end three formats coexisting in the database. The second undid half of the first, because the normalization reinterpreted the instant and moved the walk three hours. The time contract guard is what stopped it.

The PR looked like a plain bump: the library went to 1.65, the binary stayed on 1.63, and the three CI shards died before running a single test.

Three holes closed in 48 hours in the same verification engine: a null key that in Node collapsed to zero and released any score at all, an inherited environment that made the same commit produce two different results, and a timeout that killed the parent process and left four children burning CPU for over an hour.

Three suites broke on the same day in two different projects, all with the same timeout message — and none of them had a regression. The cause was a host load average of 81. What changes when a test's budget comes from observed latency instead of a number picked in a hurry.

One hundred and two tests passing, and any authenticated account could still read the live GPS of someone else's walk. A static audit of the PataPass backend found three ownership failures the suite never covered.

The banner promised 'Accept all' and stored ads: false. canShowAds() returned false every time, so no ad ever loaded even with AdSense configured. The fix took three commits over two days, and the last one was not about the bug — it was about the wrong promise on the label.

A service area drawn on the map never accepted a single point inside it. The algorithm was right in shape and wrong in its axes: it compared latitude against longitude. Two swapped lines, and PataPass geofencing simply did not exist.

Every check came out approved. Not because the system was right, but because the runner wrote PASS on the report before looking at the result. The defect stayed invisible for weeks: a gate that always agrees is not a gate, it is a rubber stamp.

Eight tests were failing and none of them was a product bug: each one measured the state of the machine instead of the contract. One of them passed alone, passed with a name filter, and failed only on the full suite — because a file left an environment variable changed at import time and nothing undid it.

A correct CSRF guard is useless at the edge if it forgets one hostname: the origin allowlist skipped www, and every signup and login POST died with a 403 before reaching the backend.

In 48 hours, two measurement instruments in the project returned the wrong answer. One listed a third of what existed by sorting text as text. The other fabricated a number where data was missing. The cure for both was the same.

The same change produced three different verdicts across three measurements. A small pilot approved it, another pilot rejected it, and only the full corpus exposed the root cause: the window diluted the embedding of the target turn.

I closed a topology leak in the API health endpoint and, in the same commit, left the Monitoring page showing an empty grid for twelve hours. The page did not break. It lied politely.

A shell block on the last line of an already published post. The same defect sat in 18 posts, and none of my gates looked at the body of the article: one checked tags, another checked PT/EN pairs. I wrote the guard that was missing — and its first version accused 9 innocent posts.

A checker that said OK and exited with code zero without declaring how many checks it had executed. On November 1st I reproduced the defect: three scenarios, three false approvals. What changes when the certificate starts demanding an explicit count.

Two images declared in the PWA manifest had dimensions completely different from what the app actually rendered — and Chrome flagged it in the rich install. The fix involved reading the PNG's binary header.

13 cross links, zero with UTM: the portfolio's measurement arrived without knowing who sent it, and the fix was a 59-line helper that tags every exit.

A transporter could finish the Stripe Connect onboarding and land on a route the app blocks for their role: the callback never ran and the account stayed unable to receive money — with no error shown anywhere.

The Portfólio e2e suite spent a day looking at a stranger's site: the cause was a reused preview on the port under test, and the fix became a port-release loop plus an identity test that fails on the very first navigation.

The router returned success, the mailbox wrote the record, and the message could still end up in a box no process ever read. Three addressing defects and one unconfirmed notice — how Yurumi's agent-to-agent path earned a receipt.

The key loader documented that the process environment beats .env files. The code did the opposite from the very first version — and the only test capable of exposing it ran solely on machines that already had the right answer.

A commit at 2:37 on a June morning made lint and tests non-blocking in the Portfolio CI, with a reasonable justification. One hundred and thirty-two days later, the repository had 14 lint errors and a broken test on master, and the pipeline was still green.

Run 2 proved that TatuEngine learned Portuguese and forgot English. The next two runs proved something else: I was measuring that forgetting on a sample whose error was larger than the effect — 16 rows out of 166.

Three 20 MB bases, 16 WAL segments and two drills that compared table inventory between production and the restored copy. What I learned by measuring recovery instead of trusting the saved file.

The Capivara suite passed in CI and broke in my terminal. The cause was not my code: it was Node 26 declaring a global localStorage that stayed undefined, and jsdom giving up on copying its own.

How Dogwalk evolved in October 2026: 1 379 tests, Stripe Connect and automatic deployment via Cloudflare Pages.

Arachne had a whole feature written, tested and green — and that was never wired up. When I fixed the mounting, it stayed broken with a 422 on every call. Two truths sailed over the same code without touching it: a test that calls the route function directly never sees the assembled app.

Three surfaces that said OK without running: the backup that skipped a 300MB database, the retention that watched half the files, and eight assertions that never executed.

Pinning sqlmodel, a single time helper, and the next-walk hero fix — the tz-aware release from 10/05.

A pip install fixed it. Then a measured bisect showed the line between working and breaking sat in a dependency, not in my code. And along the way, twelve copies of a five-line function became a single helper.

Arachne lets everyone bring their own model key. The guarantee that the key only leaves for its owner relied on an optional field that, when empty, opened the legacy path — and that path was shared per workspace. A one-line condition was the border between your key and your colleague's.

An approved PR, a merge-tree that passes, and the CI still going red afterwards: local work from another session reintroduced the very error the PR had just fixed. The guard that turns that into a rejection before the commit.

How Estudos created a system that validates fixtures against the real database schema, preventing tests from passing by deception for nine days.

A health check that only looked at the HTTP status let truncated and empty responses through for days. The probe that came after measures the body, not the envelope.

Dogwalk's /explorar page gained coverage zones, and the first version carried the link between walker and region on a public page that needs no login. The fix wasn't hiding a field: it was deciding the zone only holds aggregates — and turning the test that required the data into one that requires its absence.

The portfolio test harness measured contrast on a port. When the address came from an environment variable, the branch server never started, and something else answered there. The test never lied. It just answered a different question.

hreflang is the tag that tells Google every page also exists in another language — but it only counts if both sides point back at each other. Here, 350 of those links pointed at addresses that never existed. This post explains what the tag is, how it's supposed to work, and where our assumption broke.

A scraping bridge asked the database for a column it never had. Nine days of runs, exit code zero, nothing collected. The uncomfortable part: the test suite passed because the fixture carried the exact same mistake as the query. Fixing the bug was the easy half — the lesson was about what a green test actually proves.

A week after fixing the Dogwalk endpoints that refused to connect, I learned a repeated handshake is worse than none: the second accept() killed the connection right after it opened. And the first fix still let a dead socket into the room.

The service stopped coming up on an ordinary afternoon while the port guard swore, on every single attempt, that the port was free. 79 restarts in 24 minutes against a reservation no guest-side tool could see. The fix was to change the oracle.

For months I looked at the pipeline's green dot as if it were proof. The moment I removed the two lines that swallowed the test suite's exit code, the first honest run in the repo's history returned 8 failures out of 417 tests. The fix wasn't to hide them again: it was to give the debt an address, an owner, and an expiry date.

The tag pages shipped with no cover art at all, mixed Portuguese with English in the same listing, and threw anyone who switched language straight back to the home page. Three bugs that looked unrelated and came from the same place: the rule lived in the home page, not in the component.

Arachne's response cache returned 'Brasília' to anyone asking about France — cosine 0.8531 against a 0.85 threshold. The story of a bug born in a CI investigation, the math that hid it, and the entity guard that fixed it without sacrificing recall.

Before swapping Arachne's search engine for Yurumi, I had to prove the new one was at least as good as the old. Shadow dual-write, real traffic, overlap@10 — and the number nobody expects: 0.511.

79 tests green one day, 4 failing the next, zero production changes. Arachne stacked three different flavors of flakiness in 48 hours — and each one required a different knife: random vectors, network escaping the mock, and an assert that never ran.

A test was deleting RAG tables on teardown. One run, the environment variable pointing at the isolated database was ignored by the engine — and the DELETE landed on the production arachne.db, with the live server writing to it. What saved the data was an error message.

The status button finally worked within the session — but an F5 wiped it all: a reload brought the walker back offline for the entire system. The direct sequel to the ghost button: real persistence of the online status, from the frontend down to the profile PATCH.

The walker toggle 'Go Online' rendered, animated, and even changed color. But the handler was an empty function: the state never left 'online'. The story of how a test loop caught a ghost button and gave back real online status.

The LifeLog E2E suite started failing with 'Expected "" Received "estudos"' on 24 assertions. It wasn't a visual bug — it was the test helper drifting from production logic: it didn't filter hidden:true posts, and the frontmatter regex didn't match the CRLF line endings of the .mdx files. A story of how a test becomes the source of truth when it stops mirroring production.

After finding 28 hardcoded PT texts scattered across the Portfolio (buttons, aria-labels, modals, tracking), I created an audit test that scans components and fails CI if any Portuguese text appears outside t(). The lesson: old tests were anchored to the bug — they searched for literal PT text instead of the i18n contract.

970 commits, 1456 tests, Stripe Connect, Open Design, and the saga of turning a pet marketplace into a platform that doesn't break on deploy.

How end-to-end tests transformed quality and confidence in Dogwalk's development — the real story of a stack that learned to test itself.

From zero tests to 853: how Playwright tests became the quality backbone of Dogwalk/PataPass, what we learned and what changed in the process.