The credential nobody had ever tested

The line that was active

Every project with a captcha carries two halves of the same lock. The first is public, ships inside the HTML and any visitor can read it. The second stays on the server and confirms the challenge was actually solved. Whoever publishes the front end needs the first; whoever validates the token on the backend needs the second.

In Dogwalk those halves lived in different files from day one: the public key in the front-end environment, the secret in the file the API service reads at boot. That is where the trap sat, because the service was written to fail closed. Without the variable it refuses every captcha submission. With the wrong variable it also refuses. As far as the code can tell, “misconfigured” and “not configured” are the same state — and the user sees the same screen.

On September 15, at 15:08, one environment line went live holding the example value. It was not a secret; it was a reminder of where the secret was supposed to go. Thirty-three minutes later the service restarted and login and signup began returning 403. Not because the password was wrong, but because the captcha was never approved. Nine hours of production with the door locked and the key on the inside.

What the sweep found

We commented the line, restarted, and production recovered. But sleep was still off the table: the real secret was nowhere we could point at. What existed was a value stored in the repository vault on June 15, which had never been through a test — only through a copy and paste.

We swept twelve sources: the disks of both systems, the shell history, the git history with content search, the runner’s diagnostic directory, the backups, the exported variables, twelve edge-provider tokens on the machine — global key, service tokens, account tokens — and the metadata of the repository secrets. The conclusion was uncomfortable: the valid secret did not exist anywhere reachable.

Then came the right question. If it does not exist, what is that June value that has been sitting there for four months? It looks like a credential. It has the shape of one. Nobody ever asked whether it still did anything.

The fake pair that proves the truth

Captcha validation has a useful property: it separates the reasons for rejection. If the secret does not exist on the other side, the answer is invalid-input-secret. If the secret does exist and the token you sent is garbage, the answer is invalid-input-response. So you can ask “is this secret real?” by sending a fake token and reading which of the two errors comes back.

The trick is silly and it is exactly what we needed: it proves a credential exists without knowing it and without using it for anything. No usage record, no access to anyone’s data. Just a fake pair returning the difference between “does not exist” and “exists, but you sent garbage”.

The first fact was about size. The June value has 34 characters; the official shape for these secrets has 40. Thirty-four is not a shorter credential, it is something else — and no code ever complained, because nobody was comparing. The second fact closed the case: the verification service answered invalid-input-secret. That credential was dead, and the CI had been carrying it for four months believing it was alive.

The worse finding

With the method in hand we looked at the other half. The repository stored one public key, the deployed front end used another, and the hash of the CI value matched the test key — the one the documentation offers for development and which approves any challenge without looking. Worse: the flag the end-to-end suite uses to skip the captcha was switched on in the production environment.

That changes how the incident reads. Production was not protected and broken at the same time; it was unprotected and passing its tests precisely because of that. Any “captcha active” status report was true in name only.

What stayed

Two small tools and one rule. The first is a CI job that asks the same fake-pair question and prints only the error code and the hash of what it tested, never the value. The second is a diagnostic script that finds the widget through the provider API, checks the shape before anything else, repeats the test, and only then writes the environment file. If the test does not confirm the pair, it aborts without writing. If the file has more than one line for the same variable, it aborts too.

The rule is what closes the loop: a credential only reaches the environment after its effect has been measured. Existing in a vault is not evidence of anything. Having the right shape is not either. The only evidence is the answer from the service that will use it.

One problem stayed open: no token on this machine has read or write permission on the captcha product — they all return the same permission-denied codes. Creating the final widget and rotating the secret takes one click in the dashboard with a fresh token. Until that happens the path is mapped and the tools are planted. Which is considerably better than yesterday’s situation: a secret in the vault, an example value in the environment, and nobody able to say which of the two was lying.

# the fake pair: proves the secret exists without using it
curl -s https://challenges.cloudflare.com/turnstile/v0/siteverify \
  -d "secret=$SECRET" -d "response=fake-pair-probe" | head -c 200

# shape before anything else: 40 is the official one
printf %s "$SECRET" | wc -c

# and the real state of the service that depends on it
curl -s -o /dev/null -w "%{http_code}\n" "$API/health"