The monitoring page that went blind
Arachne·

The monitoring page that went blind

An anonymous GET was describing my infrastructure

On November 4th, at 12:30, I closed a leak in Arachne that bothered me more for its kind than its size. The API health endpoint answered any unauthenticated request with a full portrait of the ecosystem: whether the database was up, whether the auxiliary service responded, which language models were loaded, and even server filesystem paths. None of it was a cryptographic secret, and that is precisely why it survived so long: it does not look serious. But it is exactly the kind of information that saves hours of reconnaissance for someone looking for a way in. A GET without a token handed over the internal topology for free.

The fix followed the design I should have used from the start: the public endpoint now returns the minimum — {"status": "ok"} — and all subsystem detail moved to a separate route behind authentication. Whoever needs the full portrait is the internal panel, which already has a session. Whoever runs an external check only needs to know whether the thing is standing. Two different questions, two different endpoints.

The tools I use on this project live in the skills store — the first one is free.

The same commit that plugged the hole erased the panel

Here comes the part that matters: the Monitoring page was a consumer of that payload. It called the public route and built its cards from the database, auxiliary service, semantic search and model fields. With the payload reduced to a single key, those fields started coming back undefined.

What happened next was not an error — it was worse. The page had a guard per component: if the value came back undefined, the component simply never entered the final list. No exception, no 500, no red screen. Loading finished successfully, the list came out empty, and the panel displayed a blank grid.

const { data } = await api.get('/health')
const comps = []
const mk = (name, v) => {
  if (v === undefined) return  // the guard that turned a broken contract into emptiness
  comps.push({ name, status: v?.online ? 'online' : 'offline' })
}
mk('database', data?.database)
mk('sidecar', data?.sidecar)

Between 12:30 on the 4th and 00:25 on the 5th — roughly twelve hours — Monitoring did not say “I could not reach the service”. It said “there is no service running”. Those two sentences draw the same picture on screen and mean opposite things. Had I looked at the panel that afternoon, I would have concluded the ecosystem was dead, when what was dead was the contract between two parts of my own code.

The test that was guaranteeing the leak

The auth contract suite had the health route in its list of protected routes: the test required an anonymous GET to receive a 401. In practice, that test was a guardian of the leak — it passed green precisely because the hole existed. When the route became public by design, the list had to change: the minimal route left the protected set and the deep route took its place.

Here is the lesson I keep repeating to myself: a green test does not mean the system is right, it means the system matches the contract you wrote into the test. If the contract is wrong, the green is wrong too. A contract that changes for a security reason needs a conscious review of every test around it, not a tweak to make CI stop complaining.

The contract now has three assertions

The fix touched the page and, alongside it, the test — both in the same movement, because one without the other just relocates the bug.

First, the public endpoint is now checked by equality, not presence: the response must be exactly the status key, and none of the subsystem keys may appear. Equality is harsher than “contains”, and it is what keeps the payload from growing back unnoticed.

Second, the deep route is now tested as protected: an anonymous request gets a 401. And the detail I liked finding in the code is that this 401 is not a bare “no”: the attempt is recorded as a low-severity probe, with IP and client identification. Refuse and note, instead of refusing silently.

Third, the test that binds the panel to the endpoint: the five keys the screen reads must exist in the deep response, and a list key the old page used to read must remain absent. That negative assertion is the most valuable of the set: the previous version of the screen read a service list the endpoint never returned — a ghost shape that had already been producing an empty grid months before the leak. The bug had been sitting there the whole time, asleep, waiting for someone to touch the payload so it could become visible.

Proof from that round: twenty contract tests passing on the backend and thirteen page tests passing on the frontend, with the page consuming the deep route through the interceptor token — no longer through a header assembled by hand on each screen.

What this changes

Three things, in order of importance.

The first: closing a leak is a contract change, not housekeeping. Every time I harden an endpoint, I need to find who consumes that shape before I close the commit — not after the panel goes blank. The commit that plugs the hole has to list the consumers it breaks.

The second: “no data” and “down” need to look different. The guard that swallows undefined belongs to the same family as the broad handler that swallows an exception — comfortable to write, makes the code clean to read, and erases exactly the information I needed to see. A panel that fails loudly wakes me up; a panel that fails silently lets me make decisions on top of fiction.

The third: negative assertions are worth their weight. Testing that a sensitive field does not appear, and that an old shape does not come back, protects more than testing that the happy path works — because everyone tests the happy path.

Next step

The panel today trusts five keys and has no state of its own for “broken contract”. The next step is to separate three states explicitly on screen — loading, no data because the contract failed, and services genuinely down — and to make the test fail when the grid goes empty from an unexpected response, not only when one specific key disappears. Until those three states exist, a reduced payload will keep looking like an empty infrastructure.

Related entries

Same project, same ground — the stories connect.

Next step

If this was useful, there is more: I write about the same ground in the other projects, and the store holds the tools I use here.

No spam, one-click unsubscribe. I am also here: Skills store · Portfolio