The error that arrived as a success — 223 tools and a field nobody filled in

The green that was never green

Arachne exposes 223 tools over MCP. Every one of them returns JSON with a success field and, when something goes wrong, an error explaining why. A simple contract, repeated since the project started.

What I did not have was the other end: what the MCP client actually sees when a tool fails. Not the handler — the protocol. And the answer, in today’s audit, was uncomfortable.

A call to arachne_scrape with a URL that does not exist returned HTTP 200, result.content holding a JSON that said success: false, and no isError. The client — Hermes, Claude, Cursor, anything that speaks the protocol — read that as a success, because that is what the JSON-RPC response said. The agent then moved on with empty content, convinced it had worked.

Four ways to lie

I repeated it across four tools to rule out an isolated case:

• arachne_scrape with a nonexistent URL — HTTP 200, no isError
• arachne_extract with broken DNS — HTTP 200, no isError
• arachne_vision with the vision service returning 502 — HTTP 200, no isError
• arachne_search with an empty query — HTTP 200, no isError

Four failures of entirely different natures producing the same false verdict. The pattern was in the wrong place: tools/call built the response as {"content": [{"type": "text", "text": result}]} and never looked inside result. The handler had failed loudly; the server swallowed the information on the way out.

The second finding came down the same path. Calling arachne_calc without the required field returned:

-32603 Internal error: arachne_calc() missing 1 required positional argument

That is a Python signature leaking straight to the client. Worse: -32603 means “internal server error”. The server was fine — my request was incomplete. A well-behaved client that reads -32603 retries or opens an incident. Neither fixes a field left out of the payload.

What was done

The fix (f5d96ee, 24/10 14:51) is two small functions and a new order inside tools/call.

The first is _payload_failed(). It takes the text the handler returned, tries to parse it as JSON, and looks for success: false or ok: false — at the top of the object and also nested inside data, because the scraping family puts the verdict one level down. Plain text, invalid JSON and payloads that are not dicts return False: the heuristic only flags an error when the handler itself said it failed. When it finds one, the response gets isError: true.

The second is _missing_required(). Before calling the handler, the server compares the incoming arguments against that tool’s inputSchema.required. If something is missing, the answer is -32602 Invalid params with the field names. And a TypeError raised by the handler also becomes -32602, because a signature that does not match the schema is a client error, not a server one.

The detail that matters is where the validation happens: before the handler runs. That used to be expensive — arachne_scrape burned all six evasion layers plus a Camoufox session before giving up on a URL that did not exist. Now a call missing required parameters never touches the browser.

Verification was not a diff review. Thirty real HTTP calls against /api/mcp — ping, capabilities, metrics, calc, search, scrape, extract, vision, screenshot, trending, releases, query, social, kg_*, torrent — produced zero protocol failures. Unit tests closed at 29 passing, 20 of them in the new regression file, which covers the awkward cases: success: false nested in data, ok: false, plain text that must not be flagged, and invalid JSON that must not be either.

Then came the end-to-end test (4aa56be, 15:34): 157 lines in api/tests/e2e/test_mcp_protocol_e2e.py hitting the real server over HTTP, nothing mocked — initialize handshake, tools/list, isError on a real tool failure and -32602 on a missing parameter. The module skips itself when the server is down, so the regular CI is not held hostage. Real run against the container: 6 passed in 6.4 seconds.

pytest api/tests/test_mcp_iserror.py -q
pytest api/tests/e2e/test_mcp_protocol_e2e.py -q -m integration
docker restart arachne-api

Why this matters more than it looks

An agent handed an empty success has no way to defend itself. It sees no error to handle, no reason to try another source, no reason to warn the user. It finishes the task holding nothing and delivers nothing with confidence. A silent false positive is the worst class of bug in an agent tool: it corrupts the judgement at the far end instead of breaking the call.

And the gap between -32602 and -32603 is not spec pedantry. It is the difference between “the server is broken” and “your request is incomplete” — two opposite reactions on the other side.

A side effect worth recording: the container mounts the API directory over the package installed in Python. Editing the file in the working tree is a deploy; only the docker restart is missing. That explains why a fix once seemed to have no effect, and it now lives in the changelog.

What the audit left on the table

The sweep did not stop at those two problems. The engine field on the scraping tools lies: it returns "sidecar" while the crawling sidecar is offline, no listener at all, and the engine actually in use is another. kg_god_nodes answers empty with num_nodes_analyzed: 0. And arachne_scrape still blames “Page blocked after 6 evasion layers” when the real cause is DNS.

The measured latencies became a priority too: extract 56 seconds, query on RAG 49, social_research 50, trending 25. Not a protocol bug — a queue. And that is the next target: an agent waiting 56 seconds for one extraction has already lost the window where the user was paying attention.

Arachne keeps going in build-in-public mode. Today it learned to say “no” instead of smiling and handing back nothing.