
The admin claim that doubled as a key to the private network
An admin claim used to open private addresses by itself. The decision became a single fail-closed function with an explicit scope or a named allowlist.
40 posts

An admin claim used to open private addresses by itself. The decision became a single fail-closed function with an explicit scope or a named allowlist.

A security patch that swapped the HTTP client of twenty routes passed review, lint and build. The full suite reported 22 failures, 20 of them mine. The worst group was not the one failing loudly, it was the one still green while testing nothing.

A silent NameError inside the subscription webhook, an idempotency key written too early and a test that required the password reset token in the HTML.

The security scanner flagged Arachne's Dockerfile for running as root. The fix landed in the same commit that deleted the line copying the project into the container — and nobody noticed, because the build kept passing.

The security scanner flagged MD5 as a high-severity finding in Arachne. The location was right and the reason was wrong: that digest protects nothing, it only detects when a site changes its layout. Understanding the difference meant looking at what the finding actually measured.

On the same day, two commits fixed the same class of failure in Arachne: code that worked here and broke on any clean checkout, because the file had never been versioned. One froze the frontend artifact for days; the other took down nine upload routes in CI.

Two lines where one would do: how a global CSS reset and an orphan style block made the Arachne bar eat 117 pixels on mobile — and what finally measured the difference.

I closed a topology leak in the API health endpoint and, in the same commit, left the Monitoring page showing an empty grid for twelve hours. The page did not break. It lied politely.

One slow connector pushed the social research route past 200 seconds and killed the agent tool before its own timeout. The fix was not to make Reddit faster: it was a 20-second ceiling, an isolated thread, and turning the failure into honest data.

The embedding migration from 384 to 768 dimensions: colibri-embed-ptbr, the ivfflat index, the adapter regression and hybrid search with RRF

An audit of Arachne's 223 MCP tools found two protocol false positives: a tool failing on the inside reached the client as success, and a missing parameter came back as a server-side internal error. Both were fixed the same day, with a regression suite and a real end-to-end test.

The plugin was installed, the manifest was correct, and the plugin list said enabled. Yet none of its two tools showed up in any session. The bug was not in the install: it was the name of a parameter.

The build job only fired on pushes to master and the deploy job only on tags; the two conditions never crossed, so no tag ever built or shipped. Behind that, the deploy still dialed SSH to a legacy host from a machine that was already the runner itself, and CI died on the very first language setup step.

From the first iMasters scraper to a Knowledge OS with hybrid RAG, a 42-tool ReAct Agent and an MCP server — how Arachne evolved.

Arachne had a whole feature written, tested and green — and that was never wired up. When I fixed the mounting, it stayed broken with a 422 on every call. Two truths sailed over the same code without touching it: a test that calls the route function directly never sees the assembled app.

Three surfaces that said OK without running: the backup that skipped a 300MB database, the retention that watched half the files, and eight assertions that never executed.

Arachne lets everyone bring their own model key. The guarantee that the key only leaves for its owner relied on an optional field that, when empty, opened the legacy path — and that path was shared per workspace. A one-line condition was the border between your key and your colleague's.

How Arachne evolved from a scraper into a personal knowledge platform with zero‑port infra, hybrid search and bot generation.

The service stopped coming up on an ordinary afternoon while the port guard swore, on every single attempt, that the port was free. 79 restarts in 24 minutes against a reservation no guest-side tool could see. The fix was to change the oracle.

Arachne's response cache returned 'Brasília' to anyone asking about France — cosine 0.8531 against a 0.85 threshold. The story of a bug born in a CI investigation, the math that hid it, and the entity guard that fixed it without sacrificing recall.

Before swapping Arachne's search engine for Yurumi, I had to prove the new one was at least as good as the old. Shadow dual-write, real traffic, overlap@10 — and the number nobody expects: 0.511.

79 tests green one day, 4 failing the next, zero production changes. Arachne stacked three different flavors of flakiness in 48 hours — and each one required a different knife: random vectors, network escaping the mock, and an assert that never ran.

A test was deleting RAG tables on teardown. One run, the environment variable pointing at the isolated database was ignored by the engine — and the DELETE landed on the production arachne.db, with the live server writing to it. What saved the data was an error message.

The Arachne landing page loaded without CSS, the watchdog restarted the service in an invisible loop, and a zombie socat relay kept stealing the port from its rightful owner. Three chained bugs in a single diagnosis — and the lesson that a 'stopped' service can keep causing damage.

Three embedding models, two search systems, one homegrown benchmark: the study that started with a question about semantic similarity and ended up revealing the tradeoffs between colibri, bge and openai — and how each one found its place in the ecosystem.

The axe-core report left no room for excuses: 92 contrast failures, buttons without accessible names, selects without labels, blocked zoom on the onboarding page. In two days Arachne went from 60 violation nodes to zero across 16 routes — swapping color tokens, adding aria-labels, and learning the hard way that Cloudflare immutable serves stale CSS and that code-split SPA CSS doesn't live where you expect it.

Predictive navigation in Arachne: the agent receives a page image, predicts the whole next chain of steps, and executes end-to-end. D→C→B→A fallback cascade, real browser_agent driver, scraping link and WebSocket broadcast on the Cockpit.

Arachne's vision pipeline evolved: qwen2.5vl:7b on Ollama WSL, arachne-vlm-jobs queue with arachne-vlm-w1 worker, SQLite vlm_cache.db cache (sha256), polling GET /api/extract/image-jobs/{id}. Vision requires JWT, PYTHONPATH api/ and PostgreSQL not SQLite.

Arachne ran 10 services on Docker Compose. Kubernetes showed up on my resume as a desired keyword — but I didn't use it. Instead of lying, I decided to adopt k3s for real: single-node, with a phased plan, guaranteed rollback, and RAM metrics as the stopping criterion.

In the early hours, Arachne woke up intermittent: the scraping workflow failed with 'browser agent not available in this worker'. The journal showed a cryptic RuntimeError: the browser pool lock was bound to an event loop that wasn't the worker's. The cause: asyncio.Lock() created at import time. The fix: lazy init — born inside the right loop.

Arachne responded 200, but authenticated routes rendered empty. A Bug Hunter found 1,239 QueuePool errors in the journal since 02:57. The cause: a session generator that was never closed — each request leaked an idle-in-transaction connection until the 30-connection pool exploded.

777 commits, 2818 tests, hybrid RAG, ReAct Agent with 42 tools, MCP server, and the evolution from a simple scraper to a knowledge platform.

From 3 projects in June to 5 in July, the Samuel ecosystem now has 2,396 commits, 3,647+ tests, and one agent orchestrating it all.

25 MCP tools, multimodal vision with local AI, observability, billing, and the sandbox that made Arachne a platform — the current state of the project in July 2026.

Exploring hybrid search — lessons, challenges, and discoveries.

The simple scraper became a platform. Crawl4AI Sidecar, automatic 4-layer fallback, and SQLite cache with FTS5.

The architecture that actually solves web data extraction: Trafilatura, Crawl4AI SDK, Docker Sidecar, and Camoufox — how each layer works, the fallback criteria, and what I learned along the way.

Experimenting with Crawl4AI in Arachne: Docker Sidecar, Python SDK, comparison with Trafilatura, and the decision to use automatic fallback between engines.

Tired of scrapers that broke out of nowhere and expensive third-party services, I decided to build my own data extraction platform.

The real story of how a 200-line Python scraper became a platform with 740+ files, RAG, visual pipelines and 36 ReAct tools.