26 posts
Arachne had a whole feature written, tested and green — and that was never wired up. When I fixed the mounting, it stayed broken with a 422 on every call. Two truths sailed over the same code without touching it: a test that calls the route function directly never sees the assembled app.
Three surfaces that said OK without running: the backup that skipped a 300MB database, the retention that watched half the files, and eight assertions that never executed.
Arachne lets everyone bring their own model key. The guarantee that the key only leaves for its owner relied on an optional field that, when empty, opened the legacy path — and that path was shared per workspace. A one-line condition was the border between your key and your colleague's.
How Arachne evolved from a scraper into a personal knowledge platform with zero‑port infra, hybrid search and bot generation.
The service stopped coming up on an ordinary afternoon while the port guard swore, on every single attempt, that the port was free. 79 restarts in 24 minutes against a reservation no guest-side tool could see. The fix was to change the oracle.
Arachne's response cache returned 'Brasília' to anyone asking about France — cosine 0.8531 against a 0.85 threshold. The story of a bug born in a CI investigation, the math that hid it, and the entity guard that fixed it without sacrificing recall.
Before swapping Arachne's search engine for Yurumi, I had to prove the new one was at least as good as the old. Shadow dual-write, real traffic, overlap@10 — and the number nobody expects: 0.511.
79 tests green one day, 4 failing the next, zero production changes. Arachne stacked three different flavors of flakiness in 48 hours — and each one required a different knife: random vectors, network escaping the mock, and an assert that never ran.
A test was deleting RAG tables on teardown. One run, the environment variable pointing at the isolated database was ignored by the engine — and the DELETE landed on the production arachne.db, with the live server writing to it. What saved the data was an error message.
The Arachne landing page loaded without CSS, the watchdog restarted the service in an invisible loop, and a zombie socat relay kept stealing the port from its rightful owner. Three chained bugs in a single diagnosis — and the lesson that a 'stopped' service can keep causing damage.
Three embedding models, two search systems, one homegrown benchmark: the study that started with a question about semantic similarity and ended up revealing the tradeoffs between colibri, bge and openai — and how each one found its place in the ecosystem.
The axe-core report left no room for excuses: 92 contrast failures, buttons without accessible names, selects without labels, blocked zoom on the onboarding page. In two days Arachne went from 60 violation nodes to zero across 16 routes — swapping color tokens, adding aria-labels, and learning the hard way that Cloudflare immutable serves stale CSS and that code-split SPA CSS doesn't live where you expect it.
Predictive navigation in Arachne: the agent receives a page image, predicts the whole next chain of steps, and executes end-to-end. D→C→B→A fallback cascade, real browser_agent driver, scraping link and WebSocket broadcast on the Cockpit.
Arachne's vision pipeline evolved: qwen2.5vl:7b on Ollama WSL, arachne-vlm-jobs queue with arachne-vlm-w1 worker, SQLite vlm_cache.db cache (sha256), polling GET /api/extract/image-jobs/{id}. Vision requires JWT, PYTHONPATH api/ and PostgreSQL not SQLite.
Arachne ran 10 services on Docker Compose. Kubernetes showed up on my resume as a desired keyword — but I didn't use it. Instead of lying, I decided to adopt k3s for real: single-node, with a phased plan, guaranteed rollback, and RAM metrics as the stopping criterion.
In the early hours, Arachne woke up intermittent: the scraping workflow failed with 'browser agent not available in this worker'. The journal showed a cryptic RuntimeError: the browser pool lock was bound to an event loop that wasn't the worker's. The cause: asyncio.Lock() created at import time. The fix: lazy init — born inside the right loop.
Arachne responded 200, but authenticated routes rendered empty. A Bug Hunter found 1,239 QueuePool errors in the journal since 02:57. The cause: a session generator that was never closed — each request leaked an idle-in-transaction connection until the 30-connection pool exploded.
777 commits, 2818 tests, hybrid RAG, ReAct Agent with 42 tools, MCP server, and the evolution from a simple scraper to a knowledge platform.
From 3 projects in June to 5 in July, the Samuel ecosystem now has 2,396 commits, 3,647+ tests, and one agent orchestrating it all.
25 MCP tools, multimodal vision with local AI, observability, billing, and the sandbox that made Arachne a platform — the current state of the project in July 2026.
Exploring hybrid search — lessons, challenges, and discoveries.
The simple scraper became a platform. Crawl4AI Sidecar, automatic 4-layer fallback, and SQLite cache with FTS5.
The architecture that actually solves web data extraction: Trafilatura, Crawl4AI SDK, Docker Sidecar, and Camoufox — how each layer works, the fallback criteria, and what I learned along the way.
Experimenting with Crawl4AI in Arachne: Docker Sidecar, Python SDK, comparison with Trafilatura, and the decision to use automatic fallback between engines.
Tired of scrapers that broke out of nowhere and expensive third-party services, I decided to build my own data extraction platform.
The real story of how a 200-line Python scraper became a platform with 740+ files, RAG, visual pipelines and 36 ReAct tools.