The story of Arachne — the journey from scraper to knowledge platform

The scraper that became a knowledge platform

It all started with a simple need: extracting content from sites that block scraping. Arachne was born as a specialized scraper for iMasters, with four levels of evasion (Basic → Stealth → Headless → Camoufox).

But today it is a platform with hybrid RAG, a 42-tool ReAct Agent chatbot, an MCP server, a visual pipeline with 38 handlers, computer vision, audio, and monetization — 2,818 tests making sure none of it breaks.

Context — from crawler to platform

The initial commit was on June 9, 2026:

2542caf chore: checkpoint before adding RAG + chatbot

That commit shows Arachne was born already knowing it went beyond scraping. The same day brought the RAG module with semantic search and a vector store, and the chatbot engine with an API and a widget.

The product turn came with the Knowledge OS: knowledge bases with a 3-step wizard, ingestion by URL or text, and hybrid search (FTS5 + vector + RRF) with a cross-encoder reranker.

The fight — 4 evasion layers and the search that found nothing

The first battle was against Cloudflare. Each site had a different level of protection — and the pipeline needed to escalate:

# Arachne's progressive pipeline
progressive_fetch = Trafilatura → Stealth → Dynamic → Camoufox
# If everything fails: Browser Agent with 4 evasion layers

The Browser Agent became a war machine: stealth JS, fingerprint rotation, automatic consent dismissal (15+ frameworks), ad blocking with EasyList (104 thousand rules), and block detection via multilingual regex.

Then came hybrid search. FTS5 alone found too little; embeddings alone found imprecisely. The answer was to combine both with RRF (Reciprocal Rank Fusion) plus a reranker:

# Hybrid search: FTS5 + vector + RRF + cross-encoder
results = rrf_fusion(fts_results, vector_results)
results = cross_encoder_rerank(query, results, top_k=5)

Resolution — the ReAct Agent and the MCP server

The high point was when Arachne stopped being an API and became an agent. The ReAct Agent grew from 36 to 42 tools with a router planner + fallback + cache. And the MCP server (stdio + SSE) exposed everything to external agents — Cursor, Claude, Hermes.

The cherry on top: the Python SDK published on PyPI (pip install arachne-sdk), and the vision-to-text v2 pipeline — 5 extractors (PySceneDetect, OpenCV K-Means, EasyOCR, Florence-2, Whisper) that turn video into structured text for an LLM.

Metrics

Lessons learned

Arachne proved that scaling is not adding features — it is adding architecture. Every evasion level, every search layer, every agent tool was an answer to a real problem, not speculation.

And the most important lesson: once the tool becomes good enough, external agents come to it — that is how the MCP server and the SDK were born.

╔══════════════════════════════════════╗
║  Samuel Medeiros                    ║
║  Senior Software Engineer           ║
║  Stack: Python · TypeScript · Rust  ║
║  Projetos: Arachne, Dogwalk,        ║
║            Capivara, TatuEngine      ║
╚══════════════════════════════════════╝