Camoufox Research

Browser research toolkit for AI agents, exposed through MCP.

Search the web. Read JS-heavy pages. Interact with websites. Extract data. Monitor changes. Give your AI agent a real browser.

    AI Agent
       │  (tools, resources, prompts)
       ▼
      MCP
       │
       ▼
Camoufox Research   ← this server (48 tools)
       │
       ▼
   Camoufox          ← anti-detect Firefox
       │
       ▼
      Web

Most MCP servers can read the web. This one can live in it: open pages, click, type, fill forms, upload files, watch network traffic, take labeled screenshots, crawl whole sites, extract tables, monitor changes — and hand all of it to your agent through MCP (stdio, HTTP, or SSE).

Why Camoufox Research?

Most MCP browser tools give an agent isolated actions. Here the goal is different: a complete toolkit for web research — one server your agent can use end to end.

• 🔎 Search — find sources (DuckDuckGo via anti-detect browser, deep research with 20+ distinct sources)
• 🌐 Browse — read JS/SPA pages, live sessions with tabs, clicks, forms, uploads
• 👁️ Understand pages — labeled screenshots (Set-of-Mark), snapshot trees with refs
• 📊 Extract & export — fields by CSS/XPath, tables → CSV, PDF/DOCX/XLSX, JSON/Markdown files

⚡ 30-second demo

"Find all pricing pages on this website, extract the prices and save them to CSV."

Agent
 ├─ map_site     discover every /pricing page
 ├─ crawl        read them (cached)
 ├─ extract      {"plan": "css:.plan", "price": "css:.price"}
 └─ export       format=csv  →  prices.csv

No browser automation code. Just a sentence to your agent.

What it does (real scenarios)

🔎 Research — "Find information about this project, check 20 distinct sources and summarize." research → fetch_page → export

🕷 Crawl — "Walk the whole site and find every documentation page." sitemap → crawl / map_site

📊 Extract — "Collect prices from the table and save as CSV." extract / table_extract → export

👁 Vision — "Look at the page, find the Download button and press it." screenshot(som=True) → snapshot → session_click(ref="4")

📡 Monitor — "Check this page and tell me if it changed." fetch_page → page_diff (delta-read saves tokens)

One full scenario (killer demo)

git clone https://github.com/aidvizhhub/camoufox-research.git && cd camoufox-research
python3 -m venv ~/.venvs/camoufox-research
~/.venvs/camoufox-research/bin/pip install .
~/.venvs/camoufox-research/bin/python -m camoufox fetch   # download browser (once)

Then ask your agent:

"Find the latest articles about Camoufox, compare them and save the result to Markdown."

Agent
 ├─ web_search        "camoufox browser"
 ├─ research          10+ sources, dedup
 ├─ fetch_page        read the best articles
 ├─ extract           title / date / key points per source
 ├─ page_diff         skip unchanged pages
 └─ export            format=md  →  report.md

That's the whole point: your agent drives a real browser, you just describe the goal.

Deep research mode — 20+ distinct sources, not just top results

One research call, no agent loop needed:

research(
    queries=["agent observability landscape", "agentic search 2026"],
    max_results_per_query=6,
    target_domains=20,     # goal: 20 DIFFERENT websites
    domains_limit=2,       # max 2 results per site (no 15 links from one blog)
    expand=True,           # add "X comparison", "X documentation" queries
    terms_wave=True,       # 2nd wave built from rare terms of the 1st wave
    quality_first=True,    # docs / GitHub / arXiv first, forums last
    academic=True,         # arXiv + Semantic Scholar (free, no keys)
    fetch_all=True,        # read text of every collected source
    as_json=True,          # machine-readable: meta / sources / texts / notes
)

Academic channel — the vertical index industry uses to get primary sources (Exa vs Tavily: publications R@1 63.3% vs 31.8%). Both APIs are free, no keys:

paper_search("deep research agents")        # arXiv + Semantic Scholar
research(queries=["..."], academic=True)   # adds tier-0 papers to the hunt

Digests & verified (research_digest, auto after background campaigns): after the hunt the runner cuts short digests (title + first paragraph) for cheap synthesis and marks each source ✅ live / ❌ broken (verified citations gate, DEER / DeepResearch Bench pattern). The done-marker gains digests / verified / broken fields; research_report shows the status column.

One hunt at a time (guard): a new campaign starts only if no other campaign is running — 1 campaign = 1 worker = 1 browser (atomic INSERT ... WHERE NOT EXISTS, no races; Playwright EPIPE lesson).

Citation pack (citation_pack, after a campaign): verified ✅ sources with digests, one block, numbered [1]..[N] — the report citer writes with live links only (DEER / DeepResearch Bench verified-citations gate). Digests are menu-cleaned (_digest_clean: GitHub/SPA navigation junk is stripped; research_digest(camp_id, refresh) rebuilds old packs).

For automation, as_json=True returns a JSON payload instead of a text dump:

{"meta": {"sources": 31, "domains": 20, "followup_queries": ["JSON-RPC"]},
 "sources": [{"title": "...", "url": "...", "domain": "arxiv.org",
              "tier": 0, "tier_label": "первоисточник", "snippet": "..."}],
 "texts": [{"url": "...", "text": "..."}],
 "notes": []}

How it works (industry patterns, researched 27.08.2026):

• Query expansion — each query gets reformulations (comparison, documentation), which surface different domains and angles.
• Terms wave — from the 1st wave's snippets the server extracts rare terms and names (proper nouns, CamelCase) and searches them next (Open Deep Research pattern).
• Quality ranking — official docs / GitHub / arXiv rank first, forums last (gpt-researcher source ranking); you can extend the registry in camoufox_research/camoufox_sources.py.
• Second wave with pagination — if the target of distinct domains isn't reached, a final pass (pages=2) collects the rest.
• Domain dedup — docs.python.org and peps.python.org count as one source (python.org); example.co.uk handled as a 3-part domain.
• Echo of the goal in the output — доменов: N (цель 20), so you can see coverage at a glance.

Old behavior is preserved: target_domains=0, domains_limit=0, expand=False, fetch_all=False = plain top results.

Need a tool? Start here

Vision — pages with numbers

Screenshot        snapshot          agent
   │                  │               │
   ▼                  ▼               ▼
┌──────────┐    - ref: 3       session_click(ref="3")
│ [1][2][3]│    - tag: a   ───► browser clicks exact element
│ [4] [5]  │    - text: "Download"
└──────────┘

snapshot returns a compact YAML tree of interactive elements (~2–5 KB instead of 100 KB+ of HTML) with a ref on each. Click by ref, no fragile selectors.

Quick Start

# 1. Install
git clone https://github.com/aidvizhhub/camoufox-research.git && cd camoufox-research
python3 -m venv ~/.venvs/camoufox-research
~/.venvs/camoufox-research/bin/pip install .

# 2. Download the browser (once)
~/.venvs/camoufox-research/bin/python -m camoufox fetch

3. Connect to your MCP client (OpenCode / Claude Desktop / Cursor) — see Connect to MCP.
4. Ask your agent to research a website:

"Find the latest articles about Camoufox, compare them and save the result to Markdown."

Install

git clone https://github.com/aidvizhhub/camoufox-research.git
cd camoufox-research

# 1. venv + package
python3 -m venv ~/.venvs/camoufox-research
~/.venvs/camoufox-research/bin/pip install .

# 2. download the browser (once)
~/.venvs/camoufox-research/bin/python -m camoufox fetch

# 3. smoke check (stdio server, waits on stdin)
~/.venvs/camoufox-research/bin/camoufox-research

Windows: venv\Scripts\pip.exe install ., venv\Scripts\python.exe -m camoufox fetch; needs Python from python.org (not MS Store) and VC++ Redistributable.

Connect to MCP

opencode (~/.config/opencode/opencode.json):

{
  "mcp": {
    "camoufox": {
      "type": "local",
      "command": ["/path/to/venv/bin/camoufox-research"],
      "enabled": true
    }
  }
}

Claude Desktop, Cursor and others — ready-made examples in mcp/config/. No install needed: python mcp/server.py works from sources.

Check: opencode mcp list → camoufox: connected.

Tools (48)

MCP Resources & Prompts

• Resources (data readable "as files"): camoufox://stats, camoufox://cache, camoufox://session, camoufox://info
• Prompts (ready-made recipes): research_plan, extract_schema, monitor_page

Transports

stdio (default), http, sse:

camoufox-research --transport http --port 8833   # or env CAMOUFOX_PORT

Behavior

• Кампании (research_start) помнят прогресс в sqlite: счётчик РАЗНЫХ сайтов, доборка волнами, честный partial; research_resume добирает с места. Отчёт автоархивируется (CAMOUFOX_REPORT_DIR → research/ репы, по умолчанию exports кэша).
• Вторая нога охоты — фиды: research_start(feeds=[RSS/sitemap...]) собирает источники БЕЗ поисковика (queries можно опустить).
• Сторож поиска (scripts/watchdog_search.py + cron) проверяет DDG реальным путём: провал → watchdog_ALERT; research_start проверяет пульс крона и предупреждает, если тот молчит.
• Ларец не переполняется: research_index — сводка всех кампаний; scripts/campaign_cleanup.py (dry-run по умолчанию, --yes) выметает артефакты старше 30 дней. Отчёты .md метла не трогает.

Real output

Так выглядит автоархив кампании (полный файл — docs/example-report.md; добыта ТОЛЬКО фидом hnrss.org, поисковик не вызывался):

# Кампания: hacker news frontpage
- источников: 20, разных сайтов: 16/6        · статус: done

| # | источник | домен | класс |
|---|---|---|---|
| 1 | [Confdiff – semantic diff for config files](github.com/…) | github.com | первоисточник |

Publish to PyPI

Имя свободно, упаковка проверена (python -m build + twine check — PASSED). Публикация — через Trusted Publishing (OIDC, без токенов):

1. pypi.org → «Add a pending publisher»: owner aidvizhhub, repo camoufox-research, workflow release.yml, environment pypi.
2. На GitHub: Settings → Variables → PYPI_PUBLISH = yes.
3. gh release create v0.9.0 --title v0.9.0 --notes "..." — workflow соберёт и опубликует; дальше у всех: pip install camoufox-research. Без шага 1-2 джоб publish честно SKIP — CI не краснеет.

• Browser lives in a separate worker process (sync, headless) — the MCP stdio server never blocks.
• JS/SPA pages are read without preparation: content polling + scroll + stability detection; empty → retry.
• Cache: sqlite ~/.cache/camoufox-research/cache.db, TTL 24h, retry with backoff; delta=True skips re-reading unchanged pages.
• Config via environment only (see configs/example.env): CAMOUFOX_VENV, CAMOUFOX_CACHE_DIR, timeouts, proxy.

Development

See CONTRIBUTING.md: layout, adding a new tool, smoke-test ritual.

CI

GitHub Actions on every push: install on Python 3.10/3.11/3.12, import check, MCP stdio smoke (initialize → tools/list → ping). Full browser tests run locally (scripts/update_camoufox.py + manual smoke).

Experience journal

EXPERIENCE.md — verified lessons and landmines ("what not to step on"): asyncio/serve pitfalls, 403-vs-urllib, non-thread-safe Playwright, ElementTree XPath limits, and more.

Dependency licenses

• camoufox — see its repo
• mcp — MIT
• trafilatura — GPL-3.0 (optional: text extraction)