MantisFetch

English | 中文

English

MantisFetch is an Agent-native data collection and document parsing toolkit by ReadyForAI.

Why MantisFetch?

Most scrapers hand your Agent a wall of raw HTML — expensive to process and mostly noise. MantisFetch semantically distills web pages into structured sections and actions, cutting token usage by 90%+ compared to raw HTML. A three-tier loading model (digest → brief → section) lets Agents pull only what they need instead of loading entire documents into context. Diff-first incremental reads mean revisiting a page costs near-zero tokens, and table-aware extraction precomputes statistics so Agents can answer data questions without reading a single row.

Features

• Semantic distillation — web pages → structured sections + interactive actions; 4,000–8,000 chars per distill vs. full HTML
• Three-tier loading — digest (~200 tokens) → brief (~1,500 tokens) → section (on-demand); Agents load only what they need
• Diff-first — distill returns only changed sections (changed_sids); repeat visits cost near-zero tokens
• Table-aware — auto-extracts tables with precomputed stats (min/max/avg); answer data questions without reading rows
• Document parsing — PDF, DOCX, PPTX, XLSX, CSV, HTML via MarkItDown, with Gemini OCR fallback
• Multi-LLM support — Gemini (default), OpenAI, Ollama, Groq, or any OpenAI-compatible API
• MCP server — exposes /web and /doc as native MCP tools (streamable-HTTP at /mcp) for agent runtimes; loopback-only by default, with bearer-token + optional TLS for remote clients
• WebMCP client — discovers and invokes structured tools that Chrome 146+ pages expose via navigator.modelContext
• i18n — English and Chinese (set LANG=zh)

Quick Start

Docker (recommended)

git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
GEMINI_API_KEY=your_key_here docker compose up -d

# Check health
curl http://localhost:9898/health

/web and /doc are loopback-only without a token. Host requests reach the container across the Docker bridge (a non-loopback peer), so set MANTISFETCH_MCP_TOKEN=… and send Authorization: Bearer … to use them; only /health is exempt.

Python (local)

git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
pip install -r requirements.txt
playwright install chromium

export GEMINI_API_KEY=your_key_here
python mantisfetch_server.py     # listens on port 9898

Docker

The docker-compose.yml provides a single-service setup. By default, the document library is bind-mounted to the current user's ~/.mantisfetch/docs directory on the host. See DEPLOYMENT.md for container hardening, shared-volume ownership (SMB/NFS), and the single-process boundary.

Image variants (WITH_LOCAL_OCR build arg): the offline PaddleOCR stack is ~1.5 GB of the image (v1.9.1 measured: full 4.66 GB, slim 3.20 GB uncompressed; 1.21 GB vs 0.88 GB as a gzipped docker save). Build with or without it:

# Full — bundled offline OCR (default)
docker build -t readyforai/mantisfetch:latest .

# Slim — ~1.5 GB smaller, no local OCR; OCR runs via the configured LLM provider
docker build --build-arg WITH_LOCAL_OCR=false -t readyforai/mantisfetch:slim .

With WITH_LOCAL_OCR=false the image bakes MANTISFETCH_LOCAL_OCR_ENABLED=false, so all OCR routing skips the (absent) local worker and uses the LLM/vision provider — embedded images and scanned-PDF pages (which otherwise have no per-page local→LLM fallback). An LLM provider must therefore be configured for the slim image to OCR at all; startup prewarm is skipped to match. Inspect a running image with docker inspect --format '{{index .Config.Labels "com.readyforai.mantisfetch.local-ocr"}}' <image>.

# docker-compose.yml (excerpt)
services:
  mantisfetch:
    build: .
    ports:
      - "9898:9898"
    volumes:
      - ${MANTISFETCH_HOST_DOCS_DIR:-${HOME}/.mantisfetch/docs}:/root/.mantisfetch/docs

Environment variables (pass via .env or docker compose environment block):

Using a local Ollama model

MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_API_KEY=ollama \
MANTISFETCH_LLM_BASE_URL=http://host.docker.internal:11434/v1 \
MANTISFETCH_LLM_MODEL=llama3 \
docker compose up

OpenAI-compatible vendor profiles

A vendor profile supplies the base_url and the protocol quirks (image encoding, request-body extras) for an OpenAI-compatible vendor. It does not supply a model: MANTISFETCH_LLM_MODEL is required, because a model name written into the code is only correct until the vendor retires it — and then every call fails, far from the cause. Startup logs a warning naming the key, and GET /health reports which model each role resolved to. The global default OCR image input mode stays data_url for maximum compatibility.

# Zhipu: one model for text, another for OCR
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=zhipu \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<their current text model> \
MANTISFETCH_OCR_MODEL=<their current vision model> \
docker compose up

# Kimi: one multimodal model covers both (omit MANTISFETCH_OCR_MODEL)
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=kimi \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<their current multimodal model> \
docker compose up

# Aliyun Bailian
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=aliyun \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<their current text model> \
MANTISFETCH_OCR_MODEL=<their current vision model> \
docker compose up

# Volcengine Ark: set your deployed endpoint/model explicitly
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=volcengine \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=your_endpoint_or_model \
MANTISFETCH_OCR_MODEL=your_vision_endpoint_or_model \
docker compose up

Vendor-specific request fields can be injected without code changes:

MANTISFETCH_OCR_EXTRA_BODY_JSON='{"image_url_detail":"high"}'

OCR image serialization can also be switched explicitly:

MANTISFETCH_OCR_IMAGE_INPUT_MODE=plain_base64

remote_url_only is recognized but currently fails fast in the built-in OCR pipeline, because MantisFetch renders pages in-memory and does not yet upload them to a hosted URL.

Per-role providers + failover (dual-slot)

The variables above configure one provider shared by both roles (summarisation and OCR). To run the two roles on different vendors — e.g. cheap text summaries on one vendor, a strong vision model for OCR on another — and to fail over to a backup when a call fails, set MANTISFETCH_LLM_DEFAULT. That switches MantisFetch into the dual-slot scheme:

• Two credential slots, DEFAULT and EXTRA. Each slot's value is the vendor tag; its base URL and API key are read from the environment (nothing is baked into the image — point a slot at any OpenAI-compatible endpoint by setting its _BASE_URL).
• Each role picks a primary and an optional fallback model, written <vendor>/<model>. The vendor prefix routes the call to the slot with that tag.
• Failover: if the primary model's call fails, the same call is retried once against the role's _FALLBACK model. A gemini/<model> prefix uses the Gemini backend; any other prefix uses the OpenAI-compatible backend.

# Slot DEFAULT = Zhipu (strong OCR VLM); slot EXTRA = DeepSeek (cheap text)
MANTISFETCH_LLM_DEFAULT=zhipu
MANTISFETCH_LLM_DEFAULT_BASE_URL=https://open.bigmodel.cn/api/paas/v4
MANTISFETCH_LLM_DEFAULT_API_KEY=your_zhipu_key
MANTISFETCH_LLM_EXTRA=deepseek
MANTISFETCH_LLM_EXTRA_BASE_URL=https://api.deepseek.com/v1
MANTISFETCH_LLM_EXTRA_API_KEY=your_deepseek_key

# Summaries on DeepSeek, OCR on Zhipu; each with a cross-vendor fallback
MANTISFETCH_SUM_MODEL_DEFAULT=deepseek/deepseek-chat
MANTISFETCH_SUM_MODEL_FALLBACK=zhipu/glm-4.6
MANTISFETCH_OCR_MODEL_DEFAULT=zhipu/glm-4.6v
MANTISFETCH_OCR_MODEL_FALLBACK=deepseek/deepseek-chat

Failover triggers on the provider's failure signal (each provider already retries transient errors internally, then reports a failure sentinel). Leave MANTISFETCH_LLM_DEFAULT unset to keep the legacy single-provider behaviour.

API

All endpoints are served on port 9898.

Core

Browser Session API

Stateful browser sessions for multi-step web automation.

Document Library API

Access documents stored by /web/capture and /doc/parse.

MCP

MantisFetch also exposes /web and /doc as Model Context Protocol tools over streamable-HTTP at /mcp, for agent runtimes (e.g. NodalOS) that speak MCP. It is loopback-only by default; set MANTISFETCH_MCP_TOKEN (and optionally TLS) to allow remote clients.

Full API reference: see skills/mantisfetch-browser-SKILL.md, skills/mantisfetch-docreader-SKILL.md, and skills/mantisfetch-mcp-SKILL.md.

DocReader notes:

• POST /doc/parse accepts an optional metadata form field containing a JSON object; shallow scalar values are indexed for later filtering.
• POST /doc/parse also accepts an optional doc_id form field. Valid values may contain letters, digits, and internal hyphens, for example DOC-001, NBS250321, or doc001. Reusing an existing doc_id returns 409 unless replace=true.
• POST /doc/parse accepts summary_mode (sync / defer / off); with defer, the summary is generated in the background — poll GET /doc/library/{doc_id}/summary for status, and POST the same path to retry a failed one.
• POST /doc/parse accepts parse_mode (fast / accurate / full, default accurate) to tune PDF parsing intensity vs. cost.
• POST /doc/parse accepts store_only=true for the raw channel: md and png/jpg/jpeg/gif/webp are stored as the original file with no parse products at all, and read back via /doc/library/{doc_id}/source. The response and manifest carry kind: "raw" (parsed documents read "parsed"), as do search hits, the sections listing and summary status; such documents 404 on digest/brief/full, 409 on a summary retry, and do not appear in search_text. GET /doc/health lists the channel's formats as raw_formats. A 409 on this channel returns an object under detail carrying the existing document's kind; the parse channel's 409 is unchanged. The raw and parsed format sets never overlap — a parseable extension asking for store_only is a 422. Requires MANTISFETCH_STORE_SOURCE_FILES=true.
• POST /doc/parse reports duplicate content on the raw channel as dedup: "hit" plus existing_doc_id, and still creates the document under the doc_id the caller asked for — nothing is merged.
• GET /doc/library/search results carry kind as well, so a caller can tell a raw hit from a parsed one before reaching for a digest.
• GET /doc/library/{doc_id}/table/{table_id}/json returns structured cells; for tables reconstructed from scanned pages, merged cells now carry a recovered colspan (the Markdown form is unchanged).
• POST /doc/parse and POST /web/capture accept content_type, one of General, Contract, Bid, or Knowledge (case-insensitive on input; persisted in title case); the default is General.
• New ingested documents are stored under ${MANTISFETCH_DOCS_DIR}/<content_type>/<doc_id>, while legacy flat ${MANTISFETCH_DOCS_DIR}/<doc_id> documents remain readable.
• GET /doc/library/search also accepts query params prefixed with metadata. for equality-style filtering, for example metadata.customer=ACME.
• GET /doc/library/search and GET /doc/library/search_text accept content_type for category-filtered browsing.
• GET /doc/library/search_text returns snippet, sid, page_range, page_start, and page_end to support page-level follow-up actions.
• Parsed document manifests now include metadata, source_file, and enriched section page bounds.
• If MANTISFETCH_DOC_ID_STRATEGY=source_filename, uploaded files may use a sanitized source filename as the document directory name. Unsupported characters are stripped, separators such as spaces / _ / . are normalized to -, and the service falls back to DOC-xxx when no usable characters remain.

Web search (optional)

Off by default. Set MANTISFETCH_SEARCH_PROVIDER to enable two endpoints:

Providers: searxng (self-hosted, zero API cost — the recommended default), tavily, bocha (博查, China), brave. A comma-separated MANTISFETCH_SEARCH_FALLBACK switches provider only on connection error / 5xx / timeout (a 4xx config error or an empty result does not fall back). When unset, /web/search* return 404 and the MCP web_search / web_search_capture tools are not registered.

Zero-cost SearXNG in one command:

MANTISFETCH_SEARCH_PROVIDER=searxng docker compose --profile search up

For China deployments, SearXNG's default upstreams (Google, DuckDuckGo) are usually blocked — activate the China preset first (a Baidu / 360 / Sogou / Quark / Bing starting set; validate it from your own network): cp configs/searxng/settings.cn.yml configs/searxng/settings.yml.

search_and_capture reuses the /web/capture path with two layers of reuse:

1. URL cache — MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS (default 24) so repeated research queries do not re-fetch the same top hits. Independent of MANTISFETCH_CAPTURE_TTL_HOURS (still 0/off for plain /web/capture).
2. Content-hash reuse — after distill, if the body content_hash already exists in the library (same article via AMP / tracking-param URLs), the existing doc_id is returned and tags/metadata are merged (first-touch for existing metadata keys).

A cache hit keeps the document's original top-level provenance (first-touch): a URL first captured directly stays source=web_capture. Filter search-sourced documents with metadata.source=web_search, not the top-level source.

Configuration

MantisFetch is configured entirely through environment variables. See the table in the Docker section above for LLM settings. Additional variables:

For AI Agents

MantisFetch is designed to be deployed and operated autonomously by AI Agents. A self-contained deployment prompt (English + Chinese) is available at docs/agent-deployment-prompt.md — copy it directly into your Agent's system prompt, no modification needed.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for development setup, code conventions, and the PR process.

License

MIT — see LICENSE.

中文

MantisFetch 是由 ReadyForAI 开源的 Agent 原生数据采集与文档解析工具包。

为什么选择 MantisFetch？

大多数爬虫工具返回原始 HTML——对 LLM 而言噪音极多、token 消耗过高。MantisFetch 专为 AI Agent 设计：网页经过语义蒸馏，转化为结构化段落（sections）和可交互元素（actions），相比原始 HTML 节省 90%+ 的 token。三层加载模型（digest → brief → section）让 Agent 按需取用，无需把整篇文档塞进上下文。增量读取（diff-first）机制确保重复访问同一页面时的额外开销接近于零；表格自动预计算统计值，Agent 无需逐行读取即可回答数据问题。

功能特性

• 语义蒸馏 — 网页 → 结构化段落 + 可交互元素；每次 distill 仅 4,000–8,000 字符，对比原始 HTML 节省 90%+ token
• 三层加载 — digest（约 200 token）→ brief（约 1,500 token）→ section（按需）；Agent 只取所需
• 增量读取 — distill 仅返回变化段落（changed_sids），重复访问近零 token 开销
• 表格感知 — 自动提取并预计算统计值（min/max/avg），无需读取原始行即可回答数据问题
• 文档解析 — 支持 PDF、DOCX、PPTX、XLSX、CSV、HTML，基于 MarkItDown（微软），可自动 Gemini OCR 兜底
• 多 LLM 支持 — Gemini（默认）、OpenAI、Ollama、Groq，以及任意 OpenAI 兼容接口
• MCP 服务 — 把 /web 和 /doc 暴露为原生 MCP 工具（streamable-HTTP，挂载在 /mcp），供 Agent 运行时使用；默认仅 loopback，远程客户端需 bearer token + 可选 TLS
• WebMCP 客户端 — 发现并调用 Chrome 146+ 页面通过 navigator.modelContext 暴露的结构化工具
• 多语言 — 支持中英文（设置 LANG=zh 切换为中文）

快速上手

Docker（推荐）

git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
GEMINI_API_KEY=your_key_here docker compose up -d

# 检查服务状态
curl http://localhost:9898/health

Python（本地运行）

git clone https://github.com/ReadyForAI/MantisFetch.git
cd MantisFetch
pip install -r requirements.txt
playwright install chromium

export GEMINI_API_KEY=your_key_here
python mantisfetch_server.py     # 监听 9898 端口

Docker 配置

docker-compose.yml 提供单服务部署方案。默认会把文档库 bind mount 到宿主机当前用户的 ~/.mantisfetch/docs 目录。

镜像变体（WITH_LOCAL_OCR 构建参数）： 离线 PaddleOCR 栈约占镜像 1.5 GB（v1.9.1 实测：未压缩时完整版 4.66 GB、精简版 3.20 GB；docker save 再 gzip 后分别为 1.21 GB 与 0.88 GB）。可选择带或不带：

# 完整版 —— 内置离线 OCR（默认）
docker build -t readyforai/mantisfetch:latest .

# 精简版 —— 体积小约 1.5 GB，不含本地 OCR；OCR 走配置的 LLM provider
docker build --build-arg WITH_LOCAL_OCR=false -t readyforai/mantisfetch:slim .

设 WITH_LOCAL_OCR=false 时镜像会烙入 MANTISFETCH_LOCAL_OCR_ENABLED=false，所有 OCR 路由都跳过（不存在的）本地 worker、改用 LLM/vision provider——包括内嵌图片以及扫描版 PDF 页（后者本身没有逐页 local→LLM 兜底）。因此精简版必须配置 LLM provider 才能做 OCR；启动期 prewarm 也随之跳过。用 docker inspect --format '{{index .Config.Labels "com.readyforai.mantisfetch.local-ocr"}}' <image> 查看某镜像是哪种变体。

# docker-compose.yml（节选）
services:
  mantisfetch:
    build: .
    ports:
      - "9898:9898"
    volumes:
      - ${MANTISFETCH_HOST_DOCS_DIR:-${HOME}/.mantisfetch/docs}:/root/.mantisfetch/docs

环境变量（通过 .env 文件或 docker compose 的 environment 块传入）：

使用本地 Ollama 模型

MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_API_KEY=ollama \
MANTISFETCH_LLM_BASE_URL=http://host.docker.internal:11434/v1 \
MANTISFETCH_LLM_MODEL=llama3 \
docker compose up

OpenAI 兼容厂商配置

厂商 profile 提供 OpenAI 兼容厂商的 base_url 与协议细节（图片编码、请求体附加项），但不提供模型名： MANTISFETCH_LLM_MODEL 是必填的 —— 写死在代码里的模型名只在厂商下架它之前是对的，之后每次调用都会失败， 而且失败点离原因很远。启动时会打一条点名该键的告警，GET /health 也会报出每个角色实际解析到的模型。 OCR 图片输入模式的全局默认值仍固定为 data_url，优先保证兼容面。

# 智谱：文本一个模型、OCR 另一个
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=zhipu \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<该厂商当前的文本模型> \
MANTISFETCH_OCR_MODEL=<该厂商当前的视觉模型> \
docker compose up

# Kimi：一个多模态模型两用（不设 MANTISFETCH_OCR_MODEL 即可）
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=kimi \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<该厂商当前的多模态模型> \
docker compose up

# 阿里百炼
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=aliyun \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=<该厂商当前的文本模型> \
MANTISFETCH_OCR_MODEL=<该厂商当前的视觉模型> \
docker compose up

# 火山方舟：通常需要显式填写你的推理接入点 / 模型名
MANTISFETCH_LLM_PROVIDER=openai \
MANTISFETCH_LLM_VENDOR=volcengine \
MANTISFETCH_LLM_API_KEY=your_key_here \
MANTISFETCH_LLM_MODEL=your_endpoint_or_model \
MANTISFETCH_OCR_MODEL=your_vision_endpoint_or_model \
docker compose up

如果厂商需要额外请求参数，也可以直接透传 JSON：

MANTISFETCH_OCR_EXTRA_BODY_JSON='{"image_url_detail":"high"}'

也可以显式切换 OCR 图片输入模式：

MANTISFETCH_OCR_IMAGE_INPUT_MODE=plain_base64

其中 remote_url_only 目前只做了显式报错路径：MantisFetch 现在是把 PDF 页渲染到内存里直接上送，还没有内建“先上传图片再传远程 URL”的流程。

分角色 provider + 故障切换（双槽位）

上面的变量配置的是一个 provider，摘要和 OCR 共用。若想让两个角色跑在不同厂商上（比如摘要用便宜的文本模型、OCR 用能力强的视觉模型），并在调用失败时切到备用，设置 MANTISFETCH_LLM_DEFAULT 即可切换到双槽位方案：

• 两个凭据槽位 DEFAULT 和 EXTRA。槽位的值是厂商标签，其 base_url 和 API key 从环境变量读取（不写死进镜像——给槽位设 _BASE_URL 即可指向任意 OpenAI 兼容端点）。
• 每个角色各配一个主用和一个可选的备用模型，写成两段式 <厂商>/<模型>。前段厂商标签决定路由到哪个槽位。
• 故障切换：主用模型调用失败时，同一次调用会用该角色的 _FALLBACK 模型再试一次。前段是 gemini/<模型> 走 Gemini 后端，其它走 OpenAI 兼容后端。

# 槽位 DEFAULT = 智谱（视觉 OCR 强）；槽位 EXTRA = DeepSeek（文本便宜）
MANTISFETCH_LLM_DEFAULT=zhipu
MANTISFETCH_LLM_DEFAULT_BASE_URL=https://open.bigmodel.cn/api/paas/v4
MANTISFETCH_LLM_DEFAULT_API_KEY=你的智谱key
MANTISFETCH_LLM_EXTRA=deepseek
MANTISFETCH_LLM_EXTRA_BASE_URL=https://api.deepseek.com/v1
MANTISFETCH_LLM_EXTRA_API_KEY=你的deepseek_key

# 摘要走 DeepSeek，OCR 走智谱；各自配一个跨厂商备用
MANTISFETCH_SUM_MODEL_DEFAULT=deepseek/deepseek-chat
MANTISFETCH_SUM_MODEL_FALLBACK=zhipu/glm-4.6
MANTISFETCH_OCR_MODEL_DEFAULT=zhipu/glm-4.6v
MANTISFETCH_OCR_MODEL_FALLBACK=deepseek/deepseek-chat

切换在 provider 报出失败信号时触发（每个 provider 自身已先重试瞬时错误，再返回失败哨兵）。不设 MANTISFETCH_LLM_DEFAULT 则保持原有的单 provider 行为。

API 接口

所有接口均运行在 9898 端口。

核心接口

Browser Session API

有状态浏览器会话，支持多步骤网页自动化。

Document Library API

访问由 /web/capture 和 /doc/parse 存入文档库的文档。

MCP

MantisFetch 还把 /web 和 /doc 暴露为 Model Context Protocol 工具（streamable-HTTP，挂载在 /mcp），供使用 MCP 的 Agent 运行时（如 NodalOS）使用。默认仅 loopback；设置 MANTISFETCH_MCP_TOKEN（并可选启用 TLS）以允许远程客户端。

完整 API 说明见 skills/mantisfetch-browser-SKILL.md、skills/mantisfetch-docreader-SKILL.md 和 skills/mantisfetch-mcp-SKILL.md。

DocReader 补充说明：

• POST /doc/parse 支持可选 metadata 表单字段，值为 JSON object；其中浅层标量字段会进入索引，供后续过滤。
• POST /doc/parse 也支持可选 doc_id 表单字段；合法值可包含字母、数字和中间连字符，例如 DOC-001、NBS250321、doc001。复用已有 doc_id 会返回 409，除非 replace=true。
• POST /doc/parse 支持 summary_mode（sync / defer / off）；用 defer 时摘要在后台生成 —— 轮询 GET /doc/library/{doc_id}/summary 查看状态，POST 同一路径可重试失败的摘要。
• POST /doc/parse 支持 parse_mode（fast / accurate / full，默认 accurate），用于在 PDF 解析强度与成本间权衡。
• POST /doc/parse 支持 store_only=true 走原件通道：md 与 png/jpg/jpeg/gif/webp 只存原件、完全不解析，通过 /doc/library/{doc_id}/source 读回。响应与 manifest 带 kind: "raw"（解析文档为 "parsed"），搜索命中、sections 列举、摘要状态同样带；这类文档在 digest/brief/full 上 404、摘要重试上 409，也不会出现在 search_text 结果里。GET /doc/health 以 raw_formats 列出本通道的格式。本通道的 409 在 detail 里返回对象、带上已有文档的 kind；解析通道的 409 不变。原件与解析两套扩展名集合互不相交 —— 可解析的扩展名带 store_only 会返回 422。要求 MANTISFETCH_STORE_SOURCE_FILES=true。
• 原件通道遇到内容重复时返回 dedup: "hit" 与 existing_doc_id，但仍按调用方给的 doc_id 建文档 —— 只告知，不合并。
• GET /doc/library/search 的结果也带 kind，调用方在去要 digest 之前就能分辨原件文档与解析文档。
• GET /doc/library/{doc_id}/table/{table_id}/json 返回结构化单元格；对从扫描页重建的表格，合并单元格现在带恢复的 colspan（Markdown 形式不变）。
• POST /doc/parse 和 POST /web/capture 支持 content_type，可选值为 General、Contract、Bid、Knowledge（输入大小写不敏感，存储统一为首字母大写）；默认 General。
• 新入库文档会保存到 ${MANTISFETCH_DOCS_DIR}/<content_type>/<doc_id>，旧版平铺的 ${MANTISFETCH_DOCS_DIR}/<doc_id> 文档仍可读取。
• GET /doc/library/search 支持 metadata.* 形式的查询参数做等值过滤，例如 metadata.customer=ACME。
• GET /doc/library/search 和 GET /doc/library/search_text 支持 content_type，用于按分类浏览。
• GET /doc/library/search_text 会返回 snippet、sid、page_range、page_start、page_end，便于后续定位页面。
• manifest.json 现在会包含 metadata、source_file，以及补强后的 section 页码边界。
• 如果设置 MANTISFETCH_DOC_ID_STRATEGY=source_filename，上传文件会优先使用“过滤后的源文件名”作为文档目录名；空格、下划线、点会归一成 -，不支持的字符会被剔除；若过滤后没有剩余可用字符，则自动回退为 DOC-xxx。

网络搜索（可选）

默认关闭。设置 MANTISFETCH_SEARCH_PROVIDER 后启用两个端点：

Provider：searxng（自托管、零 API 成本，推荐默认）、tavily、bocha（博查，国内）、brave。逗号分隔的 MANTISFETCH_SEARCH_FALLBACK 仅在连接错误 / 5xx / 超时时切换 provider（4xx 配置错误或空结果不切换）。未设置时 /web/search* 返回 404，MCP 的 web_search / web_search_capture 工具不注册。

一条命令启动零成本 SearXNG：

MANTISFETCH_SEARCH_PROVIDER=searxng docker compose --profile search up

国内部署:SearXNG 默认上游(Google、DuckDuckGo)通常被墙——先启用国内预设(Baidu / 360 / Sogou / Quark / Bing 起步组合,请在目标网络实测校准):cp configs/searxng/settings.cn.yml configs/searxng/settings.yml。

search_and_capture 复用 /web/capture 路径，并带两层去重：

1. URL 缓存 — MANTISFETCH_SEARCH_CAPTURE_TTL_HOURS（默认 24），避免重复研究查询反复重抓同一批结果；与普通 /web/capture 的 MANTISFETCH_CAPTURE_TTL_HOURS（默认仍为 0）独立。
2. 内容哈希复用 — distill 后若库中已有相同 body content_hash（同一文章经 AMP / 跟踪参数 URL），返回已有 doc_id 并合并 tags/metadata（已有 metadata 键 first-touch 保留）。

缓存命中保留文档原有顶层来源（first-touch）：先被直接采集的 URL 仍为 source=web_capture。过滤搜索来源文档请用 metadata.source=web_search，不要用顶层 source。

配置项

MantisFetch 所有配置均通过环境变量管理。LLM 相关配置见上方 Docker 配置 一节，其他变量如下：

接入 AI Agent

MantisFetch 支持由 AI Agent 自主部署和操作。docs/agent-deployment-prompt.md 提供开箱即用的中英双语部署提示词，直接复制到 Agent 的系统提示词中即可使用，无需任何修改。

参与贡献

欢迎提交 PR！开发环境搭建、代码规范和 PR 流程请参阅 CONTRIBUTING.md。

许可证

MIT 协议，详见 LICENSE。