Webarchiver
Governed web fetch, extract, render, screenshot and link connector
The documentation site contains the full API, deployment, security, and operational reference. This README is the concise entry point for installation and the supported runtime surfaces.
Overview
webarchiver is the fleet's web fetch and scrape connector. It reads public web
pages under an explicit policy and returns typed results an agent or the graph
can ingest: raw pages, extracted main content with document metadata, rendered
DOMs, full-page screenshots and outbound links. It holds no credentials.
Key capabilities
- Governed fetch — robots.txt, a per-domain rate limit, size and content-type caps, re-checked redirects and egress pinning that refuses private and loopback destinations. Every refusal has a typed reason.
- Main-content extraction — clean text and markdown with title, author, published date, canonical URL, content digest and links.
- Rendering — headless Chromium for pages built by scripts, with ad and tracker requests aborted and fixed overlays removed.
- Screenshots — full-page PNG captures.
- Site crawl — a bounded breadth-first walk from one seed URL that returns page URLs (up to 1000) under the same robots and rate policy.
- One tool per operation —
web_fetch,web_extract,web_render,web_screenshot,web_crawlandweb_links, all read-only. - Shared web-source contract —
web_fetchandweb_extractare the agent-connector-sdk web-source contract's tools: the same request, page result and refusal reasons as every page connector, including the ArchiveBox connector. Each page names itsbackend(webarchiver).
Quick start
Install the package and run its generated entry points:
python -m pip install "webarchiver[mcp]"
webarchiver-mcp
For an on-demand run, use uvx:
uvx --from "webarchiver[mcp]" webarchiver-mcp
The networked server binds to loopback by default:
webarchiver-mcp --transport streamable-http --host 127.0.0.1 --port 8000
Architecture
The provider keeps the boundary between transport and domain code explicit:
client → MCP entry point → one tool per operation → fetch policy → web
└── skills, prompts, ontology, connector presets
The MCP surface has one typed, read-only tool per operation. Regenerate the tool
fingerprints with python scripts/generate_tool_schemas.py after changing a tool
contract.
MCP configuration
The server supports local stdio and authenticated streamable-http deployments.
Keep endpoint, credential, selector, and TLS references in SDK configuration; do not put
resolved values in client configuration.
{
"mcpServers": {
"webarchiver-mcp": {
"command": "webarchiver-mcp",
"args": [],
"env": {"SYSTEMTOOL": "true"}
}
}
}
For a networked deployment, use an authenticated TLS ingress and keep the allowlist in
MCP_ALLOWED_HOSTS. A configured remote example is https://service.example.invalid/mcp;
it is documentation-only and is never contacted by the generator.
{
"mcpServers": {
"webarchiver-mcp": {
"command": "webarchiver-mcp",
"args": ["--transport", "streamable-http", "--port", "8000"],
"env": {
"TRANSPORT": "streamable-http",
"HOST": "127.0.0.1",
"PORT": "8000",
"SYSTEMTOOL": "true"
}
}
}
}
Container builds install the published wheel for the pinned version. Publish the release first, then run from the repository root:
docker build --file docker/Dockerfile --target mcp -t webarchiver:<version>-mcp .
Deploy reviewed images by digest (webarchiver@sha256:<digest>) and include
--cap-drop=ALL in the runtime policy. The deployment guide
contains the complete transport and container examples.
Documentation
Use the Pages documentation for installation details, API capabilities, deployment profiles, platform integration, and the concept registry. The concise local references remain available at:
Development
Create an isolated branch, install the test extra, and run the complete local gate:
uv sync --extra test
pytest tests -q
pre-commit run --all-files
Keep MCP examples and environment documentation aligned with the implemented settings.
Add focused positive and adversarial tests for every behavior change. See
AGENTS.md for the current module ownership and release checks.
License
This project is distributed under the MIT license. See LICENSE for the complete terms.
1 maintainer
Metadata
1 maintainer
Release files for webarchiver 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| webarchiver-1.2.0.tar.gz | 3.8 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| webarchiver-1.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.9 MB
Release files / webarchiver-1.2.0.tar.gz
| Download URL | webarchiver-1.2.0.tar.gz |
|---|---|
| Size | 3.8 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f6e8d1b8d1f5da3af160f14928164b4a93c80c93eb1ed4434b238973e9efb4e7
|
|
BLAKE2b-256 checksum How to use checksums |
419cf65c844c80e76666c338a05c07a0bfbf1ec5ef15621104df25e270182a61
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Release files / webarchiver-1.2.0-py3-none-any.whl
| Download URL | webarchiver-1.2.0-py3-none-any.whl |
|---|---|
| Size | 36.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
183e60fe804843bb0d9aeb059bba0f90c28acd0c4133baf72ec7265b80804b04
|
|
BLAKE2b-256 checksum How to use checksums |
5d4e55378df4892e4fec0c8213245c1ad7a9529103a43553cb2b58cd88984a33
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
