Webarchiver

GitHub Repo stars GitHub forks GitHub contributors GitHub license GitHub last commit (by committer) GitHub pull requests GitHub closed pull requests GitHub issues GitHub top language GitHub language count GitHub repo size GitHub repo file count (file type) PyPI - Version PyPI - Downloads PyPI - License PyPI - Wheel PyPI - Implementation MCP Server

Governed web fetch, extract, render, screenshot and link connector

The documentation site contains the full API, deployment, security, and operational reference. This README is the concise entry point for installation and the supported runtime surfaces.

Overview

webarchiver is the fleet's web fetch and scrape connector. It reads public web pages under an explicit policy and returns typed results an agent or the graph can ingest: raw pages, extracted main content with document metadata, rendered DOMs, full-page screenshots and outbound links. It holds no credentials.

Key capabilities

Governed fetch — robots.txt, a per-domain rate limit, size and content-type caps, re-checked redirects and egress pinning that refuses private and loopback destinations. Every refusal has a typed reason.
Main-content extraction — clean text and markdown with title, author, published date, canonical URL, content digest and links.
Rendering — headless Chromium for pages built by scripts, with ad and tracker requests aborted and fixed overlays removed.
Screenshots — full-page PNG captures.
Site crawl — a bounded breadth-first walk from one seed URL that returns page URLs (up to 1000) under the same robots and rate policy.
One tool per operation — web_fetch, web_extract, web_render, web_screenshot, web_crawl and web_links, all read-only.

Quick start

Install the package and run its generated entry points:

python -m pip install "webarchiver[mcp]"
webarchiver-mcp

For an on-demand run, use uvx:

uvx --from "webarchiver[mcp]" webarchiver-mcp

The networked server binds to loopback by default:

webarchiver-mcp --transport streamable-http --host 127.0.0.1 --port 8000

Architecture

The provider keeps the boundary between transport and domain code explicit:

client → MCP entry point → one tool per operation → fetch policy → web
                                  └── skills, prompts, ontology, connector presets

The MCP surface has one typed, read-only tool per operation. Regenerate the tool fingerprints with python scripts/generate_tool_schemas.py after changing a tool contract.

MCP configuration

The server supports local stdio and authenticated streamable-http deployments. Keep endpoint, credential, selector, and TLS references in SDK configuration; do not put resolved values in client configuration.

{
  "mcpServers": {
    "webarchiver-mcp": {
      "command": "webarchiver-mcp",
      "args": [],
      "env": {"SYSTEMTOOL": "true"}
    }
  }
}

For a networked deployment, use an authenticated TLS ingress and keep the allowlist in MCP_ALLOWED_HOSTS. A configured remote example is https://service.example.invalid/mcp; it is documentation-only and is never contacted by the generator.

{
  "mcpServers": {
    "webarchiver-mcp": {
      "command": "webarchiver-mcp",
      "args": ["--transport", "streamable-http", "--port", "8000"],
      "env": {
        "TRANSPORT": "streamable-http",
        "HOST": "127.0.0.1",
        "PORT": "8000",
        "SYSTEMTOOL": "true"
      }
    }
  }
}

Container builds install the published wheel for the pinned version. Publish the release first, then run from the repository root:

docker build --file docker/Dockerfile --target mcp -t webarchiver:<version>-mcp .

Deploy reviewed images by digest (webarchiver@sha256:<digest>) and include --cap-drop=ALL in the runtime policy. The deployment guide contains the complete transport and container examples.

Documentation

Use the Pages documentation for installation details, API capabilities, deployment profiles, platform integration, and the concept registry. The concise local references remain available at:

Development

Create an isolated branch, install the test extra, and run the complete local gate:

uv sync --extra test
pytest tests -q
pre-commit run --all-files

Keep MCP examples and environment documentation aligned with the implemented settings. Add focused positive and adversarial tests for every behavior change. See AGENTS.md for the current module ownership and release checks.

License

This project is distributed under the MIT license. See LICENSE for the complete terms.

Metadata

Release files for webarchiver 1.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for webarchiver 1.1.0
File	Interpreter	ABI	Platform	Reset
webarchiver-1.1.0-py3-none-any.whl 34.8 kB Oct 18, 2026	Python 3	none	any	Details

Release files / webarchiver-1.1.0-py3-none-any.whl

Download URL	webarchiver-1.1.0-py3-none-any.whl
Size	34.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`15123c777234f07d9d130604bd07abeba271fdf2d896a8b62e874fe843025fc5`
BLAKE2b-256 checksum How to use checksums	`4c89a304ff0ed5e4ce899c64d36d43389789b7e2265f98eae520f19ddfacd271`
Upload date	Oct 18, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.4`