Skip to main content

Webarchiver

GitHub Repo stars GitHub forks GitHub contributors GitHub license GitHub last commit (by committer) GitHub pull requests GitHub closed pull requests GitHub issues GitHub top language GitHub language count GitHub repo size GitHub repo file count (file type) PyPI - Version PyPI - Downloads PyPI - License PyPI - Wheel PyPI - Implementation MCP Server

Governed web fetch, extract, render, screenshot and link connector

The documentation site contains the full API, deployment, security, and operational reference. This README is the concise entry point for installation and the supported runtime surfaces.

Overview

webarchiver is the fleet's web fetch and scrape connector. It reads public web pages under an explicit policy and returns typed results an agent or the graph can ingest: raw pages, extracted main content with document metadata, rendered DOMs, full-page screenshots and outbound links. It holds no credentials.

Key capabilities

  • Governed fetch — robots.txt, a per-domain rate limit, size and content-type caps, re-checked redirects and egress pinning that refuses private and loopback destinations. Every refusal has a typed reason.
  • Main-content extraction — clean text and markdown with title, author, published date, canonical URL, content digest and links.
  • Rendering — headless Chromium for pages built by scripts, with ad and tracker requests aborted and fixed overlays removed.
  • Screenshots — full-page PNG captures.
  • Site crawl — a bounded breadth-first walk from one seed URL that returns page URLs (up to 1000) under the same robots and rate policy.
  • One tool per operation — web_fetch, web_extract, web_render, web_screenshot, web_crawl and web_links, all read-only.
  • Shared web-source contract — web_fetch and web_extract are the scion-sdk web-source contract's tools: the same request, page result and refusal reasons as every page connector, including the ArchiveBox connector. Each page names its backend (webarchiver).

Quick start

Install the package and run its generated entry points:

python -m pip install "webarchiver[mcp]"
webarchiver-mcp

For an on-demand run, use uvx:

uvx --from "webarchiver[mcp]" webarchiver-mcp

The networked server binds to loopback by default:

webarchiver-mcp --transport streamable-http --host 127.0.0.1 --port 8000

Architecture

The provider keeps the boundary between transport and domain code explicit:

client → MCP entry point → one tool per operation → fetch policy → web
                                  └── skills, prompts, ontology, connector presets

The MCP surface has one typed, read-only tool per operation. Regenerate the tool fingerprints with python scripts/generate_tool_schemas.py after changing a tool contract.

MCP configuration

The server supports local stdio and authenticated streamable-http deployments. Keep endpoint, credential, selector, and TLS references in SDK configuration; do not put resolved values in client configuration.

{
  "mcpServers": {
    "webarchiver-mcp": {
      "command": "webarchiver-mcp",
      "args": [],
      "env": {"SYSTEMTOOL": "true"}
    }
  }
}

For a networked deployment, use an authenticated TLS ingress and keep the allowlist in MCP_ALLOWED_HOSTS. A configured remote example is https://service.example.invalid/mcp; it is documentation-only and is never contacted by the generator.

{
  "mcpServers": {
    "webarchiver-mcp": {
      "command": "webarchiver-mcp",
      "args": ["--transport", "streamable-http", "--port", "8000"],
      "env": {
        "TRANSPORT": "streamable-http",
        "HOST": "127.0.0.1",
        "PORT": "8000",
        "SYSTEMTOOL": "true"
      }
    }
  }
}

Container builds install the published wheel for the pinned version. Publish the release first, then run from the repository root:

docker build --file docker/Dockerfile --target mcp -t webarchiver:<version>-mcp .

Deploy reviewed images by digest (webarchiver@sha256:<digest>) and include --cap-drop=ALL in the runtime policy. The deployment guide contains the complete transport and container examples.

Documentation

Use the Pages documentation for installation details, API capabilities, deployment profiles, platform integration, and the concept registry. The concise local references remain available at:

Development

Create an isolated branch, install the test extra, and run the complete local gate:

uv sync --extra test
pytest tests -q
pre-commit run --all-files

Keep MCP examples and environment documentation aligned with the implemented settings. Add focused positive and adversarial tests for every behavior change. See AGENTS.md for the current module ownership and release checks.

License

This project is distributed under the MIT license. See LICENSE for the complete terms.

Metadata

Release files for webarchiver 1.2.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webarchiver 1.2.4
File Size Uploaded Provenance Actions
webarchiver-1.2.4.tar.gz 2.0 MB Without provenance Details for webarchiver-1.2.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for webarchiver 1.2.4
File Interpreter ABI Platform Actions
webarchiver-1.2.4-py3-none-any.whl Python 3 none any Details for webarchiver-1.2.4-py3-none-any.whl

Total release size: 2.0 MB

Release files / webarchiver-1.2.4.tar.gz

Download URL webarchiver-1.2.4.tar.gz
Size 2.0 MB
Tags Source
SHA-256 checksum
How to use checksums
27e02f04f7c5f809f8b4112e1511635f478b943427a22c353566a817016e8a92
BLAKE2b-256 checksum
How to use checksums
ea022c67c38e491ef0852ad6ff917355a585a00501e7c86effb474e4db925220
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Release files / webarchiver-1.2.4-py3-none-any.whl

Download URL webarchiver-1.2.4-py3-none-any.whl
Size 36.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
00363c55fb120b5f9994e92c63872a96235db7a347369601ca1ad2878e1d4f72
BLAKE2b-256 checksum
How to use checksums
8d805bef4c5b0901db740cbb86b1cb67065353a29b80b66a10764864933e9bed
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Security for webarchiver 1.2.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.4 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page