Skip to main content

Webarchiver

GitHub Repo stars GitHub forks GitHub contributors GitHub license GitHub last commit (by committer) GitHub pull requests GitHub closed pull requests GitHub issues GitHub top language GitHub language count GitHub repo size GitHub repo file count (file type) PyPI - Version PyPI - Downloads PyPI - License PyPI - Wheel PyPI - Implementation MCP Server

Governed web fetch, extract, render, screenshot and link connector

The documentation site contains the full API, deployment, security, and operational reference. This README is the concise entry point for installation and the supported runtime surfaces.

Overview

webarchiver is the fleet's web fetch and scrape connector. It reads public web pages under an explicit policy and returns typed results an agent or the graph can ingest: raw pages, extracted main content with document metadata, rendered DOMs, full-page screenshots and outbound links. It holds no credentials.

Key capabilities

  • Governed fetch — robots.txt, a per-domain rate limit, size and content-type caps, re-checked redirects and egress pinning that refuses private and loopback destinations. Every refusal has a typed reason.
  • Main-content extraction — clean text and markdown with title, author, published date, canonical URL, content digest and links.
  • Rendering — headless Chromium for pages built by scripts, with ad and tracker requests aborted and fixed overlays removed.
  • Screenshots — full-page PNG captures.
  • Site crawl — a bounded breadth-first walk from one seed URL that returns page URLs (up to 1000) under the same robots and rate policy.
  • One tool per operation — web_fetch, web_extract, web_render, web_screenshot, web_crawl and web_links, all read-only.
  • Shared web-source contract — web_fetch and web_extract are the scion-sdk web-source contract's tools: the same request, page result and refusal reasons as every page connector, including the ArchiveBox connector. Each page names its backend (webarchiver).

Quick start

Install the package and run its generated entry points:

python -m pip install "webarchiver[mcp]"
webarchiver-mcp

For an on-demand run, use uvx:

uvx --from "webarchiver[mcp]" webarchiver-mcp

The networked server binds to loopback by default:

webarchiver-mcp --transport streamable-http --host 127.0.0.1 --port 8000

Architecture

The provider keeps the boundary between transport and domain code explicit:

client → MCP entry point → one tool per operation → fetch policy → web
                                  └── skills, prompts, ontology, connector presets

The MCP surface has one typed, read-only tool per operation. Regenerate the tool fingerprints with python scripts/generate_tool_schemas.py after changing a tool contract.

MCP configuration

The server supports local stdio and authenticated streamable-http deployments. Keep endpoint, credential, selector, and TLS references in SDK configuration; do not put resolved values in client configuration.

{
  "mcpServers": {
    "webarchiver-mcp": {
      "command": "webarchiver-mcp",
      "args": [],
      "env": {"SYSTEMTOOL": "true"}
    }
  }
}

For a networked deployment, use an authenticated TLS ingress and keep the allowlist in MCP_ALLOWED_HOSTS. A configured remote example is https://service.example.invalid/mcp; it is documentation-only and is never contacted by the generator.

{
  "mcpServers": {
    "webarchiver-mcp": {
      "command": "webarchiver-mcp",
      "args": ["--transport", "streamable-http", "--port", "8000"],
      "env": {
        "TRANSPORT": "streamable-http",
        "HOST": "127.0.0.1",
        "PORT": "8000",
        "SYSTEMTOOL": "true"
      }
    }
  }
}

Container builds install the published wheel for the pinned version. Publish the release first, then run from the repository root:

docker build --file docker/Dockerfile --target mcp -t webarchiver:<version>-mcp .

Deploy reviewed images by digest (webarchiver@sha256:<digest>) and include --cap-drop=ALL in the runtime policy. The deployment guide contains the complete transport and container examples.

Documentation

Use the Pages documentation for installation details, API capabilities, deployment profiles, platform integration, and the concept registry. The concise local references remain available at:

Development

Create an isolated branch, install the test extra, and run the complete local gate:

uv sync --extra test
pytest tests -q
pre-commit run --all-files

Keep MCP examples and environment documentation aligned with the implemented settings. Add focused positive and adversarial tests for every behavior change. See AGENTS.md for the current module ownership and release checks.

License

This project is distributed under the MIT license. See LICENSE for the complete terms.

Metadata

Release files for webarchiver 1.2.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webarchiver 1.2.6
File Size Uploaded Provenance Actions
webarchiver-1.2.6.tar.gz 2.0 MB Without provenance Details for webarchiver-1.2.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for webarchiver 1.2.6
File Interpreter ABI Platform Actions
webarchiver-1.2.6-py3-none-any.whl Python 3 none any Details for webarchiver-1.2.6-py3-none-any.whl

Total release size: 2.0 MB

Release files / webarchiver-1.2.6.tar.gz

Download URL webarchiver-1.2.6.tar.gz
Size 2.0 MB
Tags Source
SHA-256 checksum
How to use checksums
c2178fb89c1eed6cd453fa4343c0083af8674b2ae69da1bc412143333e1acf39
BLAKE2b-256 checksum
How to use checksums
b3a3dd2a69efc2f25ac3d0fd4ebf44c1c992625e1b23b1789c4ffba341627a91
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Release files / webarchiver-1.2.6-py3-none-any.whl

Download URL webarchiver-1.2.6-py3-none-any.whl
Size 48.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f1856488d805e0951b59253556069d4dbfd5f2bec1de34077fb4a1e9a2b08fd2
BLAKE2b-256 checksum
How to use checksums
ce8cf789d24e759b1ae2c2c7d72319653073ebc30cf37ea889fd261e3890b754
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Security for webarchiver 1.2.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.6 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page