remove-paywall-mcp

MCP server that removes article paywalls by searching internet archives. Give it a URL, get back the article text.

How it works

1. You give it a paywalled article URL
2. Tracking params are stripped and the URL is normalized
3. It tries multiple approaches in parallel:
   • Direct fetch with Googlebot user-agent (FT, WSJ, and many others serve full content to crawlers)
   • 12ft.io proxy for hard paywalls
   • iitty textise for plain-text rendering
   • Wayback Machine CDX API, archive.ph/is mirrors, and Wayback Availability API
4. It extracts the article body with readability-lxml, stripping navigation, ads, and sidebar cruft
5. Post-extraction check: if the result still contains paywall text (e.g., the snapshot captured the paywall itself), it retries with the next archive
6. It returns clean text with the title and snapshot URL

It also learns from every attempt — success rates per domain per archive source are tracked in a local SQLite database, and archive search order is re-ranked automatically using Laplace smoothing.

Install

# zero-install (recommended — works everywhere uvx is available)
uvx remove-paywall-mcp

# from PyPI
pip install remove-paywall-mcp

# from source
pip install git+https://github.com/jasval/remove-paywall-mcp.git

# Docker
docker run -i --rm remove-paywall-mcp
docker compose up -d  # HTTP mode on port 8000

Platform configs

Once installed, add this to your MCP client config:

OpenCode

{
  "mcp": {
    "remove-paywall": {
      "type": "local",
      "command": ["uvx", "remove-paywall-mcp"],
      "enabled": true
    }
  }
}

Claude Desktop

{
  "mcpServers": {
    "remove-paywall": {
      "command": "uvx",
      "args": ["remove-paywall-mcp"]
    }
  }
}

LiteLLM

mcp_tools:
  remove_paywall:
    type: "stdio"
    command: "uvx"
    args: ["remove-paywall-mcp"]

Docker (any client)

{"command": "docker", "args": ["run", "-i", "--rm", "remove-paywall-mcp"]}

Tools

remove_paywall

Main tool. Removes a paywall from an article URL and returns clean article text.

search_archives

Search all archive sources for snapshots without extracting content. Useful to see what's available.

get_from_archive

Fetch from a specific archive source.

domain_info

Look up a domain in the knowledge base — paywall status, notes, and per-archive success rates.

add_domain

Register a domain in the knowledge base. Mark paywalled domains so archives are searched first, or non-paywalled domains so the live page is fetched directly.

Prompts

The server provides 3 prompt templates for LLMs to use the tools effectively.

remove_paywall_prompt

Full instruct for bypassing a specific URL. Tells the assistant to use remove_paywall, fall back to search_archives, and check domain_info.

bypass_paywall

Short alias — just tells the assistant to call remove_paywall on the URL.

handle_paywalls

System prompt fragment. No arguments — returns instructions for the assistant to automatically call remove_paywall whenever it encounters a paywall, login wall, or metered content. Paste this into your system prompt or load it as a prompt at session start.

Domain knowledge base

Seeded with 33 well-known paywalled domains (NYT, WSJ, Bloomberg, Medium, etc.), stored in SQLite at ~/.remove-paywall-mcp/domains.db. Tracks every archive success/failure per domain and re-ranks archive search order automatically — domains where archive.is consistently fails won't waste time on it.

Env vars

Archive sources

Priority is dynamically re-ranked per domain based on historical success rates recorded in the knowledge base.

Architecture

MCP client (Claude/OpenCode/LiteLLM)
       │  stdio or HTTP
       ▼
┌─────────────────┐
│    server.py     │  MCPServer with 5 tools
│  +3 prompts       │
└────────┬────────┘
         │
    ┌────┴────┐
    ▼         ▼
┌────────┐ ┌──────────┐
│archives│ │ extractors│
│  .py   │ │   .py     │
│        │ │           │
│googlebot│ │readability│
│ 12ft   │ │Beautiful  │
│ iitty  │ │Soup       │
│ wayback│ │           │
│ archive│ │           │
│ .is/ph │ │           │
│ wayback│ │           │
│ avail  │ │           │
└───┬────┘ └──────────┘
    │
    ▼
┌──────────────┐
│domain_store  │
│   .py        │
│              │
│ SQLite knows │
│ which domains│
│ have paywalls│
│ and which    │
│ archives work│
│ best (Laplace│
│ smoothed)    │
└──────────────┘

License

MIT