Skip to main content

webscour

A keyless web CLI for agents.
Search, page text, PDFs, sitemaps and a polite crawl. No API key. No account. No log.

PyPI MIT Python 3.11+ Linux No dependencies

wsc (the command webscour installs) asks a local SearXNG first, falls back to scrapers and structured APIs, and prints text sized for a terminal instead of raw HTML. The standard library is the whole stack; the only system binaries are the ones you choose to install for rendering and documents.

Install

pipx install webscour          # recommended: isolated, on PATH as `wsc`
uvx webscour search "pyproject vs setup.py"   # no install at all
uv tool install webscour
pip install --user webscour

Or, on Linux, the installer that also sets up the search backend:

curl -LsSf https://wsc.ashina.live/install.sh | sh
wsc setup                      # installs SearXNG as a systemd user unit
wsc doctor                     # one line: OK, DEGRADED or DOWN

Install with an agent

Paste this into Claude Code, pi, dsh, or any agent that can run a shell:

Install webscour, a keyless web CLI. Run:
  curl -LsSf https://wsc.ashina.live/install.sh | sh
Then run `wsc doctor`. A good report says lightpanda present and the SearXNG unit active, with
one VERDICT line ending in OK. If the installer is unavailable, fall back in this order:
`uv tool install webscour`, then `pipx install webscour`, then `python3 -m pip install --user
webscour` (skip the last one on Debian or Ubuntu if it refuses with externally-managed-environment).

Ask the user before running `wsc setup`, because it installs a systemd user unit on their machine.
Use exactly one install method. The skill file, if your agent supports skills, is at
https://raw.githubusercontent.com/FurBlood344324/webscour/main/skill/SKILL.md

From a checkout, the same thing without a build step:

cd webscour
python3 -m webscour.cli search "pyproject vs setup.py"

System requirements: python 3.11+, lightpanda, linux. No Python dependencies are installed, by design: dependencies = [] in pyproject.toml is a decision, not an omission. Everything else is a system binary, which pip cannot manage and should not pretend to.

  • lightpanda is required. It is the render engine, and wsc doctor fails when it is missing.
  • Linux only because wsc setup installs a systemd user unit for SearXNG. The crawler itself runs on macOS, and lightpanda ships for it too; the limit is the service, not the scraper.
  • Optional: pdftotext (poppler) for PDF, pandoc for docx/epub, markitdown as a last resort, and playwright as a render fallback.

Install lightpanda from one of these, in the order that usually applies:

yay -S lightpanda-bin                       # Arch / AUR; nightly variant: lightpanda-nightly-bin
curl -fsSL https://pkg.lightpanda.io/install.sh | bash -s "<version>"   # any Linux; needs curl, jq, sha256sum
# Debian/Ubuntu: the .deb published with each tagged release
brew install lightpanda-io/browser/lightpanda                          # Homebrew

The one-liner installs to ~/.local/bin/lightpanda; LIGHTPANDA_DIR overrides the directory. wsc setup verifies lightpanda on PATH and never installs it, because a package install that downloads and runs a binary is something a user has to audit rather than trust.

What is required, and what is optional

piece needed for installed by
python 3.11+ everything you
SearXNG search --source web wsc setup (SearXNG only)
lightpanda --render, and the retry of a hollow page you, from the list above
playwright the render fallback, only for a page lightpanda defeats wsc setup --playwright
pdftotext / pandoc / markitdown wsc pdf, and docx/epub you, optional

playwright is the one Python package this project can use, and the tool installs it into a directory it owns rather than into your environment:

wsc setup --playwright              # ~/.local/share/webscour/render-venv, then chromium
wsc setup --playwright --with-deps  # also let playwright install chromium's OS libraries (needs root)

pip install webscour never does this: setup is a command you run, not a post-install hook. The lookup order for the fallback is WEBSCOUR_PLAYWRIGHT_PYTHON, that venv, the running interpreter, then python3; it never searches your environment manager. wsc doctor reports the venv, its health, and whether its chromium build still matches the installed playwright package. A missing playwright is a note. A missing lightpanda is a failure.

wsc setup alone still means "install SearXNG"; --all does both installs in one run.

Agent skill

The repo is also a pi package and a Claude Code marketplace.

pi install npm:webscour    # adds skill/SKILL.md as a skill, no path to remember

The npm package is the skill and a small wsc shim; the tool itself is Python and is installed separately (pipx install webscour). package.json at the repo root declares the skill, so a git source works too; pi/package.json declares the same skill when pi/ is the source.

Claude Code, one step (2.1.275 or later):

/plugin install webscour --marketplace FurBlood344324/webscour

.claude-plugin/marketplace.json lists one plugin whose source is ./skill, with skills: ["."] because that directory is the skill itself rather than a folder of <name>/SKILL.md subdirectories.

First run

wsc setup      # installs the local SearXNG unit, then checks lightpanda
wsc doctor
unit      active (enabled at boot)
listen    127.0.0.1:8888
engines   10 loaded
blocked   abcnyheter, brave, quark, yacy
answered  bing, mwmbl, naver, rawweb, seznam, yacy
silent    abcnyheter, brave, quark, wikibooks
backends  lightpanda=ok  playwright=ok  pdftotext=ok  pandoc=ok  markitdown=ok
VERDICT OK (6/10 engines answering, 4 blocked)

wsc setup is idempotent: a venv under ~/.local/share/searxng, settings under ~/.config/searxng, one systemd user unit on loopback, no docker and no sudo. --check reports state without touching anything, --force rebuilds the venv, --no-start installs without starting.

Then:

wsc search "python packaging pyproject" --limit 3
1. Packaging Python Projects - Python Packaging User Guide - https://packaging.python.org/en/latest/tutorials/packaging-projects/  [naver,seznam]
   This tutorial walks you through how to package a simple Python project. It will show you how to add the necessary files and structure to create the package, how to build the package, and how to uploa…
2. PEP 609 – Python Packaging Authority (PyPA) Governance | peps.py… - https://www.python.org/dev/peps/pep-0609  [mwmbl]
   This document describes a governance model for the Python Packaging Authority (PyPA). The model is closely based on existing informal practices and the …

Each row names the engines that produced it, so a result from one weak upstream is visible as such. --json prints the same rows as objects. Exit codes: 0 answered, 2 nothing answered or a bad verb.

Verbs

Verb What it does
wsc search "<q>" one query, at most 30 rows, engine tags, optional --json, --source selects the backend
wsc fetch <url> a page as text; --max-chars, --grep (context lines around a match), --render, --render-engine, -v
wsc pdf <url|path> a paper or manual as text, through pdftotext, pandoc or markitdown; -v names the backend
wsc map <url> the URL list from sitemap, robots, llms.txt; --limit (default 200); -v names the source
wsc crawl <url> links from that seed, same origin by default, --depth, --limit, --any-origin, -v
wsc setup installs SearXNG and verifies lightpanda; --playwright installs the render fallback into its own venv, --all does both, --check reports without changing anything
wsc doctor unit, listener, engine health, render backends, the render venv, one VERDICT OK|DEGRADED|DOWN line, --json
wsc probe measures which engines this IP can use; --quality --write stores the survivors
wsc cache list|get|prune|path the on-disk captures under ./.webscour/; --save writes one, --fresh bypasses a hit
wsc mcp the same tool over stdio as a JSON-RPC MCP server, one tool named web_scour
wsc tool-argv --json '<params>' the argv builder the MCP server and the pi extension both call, so neither owns a flag string
wsc --self-test 29 offline checks of the parsers, the extraction rules, the router and the shell heuristic

Output is the answer. By default stderr carries failures only, so wsc fetch url > page.txt leaves it empty; -v explains every choice. When a page really is a JavaScript shell, plain HTTP comes back hollow and one render is attempted; that fallback prints exactly one line, render: lightpanda or render: playwright. A short page with real text is complete at any length, so example.com costs no browser launch.

Two more stderr lines a caller can rely on: route: <name> when a known site is read through its own API rather than scraped, and cache: hit (2h ago) when a capture is replayed instead of fetched again.

Exit codes: 0 the question was answered, 1 a usable answer was not produced and the reason is on stderr (a 404, a block, an empty document), 2 the command line was wrong. A non-zero exit is a real failure, never an empty result.

Sources

--source Service Notes
web (default) local SearXNG, then DuckDuckGo lite, DuckDuckGo html, Bing each row tagged [naver,bing]; a fallback prints one backend <name>: <reason> line
github api.github.com repositories and issues unauthenticated, 10 requests/min, remaining budget printed
so Stack Exchange API, site=stackoverflow answered/open, tags
hn hn.algolia.com story points, comment counts
arxiv export.arxiv.org Atom title, id URL, abstract
discourse:<base> <base>/search.json e.g. discourse:https://forums.developer.nvidia.com

A fetch on a forum topic URL (forums.developer.nvidia.com/t/...) reads the topic's .json form and prints #post author with plain text, which is far cleaner than the HTML and needs no render.

Privacy

Only the query leaves the machine, through the local SearXNG to the upstream engines. No cwd, no host, no user, no project name, no identifiers, no telemetry, and no log file: the tool writes nothing except its one-second pacing stamp in ~/.cache/web/last-live.

The SearXNG unit drops the query from its own logs. Access log off, level WARNING, autocomplete off (no keystroke leaves the box), and the bundled serve.py filters every log record so that any URL query string becomes ?<query withheld>. The journal keeps engine errors and none of the terms. That is a feature, and the reason the unit runs a wrapper instead of stock SearXNG.

This is not anonymity. Your IP talks to whichever upstream engines are enabled; a local proxy does not change that. Send generic technical terms and keep private context out of a query.

map and crawl send only a URL to the target site, never a query, and every live request is paced to one per second across processes, so a shell loop cannot look like a scraper.

How it differs from firecrawl, and from an MCP server

firecrawl is a hosted pipeline for turning sites into clean markdown at scale: crawl jobs, JS rendering farms, structured extraction, and an API key. webscour is the opposite shape, a local tool for one page or one query, with no account, no credits and no state, and it tells you which engine or backend answered instead of hiding the plumbing. An MCP server is a transport: it puts the same kind of tool behind a protocol so a chat client can call it, and then something still has to be installed, run and authenticated. wsc is a plain executable that an agent, a shell script or a human calls the same way, which is why it has no server, no registered tool, and no process to keep alive. If you already run an MCP server for this, the only thing webscour adds is the local search backend and the pacing.

Limitations

  • Linux only, because of the systemd user unit. Without the unit the scrapers answer from a residential IP, and mostly they answer CAPTCHA.
  • The scrapers degrade on residential IPs. Measured on 2026-11-13 with 30 generic queries, per engine; a meta-search needs coverage (how often an engine helps) and precision (share of its rows that are on topic), kept means coverage >= 10 % and precision >= 60 %:
engine cover prec rows engine cover prec rows
naver 90.0 % 80.1 % 403 wikibooks 20.0 % 93.8 % 16
abcnyheter 70.0 % 95.3 % 214 wiby 13.3 % 42.9 % 21 (dropped)
rawweb 53.3 % 98.8 % 82 wikiversity, wikiquote <=6.7 % 100 % 1-2 (dropped)
bing 36.7 % 60.0 % 180 encyclosearch, wikisource 3.3 % 50 % 2-14 (dropped)
yacy 30.0 % 96.7 % 90 brave 0 % this pass n/a kept by name

duckduckgo, google, mojeek, qwant, startpage, searchmysite and yandex answered CAPTCHA, "access denied" or a parsing error from a residential IP and are not in the set. Median merged query 5.0 s. Kept: naver, abcnyheter, rawweb, bing, yacy, wikibooks, brave. wsc probe --quality --write repeats the measurement on your own IP; the table above is one machine on one day, not a promise.

  • No render backend is installed by pip. lightpanda is the one to install, and playwright is the fallback for a page that defeats it. wsc doctor says which is present.
  • Scanned PDFs have no text layer. wsc pdf says so and stops; OCR is out of scope.
  • Snippets are quotes from a scraped page. Open the URL before relying on a claim, and treat a row whose title and url lack every query term as drift, which the tool drops.
  • GitHub is unauthenticated: 10 requests/min per IP, and --source github makes two calls.

Development

Everything a release needs runs on this machine, offline:

python3 -m venv .venv-dev && .venv-dev/bin/pip install ruff build twine
./scripts/check.sh          # lint, format, tests, --self-test, build, twine, guards
BENCH=1 ./scripts/check.sh  # also the extraction benchmark (needs the network)
./scripts/release.sh 0.2.0  # local: gate, stamp, changelog, tag. Never pushes or publishes

scripts/check.sh is the only definition of "releasable": it prints one PASS <name> line per check and a summary, and the workflows in .github/workflows/ are thin wrappers that call it. It runs with no network, no SearXNG and LIGHTPANDA=0.

The extraction benchmark compares our reading text with trafilatura's on six pages, from captured fixtures in tests/fixtures/bench/, and fails when a page regresses past tests/baselines/. The reference extractor is a throwaway venv, never a dependency:

python3 -m venv /tmp/tfvenv && /tmp/tfvenv/bin/pip install trafilatura
python3 tests/bench_extract.py                    # table, and a check against the baseline
python3 tests/bench_extract.py --live             # re-fetch the pages first
python3 tests/bench_extract.py --save-baseline tests/baselines/0.2.0.json

Design decisions live in docs/decisions/, one short record each, with the evidence and the condition that would reverse it.

License

MIT. See LICENSE.

Metadata

Release files for webscour 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webscour 0.1.2
File Size Uploaded Provenance Actions
webscour-0.1.2.tar.gz 119.9 kB With provenance Details for webscour-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for webscour 0.1.2
File Interpreter ABI Platform Actions
webscour-0.1.2-py3-none-any.whl Python 3 none any Details for webscour-0.1.2-py3-none-any.whl

Total release size: 215.8 kB

Release files / webscour-0.1.2.tar.gz

Download URL webscour-0.1.2.tar.gz
Size 119.9 kB
Tags Source
SHA-256 checksum
How to use checksums
241c1f46a712aafc8a7d54af84c1b30ecf6e73a46d2b51eabe9a76582e4eabb7
BLAKE2b-256 checksum
How to use checksums
12f53eddf8f2cdada6b332137bc1ca521168ee1bc80a3ca7681171539623a5fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / webscour-0.1.2-py3-none-any.whl

Download URL webscour-0.1.2-py3-none-any.whl
Size 95.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
eaa480798d6501b33f20e319a1439d35af517c0b8c72a69be1b8d065a179e51f
BLAKE2b-256 checksum
How to use checksums
df29c7557d69f25e69191d580af5ec5e1caa0a5f900048938a5c5726a4f63d07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for webscour 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for webscour-0.1.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page