sluicer 0.7.1

Web scrapers that fail loudly when a site changes,
instead of quietly returning empty values.
No model, no API key, no bill.

Documentation · Getting started · In your agent · Scoreboards · FAQ · Changelog

A real site, as the Wayback Machine kept it. The extractor learnt in January 2016 still fits in February. After the 2024 redesign it stops with exit code 3, and heal shows where the listing and most of its fields moved.

Most scrapers break without a sound. The site changes its markup, and the scraper keeps running and returns nulls, or the wrong column, for weeks before anyone notices.

Sluicer works the other way round. Show it a value on a few pages of a site (a price, a title, a date) and it learns where that value lives. On every page it reads after that, it checks the page against what it learnt. If the layout has changed, the run fails with exit code 3 and names the field that broke, and sluicer heal tells you where each field moved.

It also reads everything a page already declares about itself: JSON-LD, microdata, RDFa, OpenGraph and four more vocabularies, merged into one record per thing, with every value pointing to the exact place on the page it came from. No model reads any page, so the same page always gives the same answer.

Quick start

uv pip install "sluicer[fetch]"

# Learn an extractor from three pages built on one template, given one value from the first.
sluicer compile p1.html p2.html p3.html --want price=41.90 -o shop.json

# Replay it on any page of that template: the values as JSON, or exit 3 if the page changed.
sluicer run shop.json https://shop.example/p/7

# After a redesign: see what moved, and write the healed extractor.
sluicer heal shop.json https://shop.example/p/1 -o shop.json

Exit codes follow grep: 0 found -- a record or a summary answer, a <title> alone included -- 1 found nothing, 2 could not read the page, and 3 when a page broke its extractor's checks, a heal lost a field or left a move undecided, or an audit found a documented rule broken. diff exits 1 when something changed.

The checks are about structure: a run fails when a field is no longer where it was learnt, no longer reads the way it did, or no longer has its shape. A change that keeps all three, such as a different number in the price's place, passes: on SWDE, 17% of the extractors' wrong answers were flagged.

Reading what a page declares needs no example at all. The product page read here is examples/brake-pads.html:

>>> import sluicer
>>> page = open("examples/brake-pads.html", "rb").read()
>>> result = sluicer.extract(page, url="https://example.com/p/bp-2210")
>>> price = result.summary["price"]
>>> price.value, price.source, price.key
('41.90', 'jsonld', 'Product.offers.price')
>>> price.where
'/html/head/script[1]#/offers/price'
>>> result.normalised
{'price': '41.90', 'currency': 'EUR', 'gtin': '4006381333931'}
>>> [(answer.value, answer.source) for answer in result.conflicts[0].answers]
[('41.90', 'jsonld'), ('39.90', 'opengraph')]

The page states one price in its JSON-LD and another in its OpenGraph tags; Sluicer reports the conflict instead of picking one in silence. sluicer inspect page.html shows the same reading laid out for a person.

Install

uv pip install "sluicer[fetch,markdown,mcp]"

pip install works the same way, and uv tool install gives you the command in an environment of its own. The base install, uv pip install sluicer, reads HTML you already have, with lxml and click alone.

What you can give it

Every request names Sluicer, and robots.txt and Crawl-delay are obeyed; a crawl waits when a site asks it to. Only a command that reads single pages or learns an extractor can be told otherwise, with --stealth or --no-robots; map, crawl and batch cannot. sluicer --help lists every command, and the command line reference explains each one.

In your agent

claude mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp   # Claude Code
codex mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp    # Codex

The MCP server has ten read-only tools, among them extract_declared, compile_extractor, run_extractor and heal_extractor. Every answer carries ok, true only when it can be used as it is. The server does not fetch localhost, private networks or cloud metadata addresses unless it is started with SLUICER_ALLOW_PRIVATE=1. In your agent covers Cursor, VS Code, Gemini CLI, Claude Desktop, Zed, LangChain, the OpenAI Agents SDK and Pydantic AI. For any other language, sluicer serve offers the same tools over HTTP (HTTP API).

Measured, losses included

Every number below comes from a public test set, and every scoreboard gives the command that produces it again. Where another tool does better on what a row measures, the row shows it.

Sluicer reads only what a page states in its markup, so on titles, authors and dates it answers less often than tools that also read the visible text, and it invents far fewer dates. Its rules were written while reading the pages of these scoreboards, so the numbers show how it does on pages it was tuned on. The one held-out test is the half of SWDE's sites whose pages and errors were not read while making the rules; its numbers were looked at three times, once to decide whether to keep a rule. There it scores 0.845, including five camera sites that were read before the split and so are not a clean test. bench/PREREG.md records which pages each rule was made on.

When not to use Sluicer

• You need authors or dates that pages do not declare. trafilatura reads them from the visible text and finds more of them. Sluicer's --visible option guesses them too, but it is new and not yet measured on the scoreboards.
• You need an article's full text. sluicer markdown uses trafilatura for it; if you need trafilatura's options or other output formats, use it directly.
• The site blocks bots. Sluicer is not built to get past bot protection: every request names it, unless a single-page command is given --stealth.

Why Sluicer compares it with extruct, trafilatura, Scrapling, Crawl4AI and Firecrawl, and says when each is the better choice.

Learn more

• Getting started, a ten-minute tour, and Extractors, how learning, checking and healing work.
• FAQ, Known limits and What is stable before 1.0.
• Moving from extruct: from sluicer.compat import extruct answers extruct's own calls.
• The full documentation is at https://gi0tto.github.io/sluicer/.

Community

Questions, ideas and what you built go to Discussions. A page Sluicer read wrong is an issue: attach the page, so the fix comes with a test. Report a vulnerability privately, as SECURITY.md explains, and see CONTRIBUTING.md to set up a checkout.

Sluicer is built and maintained by one person. If it saves you time or money, sponsoring it pays for keeping the scoreboards measured and the extractors working as the web changes.

Licence

MIT, except two data files under their own licences: schema.org's type names (CC BY-SA 3.0) and CLDR's month and weekday names (Unicode License v3); the package's licence expression is MIT AND CC-BY-SA-3.0 AND Unicode-3.0. The base install needs lxml and click, both BSD-3-Clause. The extras pull a wider tree that is not all permissive: tld is MPL-1.1, GPL-2.0-only or LGPL-2.1-or-later, orjson is MPL-2.0 alongside Apache-2.0 or MIT, and certifi is MPL-2.0. CI lists every licence in that tree and fails on one nobody has read; NOTICE says more.