sluicer 0.7.0

Turn any web page, or a whole site, into structured data.
No model, no API key, no bill.

Documentation · In your agent · Scoreboards · Why Sluicer · Discussions · Changelog

Sluicer reads everything a web page declares about itself -- products, articles, recipes, events, people, prices, dates, in JSON-LD, microdata, RDFa, OpenGraph and four more vocabularies -- and merges it into one record per thing, every value naming the vocabulary and the place on the page it came from. Where a page declares nothing, show it one example of the value you want and it learns where that value sits on every page of the site, then says so when the site changes. One page, a list of URLs, or a whole site: no model reads any of them, so the same page always gives the same answer.

📏 Why the scoreboards below talk about title, author and date. Those are the fields public test sets label by hand, so they are what can be scored against a right answer somebody else wrote. They measure a part of what Sluicer reads, not all of it. The products scoreboard measures prices and availability, and SWDE the fields you teach an extractor: 32 kinds, from prices, ISBNs and engines to phone numbers, addresses and job locations, on 80 sites.

SWDE: cars, books, cameras, jobs, films, NBA players, restaurants and universities, 32 kinds of field on 80 sites. On the half of the sites never read while the rules were made, Sluicer scores 0.845. Every number, and how it was made.

Which way in

Highlights

• 🧩 Eight vocabularies, one record. JSON-LD, microdata, RDFa, Dublin Core, OpenGraph, the Twitter card, HTML's meta names and microformats2, merged into one record per thing.
• 🌐 A page or a whole site. map reads a site's sitemaps, crawl follows its links, batch reads a list, feed and warc read feeds and web archives: every page comes out with all it declares, one JSON line each.
• 📍 Provenance for every value. The vocabulary, the key, and the place on the page: an XPath, and inside JSON-LD a pointer to the very value.
• ❓ A summary of 25 questions -- title, author, date, price, currency, availability, GTIN and the rest -- each answer naming where it came from, and a conflict reported when the page answers one of them two ways.
• 🛡️ Extractors that check every page. Learn one from a few pages of a template; a page that drifted exits 3 instead of returning nulls for weeks, and heal says what moved where. On 44 real redesigns none failed silently. On pages of one template they are right on 97% of their answers, and most of the 3% they get wrong pass their checks: the SWDE scoreboard says which.
• ⚡ Deterministic and light. No model and no key: the 511 pages of the WCXB test set are read in 1.4 s, and the base install is three packages.
• 👀 What the page shows, when you ask. --visible guesses the heading, byline and dates a page shows a reader, by rules and no model, each guess naming its element and rule and kept apart from what is declared.
• 🤝 Polite by construction. It announces itself, obeys robots.txt and Crawl-delay, waits a site's Retry-After in a crawl, and honours TDMRep reservations when asked.
• 🤖 Made for agents. An MCP server with ten read-only tools, tried in Claude Code, Codex and Gemini CLI, and the same tools over HTTP for any language.
• 📊 Measured in public, losses included. Six scoreboards against trafilatura, newspaper4k, metascraper, extruct, Scrapling, Zyte and Diffbot.

Install

uv pip install "sluicer[fetch,markdown,mcp]"

With pip, pip install "sluicer[fetch,markdown,mcp]"; as a command in an environment of its own, uv tool install "sluicer[fetch,markdown,mcp]". The base install, uv pip install sluicer, reads HTML you already have with lxml and click alone. Each extra adds one job:

For the browser rung, once: uvx --from "sluicer[fetch]" scrapling install. Without it, plain HTTP still works, and a page that wanted a browser comes back from the HTTP rung with the failed climb recorded.

Quick start

The product page read here is examples/brake-pads.html.

>>> import sluicer
>>> page = open("examples/brake-pads.html", "rb").read()
>>> result = sluicer.extract(page, url="https://example.com/p/bp-2210")
>>> price = result.summary["price"]
>>> price.value, price.source, price.key
('41.90', 'jsonld', 'Product.offers.price')
>>> price.where
'/html/head/script[1]#/offers/price'
>>> result.normalised
{'price': '41.90', 'currency': 'EUR', 'gtin': '4006381333931'}
>>> [(answer.value, answer.source) for answer in result.conflicts[0].answers]
[('41.90', 'jsonld'), ('39.90', 'opengraph')]

The page describes one product in three vocabularies; result.records holds it once, each field with its source and place. The same reading from the command line is sluicer extract for JSON, or sluicer inspect for this:

See it meet a redesign

A real site's software directory, as the Wayback Machine kept it. An extractor learnt from two pages of January 2016 replays a page of February 2016 and exits 0; on the page of June 2024, after the site's redesign, it fails loudly and exits 3, where a selector would have returned nulls. heal then says where the listing and each field went, and on how many of the values it learnt each move rests. Every command ran for real; scripts/demo.py records them again.

Use it

From the command line

sluicer extract page.html                           # a file, a URL, or - for stdin
sluicer inspect https://example.com/product         # the same, for a person to read
sluicer markdown https://example.com/article        # the readable content
sluicer compile page1.html page2.html -o shop.json  # learn an extractor
sluicer run shop.json https://shop.example/c?p=7    # replay it, checked
sluicer heal shop.json https://shop.example/c -o shop.json  # after a redesign

sluicer extract listing.html --induce               # rows of a page that declares nothing
sluicer diff yesterday.html https://shop.example/p  # what changed, and where from
sluicer diff URL URL --at 2024-01                   # since the Wayback Machine's capture
sluicer extract URL --cache ~/.cache/sluicer        # ask the site if it changed (304)
sluicer audit https://example.com/product           # its markup against Google's documentation
sluicer compile p1.html p2.html -o shop.json --want price=41.90 --want title="Brake pads"
sluicer map https://shop.example/                   # a site's addresses, from its sitemaps
sluicer crawl https://shop.example/ -o shop.jsonl   # follow its links, politely; --resume
sluicer batch urls.txt -o pages.jsonl               # read a list, one JSON line per page
sluicer warc crawl.warc.gz > pages.jsonl            # the pages a web archive holds
sluicer feed https://blog.example/                  # a feed's items, from the page that declares it

Exit codes follow grep: 0 found, 1 nothing declared, 2 could not read, and 3 for a page that broke its extractor, a heal that lost a field, or an audit that found a documented rule broken. A drifted page never exits 0.

In your agent

claude mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp   # Claude Code
codex mcp add sluicer -- uvx --with "sluicer[mcp]" sluicer mcp    # Codex

Cursor, VS Code, Gemini CLI, Claude Desktop and Zed, and LangChain, the OpenAI Agents SDK and Pydantic AI, are in In your agent. The repository is also a Claude Code plugin, with a skill in the open Agent Skills format that Codex reads too. The server has ten tools, all read-only.

Every answer carries ok, true exactly when it can be used as it is, and an output schema; every tool says in its annotations that it only reads. The server fetches nothing on localhost, a private network or a cloud's metadata endpoint -- redirects and a browser's requests included -- unless started with SLUICER_ALLOW_PRIVATE=1.

From any other language

sluicer serve answers the same tools over HTTP, POST /v1/tools/<name> with the tool's arguments as JSON, and describes them at /openapi.json. It listens on loopback; anywhere else it needs SLUICER_API_TOKEN. See the HTTP API.

sluicer serve                                       # 127.0.0.1:8000
curl -s http://127.0.0.1:8000/v1/tools/extract_declared \
  -H 'Content-Type: application/json' -d '{"html_or_url": "https://example.com/p"}'

How it compares

from sluicer.compat import extruct answers extruct's own calls in its own shapes, for code written against it: see moving from extruct. Why Sluicer has the full comparison, and says when another tool is the better choice.

Measured, losses included

Five of these scoreboards measure title, author and date, because those are the fields their test sets label; products measure price and availability, and SWDE the fields you teach an extractor. On the first five Sluicer reads only what a page declares, so it answers less often than tools that also read the visible page, and is wrong less often when it answers. A hit rate is right answers over the pages that carry a label; an invention is an answer on a page whose label is empty.

Every scoreboard says how it was made and the command that makes it again; bench/PREREG.md says what each fixes before it is run.

FAQ

Principles

• No LLM call, anywhere in the path. A test fails the build if a model client is ever imported.
• No paid API. A feature that needs somebody's key does not ship.
• Deterministic. The same page always gives the same answer, which is what makes the scoreboards reproducible.
• Honest about failure. A page that cannot be read says so, and nothing returns a plausible answer where the truth was unavailable.

Documentation

At https://gi0tto.github.io/sluicer/, or in the repository: Why Sluicer · Extractors · In your agent · HTTP API · Audit · Crawling · Scoreboard · Scoreboard, as served · Scoreboard, products · Scoreboard, news · Scoreboard, trafilatura's set · Drift · Moving from extruct · Known limits · Design notes · Examples · Roadmap · Changelog · Security · Contributing

Community

Questions, ideas and what you built with Sluicer go to Discussions. A page Sluicer read wrong is an issue, with the page attached, so that the fix comes with a test. A vulnerability is reported privately, as SECURITY.md says. CONTRIBUTING.md says how to set up, what CI checks and where each part of the code lives.

Sluicer is built and kept up by one person. If it saves you time or a bill, sponsoring it keeps the scoreboards measured and the extractors honest as the web changes.

Licence

MIT, with no vendored code, and two exceptions: sluicer/audit/schema_org.py holds schema.org's type and enumeration names, which schema.org publishes under CC BY-SA 3.0, and sluicer/calendar_names.py holds CLDR's month and weekday names, which Unicode publishes under the Unicode License v3; each of the two files is distributed under its own (the package's licence expression is MIT AND CC-BY-SA-3.0 AND Unicode-3.0). The base install needs lxml and click, both BSD-3-Clause. The extras pull a wider tree that is not all permissive: tld is tri-licensed MPL-1.1, GPL-2.0-only or LGPL-2.1-or-later, orjson is MPL-2.0 alongside Apache-2.0 or MIT, and certifi is MPL-2.0. CI lists every licence in that tree and fails on one nobody has read; see the licence notes.