Frostwork

Fast HTML extraction for Python and Rust, without a DOM.

Frostwork compiles a set of CSS or XPath selectors, scans an HTML response once, and emits only the requested values. It does not build a document tree, so working memory tracks parser state and pending matches rather than the whole page. Supported results are continuously checked against lxml; the exact coverage and known differences — in both directions — are listed in the compatibility contract.

~14× faster than Parsel (what Scrapy uses) and ~8× faster than lxml at the median on the measured production-selector corpus, and ~7× faster than selectolax/lexbor on the workload both can express. Often much faster on large, selector-rich product and listing pages, where each of them must traverse a DOM per field.

Because Frostwork never builds that DOM, working memory stays essentially constant as page size grows for a fixed-output schema; it scales with parser state and returned values instead of the page tree. Results depend on page shape, selector count and output volume; BENCHMARKS.md has the full methodology and performance boundaries.

Frostwork deliberately supports a focused subset of CSS and XPath. Python fails before scanning when a selector is unsupported; strict=False opts into an empty column instead. Unsupported selectors never fall back to another parser or produce guessed results. If an application needs arbitrary DOM access, use lxml — Frostwork is for schemas known in advance.

Install

Until packages are published, install from the repository. Building the extension needs a Rust toolchain (stable) alongside Python ≥ 3.9:

python -m venv .venv
.venv/bin/pip install maturin
.venv/bin/maturin develop --release --extras=webpoet   # compiles the Rust core; needs cargo/rustc on PATH

--extras=webpoet installs the supported web-poet release; drop it if you only need extract/Page. That extra requires Python ≥ 3.10, while the core supports Python ≥ 3.9.

Extracting values

The primitive API takes the response body and all selectors together, and answers them in one scan:

from frostwork import extract

html = b"<h1>Widget</h1><span class=price>$9</span><a href=/p/1>buy</a>"
title, price, link = extract(html, [
    "h1::text",
    ".price::text",
    "a::attr(href)",
])

assert title == ["Widget"]
assert price == ["$9"]
assert link == ["/p/1"]

extract(..., encoding="windows-1252") accepts the charset label a Scrapy response supplies; without one, Frostwork checks the BOM and <meta> declarations before defaulting to UTF-8. frostwork.Page adds names and per-field cardinality for applications that do not use web-poet.

The same engine is a Rust library — frostwork::extract(html, &queries, None), with Page/Plan for named fields and compile-once reuse. See the runnable example.

Scrapy and web-poet

A page object declares its selectors; Frostwork fills every field from one scan of the response:

from frostwork.webpoet import FrostPage, field

class ProductPage(FrostPage):
    name = field("h1::text")
    price = field(".price::text")
    images = field("img::attr(src)", all=True)
    specs = field(".spec ::text", join=" ")
    brand = field("//meta[@itemprop='brand']/@content")

to_item() returns a dict here. Add Returns[YourItem] for a typed item, as shown in the Python guide.

Install scrapy-poet and enable it in settings.py with ADDONS = {"scrapy_poet.Addon": 300} (Scrapy ≥ 2.10; see scrapy-poet's setup guide for older versions). It then builds the page object from the callback's annotation:

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalogue/"]

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product)

    async def parse_product(self, response, page: ProductPage):
        yield await page.to_item()

Requests with callback=None — including those created from start_urls — do not reliably receive dependencies in parse. Use an explicitly assigned callback for injected page objects. Outside Scrapy, construct the page directly: item = await ProductPage(response=http_response).to_item().

This repository does not pin or test a Scrapy/Twisted matrix; use scrapy-poet's documentation for setup details. Frostwork does gate the injection boundary: every shipped page base is injectable and can be planned as a callback dependency. Field processors, groups, response types and schema auditing are covered in the Python guide.

How it differs from lxml

Frostwork is not a subset of lxml, and not a superset — the set difference runs both ways, and both halves are proven by the same gates.

What it does not answer. Supported CSS is tags, IDs, classes, attribute operators, descendant/child/sibling combinators, :not(), :is()/:where(), subject :has(), :contains(), and structural positions such as :nth-child()/:last-of-type; supported XPath is downward paths, attribute and text predicates, unions, positional predicates, following-sibling::, ancestor::, parent:: and top-level normalize-space(); values are text, attributes, descendant attributes, joined text and raw outer HTML. Anything that cannot be answered without retaining more tree state stays unsupported, and reverse positions and :has() have placement restrictions. An unsupported selector never falls back and never guesses — check() reports the verdict before a scrape, and frostwork-audit --scan myproject/spiders/ classifies selector literals in existing code without importing it.

What it answers and lxml does not. A dozen constructs run here and refuse, truncate or mis-decode there:

Valid CSS cssselect rejects — div:has([data-x]), div:has(a, img), p:not(.a, .b), [type=submit i]. Parsel raises SelectorSyntaxError; Frostwork matches them with the semantics the spec defines.
Values libxml2 drops — names longer than its 100-byte buffer, and everything after a </html>, which real pages put before the content that matters (one crawled page keeps 14 of its 17 KB there).
Encoding, the widest gap — and the one place the browser, not lxml, is the standard. parsel.Selector(body=…) never sniffs <meta charset>; frostwork.detect_encoding does, at any depth and past the <body> and unsupported-label cut-offs where w3lib gives up. It reads a UTF-16 body, which lxml's HTML parser cannot parse at all, and decodes with the WHATWG indexes browsers use — where Python's legacy codecs return U+FFFD for 457 euc-jp and 192 big5 sequences of ordinary prose. The rule is never diverge from the browser: a price is right when it matches what the site shows a person, so where Frostwork's text differs from Scrapy's on an oddly-encoded page, Frostwork is the one agreeing with the browser. Every such difference is tabulated with its reason.
A schema verdict before the scrape, and one pass for the whole schema — a page object of single-valued fields stops scanning once every field has a value, which a tree parser cannot do.

Three of those return extra values, the one direction that can surprise a port. The compatibility contract lists supported, divergent, beyond and unsupported forms with examples, the gate behind each, and the migration caveats.

Build, test, benchmark

make bootstrap     # create .venv and install the pinned Python test toolchain
make ci            # Rust, Python, lxml differential, encoding, and fuzz gates
make bench         # benchmark matrix against Parsel
make soak          # multi-seed differential and fuzz soak

make help lists the individual targets. TESTING.md explains what each gate proves and what remains outside it.

Frostwork is usable from source but not yet published to PyPI.

License

Apache-2.0. See LICENSE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

frostwork-0.1.0.tar.gz (606.2 kB view details)

Uploaded Aug 19, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

frostwork-0.1.0-cp310-abi3-win_arm64.whl (519.0 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+Windows ARM64

frostwork-0.1.0-cp310-abi3-win_amd64.whl (538.5 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+Windows x86-64

frostwork-0.1.0-cp310-abi3-win32.whl (509.3 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+Windows x86

frostwork-0.1.0-cp310-abi3-musllinux_1_2_x86_64.whl (881.2 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+musllinux: musl 1.2+ x86-64

frostwork-0.1.0-cp310-abi3-musllinux_1_2_riscv64.whl (832.7 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+musllinux: musl 1.2+ riscv64

frostwork-0.1.0-cp310-abi3-musllinux_1_2_ppc64le.whl (848.3 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+musllinux: musl 1.2+ ppc64le

frostwork-0.1.0-cp310-abi3-musllinux_1_2_armv7l.whl (930.0 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+musllinux: musl 1.2+ ARMv7l

frostwork-0.1.0-cp310-abi3-musllinux_1_2_aarch64.whl (842.7 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+musllinux: musl 1.2+ ARM64

frostwork-0.1.0-cp310-abi3-manylinux_2_31_riscv64.whl (659.9 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+manylinux: glibc 2.31+ riscv64

frostwork-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (658.5 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

frostwork-0.1.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (716.4 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+manylinux: glibc 2.17+ ppc64le

frostwork-0.1.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (654.5 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+manylinux: glibc 2.17+ ARMv7l

frostwork-0.1.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (664.3 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+manylinux: glibc 2.17+ ARM64

frostwork-0.1.0-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl (674.7 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+manylinux: glibc 2.5+ i686

frostwork-0.1.0-cp310-abi3-macosx_11_0_arm64.whl (618.7 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+macOS 11.0+ ARM64

frostwork-0.1.0-cp310-abi3-macosx_10_12_x86_64.whl (632.1 kB view details)

Uploaded Aug 19, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file frostwork-0.1.0.tar.gz.

File metadata

Download URL: frostwork-0.1.0.tar.gz
Upload date: Aug 19, 2026
Size: 606.2 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for frostwork-0.1.0.tar.gz
Algorithm	Hash digest
SHA256	`42066ebee9404155015cd71fe8a76c42def0aa993177e0823b68492302275a41`
MD5	`2038574df87cf8c4393f1beb8bf70eec`
BLAKE2b-256	`a4b325380caf1829cd997a5b951f601e229c13045cfc8620723ef23421c48438`

Algorithm	Hash digest
SHA256	`ba960c85a0e84143b9a7eaadd2a519b05826bdf45eb7c817cb0d04781c2952ee`
MD5	`b7a5bdfb124058d2a08ff79cad7c7391`
BLAKE2b-256	`5ae97da2a8269a8deea0cafd6992572f559ace9026b9cf36a71ab8a030b66142`

Algorithm	Hash digest
SHA256	`4f3dc4b286dbb5de50b1cf8e4dea4d70cf169691d8fd892271240f590641e40a`
MD5	`f11118673ce572dd1fc4e7d7b887c8f7`
BLAKE2b-256	`7a5c635fcb5d32f3e4c053652540a4e3039321b62a33d2969e96e7a73ab8b888`

Algorithm	Hash digest
SHA256	`16218c8ebd52642cf70c209208cc2561d28bee4eb1c12861ae477ea372013f63`
MD5	`4f981c61638060933a78d42572cae9e4`
BLAKE2b-256	`9256957b8c28259b9ad591c5fb1a7ec6014331e15d35e015045f424d834a462e`

Algorithm	Hash digest
SHA256	`ae0ef11bbae688d07d13e62fddc9bc2215d8d8912088155d45a596f6d72cd464`
MD5	`3028778d08a300b7a18e30a62132fe9d`
BLAKE2b-256	`b3e7f259fb5cea5faeead4cc60dca6c099b94962b2a6f9521fad8648b527bcf9`

Algorithm	Hash digest
SHA256	`dd574a032d7475b048578547fb7428e3d62d81b0e627fb205378d26238c2b5fc`
MD5	`072e63a3a3e37a9cb5fb391d5d7ebcd0`
BLAKE2b-256	`c6aa8dcb1f63b7d424754987466deae337d944001a53b2723ffe696931aac52d`

Algorithm	Hash digest
SHA256	`feb77d617c4637705e07eb9dba2864aa3c3c4931e4a7cb6e47d3eb294fb9521b`
MD5	`edfa654bdba91cf67828e557118089c0`
BLAKE2b-256	`37cb8c99b3db92f7afb3cdd7ff88a9d1b30032e64cf5f0d24527967ec5fc780d`

Algorithm	Hash digest
SHA256	`d74d4e154a720fd09f40f4e3743905e7b7f2a53db805d356b7548f7e99685847`
MD5	`4974a2f0a4f4e8868847c4cf18081147`
BLAKE2b-256	`129ade174d5717e51fcfa6ecb763e21a8ca51055f4b8c20aa9e557b1ec8fe9dc`

Algorithm	Hash digest
SHA256	`ba49e6a16965e0a9bf87783241b4abc8fd441cdc3ed8a2c77443a831bd017995`
MD5	`ca699aba725ce783bb633bcc6d84641b`
BLAKE2b-256	`72e3380cf68942da79a4dc598b580e95e7ff271e2d2369749ce65c8c748d4033`

Algorithm	Hash digest
SHA256	`ae703ba4137eba3cb1e04f7b3df934a1daefdc12fdbd34a779dae7294c8e0c51`
MD5	`3b5d34022a3c9597200854e973f9f9fb`
BLAKE2b-256	`344c8e769782f94f796f63dd08f6d9f177bc3ed6e4e0b074cfdb963746cf9545`

Algorithm	Hash digest
SHA256	`d36dd3b1671ce1e483623b0a719479c743738c387dced8fb0cfb77639df084dd`
MD5	`9038055553a3c18bf1dabe9562e4d64a`
BLAKE2b-256	`aa86ca785a7b8e9cfb2b3dd599e16b4f16c8f793bf75810148158ea85a4d923a`

Algorithm	Hash digest
SHA256	`a1c6a8674bab1dcdda79d0d7d623b71bbd94bf30300393f52af6c41e891ed08f`
MD5	`5f0e350d464424eab9caa6753a65e324`
BLAKE2b-256	`3f871561ec4b256377b10b09634fcc10bd7fddb7211816a4b973c4aa26dd1d08`

Algorithm	Hash digest
SHA256	`58fd01d5bf3126922cba2566ab425cb40e39bb9de5d915b9f8e7d4433febbc9b`
MD5	`286dc11719b1292793d7a1d6846f76f5`
BLAKE2b-256	`9d51c9e2f78a9d04e967aabfd601f9f304f71b716918889714f36a51f8da733e`

Algorithm	Hash digest
SHA256	`b6a638580ff7f7c6496152163cda7e7721ed72474e71bf74a637eb889191470d`
MD5	`d998de7fa368f024e44ace2326847330`
BLAKE2b-256	`fc58666e85bda871ef86ca4fa4e1faea81a0ebdbc01437767bb9e11897dc8985`

Algorithm	Hash digest
SHA256	`9afd62ae610203010274a450c600ad0dd00ec88f6f9b998e20b8e72b40499d69`
MD5	`d00a14be1fbf131e5bc64505f9f16a6d`
BLAKE2b-256	`890baabdd415bfbea8db681d836ef010064cf4109028b8228162a7b139336c13`

Algorithm	Hash digest
SHA256	`7186a2d221db97240e88083e3f4198f882667a47080d08c43a7fd30b12b04998`
MD5	`d1c7b3dcfb281948b8b1d3a31f540e38`
BLAKE2b-256	`800b380c1ee59b64afdf0a449bf1bcc8823a485ebd8676f915144017ddd1ca70`

Algorithm	Hash digest
SHA256	`fbbae1565769293ea07be7a3e239d81e861a6536427ba57fc24e42c73b89680d`
MD5	`da510bee96bef5c76749e849530551ae`
BLAKE2b-256	`9e00722a4c25e305bd06f8e376d4c855d47538673d8618a33425acdbb07e5abf`

frostwork 0.1.0

Frostwork

Install

Extracting values

Scrapy and web-poet

How it differs from lxml

Build, test, benchmark

More

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata