Frostwork

Fast HTML extraction for Python and Rust, without a DOM.

Frostwork compiles a set of CSS or XPath selectors and extracts their values in one scan of an HTML response. Reuse the schema across Scrapy callbacks, web-poet pages or standalone code. Working memory tracks parser state, pending matches and returned values, without storing a document tree.

Frostwork supports a focused selector subset, continuously checked against Parsel/lxml. Python rejects unsupported selectors before scanning; strict=False requests empty columns. There is no parser fallback. Check the compatibility contract before migrating code that needs arbitrary tree traversal.

The benchmarks compare throughput, memory and selector coverage with Parsel, lxml and other engines, including workloads where Frostwork loses. Performance depends on page shape, selector count and output volume.

Install

Frostwork requires Python ≥ 3.10. Install the core package, or include the web-poet integration:

pip install frostwork
pip install "frostwork[webpoet]"

Published wheels contain the Rust extension, so installing from PyPI does not need a Rust toolchain. Building an editable checkout does; see the Python development guide.

Extract a named item

Build a Page once, then reuse it for each response:

from frostwork import Page

PRODUCT = (Page()
           .field("name", "h1::text")              # first match, or None
           .field("price", ".price::text")
           .field_all("images", "img::attr(src)")) # every match, or []

html = b"<h1>Widget</h1><span class=price>$9</span><img src=/a.png>"
assert PRODUCT.extract(html).to_dict() == {
    "name": "Widget", "price": "$9", "images": ["/a.png"],
}

In an ordinary Scrapy callback, pass the original bytes and the response's encoding:

def parse_product(self, response):
    yield PRODUCT.extract(response.body, encoding=response.encoding).to_dict()

Use field_all whenever you need every match, and field_join to join text nodes. Without an explicit encoding, Frostwork checks the BOM and declarations before defaulting to UTF-8. For positional columns without field names, use the primitive extract API.

The same engine is a Rust library — frostwork::extract(html, &queries, None), with Page/Plan for named fields and compile-once reuse. See the runnable example.

Scrapy and web-poet

For a runnable Scrapy project, start with frostwork-demo. Its local storefront examples cover ordinary callbacks, ItemLoader, web-poet, nested products and browser rendering, with tests that run real crawls and compare exported items against Parsel. For existing spiders, follow the migration workflow to audit complete schemas, verify saved responses and monitor extracted items.

A page object declares its selectors; Frostwork fills every field from one scan of the response:

from frostwork.webpoet import FrostPage, field

class ProductPage(FrostPage):
    name = field("h1::text")
    price = field(".price::text")
    images = field("img::attr(src)", all=True)
    specs = field(".spec ::text", join=" ")
    brand = field("//meta[@itemprop='brand']/@content")

to_item() returns a dict here. Add Returns[YourItem] for a typed item, as shown in the Python guide.

For injection, install and configure scrapy-poet using its setup guide, or copy the working settings and explicitly assigned callbacks from the demo. Outside Scrapy, construct the page directly: item = await ProductPage(response=http_response).to_item().

Before migrating a spider

Run frostwork-audit --scan myproject/spiders/ to inspect selector literals without importing the spider. Once you have a schema, PRODUCT.check().raise_for_status() checks its selectors and shared budgets. Then compare complete items on saved responses before switching a callback.

The main compatibility boundaries are:

Selector context: grouped fields support fewer shapes than flat fields. Audit the actual schema.
HTML values: bare-element selectors return source HTML, which can differ from lxml serialization.
Encoding: decoding follows browser behavior, including documented differences from Parsel/w3lib.

The compatibility contract contains the exact limits and cases where Frostwork returns values Parsel cannot. The Python guide covers groups, processors, response adapters and runtime item validation.

Build, test, benchmark

make bootstrap     # create .venv and install the pinned Python test toolchain
make ci            # Rust, Python, lxml differential, encoding, and fuzz gates
make bench         # benchmark matrix against Parsel
make soak          # multi-seed differential and fuzz soak

make help lists the individual targets. TESTING.md explains what each gate proves and what remains outside it.

License

Apache-2.0. See LICENSE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

frostwork-0.1.4.tar.gz (650.4 kB view details)

Uploaded Sep 5, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

frostwork-0.1.4-cp310-abi3-win_arm64.whl (526.1 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+Windows ARM64

frostwork-0.1.4-cp310-abi3-win_amd64.whl (546.1 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+Windows x86-64

frostwork-0.1.4-cp310-abi3-win32.whl (517.4 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+Windows x86

frostwork-0.1.4-cp310-abi3-musllinux_1_2_x86_64.whl (888.9 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+musllinux: musl 1.2+ x86-64

frostwork-0.1.4-cp310-abi3-musllinux_1_2_riscv64.whl (841.7 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+musllinux: musl 1.2+ riscv64

frostwork-0.1.4-cp310-abi3-musllinux_1_2_ppc64le.whl (855.7 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+musllinux: musl 1.2+ ppc64le

frostwork-0.1.4-cp310-abi3-musllinux_1_2_armv7l.whl (943.1 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+musllinux: musl 1.2+ ARMv7l

frostwork-0.1.4-cp310-abi3-musllinux_1_2_aarch64.whl (850.2 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+musllinux: musl 1.2+ ARM64

frostwork-0.1.4-cp310-abi3-manylinux_2_31_riscv64.whl (667.3 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+manylinux: glibc 2.31+ riscv64

frostwork-0.1.4-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (667.5 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

frostwork-0.1.4-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (722.4 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+manylinux: glibc 2.17+ ppc64le

frostwork-0.1.4-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (665.5 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+manylinux: glibc 2.17+ ARMv7l

frostwork-0.1.4-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (672.5 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+manylinux: glibc 2.17+ ARM64

frostwork-0.1.4-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl (684.7 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+manylinux: glibc 2.5+ i686

frostwork-0.1.4-cp310-abi3-macosx_11_0_arm64.whl (625.1 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+macOS 11.0+ ARM64

frostwork-0.1.4-cp310-abi3-macosx_10_12_x86_64.whl (639.2 kB view details)

Uploaded Sep 5, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file frostwork-0.1.4.tar.gz.

File metadata

Download URL: frostwork-0.1.4.tar.gz
Upload date: Sep 5, 2026
Size: 650.4 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for frostwork-0.1.4.tar.gz
Algorithm	Hash digest
SHA256	`09bdd969d63e143e6a2cbf40b93a6df4cd458cf0fe7735b689d8c569c359815f`
MD5	`fc78516e8d2e2ca75df75eb2576e3d05`
BLAKE2b-256	`2be9b92b869d6a5455ddb7fcc28b971fa5d0b8e5cfbc74d68f777d292c8a36d2`

Algorithm	Hash digest
SHA256	`7de2abbd54a4122b3f7f46a0be4fa7811810e782c627b217848a33d6be811640`
MD5	`07646e342569171af4c01f1ef715667b`
BLAKE2b-256	`8f6f1c1d24c90efb78d92567f9c58fed7e2b22399358f8e3968f71f57d53a32b`

Algorithm	Hash digest
SHA256	`825a9724caee1d53859031c0b7e40edbb177a270a844b71da618a53af927c544`
MD5	`5089fc11f9b2f67729701cf5ac21fb34`
BLAKE2b-256	`f1196003d79e26eaef6079f45964f4425070877b85e5f12f1ec2165c5925d38b`

Algorithm	Hash digest
SHA256	`1ac62909e36cd89cd3f361ffdec8d61ea60b17e3d5f343ab5966669feb01b0a0`
MD5	`44ed8ca385ebdb970e82a351ea75877d`
BLAKE2b-256	`34667cba0bb604eb6888e8a1692450cb9524aa945dbf63a0527c2cf0d19dbbcc`

Algorithm	Hash digest
SHA256	`0d5aa747fc4f3ae8017896a54eb58fa63f198dab5650f501a95a59b92aa4a912`
MD5	`960d99dc10e9bb6ba1947678f79f5e0a`
BLAKE2b-256	`59016c17311582111d26e4345d79462b923daea2fe6c8ce0b5242d442466d788`

Algorithm	Hash digest
SHA256	`185316046c32612073b9b3a3b117785b258cb0da250118189df7736ecd6efc0a`
MD5	`be278eaf1d7a52c779988cb3675eeeb2`
BLAKE2b-256	`e2a09c398842f2514d9f802e436d90ca893b2b0242c31f971f461ba77d36451a`

Algorithm	Hash digest
SHA256	`96a91e54fa311c363d1a376fa9bb9425e430160dc04a000ddfaa291a20ec575e`
MD5	`c51a16542d8ceed4e1a9829a41a90a82`
BLAKE2b-256	`d59b1f968b4a5b696c5ebb405bfe06a49d98ce58bb7e157a2c62a6f7edda0ab7`

Algorithm	Hash digest
SHA256	`2533fa00c95111a5bdb402b0d85bf029129400a032862b14ed32b5a64963fb33`
MD5	`77717c52e4dc27f02044bd60d52d75b3`
BLAKE2b-256	`472cd55ff54a6ed382978014c04ef72521e45b6f7e172e54c39e77744bf2d0ac`

Algorithm	Hash digest
SHA256	`1cbe14ba9027b3e829216ed2ff040bb69dd7ad33288b7a282cdd566eb31668b6`
MD5	`4d4612b8aa35cf9b77586bf1c24cf722`
BLAKE2b-256	`b498c512b56e3cb0a9be05fc2b2d54632f3cfac81e8202a80ee95d06b5af65e7`

Algorithm	Hash digest
SHA256	`5078b384960cf8d90196695e197025b511e049122a79697f1ea6927388d5ae4a`
MD5	`897ba0bf4132a03ca4555d5b33f7d49c`
BLAKE2b-256	`55fb38878bc5fdc849aeb317961c246ce008ad5f8d1cf78ea115902fbdb77044`

Algorithm	Hash digest
SHA256	`7d652c297e766e7da3162b0109cb668a60e231b16ae88fec4e53ecf08c992b9d`
MD5	`b95fa686e59126565d1045656b4f8c98`
BLAKE2b-256	`dc47961c861298346bdb1fee62b25f5956c563de7f79732177693af2d068a41a`

Algorithm	Hash digest
SHA256	`0b658cbbbbc9f56b7fce818c40651e9275790c48c7d99ca3d3b5ba943f6952f2`
MD5	`84dc40454225451443eb3032fd0b23f4`
BLAKE2b-256	`c6d298de7f448f3440fa1a4da516b0e27b814d1d12c30c327aacf760722bc7fb`

Algorithm	Hash digest
SHA256	`0fa989596ec1a4deff2c02ca0e0476a394315c7799b2885f6fdb8b53c6adefb3`
MD5	`400b7b34b4235063336a8b2dfd031346`
BLAKE2b-256	`7a4ff5cab75a8afa3c8b2583f68e740791f7b0dd011d5ebf4dffc9342b2f439b`

Algorithm	Hash digest
SHA256	`09809f1847605f3b60602a5905b89da656344ddf3d8cc7d335e3c04ce9ed085d`
MD5	`7b31e83daad70b3931ec5baa2a0e559c`
BLAKE2b-256	`7f6ec135521615a7bf6ff0c1a2b369215c898bb300b87a312913a17c2b557ae4`

Algorithm	Hash digest
SHA256	`6e3b7b2ee418a2d54a7f22af96aadce51138924e4346aa83d739dc43357020a9`
MD5	`557af1cd7f1843ebf44ebdccf4a5cdeb`
BLAKE2b-256	`74303f400f75b9b2b740b2002ee156613df9a72905d3d2b1ccb7b62f037beab4`

Algorithm	Hash digest
SHA256	`3dac51448bb62b644b3e358263d912010fced95364f5ae0de861e25278cf1914`
MD5	`fc9811785ca756d1a7e167614e9c0bf5`
BLAKE2b-256	`d73bdd0e55246a547dc2acaedf71361638d3508c1b8df210351d297b6639d888`

Algorithm	Hash digest
SHA256	`929d8d42fdf9f90e90f4254f3e082637c1f6e76eda4c52cf96c3804b3360badf`
MD5	`5f71ffdda682a19c26d0340706be5493`
BLAKE2b-256	`7cfcceca8b685a7f29038b56b327e755a873b8c82d1266a2a1d28f855d4ec4c3`

frostwork 0.1.4

Frostwork

Install

Extract a named item

Scrapy and web-poet

Before migrating a spider

Build, test, benchmark

More

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata