silkworm-rs
Silkworm is an async-first Python web scraping framework built on wreq and scraper-rs. It combines a small, typed Spider/Request/Response API with middleware, output pipelines, and production crawl controls.
Documentation · Getting started · Examples · API reference
Highlights
- Async crawling with concurrency, priorities, request deduplication, timeouts, and deadlock-free queue backpressure.
- Browser-impersonating HTTP through wreq, with redirects, proxies, cookies, retries, throttling, and robots.txt support.
- Push-style typed callbacks using
await self.emit(...)andawait response.follow(...). - Async CSS/XPath selection and optional declarative
Item,Text, andAttrextraction. - File, database, cloud, queue, and message-stream pipelines, with batch processing available on every built-in pipeline.
- Production controls for failure policies, stop limits, pause/resume, caching, graceful shutdown, metrics, and structured crawl statistics.
- Optional CDP, Servo, and OnionLink clients for rendered or onion-service pages.
Install
Silkworm supports Python 3.11–3.15, including PyPy 3.11 and PyPy 3.12 (beta) on Linux x86-64.
pip install silkworm-rs
With uv:
uv add silkworm-rs
Integrations are installed as optional extras. For example:
pip install "silkworm-rs[rsloop,polars]"
See Getting Started for the complete extras and Python-version compatibility table.
Quick start
from silkworm import HTMLResponse, Response, Spider, run_spider
from silkworm.pipelines import JsonLinesPipeline
class QuotesSpider(Spider):
name = "quotes"
start_urls = ("https://quotes.toscrape.com/",)
async def parse(self, response: Response) -> None:
if not isinstance(response, HTMLResponse):
return
for quote in await response.select(".quote"):
text = await quote.select_first(".text")
author = await quote.select_first(".author")
if text is not None and author is not None:
await self.emit({"text": text.text, "author": author.text})
next_link = await response.select_first("li.next > a")
if next_link is not None and (href := next_link.attr("href")):
await response.follow(href, callback=self.parse)
run_spider(
QuotesSpider,
item_pipelines=[JsonLinesPipeline("data/quotes.jl")],
)
Callbacks return None; they report items and requests with emit and follow,
which apply backpressure while the callback is running.
Command line
Run a spider module or test its parser without writing a runner script:
silkworm crawl examples/quotes_spider.py -o data/quotes.jl -s max_items=100
silkworm parse https://quotes.toscrape.com/ --spider examples/quotes_spider.py
See the CLI reference for configuration, output formats, and exit codes.
Documentation
| Topic | Guide |
|---|---|
| Installation, extras, and first spider | Getting Started |
| Spider, request, response, selectors, and callbacks | Core Concepts |
| Declarative item extraction | Declarative Extraction |
| Engine, HTTP, CDP, Servo, and OnionLink | Engine and HTTP Client |
| Request and response middleware | Middlewares |
| Output destinations and batch processing | Pipelines |
| asyncio, rsloop, uvloop, winloop, and Trio | Runners |
| Resumable and observable crawls | Production Crawling |
| Version changes | Migration Guide |
| Constraints and workarounds | Limitations |
Development
uv venv --python python3.13
uv sync --group dev
just fmt && just lint && just typecheck && just test
See the development workflow and open an issue or pull request on GitHub.
Acknowledgements
Silkworm builds on wreq, scraper-rs, fast-h2m, turboxml, and optional integrations including OnionLink and Servo. Thank you to their maintainers and contributors.
License
MIT. See LICENSE.
1 maintainer
Metadata
1 maintainer
Release files for silkworm-rs 0.16.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| silkworm_rs-0.16.0.tar.gz | 562.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| silkworm_rs-0.16.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 753.8 kB
Release files / silkworm_rs-0.16.0.tar.gz
| Download URL | silkworm_rs-0.16.0.tar.gz |
|---|---|
| Size | 562.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cdeb968acd8b1709d552460d63ce89830d0e5e296b14f03b8eed5e42f68bd7b6
|
|
BLAKE2b-256 checksum How to use checksums |
8d1a7bd2c3158cd4e16acdfa934af9628fd3791e89f772d96d69cb46dcbb9bbe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 7, 2026.
Transparency logRelease files / silkworm_rs-0.16.0-py3-none-any.whl
| Download URL | silkworm_rs-0.16.0-py3-none-any.whl |
|---|---|
| Size | 191.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1b5bc0b5b70329fc39febeb2f25458133bb7d91174dc77c735f63f812f673fc6
|
|
BLAKE2b-256 checksum How to use checksums |
9ce41af82ec74f37b4050ff990042e5d25c4ebb01bdf1c234573e4dc3ef17260
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 7, 2026.
Transparency log
