pagesift (Python)

1.2.0: see the release notes.

Python bindings for pagesift: turn web pages into clean Markdown for LLMs. Fetching, robots.txt, rate limiting, content extraction, PDF parsing and browser rendering all run in Rust. No Playwright, no Rust toolchain needed to install.

pagesift obeys robots.txt and Crawl-delay and rate-limits per host; the per-site budget is shared by every egress. A blocked or challenge page is reported as an error (AccessBarrier). When two or more egresses are configured ([[egress]]), a page denied through one egress (an IP block or a challenge page) can be retried through another; rate limiting (429, Retry-After) only backs off.

Quick start

import pagesift

doc = pagesift.scrape("https://example.com/post")
print(doc.markdown, doc.metadata["title"], doc.status_code)

# Many pages: one call, concurrent in Rust, results as they finish
with pagesift.Scraper() as scraper:          # keep one Scraper for the life of the program
    for doc in scraper.scrape_iter(urls, concurrency=16):
        if doc.success:
            save(urls[doc.index], doc.markdown)
        else:
            print(doc.url, doc.error_code, doc.error_message)

Async:

async with pagesift.Scraper() as scraper:
    doc = await scraper.ascrape("https://example.com")
    async for doc in scraper.ascrape_many(urls, concurrency=16):   # completion order
        ...

Convert HTML you already have (no network):

doc = pagesift.convert(html, url="https://example.com/post", only_main_content=True)

Designed for throughput

• Batch first. scrape_many, scrape_iter and ascrape_many take the whole list in one call; the concurrency runs on Rust's async runtime, not in Python threads or Python tasks.
• Stream in completion order. scrape_iter / ascrape_many hand you each page as soon as it is done, so one slow site does not hold back the rest. doc.index maps a result back to its URL.
• Copy only what you read. A Document keeps its fields as Rust values. A field becomes a Python object the first time you read it (strings are then cached), so raw_html you never touch costs nothing. Failed items build their exception object only when you read doc.error.
• Request only what you need. Formats you do not ask for (html, raw_html, links, images) are never computed.
• Reuse one Scraper. It owns the connection pool, the robots.txt cache, rate-limit state and the browser pool. The module-level functions share one default Scraper per process.
• The GIL is released while pages are fetched and processed, so even plain Python threads scale.

Benchmarks

python crates/pagesift-py/bench/bench.py (release build, local HTTP server in a separate process, no internet). 16-thread Linux machine, Python 3.14; the machine was also running other builds (load average ≈ 15), so absolute numbers are noisy, the comparisons are what matter.

1. 200 pages, 50 ms server latency each

One scrape_many call is 15× faster than a loop and as fast as a hand-built thread pool or asyncio.gather, with no threads, semaphores or task bookkeeping on the Python side. At this latency the server is the limit (16 × 1000 / 50 ms = 320 pages/s at most).

2. Time to first result: 1 slow page (2 s) + 31 fast pages

3. Lazy fields: 100 pages of ~200 KB, all formats requested

Converting everything up front would cost every caller the last row; with lazy fields you pay for what you read.

4. Reusing one Scraper vs a new one per page (40 pages, sequential)

On a real network the gap is larger: a new Scraper also redoes DNS, TCP and TLS handshakes and refetches robots.txt.

5. convert on ~200 KB pages with Python threads (GIL released)

API

In batch calls one failed URL never fails the batch: its Document has success=False and error set. A Batch starts scraping as soon as it is created, can be iterated once (with for or async for), supports len(), and batch.cancel() (or dropping it) cancels unfinished pages. concurrency controls batch tasks; per-host concurrency and request spacing are separate limits, and a robots.txt Crawl-delay takes precedence over the configured request interval. Use engine="http" for HTTP-only batches; the default engine="auto" preserves browser fallback for short or client-rendered pages.

Crawl, map, parse

crawl starts as soon as it is called and yields pages as they finish. Failed pages are yielded too (success=False, same shape as failed batch items; index is None); an error in the crawl itself (for example an invalid include_paths regex) is raised from the iteration. Results go through a bounded channel whose capacity is the crawl's concurrency: when the consumer is slower, the crawl waits instead of dropping pages. break (dropping the iterator) or crawl.cancel() stops the crawl. After the iteration ends, crawl.summary is a dict with stop_reason, pages, coverage and term_evidence; crawls with query get it from the engine, other crawls report limit_reached, exhausted or cancelled from the page counts. crawl.total, crawl.pages and crawl.errors are live counters.

map orders links that contain more search terms first and keeps the rest after them. parse detects the type from the content first, then from the file name extension and content_type; unknown types raise UnsupportedOption, as do options that only make sense for URLs (engine="browser", wait_for, mobile, conditional requests). url resolves relative links in HTML; it is never fetched. A parsed Document has the file name as url.

Options

Pass a ScrapeOptions, keyword arguments, or both (keywords override fields of the object). Invalid values raise immediately (InvalidRequest / UnsupportedOption, both also ValueError); unknown keywords raise TypeError.

Scraper(...)

concurrency= on batch calls caps pages in flight for that call; the per-host limits above still apply, so a batch against one host is as polite as single requests.

Document

to_dict() converts every field at once; reading attributes is cheaper when you need only some.

Errors

Every error code has its own exception class; all derive from pagesift.PagesiftError and carry code, message, details (dict), retry_after (seconds or None) and url.

Development

python3 -m venv .venv && . .venv/bin/activate
pip install maturin pytest
maturin develop -m crates/pagesift-py/Cargo.toml
pytest crates/pagesift-py/tests

The tests and the benchmark start a local HTTP server and never touch the internet.

License

AGPL-3.0-only, with a commercial license available for use without AGPL obligations. See the repository's LICENSE and COMMERCIAL.md.