pagesift (Python)

Python bindings for pagesift: turn web pages into clean Markdown for LLMs. Fetching, robots.txt, rate limiting, content extraction, PDF parsing and browser rendering all run in Rust. No Playwright, no Rust toolchain needed to install.

pagesift follows the rules: it obeys robots.txt and Crawl-delay, rate-limits per host, and never bypasses anti-bot protection. A blocked page is reported as an error (AccessBarrier), not retried with tricks.

Quick start

import pagesift

doc = pagesift.scrape("https://example.com/post")
print(doc.markdown, doc.metadata["title"], doc.status_code)

# Many pages: one call, concurrent in Rust, results as they finish
with pagesift.Scraper() as scraper:          # keep one Scraper for the life of the program
    for doc in scraper.scrape_iter(urls, concurrency=16):
        if doc.success:
            save(urls[doc.index], doc.markdown)
        else:
            print(doc.url, doc.error_code, doc.error_message)

Async:

async with pagesift.Scraper() as scraper:
    doc = await scraper.ascrape("https://example.com")
    async for doc in scraper.ascrape_many(urls, concurrency=16):   # completion order
        ...

Convert HTML you already have (no network):

doc = pagesift.convert(html, url="https://example.com/post", only_main_content=True)

Designed for throughput

• Batch first. scrape_many, scrape_iter and ascrape_many take the whole list in one call; the concurrency runs on Rust's async runtime, not in Python threads or Python tasks.
• Stream in completion order. scrape_iter / ascrape_many hand you each page as soon as it is done, so one slow site does not hold back the rest. doc.index maps a result back to its URL.
• Copy only what you read. A Document keeps its fields as Rust values. A field becomes a Python object the first time you read it (strings are then cached), so raw_html you never touch costs nothing. Failed items build their exception object only when you read doc.error.
• Request only what you need. Formats you do not ask for (html, raw_html, links, images) are never computed.
• Reuse one Scraper. It owns the connection pool, the robots.txt cache, rate-limit state and the browser pool. The module-level functions share one default Scraper per process.
• The GIL is released while pages are fetched and processed, so even plain Python threads scale.

Benchmarks

python crates/pagesift-py/bench/bench.py (release build, local HTTP server in a separate process, no internet). 16-thread Linux machine, Python 3.14; the machine was also running other builds (load average ≈ 15), so absolute numbers are noisy, the comparisons are what matter.

1. 200 pages, 50 ms server latency each

One scrape_many call is 15× faster than a loop and as fast as a hand-built thread pool or asyncio.gather, with no threads, semaphores or task bookkeeping on the Python side. At this latency the server is the limit (16 × 1000 / 50 ms = 320 pages/s at most).

2. Time to first result: 1 slow page (2 s) + 31 fast pages

3. Lazy fields: 100 pages of ~200 KB, all formats requested

Converting everything up front would cost every caller the last row; with lazy fields you pay for what you read.

4. Reusing one Scraper vs a new one per page (40 pages, sequential)

On a real network the gap is larger: a new Scraper also redoes DNS, TCP and TLS handshakes and refetches robots.txt.

5. convert on ~200 KB pages with Python threads (GIL released)

API

In batch calls one failed URL never fails the batch: its Document has success=False and error set. A Batch starts scraping as soon as it is created, can be iterated once (with for or async for), supports len(), and batch.cancel() (or dropping it) cancels unfinished pages.

Options

Pass a ScrapeOptions, keyword arguments, or both (keywords override fields of the object). Invalid values raise immediately (InvalidRequest / UnsupportedOption, both also ValueError); unknown keywords raise TypeError.

Scraper(...)

concurrency= on batch calls caps pages in flight for that call; the per-host limits above still apply, so a batch against one host is as polite as single requests.

Document

to_dict() converts every field at once; reading attributes is cheaper when you need only some.

Errors

Every error code has its own exception class; all derive from pagesift.PagesiftError and carry code, message, details (dict), retry_after (seconds or None) and url.

Development

python3 -m venv .venv && . .venv/bin/activate
pip install maturin pytest
maturin develop -m crates/pagesift-py/Cargo.toml
pytest crates/pagesift-py/tests

The tests and the benchmark start a local HTTP server and never touch the internet.

License

AGPL-3.0-only, with a commercial license available for use without AGPL obligations. See the repository's LICENSE and COMMERCIAL.md.