Bitscrape

Bitscrape

A modular, async Python web scraping framework — simple for a first spider, capable of distributed crawling, storage, search-ranking, and observability for everything after that.

It provides fast async networking, powerful HTML parsing, configurable pipelines, distributed crawling support, and optional browser rendering with Playwright.

Install

pip install bitscrape

or

uv add bitscrape

Documentation

• Repository: https://github.com/Sudharsansm/Bitscrape
• Issues: https://github.com/Sudharsansm/Bitscrape/issues

import bitscrape

@bitscrape.spider(name="quotes", start_urls=["https://example.com"])
async def parse(response):
    for quote in response.css("div.quote"):
        yield {
            "text": quote.css("span.text::text").get(),
            "author": quote.css("small.author::text").get(),
        }

bitscrape.run(parse, output="quotes.jsonl")

That same call scales to a Redis-backed, multi-worker distributed crawl by changing one argument — the spider code doesn't change:

bitscrape.run(parse, settings=bitscrape.Settings(
    scheduler_use_redis=True,
    distributed_throttle_enabled=True,
))

Why Bitscrape

• One function call to run a spider, whether it's a plain decorated function or a full class with multiple callbacks.
• One Settings object controls everything else — proxy rotation, session pooling, JS rendering, distributed crawling, live monitoring, Prometheus metrics — all opt-in toggles, nothing extra loads unless you ask for it.
• Respects robots.txt and meta-robots by default, and fails safe (blocks) rather than open (allows) if robots.txt can't be fetched.
• Every documented feature has a real test behind it — a real local Redis, a real local HTTP server, a real on-disk SQLite database, a real S3 API emulation, real Prometheus/OpenTelemetry libraries. Where something couldn't be verified for real (a live PostgreSQL/MongoDB server, a Kubernetes cluster), that's stated explicitly rather than implied — see CHANGELOG.md and docs/ for exactly what's verified vs. documented.

What's included

Explicitly not included, by design: CAPTCHA-solving, browser/TLS fingerprint spoofing, or any other bot-detection evasion tooling. See docs/security/ for why.

Install

python3 -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\Activate.ps1
pip install -e ".[all,dev,cli]"
bitscrape --version

See docs/installation/ for extras, system requirements, and troubleshooting.

Quickstart

bitscrape genspider quotes example.com
bitscrape crawl spiders/quotes.py -o quotes.jsonl

Full walkthrough: docs/quickstart/.

Documentation

Full documentation lives in docs/:

• Getting Started · Quickstart · Tutorials
• User Guide · Developer Guide · Architecture
• API Reference · CLI Reference
• Plugins · Browser Rendering · Crawling
• Scheduler · Parser · Extractors
• AI / Search Infra · Storage · Deployment
• Security · Monitoring · Examples
• Troubleshooting · FAQ · Roadmap · Changelog

Also at the repository root: ARCHITECTURE.md (condensed system design), API_REFERENCE.md (flat single-page API reference), ROADMAP.md, CHANGELOG.md.

Running the test suite

pytest -q

Expect 440 passed. A handful of tests need a local Redis on port 6390 and a local PostgreSQL server (database bitscrape_test, user/password postgres/postgres) — if it's not running, only those specific tests fail, everything else still passes. See docs/developer-guide/.

Contributing

See CONTRIBUTING.md. Security issues: see SECURITY.md rather than a public issue. Community standards: CODE_OF_CONDUCT.md.

License

MIT — see LICENSE.