Most scrapers are throwaway scripts. They re-download everything whenever a parser changes, start over after a crash, hammer servers until they get blocked, and produce data nobody can trace back to its source.

harvester treats harvesting as a data pipeline. Every response goes into a content-addressed store, so parsers can be fixed and re-run offline. The crawl queue lives on disk, so interrupted runs resume where they stopped. Politeness is built into the engine, not left as an afterthought. Every record carries its provenance and licence.

It uses Scrapling for fetching and parsing, and adds the engine around it.

┌──────────────────────────── quotes | crawl ─────────────────────────────┐
│      60   pages        150   records      60   fetched   0   from cache │
│       0   queued         0   retries       0   failed    0   blocked    │
│ 272.5 KB  downloaded    93   pages/min   39s   elapsed   0   skipped    │
└─────────────────────────────────────────────────────────────────────────┘
complete run 20261003T044028Z
  records: 50 author, 100 quote

Highlights

Quick start

pip install harvester-kit       # installs the `harvester` command and package

harvester run examples/quotes.py          # crawl the scraping sandbox
harvester reparse examples/quotes.py      # rebuild every record offline
harvester status examples/quotes.py       # queue, store and run history

Output lands in .harvester/<source>/runs/<run-id>/records.jsonl, one record per line:

{
  "kind": "author",
  "id": "Albert-Einstein",
  "data": {"name": "Albert Einstein", "born": "March 14, 1879", "born_in": "in Ulm, Germany"},
  "provenance": {
    "source": "quotes",
    "url": "https://quotes.toscrape.com/author/Albert-Einstein/",
    "fetched_at": "2026-10-03T04:41:36Z",
    "sha256": "9d1c…",
    "status": 200,
    "parser_version": "1",
    "harvester_version": "0.1.0",
    "license": {"name": "Scraping sandbox by Zyte; sample data"}
  }
}

Writing a source

from harvester import CachePolicy, Page, Request, Source


class Quotes(Source):
    name = "quotes"
    start_urls = ("https://quotes.toscrape.com/",)
    allowed_domains = ("quotes.toscrape.com",)

    def start(self):
        # Listings change, so revalidate them on each pass.
        yield Request(url=self.start_urls[0], cache=CachePolicy.REVALIDATE)

    def parse(self, page: Page):
        for quote in page.css("div.quote"):
            yield page.record(
                "quote",
                id=quote.css("span.text::text").get(),
                author=quote.css("small.author::text").get(),
                tags=quote.css("a.tag::text").getall(),
            )
        if next_href := page.css("li.next a::attr(href)").get():
            yield page.follow(next_href, cache=CachePolicy.REVALIDATE)

harvester new my-site scaffolds one. Callbacks receive a Page, which offers CSS/XPath via Scrapling, .json() and .text. They yield Requests to follow and Records to keep. The one rule: callbacks must not do their own I/O. Keeping them pure is what makes offline re-parsing exact.

See docs/writing-sources.md for options, multiple callbacks, file downloads and packaging a source as a plugin.

How it works

flowchart LR
    S[Source.start] --> F[(Frontier<br/>SQLite queue)]
    F -->|claim| R[Scope + robots.txt check]
    R --> C{In raw store?}
    C -->|PREFER| PG[Page]
    C -->|REVALIDATE / miss| T[Host throttle] --> H[Scrapling fetch]
    H -->|200| ST[(Raw store<br/>SHA-256 blobs)] --> PG
    H -->|304| PG
    H -->|429 / 503 / 5xx| B[Back off + retry] --> F
    PG --> CB[Parse callback]
    CB -->|Request| F
    CB -->|Record + provenance| O[(records.jsonl<br/>+ manifest)]

.harvester/<source>/
├── frontier.sqlite        # queue: pending / done / failed / skipped, retries, priorities
├── store/
│   ├── index.sqlite       # latest response per URL + full fetch history
│   └── blobs/ab/cd/…gz    # bodies, content-addressed and deduplicated
└── runs/<run-id>/
    ├── records.jsonl      # output
    └── manifest.json      # settings, stats, licence, outcome

Built-in sources

About India Code. harvester was demonstrated and tested against India Code, the Government of India's public repository of legislation, through its public DSpace REST API. The test suite includes a few small sample API responses (see tests/fixtures/india_code/). No harvested dataset is distributed with this project. harvester is an independent project and is not affiliated with or endorsed by India Code or the Government of India.

CLI

SOURCE is an installed source name, path/to/file.py, or path/to/file.py:ClassName.

Politeness, precisely

harvester is meant for collecting data you are entitled to collect, without being a burden on the sites that host it.

• robots.txt (RFC 9309).
  • 2xx: rules and Crawl-delay are obeyed.
  • 4xx: no restrictions.
  • 5xx or network error: complete disallow.
  • A source may opt out of that last rule only by declaring robots_unavailable = "allow" with a written reason. The choice is recorded in every run manifest.
• Rate limits. Per-host minimum delay (default 1 s) and concurrency (default 1). On 429/503 the delay doubles, honouring Retry-After. It recovers by 10% per healthy response.
• Identity. harvester/<version> (+https://github.com/sarthak213/harvester). Change it with --user-agent, but keep a contact URL in it.
• No evasion. No browser fingerprint spoofing, no CAPTCHA solving, no proxy rotation to get around blocks. If a site says no, harvester listens.

Development

git clone https://github.com/sarthak213/harvester && cd harvester
python -m venv .venv && . .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytest                   # 43 tests, including a real HTTP server for end-to-end runs
ruff check . && ruff format --check . && mypy src

Roadmap

• Browser rendering for JavaScript-only pages (Scrapling DynamicFetcher), opt-in per request
• Sitemap and RSS/Atom discovery sources
• Parquet / SQLite export with record de-duplication across runs
• Scheduled incremental runs and change notifications
• More open-data connectors: CKAN, Zenodo, S3 open-data buckets

License

MIT. See LICENSE. The licence covers harvester's code. The data you harvest has its own terms; sources declare them, and harvester records them in every output.