forktex-scraping

Acquire public data, and be able to prove what you acquired.

pip install forktex-scraping[http]

An acquisition framework, not a scraper. Scraping a rendered page is its lowest tier, used when nothing better exists — and across the four sources it was built against, two are official JSON APIs and only one needs a browser at all.

A remote that answers 200 to a request it did not understand is the adversary. Everything here exists to turn a silently wrong answer into a loud one.

The problem it solves

A public procurement portal was measured to do all of this at once:

• reject any request without a Referer, with a message and no status code to match;
• answer every unfiltered query with total: 3000, searchTooLong: true — a cap, not a count;
• accept a filter field it did not recognise, ignore it, and return 200 with the capped, unfiltered set — byte-identical to sending no filter at all.

That last one is the reason this library exists. A guessed field name does not fail. It produces a confidently wrong dataset, which is worse than an error, and no amount of care in the calling code detects it.

What it does about that

A request contract is captured, not guessed. A browser drives the site's own filter controls once, and the wire name of every filter it honours is read off the traffic. That contract is data; replaying it never needs a browser again.

Every filter is verified on every page. A contract's filter declares where its effect must appear in a returned row. A row that violates it raises ContractDriftError — the remote answered a different question than the one asked, and that is not something to patch client-side.

Coverage is a claim that has to be earned. A capped query is split along its axis until the remote answers honestly. A window that cannot be split further and is still capped is neither dropped nor quietly accepted:

coverage = await harvest(source, contract, transport=..., politeness=..., sink=..., run_id=...)

if not coverage.complete:
    for window in coverage.truncated:   # named, not silently missing
        ...

Politeness is a refusal, not a log line. robots.txt is honoured — and it fails open, since absence is not prohibition. The rate limiter is per-origin and shared across callers, so eight concurrent workers at "one request per second" produce one request per second, not eight. Retry-After wins over the computed backoff.

There is no stealth, no user-agent rotation and no proxy pool. The agent string says who you are.

The seam: stateless versus stateful

The central distinction is not HTTP-versus-browser; it is whether a thing can be replayed.

The rule that follows: a session's output is never a result. It is data — a RequestContract or a SourceDefinition — which the stateless path then replays forever. An agent explores a site statefully; what comes out is never records. That is what keeps a harvest reproducible.

Sources are data

A source is a JSON file naming the model it loads into. Adding a source is a data change; adding a kind is a code change, because a kind with no runner acquires nothing, silently.

{ "$model": "ApiSource",
  "id": "source.example.notices",
  "tier": "captured_contract",
  "contract": "example/notices.json",
  "partition": {"axis": "publishedAt", "start_field": "from", "end_field": "to",
                "initial": "P1M", "floor": "P1D", "end_bound": "inclusive_day"},
  "politeness": {"rate_per_second": 2.0, "respect_robots": true} }

Five kinds, each earned by a real source: ApiSource, BatchKeySource, CatalogSource, FormStateSource, SitemapSource.

Driving it from an agent

forktex_scraping.operations describes the library as typed operations whose JSON Schemas come free from Pydantic — with no MCP, agent-framework or SDK dependency. A harness binds them:

from forktex_scraping import OPERATIONS, schemas

for op in OPERATIONS:
    register(name=op.name, description=op.summary, parameters=op.schema())
    # op.holds_resource marks the session group — scope those, or you leak browsers

Install

Extras import lazily: import forktex_scraping works with none of them installed, and asking for one you did not install raises an ImportError that names the extra.

Documentation

One page per module in docs/, plus DESIGN.md for why it is shaped this way and docs/development.md for the gate.

Licence

Dual-licensed: AGPL-3.0-or-later, or a commercial licence for use in proprietary products and SaaS deployments where AGPL obligations cannot be met — info@forktex.com. See LICENSE and NOTICE.