reapx 0.1.0

A tiny, dependency-free client over reapx's public open data: sources, entity pages and their Hugging Face mirrors.

reapx

A tiny, dependency-free Python client over reapx's public open data.

reapx.dev publishes one page per entity returned by a public-source scraper - one page for apache/airflow, one for Apple's SEC filer record, one for every arXiv paper that a run actually returned. Each page carries the observations behind it and the id of the run that produced them. This package reads that surface from Python: it lists the sources, searches the entities under one, and hands you back the rows.

Everything is standard library. No requests, no BeautifulSoup, no lxml, nothing to resolve. Python 3.9 and up.

Install

pip install reapx

Quickstart

import reapx

for source in reapx.sources():
    print(source.slug, source.pages, source.key)

hits = reapx.search("sec-edgar-scraper", "apple", limit=3)
record = reapx.get("github-repo-scraper", "airflow")
print(record.observations[0]["language"])

The three functions

sources()

Reads the machine index at reapx.dev/llms.txt, which is the site's own declaration of what exists, and returns a Source for each one.

>>> s = [x for x in reapx.sources() if x.slug == "sec-edgar-scraper"][0]
>>> s.title
'SEC EDGAR Scraper — Filings, Financials & Full-Text Search'
>>> s.key, s.pages, s.rows
('cik', 771, 4125)
>>> s.huggingface
'reapxdev/sec-edgar-scraper'

Source carries slug, title, description, url, pages, rows, key (the field the entity pages are keyed on), newest, actor_url, and two derived properties, huggingface and huggingface_url, which point at that source's mirror on the Hugging Face Hub.

search(source, query, limit=20)

Case-insensitive substring match over the entity slugs and labels under one source. Entity names are identifiers - tickers, package names, repo slugs, CIK numbers - so a substring is the right tool; there is no ranking and none is pretended.

>>> for h in reapx.search("sec-edgar-scraper", "apple", limit=3):
...     print(h.entity, "|", h.label)
0000320193 | Apple Inc.
0001938109 | Pineapple Financial Inc.
aapl | Apple Inc.

Hubs are paginated at 250 entities per page, and search walks them lazily through rel="next", stopping the moment it has limit matches. An empty query returns the first limit entities in the source's own order. If you want to iterate the whole source rather than search it, reapx.entities(source) is a generator over the same walk.

get(source, entity)

Fetches one entity page and returns a Record: the metadata, the observations, and the run ids behind them.

>>> r = reapx.get("coingecko-scraper", "bch")
>>> r.name
'Bitcoin Cash — coingecko scraper'
>>> len(r.fields)
25
>>> r.observations[0]["currentPrice"]
'1067.98'
>>> r.runs
['0NNZkkxvV7oBS0y0a', 'EHkLvzXn94bZ4hDWd', 'Ei5mJGc17bd3fuCfU', 'P7kQr8Wm5wbEy9nIi', ...]

That runs list is the point of the whole thing. An entity page is the union of every run that ever returned rows for that entity, so Bitcoin Cash is backed by thirteen separate runs and says so. Nothing on the page is estimated, modelled or filled in; if a figure is there, a run returned it.

Record fields: source, entity, name, url, description, fields (every measured field the source declares), observations (the rows shown on the page, as dicts), runs, updated, license, keywords, temporal_coverage, actor_url, plus huggingface and huggingface_url. record.to_dict() gives you a JSON-serialisable dict.

Command line

The package installs a reapx command, and python -m reapx does the same thing.

$ reapx sources
app-store-reviews-scraper           166 pages    22,375 rows  key=appId
arbeitsagentur-scraper            3,555 pages    23,618 rows  key=city
arxiv-papers-scraper             10,594 pages    21,922 rows  key=arxivId
clinicaltrials-scraper            8,175 pages    11,191 rows  key=nctId
coingecko-scraper                 4,828 pages     9,951 rows  key=id
crossref-scraper                  2,492 pages     2,550 rows  key=doi
discogs-scraper                   3,013 pages    14,515 rows  key=handle
docker-hub-scraper                2,805 pages     2,835 rows  key=slug
federal-register-scraper          6,210 pages     9,784 rows  key=citation
github-repo-scraper               2,071 pages     2,481 rows  key=slug
...

$ reapx search coingecko-scraper bitcoin -n 4
0xBitcoin
    0xbtc
    https://reapx.dev/data/coingecko-scraper/0xbtc/
BitcoinII
    bc2
    https://reapx.dev/data/coingecko-scraper/bc2/
Bitcoin Atom
    bca
    https://reapx.dev/data/coingecko-scraper/bca/
Bitcoin Cash
    bch
    https://reapx.dev/data/coingecko-scraper/bch/

4 match(es)

$ reapx get github-repo-scraper airflow
apache/airflow — github repo scraper
https://reapx.dev/data/github-repo-scraper/airflow/
updated  2026-08-03T22:13:12+00:00
runs     k6WwXwZp1DPqPimjS
covers   2015-04-13/2026-08-03
mirror   https://huggingface.co/datasets/reapxdev/github-repo-scraper

1 observation(s):
  createdAt              2015-04-13T18:04:58Z
  defaultBranch          main
  description            Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
  forks                  17514
  fullName               apache/airflow
  htmlUrl                https://github.com/apache/airflow
  isArchived             False
  isDisabled             False
  isFork                 False
  language               Python
  license                apache-2.0
  licenseName            Apache License 2.0

Add --json to any command to get machine-readable output instead:

reapx --json get sec-edgar-scraper 0000320193 | jq '.observations[0]'

Bulk data

This client reads one page at a time, which is the right shape for looking something up and the wrong shape for training on a whole source. For bulk, every source is mirrored as a dataset under the reapxdev organisation on Hugging Face, and each Source tells you which one:

>>> import reapx
>>> reapx.sources()[0].huggingface_url
'https://huggingface.co/datasets/reapxdev/app-store-reviews-scraper'

Each mirror holds <slug>.jsonl and <slug>.csv, so pulling a whole source needs nothing beyond the standard library either:

>>> import json, urllib.request, reapx
>>> s = [x for x in reapx.sources() if x.slug == "github-repo-scraper"][0]
>>> url = s.huggingface_url + "/resolve/main/" + s.slug + ".jsonl"
>>> with urllib.request.urlopen(url) as r:
...     rows = [json.loads(line) for line in r.read().decode().splitlines() if line.strip()]
>>> len(rows)
2481
>>> rows[0]["fullName"]
'tailwindlabs/tailwindcss'

That 2481 is the same row count sources() reports for the source, which is the point: the mirror and the pages are built from the same runs. If you already use the Hugging Face tooling, load_dataset("reapxdev/github-repo-scraper", data_files="github-repo-scraper.jsonl") reads the same file.

If you would rather collect fresh rows than read published ones, each source is a scraper you can run yourself; Source.actor_url and Record.actor_url link to it.

Caching and politeness

reapx.dev is a static site behind a CDN, but it is still somebody else's bandwidth. Every response is cached on disk for an hour, so a repeated search over the same source costs nothing after the first walk.

• Cache location: $REAPX_CACHE_DIR, else $XDG_CACHE_HOME/reapx, else ~/.cache/reapx.
• Disable entirely with REAPX_NO_CACHE=1, or per call with cache=False.
• Change the TTL per call with ttl=, or pass ttl=-1 to keep entries forever.

Requests carry a reapx-python/<version> user agent, retry with backoff on 429 and 5xx, and give up after three attempts.

Errors

Everything raised inherits reapx.ReapxError.

exception	when
SourceNotFound	no source has that slug; call sources() for the list
EntityNotFound	that source publishes no page for that entity
NotFound	a URL returned 404
HTTPError	any other unsuccessful status, or the host was unreachable
ParseError	a page was fetched but did not have the expected shape

Data licence and provenance

The published data is CC BY 4.0. Each entity page names the runs that produced it, and Record.runs gives you those ids, so any figure you take from this package can be traced back to the run that returned it. The underlying sources are public APIs and public websites; each source page on reapx.dev names which.

This package is a client, not the data. It reads the same public URLs your browser would.

Tests

The test suite runs against the live site on purpose - a parser that passes fixtures and fails production is worthless.

python -m unittest discover -s tests -v
REAPX_SKIP_LIVE=1 python -m unittest discover -s tests   # offline: parsers only

Licence

MIT.