toapi

Turn any website into a JSON API — declaratively.

toapi lets you point at a web page, declare the fields you want with CSS selectors, and get back a clean JSON API. No crawler to babysit, no database to maintain — pages are fetched and parsed on demand, with built‑in caching.

Install

pip install toapi

Requires Python 3.10+.

Quickstart

from htmlparsing import Attr, Text
from toapi import Api, Item

api = Api()


@api.site("https://news.ycombinator.com")
@api.list(".athing")
@api.route("/posts", "/news")
@api.route("/posts?page={page}", "/news?p={page}")
class Post(Item):
    title = Text(".titleline > a")
    url = Attr(".titleline > a", "href")


api.run(host="127.0.0.1", port=5000)

Run it:

python app.py

Then visit http://127.0.0.1:5000/posts and you get:

{
  "Post": [
    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://www.quantamagazine.org/..."},
    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://jalopnik.com/..."}
  ]
}

How it works

   ┌────────────┐    ┌────────────┐    ┌────────────┐
   │  /posts    │ ─▶ │  fetch     │ ─▶ │  parse     │ ─▶  JSON
   │  (route)   │    │  (cache)   │    │  (Item)    │
   └────────────┘    └────────────┘    └────────────┘

Route — @api.route("/posts", "/news") maps your API path to a source URL.
Fetch — pages are fetched with requests (or a headless browser if you pass browser=) and cached in memory.
Parse — each Item extracts fields with CSS selectors via htmlparsing.
Serve — Flask returns the result as JSON; subsequent calls hit the cache.

Features

Declarative — describe data, not scraping logic.
Routes — map clean API paths to messy source URLs with {param} placeholders.
Multi-site — merge several websites behind one API.
Cleaning hooks — define clean_<field> methods to post-process values.
Caching — pages and parsed results are cached automatically.
Headless browser — pass Api(browser="/path/to/geckodriver") for JS-heavy sites.

Cleaning values

Add a clean_<fieldname> method on the Item to transform a value before it's returned:

@api.site("https://news.ycombinator.com")
@api.route("/posts", "/news")
class Page(Item):
    next_page = Attr(".morelink", "href")

    def clean_next_page(self, value):
        return f"/posts?{value.split('?', 1)[1]}"

Development

git clone https://github.com/elliotgao2/toapi.git
cd toapi
uv sync          # install deps into .venv
uv run pytest    # run tests
uv run ruff check .

We use uv for packaging and ruff for lint + format. Pre-commit hooks keep both clean:

uv run pre-commit install

Contributing

Pull requests are welcome. For non-trivial changes, please open an issue first to discuss what you'd like to change. Make sure uv run pytest and uv run ruff check . pass before submitting.

License

MIT © Elliot Gao

Metadata

Release files for toapi 2.2.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for toapi 2.2.4
File	Size	Uploaded
toapi-2.2.4.tar.gz	422.0 kB	May 22, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for toapi 2.2.4
File	Interpreter	ABI	Platform	Reset
toapi-2.2.4-py3-none-any.whl 7.2 kB May 22, 2026	Python 3	none	any	Details

Total release size: 429.2 kB

Release files / toapi-2.2.4.tar.gz

Download URL	toapi-2.2.4.tar.gz
Size	422.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`418afa947e4ec0e5d8fa4c5349252c1ff10d66824ba0e6548f3b1316356ac755`
BLAKE2b-256 checksum How to use checksums	`bdb7db48d467983dc6c2cb6badad4add17b45218c694877e855e2743c9a9ef31`
Upload date	May 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.5.9`

Release files / toapi-2.2.4-py3-none-any.whl

Download URL	toapi-2.2.4-py3-none-any.whl
Size	7.2 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`344a1cd42c93045d2ffa0e0c9355dc7b8a7aed44c9995e0d6915bd347117fd77`
BLAKE2b-256 checksum How to use checksums	`59df7c022494741e4d7cdec79901e2fb063e3a4a97d8bd797a1df622850ec555`
Upload date	May 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.5.9`