CLI tool for fetching paginated JSON from a URL
Project description
paginate-json
CLI tool for retrieving JSON from paginated APIs.
Currently works against APIs that use the HTTP Link header for pagination. The GitHub API is the most obvious example.
Usage: paginate-json [OPTIONS] URL
Fetch paginated JSON from a URL
Options:
--version Show the version and exit.
--nl Output newline-delimited JSON
--jq TEXT jq transformation to run on each page
--accept TEXT Accept header to send
--sleep INTEGER Seconds to delay between requests
--silent Don't show progress on stderr
--show-headers Dump response headers out to stderr
--header <TEXT TEXT>... Send custom request headers
--help Show this message and exit.
The --jq
option only works if you install the optional pyjq dependency.
Works well in conjunction with sqlite-utils. For example, here's how to load all of the GitHub issues for a project into a local SQLite database.
paginate-json \
"https://api.github.com/repos/simonw/datasette/issues?state=all&filter=all" \
--nl | \
sqlite-utils upsert /tmp/issues.db issues - --nl --pk=id
You can then use other features of sqlite-utils to enhance the resulting database. For example, to enable full-text search on the issue title and body columns:
sqlite-utils enable-fts /tmp/issues.db issues title body
You can use the --header
option to send additional request headers. For example, if you have a GitHub OAuth token you can pass it like this:
paginate-json https://api.github.com/users/simonw/events \
--header Authorization "bearer e94d9e404d86..."
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for paginate_json-0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61d6de2d653c03fb7297219561216f7511df38025114bacaca1304a36bc4b731 |
|
MD5 | 9e8ca857e0664eca4fca4293842eaf95 |
|
BLAKE2b-256 | 8f206c303d0258f2abbeedd963a98ee1a678bd50a459d93a31c7b4c0dc829a06 |