Skip to main content

CLI tool for fetching paginated JSON from a URL

Project description


PyPI CircleCI License

CLI tool for retrieving JSON from paginated APIs.

Currently works against APIs that use the HTTP Link header for pagination. The GitHub API is the most obvious example.

Usage: paginate-json [OPTIONS] URL

  Fetch paginated JSON from a URL

  --version                Show the version and exit.
  --nl                     Output newline-delimited JSON
  --jq TEXT                jq transformation to run on each page
  --accept TEXT            Accept header to send
  --sleep INTEGER          Seconds to delay between requests
  --silent                 Don't show progress on stderr
  --show-headers           Dump response headers out to stderr
  --header <TEXT TEXT>...  Send custom request headers
  --help                   Show this message and exit.

The --jq option only works if you install the optional pyjq dependency.

Works well in conjunction with sqlite-utils. For example, here's how to load all of the GitHub issues for a project into a local SQLite database.

paginate-json \
    "" \
    --nl | \
    sqlite-utils upsert /tmp/issues.db issues - --nl --pk=id

You can then use other features of sqlite-utils to enhance the resulting database. For example, to enable full-text search on the issue title and body columns:

sqlite-utils enable-fts /tmp/issues.db issues title body

You can use the --header option to send additional request headers. For example, if you have a GitHub OAuth token you can pass it like this:

paginate-json \
  --header Authorization "bearer e94d9e404d86..."

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for paginate-json, version 0.3
Filename, size File type Python version Upload date Hashes
Filename, size paginate_json-0.3-py3-none-any.whl (7.9 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page