Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

CLI tool for fetching paginated JSON from a URL

Project description

paginate-json

PyPI CircleCI License

CLI tool for retrieving JSON from paginated APIs.

Currently works against APIs that use the HTTP Link header for pagination. The GitHub API is the most obvious example.

$ paginate-json --help
Usage: paginate-json [OPTIONS] URL

  Fetch paginated JSON from a URL

Options:
  --version  Show the version and exit.
  --nl       Output newline-delimited JSON
  --jq TEXT  jq transformation to run on each page
  --help     Show this message and exit.

The --jq option only works if you install the optional pyjq dependency.

Works well in conjunction with sqlite-utils. For example, here's how to load all of the GitHub issues for a projects into a local SQLite database.

paginate-json \
    "https://api.github.com/repos/simonw/datasette/issues?state=all&filter=all" \
    --nl | \
    sqlite-utils upsert /tmp/issues.db issues - --nl --pk=id

You can then use other features of sqlite-utils to enhance the resulting database. For example, to enable full-text search on the issue title and body columns:

sqlite-utils enable-fts /tmp/issues.db issues title body

Project details


Release history Release notifications

This version
History Node

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
paginate_json-0.1-py3-none-any.whl (7.3 kB) Copy SHA256 hash SHA256 Wheel py3

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page