Tool for posting JSON to an API, broken into pages
Project description
json-post
Tool for posting JSON to an API, broken into pages
Why would you need this?
This tool is for a very specific use-case. Let's say you have 80MB of data that you want to send to an API. The data is a big JSON file with thousands of items in it - but the API you are using can only accept 50 items at a time.
json-post
can take that big JSON file and break it up into batches, then send each batch to the API in turn.
json-post my-big-json-file.json https://example.com/my-api --batch-size 50
Installation
Install this tool using pip
:
$ pip install json-post
Options
-h
or --header KEY VALUE
Takes two arguments to specify an HTTP header, for example to send an authorization token you might use:
json-post d.json https://example.com/api --header Authorization "Bearer x...."
--log FILENAME
A filename to log the JSON responses from the API to, as newline-delimited JSON.
--batch-size N
The batch size to use. Omit this and the data will be sent in a single request.
--stop-after N
Send this many items and then stop. Useful for debugging.
--reverse
Send the items from the file in reverse order.
--shuffle
Send the items from the file in random order.
--http-read-timeout N
Timeout (in seconds) for network read operations.
--filter 'item.get("field")'
Filter the items in the array using this Python expression before sending them. The variable item
will refer to each item in the list.
--count
Output the number of items that would be sent and then exit. Useful for testing that your --filter
operation works as expected.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd json-post
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv
:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.