CLI tool for fetching data using HTTP conditional get
Project description
conditional-get
CLI tool for fetching data using HTTP conditional get.
Installation
pip install conditional-get
Usage
The first time you run this command it will download the file and store the ETag (if one was returned) in a file called etags.json
.
The second time you run this command against the same URL it will use that ETag, potentially resulting in a 304 Not Modified
response which saves bandwidth by not re-downloading the file.
# First run - will fetch the file
conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png
# Second run - will only fetch the file if it has changed
conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png
The filename will be derived from the URL. You can customize the filename using the -o
option:
conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png -o til.png
By default the ETags for the retrieved URLs will be stored in a etags.json
file in the current directory. You can use the --etags otherfile.json
to store that file somewhere else:
conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png --etags my-etags.json
Use the -v
option to get debug output showing what is happening:
$ conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png -v
Response status code: 200
[####################################] 100%
$ ls
Simon_Willison__TIL.png etags.json
$ cat etags.json
{
"https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png": "\"d65b78782dfa93213c99099e0e2181d8\""
}
$ conditional-get https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png -v
Existing ETag: "d65b78782dfa93213c99099e0e2181d8"
Response status code: 304
The key used to store the ETag in etags.json
defaults to the URL. You can specify a custom key using the --key
option:
conditional-get --key til https://static.simonwillison.net/static/2020/Simon_Willison__TIL.png
cat etags.json
{
"til": "\"d65b78782dfa93213c99099e0e2181d8\""
}
This is useful if the URL to the file changes even though the file contents stays the same - for example if you are downloading files from URLs that include an expiring signature.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for conditional_get-0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | beb6a10ee1c691445a05c7f454b2ecb91b0b9cbd6d44d0b821070e514d3c0479 |
|
MD5 | 588958dfee189791a7678ef1fb32bad4 |
|
BLAKE2b-256 | b1d6d8498f008d828390f3a09c4fca48a32b66d2e44d373b89a9c0857a72fa7d |