A command line tool to minimize curl http requests.
Project description
http_shrinkwrap - Minimizes curl HTTP commands
In a nutshell
http_shrinkwrap is a command line tool that removes all obsolete HTTP headers from a curl HTTP request.
- All headers that have no apparent effect on the response obtained from the webserver are removed.
- Long Cookies and some other header values are also shortened.
Since the Chrome network inspector has a nifty "Copy as cURL", this tool is useful for minimizing the recreated browser requests in your shell. The tool is written in python and based on uncurl.
Example
Example ipinfo
turns this:
curl 'https://ipinfo.io/' -H 'authority: ipinfo.io' -H 'cache-control: max-age=0' -H 'dnt: 1' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/82.0.2240.398 Safari/534.16' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' -H 'sec-fetch-site: none' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'sec-fetch-dest: document' -H 'accept-language: en-US,en-GB;q=0.9,en;q=0.8,pt-PT;q=0.7,pt;q=0.6,de;q=0.5' -H 'sec-gpc: 1' --compressed
into this:
curl -X GET -H 'user-agent: Mozilla/5.0' https://ipinfo.io/
Example heise.de
turns this:
curl 'https://www.heise.de/' -H 'authority: www.heise.de' -H 'cache-control: max-age=0' -H 'dnt: 1' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/82.0.2240.398 Safari/137.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' -H 'sec-fetch-site: none' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'sec-fetch-dest: document' -H 'accept-language: en-US,en-GB;q=0.9,en;q=0.8,pt-PT;q=0.7,pt;q=0.6,de;q=0.5' -H 'cookie: wt_nv_s=1; wt3_sid=%3B288689636920174%3B589751618140993; wt_ttv2_e_288689636920174=meldung.newsticker.bottom.kommentarelesen*4439526%3ASmart%20Home%3A%20Innenminister%20planen%20Zugriff%20auf%20Daten%20von%20Alexa%20%26%20Co.**meldung.newsticker.bottom*******2*2*; wt_ttv2_c_288689636920174=meldung.newsticker.bottom.kommentarelesen*4428549%3AAntergos-Entwickler%20stellen%20Linux-Projekt%20ein**meldung.newsticker.bottom*******2*2*~meldung.newsticker.bottom.kommentarelesen*4432329%3AEuropa-Wahl%3A%20Schwere%20Schlappe%20f%C3%BCr%20deutsche%20Koalitionsparteien%2C%20Erfolge%20f**meldung.newsticker.bottom*******2*2*~meldung.newsticker.bottom.kommentarelesen*4436209%3AHuawei-Konflikt%3A%20China%20k%C3%BCndigt%20eigene%20schwarze%20Liste%20an**meldung.newsticker.bottom*******2*2*~meldung.newsticker.bottom.kommentarelesen*4439526%3ASmart%20Home%3A%20Innenminister%20planen%20Zugriff%20auf%20Daten%20von%20Alexa%20%26%20Co.**meldung.newsticker.bottom*******2*2*; volumeControl_volumeValue=100; wt_nv=1; wt_ttv2_s_288689636920174=9700; wt_ttv2_s_288689636920174=9700; wt3_eid=%3B288689636920174%7C2155707251500935604%232159741754555500039%3B589751618140993%7C2155796048217456639%232155796801880162886' -H 'sec-gpc: 1' -H 'if-modified-since: Wed, 18 Nov 2020 21:57:08 GMT' --compressed
into this:
curl -X GET https://www.heise.de/
Usage
There are tree main ways to invoke http_shrinkwrap hsw
- By passing a
file
as an argument - Via piping a curl command from
stdin
- By calling
hsw
from insdevim
(orfc
)
Use --bust
to avoid having the web server refer the client back to the cache with a 304 by
removing the according headers
Via file
hsw file_containing_curl_cmd
eg:
- in Chrome/Mozilla dev tools > "copy request as curl" & paste to some_file
hsw some_file
Via stdin
pipe curl command to hsw
eg:
- in Chrome/Mozilla dev tools > "copy request as curl"
echo "curl http://foo.com -H 'some thing'" | hsw
Note:
- wrap the curl command in double quotes
- this will not work if the curl command has single and double quotes or other sepcial chars. Use the file method in these cases.
From fc & vim
given export EDITOR="vim"
- in Chrome/Mozilla dev tools > "copy request as curl"
- paste and execute curl command in terminal
- run
fc
- now inside vim run
:%! hsw
- then save output if needed
:w outfile_name
Install
pip3 install -i https://test.pypi.org/simple/ http-shrinkwrap
Run without install
git clone https://github.com/zrthstr/http_shrinkwrap
cd http_shrinkwrap
pip install -r requirements.txt
echo 'some curl cmd ' | python -m http_shrinkwrap.bin
# or
python -m http_shrinkwrap.bin some_file_containing_a_curl_cmd
Development
Debugging
export DEBUG=TRUE
even more info
export DEBUG=TRACE
Testing
make test
License
Apache License 2.0
Develpement
Written by zrth1k@gmail.com
Thanks to Lars Wirzenius for reviewing the README!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for http_shrinkwrap-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 688fd6941e85b55e0600502773a889696e60256490e6d5474dcee7b2f67ae46d |
|
MD5 | 05a55099c0adf9012beb136a278f9616 |
|
BLAKE2b-256 | 3edd46890de65120489cda5123bf7aa72f983be62d68aaaf9c5cf7b632b9fd3f |