Skip to main content

A command line tool to minimize curl http requests.

Project description

http_shrinkwrap - Minimizes curl HTTP commands

In a nutshell

http_shrinkwrap is a command line tool that removes all obsolete HTTP headers from a curl HTTP request.

  • All headers that have no apparent effect on the response obtained from the webserver are removed.
  • Long Cookies and some other header values are also shortened.

Since the Chrome network inspector has a nifty "Copy as cURL", this tool is useful for minimizing the recreated browser requests in your shell. The tool is written in python and based on uncurl.

Example

Example ipinfo

turns this:

curl 'https://ipinfo.io/' -H 'authority: ipinfo.io' -H 'cache-control: max-age=0' -H 'dnt: 1' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/82.0.2240.398 Safari/534.16' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' -H 'sec-fetch-site: none' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'sec-fetch-dest: document' -H 'accept-language: en-US,en-GB;q=0.9,en;q=0.8,pt-PT;q=0.7,pt;q=0.6,de;q=0.5' -H 'sec-gpc: 1' --compressed

into this:

curl -X GET -H 'user-agent: Mozilla/5.0' https://ipinfo.io/

Example heise.de

turns this:

curl 'https://www.heise.de/'   -H 'authority: www.heise.de'   -H 'cache-control: max-age=0'   -H 'dnt: 1'   -H 'upgrade-insecure-requests: 1'   -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/82.0.2240.398 Safari/137.36'   -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'   -H 'sec-fetch-site: none'   -H 'sec-fetch-mode: navigate'   -H 'sec-fetch-user: ?1'   -H 'sec-fetch-dest: document'   -H 'accept-language: en-US,en-GB;q=0.9,en;q=0.8,pt-PT;q=0.7,pt;q=0.6,de;q=0.5'   -H 'cookie: wt_nv_s=1; wt3_sid=%3B288689636920174%3B589751618140993; wt_ttv2_e_288689636920174=meldung.newsticker.bottom.kommentarelesen*4439526%3ASmart%20Home%3A%20Innenminister%20planen%20Zugriff%20auf%20Daten%20von%20Alexa%20%26%20Co.**meldung.newsticker.bottom*******2*2*; wt_ttv2_c_288689636920174=meldung.newsticker.bottom.kommentarelesen*4428549%3AAntergos-Entwickler%20stellen%20Linux-Projekt%20ein**meldung.newsticker.bottom*******2*2*~meldung.newsticker.bottom.kommentarelesen*4432329%3AEuropa-Wahl%3A%20Schwere%20Schlappe%20f%C3%BCr%20deutsche%20Koalitionsparteien%2C%20Erfolge%20f**meldung.newsticker.bottom*******2*2*~meldung.newsticker.bottom.kommentarelesen*4436209%3AHuawei-Konflikt%3A%20China%20k%C3%BCndigt%20eigene%20schwarze%20Liste%20an**meldung.newsticker.bottom*******2*2*~meldung.newsticker.bottom.kommentarelesen*4439526%3ASmart%20Home%3A%20Innenminister%20planen%20Zugriff%20auf%20Daten%20von%20Alexa%20%26%20Co.**meldung.newsticker.bottom*******2*2*; volumeControl_volumeValue=100; wt_nv=1; wt_ttv2_s_288689636920174=9700; wt_ttv2_s_288689636920174=9700; wt3_eid=%3B288689636920174%7C2155707251500935604%232159741754555500039%3B589751618140993%7C2155796048217456639%232155796801880162886'   -H 'sec-gpc: 1'   -H 'if-modified-since: Wed, 18 Nov 2020 21:57:08 GMT'   --compressed

into this:

curl -X GET https://www.heise.de/

Usage

There are tree main ways to invoke http_shrinkwrap hsw

  • By passing a file as an argument
  • Via piping a curl command from stdin
  • By calling hsw from insde vim (or fc)

Use --bust to avoid having the web server refer the client back to the cache with a 304 by removing the according headers

Via file

hsw file_containing_curl_cmd

eg:

  • in Chrome/Mozilla dev tools > "copy request as curl" & paste to some_file
  • hsw some_file

Via stdin

pipe curl command to hsw eg:

  • in Chrome/Mozilla dev tools > "copy request as curl"
  • echo "curl http://foo.com -H 'some thing'" | hsw

Note:

  • wrap the curl command in double quotes
  • this will not work if the curl command has single and double quotes or other sepcial chars. Use the file method in these cases.

From fc & vim

given export EDITOR="vim"

  • in Chrome/Mozilla dev tools > "copy request as curl"
  • paste and execute curl command in terminal
  • run fc
  • now inside vim run :%! hsw
  • then save output if needed :w outfile_name

Install

pip3 install -i https://test.pypi.org/simple/ http-shrinkwrap

Run without install

git clone https://github.com/zrthstr/http_shrinkwrap
cd http_shrinkwrap
pip install -r requirements.txt

echo 'some curl cmd ' | python -m http_shrinkwrap.bin
# or
python -m http_shrinkwrap.bin some_file_containing_a_curl_cmd

Development

Debugging

export DEBUG=TRUE even more info export DEBUG=TRACE

Testing

make test

License

Apache License 2.0

Develpement

Written by zrth1k@gmail.com

Thanks to Lars Wirzenius for reviewing the README!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

http_shrinkwrap-0.1.1.tar.gz (10.7 kB view hashes)

Uploaded source

Built Distribution

http_shrinkwrap-0.1.1-py3-none-any.whl (15.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page