This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Based on the excellent and simple pymiproxy by Nadeem Douba. https://github.com/allfro/pymiproxy

License: because pymiproxy is GPL and warcprox is a derivative work of pymiproxy, warcprox is also GPL.

Install

Warcprox runs on python 3.4.

To install latest release run:

pip install warcprox

You can also install the latest bleeding edge code:

pip install git+https://github.com/internetarchive/warcprox.git

Trusting the CA cert

For best results while browsing through warcprox, you need to add the CA cert as a trusted cert in your browser. If you don’t do that, you will get the warning when you visit each new site. But worse, any embedded https content on a different server will simply fail to load, because the browser will reject the certificate without telling you.

Usage

usage: warcprox [-h] [-p PORT] [-b ADDRESS] [-c CACERT]
                [--certs-dir CERTS_DIR] [-d DIRECTORY] [-z] [-n PREFIX]
                [-s SIZE] [--rollover-idle-time ROLLOVER_IDLE_TIME]
                [-g DIGEST_ALGORITHM] [--base32] [-j DEDUP_DB_FILE]
                [-P PLAYBACK_PORT]
                [--playback-index-db-file PLAYBACK_INDEX_DB_FILE] [--version]
                [-v] [-q]

warcprox - WARC writing MITM HTTP/S proxy

optional arguments:
  -h, --help            show this help message and exit
  -p PORT, --port PORT  port to listen on (default: 8000)
  -b ADDRESS, --address ADDRESS
                        address to listen on (default: localhost)
  -c CACERT, --cacert CACERT
                        CA certificate file; if file does not exist, it will
                        be created (default: ./desktop-nlevitt-warcprox-
                        ca.pem)
  --certs-dir CERTS_DIR
                        where to store and load generated certificates
                        (default: ./desktop-nlevitt-warcprox-ca)
  -d DIRECTORY, --dir DIRECTORY
                        where to write warcs (default: ./warcs)
  -z, --gzip            write gzip-compressed warc records (default: False)
  -n PREFIX, --prefix PREFIX
                        WARC filename prefix (default: WARCPROX)
  -s SIZE, --size SIZE  WARC file rollover size threshold in bytes (default:
                        1000000000)
  --rollover-idle-time ROLLOVER_IDLE_TIME
                        WARC file rollover idle time threshold in seconds (so
                        that Friday's last open WARC doesn't sit there all
                        weekend waiting for more data) (default: None)
  -g DIGEST_ALGORITHM, --digest-algorithm DIGEST_ALGORITHM
                        digest algorithm, one of sha384, sha512, md5, sha224,
                        sha256, sha1 (default: sha1)
  --base32              write digests in Base32 instead of hex (default:
                        False)
  -j DEDUP_DB_FILE, --dedup-db-file DEDUP_DB_FILE
                        persistent deduplication database file; empty string
                        or /dev/null disables deduplication (default:
                        ./warcprox-dedup.db)
  -P PLAYBACK_PORT, --playback-port PLAYBACK_PORT
                        port to listen on for instant playback (default: None)
  --playback-index-db-file PLAYBACK_INDEX_DB_FILE
                        playback index database file (only used if --playback-
                        port is specified) (default: ./warcprox-playback-
                        index.db)
  --version             show program's version number and exit
  -v, --verbose
  -q, --quiet

To do

  • (partly done) integration tests, unit tests
  • (done) url-agnostic deduplication
  • unchunk and/or ungzip before storing payload, or alter request to discourage server from chunking/gzipping
  • check certs from proxied website, like browser does, and present browser-like warning if appropriate
  • keep statistics, produce reports
  • write cdx while crawling?
  • performance testing
  • (done) base32 sha1 like heritrix?
  • configurable timeouts and stuff
  • evaluate ipv6 support
  • (done) more explicit handling of connection closed exception during transfer
  • dns cache?? the system already does a fine job I’m thinking
  • keepalive with remote servers?
  • (done) python3
  • special handling for 304 not-modified (write nothing or write revisit record… and/or modify request so server never responds with 304)
  • (done) instant playback on a second proxy port
  • special url for downloading ca cert e.g. http(s)://warcprox./ca.pem
  • special url for other stuff, some status info or something?
  • browser plugin for warcprox mode
    • accept warcprox CA cert only when in warcprox mode
    • separate temporary cookie store, like incognito
    • “careful! your activity is being archived” banner
    • easy switch between archiving and instant playback proxy port

To not do

The features below could also be part of warcprox. But maybe they don’t belong here, since this is a proxy, not a crawler/robot. It can be used by a human with a browser, or by something automated, i.e. a robot. My feeling is that it’s more appropriate to implement these in the robot.

  • politeness, i.e. throttle requests per server
  • fetch and obey robots.txt
  • alter user-agent, maybe insert something like “warcprox mitm archiving proxy; +http://archive.org/details/archive.org_bot”
Release History

Release History

2.0b2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.0b1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

2.0.dev9

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.4

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
warcprox-1.4.tar.gz (18.7 kB) Copy SHA256 Checksum SHA256 Source Jun 12, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting