This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!

A caching proxy for package downloads

Project Description

Package-cache is a simple caching proxy for package downloads. Just configure a list of upstream sources (with the --source option) and point clients at the package-cache server. The first time a package is requested, we download that package from one of the sources and cache it locally, while also streaming it to the client. Future requests for that package are streamed directly from the local cache. This helps reduce the load on the network and source servers, if you have a number of local clients that will repeatedly request the same files (e.g. Gentoo’s distfiles).

We don’t do anything fancy with Cache-Control headers, since package source files should include the version stamp in the filename itself (e.g. my-package-0.1.2.tar.gz). Files are cached after the first request, and stored forever. This means that every package you’ve ever requested will still be there if you need it later. That’s nice, but it will end up consuming a fair amount of disk space. You might want to periodically cull the cache, using access times to see which files you are unlikely to want in the future.

Package-cache is written in Python, and has no dependencies outside the standard library.

Running package-cache

By default, we’ll use Python’s reference WSGI implementation to run our application:

$ package-cache --source

For other command-line options, see:

$ package-cache --help

If you need a more perfomant backend, you might try Gunicorn.


There’s an OpenRC init script in contrib/openrc/init.d, and a package-cache ebuild in my wtk overlay. To use package-cache as a caching proxy for your distfiles downloads, add the wtk overlay to layman and run:

# emerge --ask --verbose net-proxy/package-cache
# rc-update add default net-proxy/package-cache
# rc-service package-cache start

You can tweak the parameters by setting variables in /etc/conf.d/package-cache (PC_USER, PC_GROUP, CACHE_DIR, HOST, PORT, SOURCES, and PC_OPTS). See the init script for default values.

Once you’ve setup your package-cache service, just point your local package manager to the cache server instead of the usual mirror. For Portage, that’s going to be something like:


in your /etc/portage/make.conf.

If you don’t want to tweak your clients (perhaps there are many of them, or they are out of your direct control), you can add some firewall rules to your router to transparently proxy specific Gentoo mirrors. With an internal eth1 and an internal proxy on, that looks something like:

# for SOURCE_IP in $(dig +short;
> do
>   iptables --table nat --append PREROUTING --protocol tcp \
>     --in-interface eth1 ! --source "${CACHE_IP}" \
>     --destination "${SOURCE_IP}" \
>     --match tcp --destination-port 80 \
>     --jump DNAT --to-destination "${CACHE_IP}:4000" ;
> done

To remove those entries later, repeat the command with --delete instead of --append. You may need to list the SOURCE_IP values explicitly if the DNS entries have changed. Run:

# iptables --table nat --list PREROUTING --numeric

to list the entries. See iptables(8) and iptables-extensions(8) for more details.

Release History

Release History

This version
History Node


History Node


Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
package-cache-0.2.tar.gz (18.5 kB) Copy SHA256 Checksum SHA256 Source Feb 22, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting