package-cache 0.1

A caching proxy for package downloads

Package-cache is a simple caching proxy for package downloads. Just
configure a list of upstream sources (with the ``--source`` option)
and point clients at the package-cache server. The first time a
package is requested, we download that package from one of the sources
and cache it locally, while also streaming it to the client. Future
requests for that package are streamed directly from the local cache.
This helps reduce the load on the network and source servers, if you
have a number of local clients that will repeatedly request the same
files (e.g. `Gentoo's distfiles`_).

We don't do anything fancy with `Cache-Control headers`_, since
package source files should include the version stamp in the filename
itself (e.g. ``my-package-0.1.2.tar.gz``). Files are cached after the
first request, and stored forever. This means that every package
you've ever requested will still be there if you need it later.
That's nice, but it will end up consuming a fair amount of disk space.
You might want to periodically cull the cache, using access times to
see which files you are unlikely to want in the future.

Package-cache is written in Python, and has no dependencies outside
the standard library.

Running package-cache
=====================

By default, we'll use `Python's reference WSGI implementation`_ to run
our application::

$ package-cache --source http://distfiles.gentoo.org/

For other command-line options, see::

$ package-cache --help

If you need a more perfomant backend, you might try Gunicorn_.

Gentoo
======

There's an OpenRC_ init script in ``contrib/openrc/init.d``, and a
package-cache ebuild in my `wtk overlay`_. To use package-cache as a
caching proxy for your distfiles downloads, add the ``wtk`` overlay to
layman_ and run::

# emerge --ask --verbose net-proxy/package-cache
# rc-update add default net-proxy/package-cache
# rc-service package-cache start

You can tweak the parameters by setting variables in
``/etc/conf.d/package-cache`` (``PC_USER``, ``PC_GROUP``,
``CACHE_DIR``, ``HOST``, ``PORT``, ``SOURCES``, and ``PC_OPTS``). See
the init script for default values.

Once you've setup your package-cache service, just point your local
package manager to the cache server instead of the usual mirror. For
Portage_, that's going to be something like:

GENTOO_MIRRORS="http://my-package-cache-server.net:4000/"

in your ``/etc/portage/make.conf``.

.. _Gentoo's distfiles:
https://wiki.gentoo.org/wiki/Project:Infrastructure/Source_mirrors
.. _Cache-Control headers:
https://tools.ietf.org/html/rfc2616#section-14.9
.. _Python's reference WSGI implementation:
http://docs.python.org/3/library/wsgiref.html
.. _Gunicorn:
http://gunicorn-docs.readthedocs.org/en/latest/run.html#gunicorn
.. _OpenRC: http://www.gentoo.org/proj/en/base/openrc/
.. _wtk overlay: http://blog.tremily.us/posts/Gentoo_overlay/
.. _layman: https://wiki.gentoo.org/wiki/Layman
.. _Portage: http://wiki.gentoo.org/wiki/Project:Portage