This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

squid_dedup is a squid proxy helper, helping to reduce cache misses when identical content is accessed using different URLs (aka CDNs).

This helper implements the squid StoreID protocol, as found in squid 3 onwards. URL patterns, specified in config files, are rewritten to a presumably unique internal address. Further accesses, modified in the same way, map to already stored objects, even if using different URLs.

Global configuration options are specified in the primary config file, which must exist. A template can be created with the –extract command line switch in the current directory.

CDN match/replacement parameter are specified in additional config files.

Installation

$ python3 setup.py install

Create directory for custom config files:

$ mkdir /etc/squid/dedup

Create primary config file template /etc/squid/squid_dedup.conf:

$ cd /etc/squid
$ squid_dedup -X

openSUSE CDN:

$ gen_openSUSE_dedups

creates /etc/squid/dedup/opensuse.conf.

Activation

Add similar values to /etc/squid/squid.conf:

store_id_program /usr/bin/squid_dedup
store_id_children 20 startup=10 idle=5 concurrency=0

acl getmethod method GET
store_id_access deny !getmethod
store_id_access allow all

That’s it.

Configuration

The primary configuration is located in /etc/squid/squid_dedup.conf, and defines the general behaviour.

Additional config files should be stored in /etc/squid/dedup, e.g.:

[sourceforge]
match: http:\/\/[a-zA-Z0-9\-\_\.]+\.dl\.sourceforge\.net\/(.*)
replace: http://dl.sourceforge.net.%(intdomain)s/\1
fetch: false

Here, any URL pointing to a sub domain of dl.sourceforge.net, is mapped to dl.sourceforge.net.%(intdomain)s, where %(intdomain)s is replaced according to the value of intdomain in /etc/squid/squid_dedup.conf.

match is a list of regular expressions matching URLs, separated by newlines, with all subsequent URLs indented.

replace is a single replacement value.

fetch is an optional boolean flag. If fetch is enabled, the object is fetched also (with a certain delay). This is useful for clients, that download byte ranges only from multiple sources. That behavior results in uncachable objects otherwise. Care is taken for not fetching objects more than once.

Changes to the config files result in an automatic reload by default.

Watch

You might want to increase the log level in /etc/squid/squid_dedup.conf.:

$ less +F /var/log/squid/dedup.log

Notes

The gen_openSUSE_dedups utility is meant to be executed as a user by crontab, e.g.:

$ touch /etc/squid/dedup/opensuse.conf
$ chown user:group /etc/squid/dedup/opensuse.conf
$ chmod 644 /etc/squid/dedup/opensuse.conf
$ su - user
> crontab -e

Add a line similar to:

0 6 * * * /usr/bin/gen_openSUSE_dedups -vs

Credits

The basic idea and a reference implementation in PHP was done by Per Jessen.

Per, thank you for the valuable discussions on this topic.

Release History

Release History

0.0.3

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
squid_dedup-0.0.3.tar.gz (31.3 kB) Copy SHA256 Checksum SHA256 Source May 13, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting