plone.recipe.varnish 2.0a2

Build and/or configure Varnish Cache with zc.buildout

Varnish recipe for buildout

plone.recipe.varnish is a zc.buildout recipe to install Varnish. Even though the name contains the name Plone, there is nothing Plone-specific about this recipe: it works for non-Zope sites just as well.

Configuring it is very simple. For example:

[varnish-build]
recipe = plone.recipe.varnish:build

[varnish-configuration]
recipe = plone.recipe.varnish:configuration
backends = 127.0.0.1:8081

[varnish-script]
recipe = plone.recipe.varnish:script
bind = 127.0.0.1:8000
cache-size = 512M

This configures three buildout parts:

varnish-build

which will download, compile and install varnish,

varnish-configuration

which generates the VCL configuration file, sending requests to a backend at 127.0.0.1:8081, and

varnish-script

which runs Varnish, configured to listen on 127.0.0.1:8000 for requests, using a 512 megabyte cache.

A wrapper script for the varnish startup command is created in the bin directory of your buildout.

Please note that the configuration generated by this recipe requires Varnish 4.0.x or later. The URL provided by the recipe points to the latest tested Varnish release compatible with the generated configuration.

Virtual hosting

Varnish supports virtual hosting by selecting a different backend server based on headers on the incoming request. You can configure the backends through the backends option:

[varnish-instance]
backends =
   plone.org:127.0.0.1:8000
   plone.net:127.0.0.1:9000

This will generate a configuration which sends all traffic for the plone.org host to a backend server running on port 8000 while all traffic for the plone.net host is send to port 9000.

Zope 2 hosting (with Virtual Host Monster)

If you are using Zope 2 as backend server you will need to rewrite the URL so the Zope Virtual Host Monster (VHM) can generate correct links for links in your pages. This can be done either by a web server such as Apache or nginx (placed either in front or behind Varnish) but can also be done by Varnish itself.

The three options are described below.

Option 1 (rewrites after Varnish)

If generating these VHM-style URLs in a proxy behind Varnish (or if using VHM’s ‘mapping’ feature), no extra Varnish configuration is needed. Just make sure the backends option directs the traffic to the proxy.

Option 2 (rewrites before Varnish)

If generating these VHM-style URLs in a proxy in front of Varnish, no extra Varnish configuration is needed as long as the original hostname is still retained in the URL. If the hostname is not retained, you can tell Varnish to direct requests based on the “path” instead of the hostname. For example:

[varnish-instance]
backends =
  /VirtualHostBase/http/plone.org:80/Plone:127.0.0.1:8000
  /VirtualHostBase/http/plone.net:80/Plone:127.0.0.1:9000

This will generate a configuration which sends all traffic for any request whose path starts with /VirtualHostBase/http/plone.org:80/Plone to a backend server running at 127.0.0.1 on port 8000, while request paths starting with /VirtualHostBase/http/plone.net:80/Plone are sent to port 9000.

Option 3 (rewrites within Varnish)

To have Varnish generate these VHM-style URLs, you can use the zope2_vhm_map option. Here is an example:

[varnish-instance]
zope2_vhm_map =
    plone.org:/plone
    plone.net:/plone

This tells us that the domain plone.org should be mapped to the location /plone in the backend. By combining this with the information from the backends option a varnish configuration will be generated that maps URLs correctly.

Load Balancing

Varnish supports load balancing by configuring a director for a pool of backends. This director sends the incoming requests that cannot be fulfilled by varnish to backends in the pool in either random or round robin fashion. You can configure the director via the balancer option:

[varnish-instance]
balancer = random

This will generate a configuration which sends all traffic to the director, which will choose a ‘random’ backend server to fulfill the request if the content requested is not cached by varnish itself.

plone.recipe.varnish reference

The plone.recipe.varnish recipe does one or more of the following:

plone.recipe.varnish:build

compiles varnish from sources

plone.recipe.varnish:configuration

generates a VCL-configuration file

plone.recipe.varnish:script

generates a wrapper script inside your buildout that will start Varnish with the correct configuration.

Please note that this recipe requires Varnish 4.0.x or later.

Build varnish from sources

build is based on zc.recipe.cmmi - so all parameters from there are available here too (but are rarely used).

Two parameters are different/ extra:

url

Location used for download of varnish sources. Defaults to a version tested for the selected varnish_version.

jobs

Passes the number of parallel jobs to make, defaults to 4. Adjust as needed to your CPU resources.

varnish_version

Varnish target version. Default is 4.

VCL Configuration Generator

backends

Specifies the backend or backends which will process the (uncached) requests. The syntax for backends:

[<hostname>][/<path>]:<ip address>:<port>

The optional hostname and path allows you to do virtual hosting. If multiple backends are specified then each backend must include either a hostname or path (or both) so that Varnish can direct the matching request to the appropriate backend. Defaults to 127.0.0.1:8080.

balancer

If included and set to either random or round-robin, this option configures varnish to load balance the servers specified by the backends directive. Possible values: none (default), round-robin or random.

between-bytes-timeout

If specified, this option configures the timeout (in seconds) for Varnish waiting between bytes when receiving data from a backend. Varnish will only wait this many seconds between bytes before giving up. A value of 0s means this will never time out. Defaults to 60s, as per Varnish’s default settings.

bind

Hostname and port on which Varnish will listen for requests. Defaults to 127.0.0.1:8000.

connect-timeout

If specified, this option configures the connection timeout (in seconds) for Varnish connecting to a backend server. Varnish will only try to connect to a given backend for this many seconds before giving up. Defaults to 0.4s, as per Varnish’s default settings.

cookie-whitelist

After the cookie-pass was processed this list is used to to sanitize cookie data on the request. Cookie data to be sent to the backend includes only cookies with the given namens. Goal is to work better with the backend, i.e. detect if user is logged in and adjust caching to ensure no authenticated pages get cached. Defaults are optimized for Zope2/Plone: statusmessages __ac _ZopeId __cp

cookie-pass

This list consists of lines with a cookie-match and urlexclude in the form: "cookiematch":"urlexcludes". If cookiematch applies for the cookiename and the current url does not match urlexcludes, the request is passed directly to the configured backend bypassing any caching. But if cookie applies and url matches urlexcludes, then a lookup is forced. Defaults are optimized for Plone, one line: "__ac(|_(name|password|persistent))=":"\.(js|css|kss)"

first-byte-timeout

If specified, this option configures the timeout (in seconds) for Varnish receiving the first byte from a backend. Varnish will only wait for this many seconds before giving up. A value of 0s means Varnish will never time out. Defaults to 300s.

purge-hosts

Specifies hostnames or IP addresses for purge ACL. By default localhost and the backends are allowed to purge. Additional allowed hosts are listed here.

vcl_recv, vcl_hit, vcl_miss, vcl_fetch, vcl_deliver, vcl_pipe

Insert arbitrary VCL code into the generated config.

verbose-headers

Enable sending extra headers in responses that expose what varnish did with the request and the cache status. Useful for debugging cache settings and optimizations. Possible values: on or off (default).

zope2_vhm_map

Defines a virtual host mapping for Zope servers. This is a list of hostname:ZODB location entries which specify the location inside Zope where the website for a virtual host lives.

zope2_vhm_port

Defines a virtual host mapping port to use in the VHM URL to send back to clients. Useful if there is another port mapping in front of varnish, such as haproxy. Defaults to bind port.

zope2_vhm_ssl

If specified, this maps VHM URLs to https for all requests. Possible values: on or off (default).

zope2_vhm_ssl_port

Defines a virtual host mapping port to use in the VHM URL to send back to clients. Useful if there is another port mapping in front of varnish, such as haproxy. Defaults to 443.

varnish_version

Varnish target version. If not given, it looks for the varnish_version setting in the build part (see the build-part setting). If not given there either, it defaults to 4.

To test the generated configuration for syntactic correctness, run varnishd -C -f ./parts/varnish-configuration/varnish.vcl.

Create script to start varnish

Start varnish as a daemon or in foreground with the given settings.

bind

Hostname and port on which Varnish will listen for requests. Defaults to 127.0.0.1:8000.

build-part

References the buildout part in order to get settings from there. Defaults to varnish-build. Set it to false in order to switch it off.

cache-location

Customise the location for the Varnish file storage. Option only applicable when used with file or persistent cache-type options. Defaults to using a file named storage inside the relevant parts directory (eg parts/varnish/storage). Changing the default location can be useful in putting the storage somewhere with quicker read speeds (e.g. RAM disk).

cache-size

The size of the cache (limited to 2G on 32bit systems). Defaults to 256M.

cache-type

Specify the type of cache storage to use with Varnish. Possible values: file (storage for each object is allocated from an arena backed by a file), malloc (storage for each object is allocated with malloc; in memory), or persistent (experimental as at Varnish 2.1.4). Defaults to file.

configuration-file

Path to a Varnish VCL configuration file to use. Defaults to the generated file from the configuration-part setting. If no configuration was generated, this setting is mandatory.

configuration-part

Names the buildout part to get settings from. Defaults to varnish-configuration.

daemon

The file and path of the varnish daemon varnishd to use. If not given, it looks for the build part (see the build-part setting) and uses its location setting plus the string /sbin/varnishd. If there is no build part, it defaults to /usr/sbin/varnishd - the most common place where it’s found on many Unix systems. Adjust it if needed.

grace-healthy

Grace in the context of Varnish means delivering otherwise expired objects when circumstances call for it. This can happen because: (1) the backend-director selected is down, or (2) a different thread has already made a request to the backend that’s not yet finished.

If the backend is healthy, accept objects that are this number of seconds old. Clients will be delivered content that is no more than number of seconds past its TTL.

Format: number followed by a time unit: ms, s, m, h.

Defaults to None. If this is set to None the grace feature is disabled.

grace-sick

If the backend is sick, accept objects that are this old. See also grace-healthy.

Format: number followed by a time unit: ms, s, m, h.

Defaults to 600s. Should be greater than grace-healthy.

group

The name of the group that varnish should switch to before accepting any request. This defaults to the main group for the specified user.

mode

Specify whether the varnish daemon should run in daemon or foreground mode. The latter is useful when varnish is run by service supervision tools like daemontools or runit. Defaults to daemon.

name

If specified this sets the name of the varnish instance (defaults to the host name).

From varnishd’s manpage:

Amongst other things, this name is used to construct the name of the directory in which varnishd keeps temporary files and persistent state. If the specified name begins with a forward slash, it is interpreted as the absolute path to the directory which should be used for this purpose.

runtime-parameters

Runtime parameter configuration options. The full list of available options can be found in the manpage varnishd(1) for your version of varnish. Examples include thread_pool_max, thread_pool_min, sess_timeout.

telnet

If specified sets the hostname and port on which Varnish will listen for commands using its telnet interface.

user

The name of the user varnish should switch to before accepting any requests. Defaults to nobody.

varnish_version

Varnish target version. If not given, it looks for the varnish_version setting in the build part (see the build-part setting). If not given there either, it defaults to 4.

Examples:

Use system varnish at /usr/sbin/varnishd, generate start script in ./bin/varnishd using a VCL-file in ./parts/varnish-configuration/varnish.vcl:

[buildout]
parts =
    varnish-script
    varnis-configuration

[varnish-script]
recipe = plone.recipe.varnish:script

[varnish-configuration]
recipe = plone.recipe.varnish:configuration

Changelog

2.0a2 (2015-12-22)

• Fix daemon location of script part of the recipe (/usr/bin/varnishd was always used. [fredvd]

• Fix tests, download Varnish 4.0.3 as download. [fredvd]

2.0a1 (2015-03-02)

• refactoring and cleanup of the whole recipe and vcl generation:

  • skip support of varnish < v4.0, use 1.x series for older varnish support.

  • do not generate vcl code in python

  • use jinja2 templates for vcl

  • refactor vcl generation out in own testable class

  • change fixup cookies into a cookie whitelist

  • split up recipe in 3 parts: build, configuration generation and script generation.

  [jensens]

1.4 (unreleased)

• Fix test for running in the Varnish 2 or later. [cleberjsantos]

• Fixup VCL template Varnish 3. [cleberjsantos]

• add saint-mode for varnish_version 3. [cleder, cleberjsantos]

• set a default download-url for varnish_version 3. [cleder]

• Fixup string concat for varnish_version 3. [damaestro]

• Add zope2_vhm_port to be able to explicitly define a response port in VHM URLs. [damaestro]

• Add zope2_vhm_ssl to use VHM to render https urls. [damaestro]

• Add zope2_vhm_ssl_port to be able to explicitly define a response port in VHM URLs for ssl. [damaestro]

• Update verbose-headers to use upstream debug example: https://www.varnish-cache.org/trac/wiki/VCLExampleHitMissHeader [damaestro]

• Add cookie-fixup to better support caching of plone conent and to ensure no authenticated content gets cached. http://developer.plone.org/hosting/varnish.html [damaestro]

• Update VCL templates to be more flexible. [damaestro]

1.3 (2013-08-21)

• Add varnish_version option in order to control vcl generation for varnish version >= 3 [rnix]

1.2.2 (2012-10-14)

• Moved to https://github.com/collective/plone.recipe.varnish [maurits]

1.2.1 (2011-05-13)

• Update known good Varnish to 2.1.5. [elro]

• Add vcl_recv, vcl_hit, vcl_miss, vcl_fetch, vcl_deliver, vcl_pipe options to insert arbitrary vcl. [elro]

1.2 (2011-01-11)

• Added new options cache-type, cache-location for specifying type of Varnish storage (such as using malloc for alternative storage) and setting a custom location for said storage [davidjb]

• Added additional unit tests to check Varnish initialisation script [davidjb]

• Added new option ‘purge-hosts’. Enables additional addresses allowed to purge. [jensens]

• Added the name option to be able to define the directory varnishd puts temporary files to and identify the instance when using varnishlog or varnishstat. [fRiSi]

• fixed configuration for verbose-headers=on (context in vlc_fetch is bresp instead of obj in newer varnish versions) [fRiSi]

1.1 (2010-08-05)

• Changed the default cache size to 256M from 1G. [hannosch]

• Updated Varnish to 2.1.3. [hannosch]

1.1b1 (2010-04-25)

• Updated advertised Varnish version to 2.1 and adjusted config. [hannosch]

• Correct documentation for the daemon setting and remove the default. [hannosch]

• Removed the deprecated build recipe. [hannosch]

• Added basic test infrastructure and a test for the simple buildout. [hannosch]

• Use the built-in set type instead of the deprecated sets module. This recipe now requires at least Python 2.4. [hannosch]

• Added the ability to configure runtime parameters in the varnish runner configuration and added information to the documentation for it. [benliles]

• Improve readability of the generated config. [ldr]

1.0.2 (2010-01-18)

• Update proposed Varnish to 2.0.6. [hannosch]

• Further documentation cleanup. [hannosch, vincentfretin]

1.0.1 (2009-11-27)

• Expose the download-url of a known-good Varnish release that works with the configuration produced by the instance recipe. [hannosch]

• Consistently use tabs in the generated vcl file. [hannosch]

• Whitespace and documentation cleanup. [hannosch]

1.0 (2009-08-27)

• Made the vcl template build its acl purge section. At present, the vcl will only allow purges coming from the local host. If we have multiple hosts that are separate from localhost, any PURGE requests will be denied without this. See http://varnish.projects.linpro.no/wiki/VCLExamplePurging [rockdj]

• Added ability to set various Varnish timeouts (connect_timeout, first_byte_timeout, and between_bytes_timeout) from each option in the buildout. Default values are set at Varnish defaults of 0.4s for connect_timeout, and 60s for between_bytes_timeout. Time for first_byte_timeout is set at 300s as per plone.recipe.varnish 1.0rc9. [rockdj]

• Set req.http.host for incoming virtual hosted URLs. Without setting this, purge requests sent from hosts other than localhost (the only host in the acl purge list) will result in a 404 message. See http://davidjb.com/blog/2009/01/plone-varnish-configuration-cache-hits-purge-fails [rockdj]

1.0rc11 (2009-06-27)

• Reintroduced grace options. What the varnish documentation say about grace: “varnish serves stale (but cacheable) objects while retrieving object from backend”. The problem is “default_ttl” value is 120s (see bin/varnishd/mgt_param.c in varnish 2.0.4). Added a special rule for createObject url to not look up in the cache. [vincentfretin]

1.0rc10 (2009-06-26)

• 1.0rc9 generated broken configuration with balancer=none [vincentfretin]

1.0rc9 (2009-06-25)

• Do not set req.grace and obj.grace. See http://vincentfretin.ecreall.com/articles/varnish-user-be-careful [vincentfretin, maurits]

• Removed header_hit_deliver and header_hit_notcacheable debug messages from default template. It is not safe to assign to the object during vcl_hit until http://varnish.projects.linpro.no/ticket/310 is not fixed. See also http://kristian.blog.linpro.no/2009/05/25/common-varnish-issues. [hannosch]

• Updated to refer to Varnish 2.0.4. Added a first_byte_timeout value of 300 seconds to the backend definitions. This is a new option since Varnish 2.0.3 and by default set to 60 seconds. This is arguably too low for certain edit operations in Plone sites. [hannosch]

1.0rc8 (2008-02-12)

• Remove the custom vcl_hash from the template. Adding the Accept-Encoding header to the cache break effectively breaks purging since nobody will ever include those headers in a PURGE request. To make this safe we just remove the Accept-Encoding header from all incoming requests as well. [wichert]

1.0rc7 (2008-11-26)

• Be more explicit about deprecating the :build entry point. [wichert]

• Make the :instance specifier optional: after :build has been removed we can deprecate :instance as well. [wichert]

1.0rc6 (2008-09-22)

• Deprecate plone.recipe.varnish:build in favour of zc.recipe.cmmi: it does not make sense to duplicate its logic here. [wichert]

• Add feature to enable verbose headers in varnish.vcl. This is primary interesting for debugging of cache-settings. See README.txt. [jensens]

• Deal better with sources which do not have executable-bits set or are svn exports. [wichert]

• The 1.0rc5 release was broken and has been retracted. Currently the trunk is only usable with the Varnish 2.0-beta1 and later. [hannosch]

1.0rc5 (2008-04-27)

• Pipe is evil: it pipes the whole connection to the backend which means varnish will no longer process any further requests if HTTP pipelining is used. Switch to using pass instead. [wichert]

• Add a default_ttl of zero seconds to the Varnish runner to avoid a Varnish bug with the handling of an Expires header with a date in the past. [newbery]

• Merged branches/newbery-hostnamepath. [newbery]

• We don’t need to include Accept-Encoding in the hash. Varnish takes care of Vary negotiation already. [newbery]

1.0rc4 (2008-03-18)

• Fixed typos / whitespace. [hannosch]

• Varnish 1.1.2 is out. [wichert]

• Merged witsch-foreground-support back to trunk. [witsch]

• Use a pidfile. [wichert]

1.0rc3 (2007-09-02)

• Fixed a bug where options[“location”] was being used before it was being set. [rocky]

• Made the module name determination a little more robust during createVarnishConfig so that recipes that specify version deps still work. [rocky]

• Do not use defaults for user and group. [wichert]

• We do need the parts: we use it for the file storage. [wichert]

1.0rc2 (2007-08-29)

• Add an option to use an existing configuration file. [wichert]

• Remove hardcoded caching for images, binaries, CSS and javascript. This should be done by the backend server or a custom varnish configuration. [wichert]

• Add Accept-Encoding to the cache key so we can handle compressed content. [wichert]

• Test if a bin-directory exists. This allows us to compile varnish 1.0 which does not have an sbin directory. [wichert]

1.0rc1 (2007-08-27)

• Document the OSX bugfix we apply when building varnish. [wichert]

• Add a dummy update method to prevent needless recompiles. [wichert]

• Update for Varnish 1.1.1. [wichert]

1.0b2 (2007-08-25)

• When building from svn, we need to run autogen.sh. [optilude]

• Refactor the recipe: there are now separate recipes to build and configure Varnish. This makes it possible to reconfigure varnish without having to recompile with as well as using an already installed varnish. [wichert]

• Move the OSX patching code into a separate method. [wichert]

• Use pass for non-GET/HEAD requests. This makes a bit more sense and fixes a login problem for Plone sites. [wichert]

• Reorganize a bit for readability. [wichert]

• Support Python 2.3 as well. [wichert]

• Make it possible to specify the user and group as well. [wichert]

• Do not create the source directory - we move the extracted source in its place later. [wichert]

• If running on OS X, patch libtool as described in http://varnish.projects.linpro.no/ticket/118 and http://thread.gmane.org/gmane.comp.web.varnish.misc/668/focus=669. [optilude]

• VCL is not C. You need the curlies even on single-line if statements. [optilude]

• This rewriting style only works on Zope 3 - Zope 3 reinvented that wheel. [wichert]

• Add support for If-Modified-Since and If-None-Match requests. Thanks to newbery for the suggstions. [wichert]

• Explicitly mention that there is nothing Plone or Zope specific about this recipe. [wichert]

1.0b1 (2007-08-04)

• More documentation. [wichert]

• Ignore the port information in the host header. [wichert]

• Use the port varnish is bound to in the VHM mapping. [wichert]

• Define all default values centrally. [wichert]

• Add support for Zope virtual hosts. [wichert]

• Add support for virtual hosting. [wichert]

• Initial import of Varnish recipe. [wichert]