varnishlog-to-elasticsearch 0.1.0

A tool to parse Varnish HTTP logs and send them to Elasticsearch

varnishlog-to-elasticsearch

A lightweight, stream-processing tool that parses Varnish HTTP logs and sends structured JSON documents directly to Elasticsearch.

This tool is designed for simplicity, resilience, and easy integration into production environments. It supports both development and systemd-based deployments, and includes full support for runtime configuration via environment variables.

🚀 Features

• Parses full Varnish request lifecycle (-g request)
• Captures client and backend metadata
• Automatically determines production/staging mode via headers
• Sends structured JSON documents to Elasticsearch
• Built-in support for environment-based configuration
• Production-ready systemd service example included
• Efficient bulk indexing with configurable buffering
• Secure credential management via environment variables

📦 Installation

From PyPI (Recommended)

pip install varnishlog-to-elasticsearch

From Source

1. Clone this repository:

git clone https://github.com/NIXRUK/varnishlog-to-elasticsearch.git
cd varnishlog-to-elasticsearch

2. Install in development mode:

pip install -e .

⚙️ Configuration

All configuration is done through environment variables. The following variables are supported:

Core Configuration

Variable	Description	Default Value
ES_BASE_URL	Elasticsearch base URL	http://localhost:9200
ES_USERNAME	Elasticsearch username	elastic
ES_PASSWORD	Elasticsearch password	(empty)
ES_VERIFY_SSL	Verify SSL certs (true, false, etc.)	true

Buffering Configuration

Variable	Description	Default Value
ES_BUFFER_SIZE	Maximum number of documents to buffer	100
ES_FLUSH_INTERVAL	Maximum time (seconds) between flushes	5

Set them manually like this:

export ES_BASE_URL=https://10.99.99.4:9200
export ES_USERNAME=elastic
export ES_PASSWORD=your_secure_password
export ES_VERIFY_SSL=true
export ES_BUFFER_SIZE=200
export ES_FLUSH_INTERVAL=10

Or define them in a system-wide config file for systemd (see below).

🧪 Usage

Basic Usage

You can pipe Varnish output directly:

/usr/bin/varnishlog -g request | varnishlog-to-es

Or test with a sample log file:

cat example.log | varnishlog-to-es

Python API

You can also use the tool programmatically:

from varnishlog_to_elasticsearch.parser import main_loop

# Process logs from a file
with open('varnish.log', 'r') as f:
    main_loop(f)

# Or process from stdin
import sys
main_loop(sys.stdin)

🛠️ Running as a systemd Service

This project includes a ready-to-use systemd service file.

1. Create the environment file

Create a file at /etc/varnishlog-to-es.env with your credentials:

ES_BASE_URL=https://10.99.99.4:9200
ES_USERNAME=elastic
ES_PASSWORD=your_secure_password
ES_VERIFY_SSL=true
ES_BUFFER_SIZE=200
ES_FLUSH_INTERVAL=10

Protect it:

sudo chmod 640 /etc/varnishlog-to-es.env
sudo chown root:varnish /etc/varnishlog-to-es.env

2. Create the systemd unit file

Save this to /etc/systemd/system/varnishlog-to-es.service:

[Unit]
Description=Varnishlog to Elasticsearch
After=network.target

[Service]
Type=simple
User=varnish
Group=varnish
EnvironmentFile=/etc/varnishlog-to-es.env
ExecStart=/usr/bin/varnishlog -g request | varnishlog-to-es
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

3. Enable and start the service

sudo systemctl daemon-reload
sudo systemctl enable varnishlog-to-es
sudo systemctl start varnishlog-to-es

4. View logs

journalctl -u varnishlog-to-es -f

🔒 Security Notes

• Always use HTTPS for Elasticsearch, especially in production.
• Never commit passwords to source code — use environment variables or secrets managers.
• Use .env or systemd's EnvironmentFile to manage runtime secrets securely.
• SSL verification is enabled by default (ES_VERIFY_SSL=true). Only disable it in development environments.
• The buffer size and flush interval can be tuned based on your security and performance requirements.

🧹 Future Ideas

• Docker container
• GeoIP enrichment
• Log rotation / buffering
• Index lifecycle policy automation
• Retry mechanism for failed bulk operations
• Metrics and monitoring integration

📜 License

MIT © Will Riches — open to contributions and extensions.