waf-downloader 0.1.5

Web Application Firewall logs downloader.

Cloudflare Web Application Firewall downloader

A library, CLI, and docker image that downloads Cloudflare WAF logs for a specified zone_id and time frame.

Schema

See src/waf_logs/resources/db/ for a list of schemas that are auto-applied at start. This can be disabled by passing --ensure_schema False.

Quickstart

The project is published at https://pypi.org/project/waf-downloader/.

Install it via:

pip install waf-downloader

# or alternatively, directly from git
pip install "git+https://github.com/MihaiBojin/waf-downloader@main"

Or with Docker:

docker pull ghcr.io/mihaibojin/waf-downloader:latest

The list of published images can be found at: https://github.com/MihaiBojin/waf-downloader/pkgs/container/waf-downloader

Build and run with Docker

Define secrets in an .env file (do not quote values):

CLOUDFLARE_TOKEN=...
DB_CONN_STR=...

The Cloudflare token is required, but the connection string is optional. If skipped, it will result in logs being printed to stdout.

IMPORTANT: This project uses taskfile.dev, which you will need to install for running the following commands:

# Build
task docker-build

# Load all logs in zone, starting 5 minutes prior
task docker-run ARGS="--zone_id [CLOUDFLARE_ZONE_ID] --start_minutes_ago -5"

# And alternatively, only output the logs
task docker-run ARGS="--zone_id [CLOUDFLARE_ZONE_ID] --start_minutes_ago -5" 2>/dev/null

Publishing to PyPI

GitHub-based version publishing

The simplest way to publish a new version (if you have committer rights) is to tag a commit and push it to the repo:

# At a certain commit, ideally after merging a PR to main
git tag v0.1.x
git push origin v0.1.x

A GitHub Action will run, build the library and publish it to the PyPI repositories.

Manual publish

These steps can also be performed locally. For these commands to work, you will need to export two environment variables (or define them in .env):

export TESTPYPI_PASSWORD=... # token for https://test.pypi.org/legacy/
export PYPI_PASSWORD=... # token for https://upload.pypi.org/legacy/

First, publish to the test repo and inspect the package:

task publish-test

If correct, distribute the wheel to the PyPI index:

task publish

Verify the distributed code

task publish-verify

Cloudflare WAF documentation

• https://developers.cloudflare.com/analytics/graphql-api/tutorials/querying-firewall-events
• https://developers.cloudflare.com/analytics/graphql-api/tutorials/export-graphql-to-csv/