Cloudflare Web Application Firewall downloader
A library, CLI, and docker image that downloads Cloudflare WAF logs for a specified zone and time range.
Schema
See src/waf_logs/resources/db/ for a list of schemas that are auto-applied at start. This can be disabled by passing --ensure_schema False.
Quickstart
The project is published at https://pypi.org/project/waf-downloader/.
Install it via:
pip install waf-downloader
# or alternatively, directly from git
pip install "git+https://github.com/MihaiBojin/waf-downloader@main"
Or with Docker:
docker pull docker.io/mihaibojin/waf-downloader:latest
The list of published images can be found at: https://github.com/MihaiBojin/waf-downloader/pkgs/container/waf-downloader
and
https://hub.docker.com/repository/docker/mihaibojin/waf-downloader/tags
Or with Helm
See charts/waf-downloader/README.md for more details.
Development
This project uses uv. Install it, then one command gets you a working development environment:
uv sync --all-extras
That creates .venv/, installs everything from the committed uv.lock, and
provisions a Python interpreter if you do not have a suitable one. Run
commands through uv run (uv run pytest tests) or activate the environment
with source .venv/bin/activate.
To also install the git hooks, run task setup.
Dependencies
Declare them in pyproject.toml, under [project] dependencies or
[project.optional-dependencies], then refresh the lock file:
uv lock
Commit uv.lock alongside the pyproject.toml change; CI runs
uv sync --locked and fails if the two disagree.
Build and run with Docker
Define secrets in an .env file (do not quote values):
CLOUDFLARE_API_TOKEN=...
DB_CONN_STR=...
The Cloudflare token is required (see required permissions), but the connection string is optional. If skipped, it will result in logs being printed to stdout.
IMPORTANT: This project uses taskfile.dev, which you will need to install for running the following commands:
# Build
task docker-build
# Load all logs in zone, starting 5 minutes prior
task docker-run -- --zone_id zoneid1 --start_minutes_ago 5
# And alternatively, only output the logs
task docker-run -- --zone_id zoneid1 --start_minutes_ago 5 2>/dev/null
# Do not specify a start time, relying on a starting timestamp stored in the database
# If a timestamp is not found in the database, or specified with --start_minutes_ago, the downloader will start 5 minutes prior
# This functionality makes it easy to run waf-downloader as a cron job
# NOTE: specifying --start_minutes_ago will always override the timestamp stored in the database, causing potential gaps in the data
task docker-run -- --zone_id zoneid1
# Do not exit and keep downloading new logs forever
# These will be recent up to the last minute
task docker-run -- --zone_id zoneid1 --follow
# Multiple zones can be specified via a comma-separated string
task docker-run -- --zone_id zoneid1,zoneid2,zoneid3,etc
# Or by repeating the flag
task docker-run -- --zone_id zoneid1 --zone_id zoneid2 --zone_id zoneid3 ...
Publishing to PyPI
GitHub-based version publishing
The simplest way to publish a new version (if you have committer rights) is to tag a commit and push it to the repo:
# At a certain commit, ideally after merging a PR to main
git tag v0.1.x
git push origin v0.1.x
A GitHub Action will run, build the library and publish it to the PyPI repositories.
Manual publish
These steps can also be performed locally. For these commands to work, you will need to export two environment variables (or define them in .env):
export TESTPYPI_PASSWORD=... # token for https://test.pypi.org/legacy/
export PYPI_PASSWORD=... # token for https://upload.pypi.org/legacy/
The tasks pass these to uv publish as UV_PUBLISH_TOKEN; the test index is
defined as testpypi under [[tool.uv.index]] in pyproject.toml.
First, publish to the test repo and inspect the package:
task publish-test
If correct, distribute the wheel to the PyPI index:
task publish
Verify the distributed code
task publish-verify
Cloudflare WAF documentation
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file waf_downloader-0.3.1.tar.gz.
File metadata
- Download URL: waf_downloader-0.3.1.tar.gz
- Upload date:
- Size: 217.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
08777bfa8790f50f3594cad919e28dbdac2b375cea8f92df77b53eb40a65710e
|
|
| MD5 |
5a97f86d863c20bd8007877f115defaa
|
|
| BLAKE2b-256 |
d58ec9f2f7ea49418e9e9b484c0d2ea5abe38434976cfcf450b7ed44bc33b548
|
File details
Details for the file waf_downloader-0.3.1-py3-none-any.whl.
File metadata
- Download URL: waf_downloader-0.3.1-py3-none-any.whl
- Upload date:
- Size: 21.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
eca763ba09285781cd48bae0ffdf249cabceffd493e20fe515291561705ef0cb
|
|
| MD5 |
f112973b0d2d4615cd6a5480b4863ade
|
|
| BLAKE2b-256 |
8b1654a2d064837b7205b35d08a6b371d3890508a4ab67f89d5c74431c7da94f
|
