Cloudflare Web Application Firewall downloader

Build Status

A library, CLI, and docker image that downloads Cloudflare WAF logs for a specified zone and time range.

Logo

Schema

See src/waf_logs/resources/db/ for a list of schemas that are auto-applied at start. This can be disabled by passing --ensure_schema False.

Quickstart

The project is published at https://pypi.org/project/waf-downloader/.

Install it via:

pip install waf-downloader

# or alternatively, directly from git
pip install "git+https://github.com/MihaiBojin/waf-downloader@main"

Or with Docker:

docker pull docker.io/mihaibojin/waf-downloader:latest

The list of published images can be found at: https://github.com/MihaiBojin/waf-downloader/pkgs/container/waf-downloader

and

https://hub.docker.com/repository/docker/mihaibojin/waf-downloader/tags

Or with Helm

See charts/waf-downloader/README.md for more details.

Development

Build and run with Docker

Define secrets in an .env file (do not quote values):

CLOUDFLARE_API_TOKEN=...
DB_CONN_STR=...

The Cloudflare token is required (see required permissions), but the connection string is optional. If skipped, it will result in logs being printed to stdout.

IMPORTANT: This project uses taskfile.dev, which you will need to install for running the following commands:

# Build
task docker-build

# Load all logs in zone, starting 5 minutes prior
task docker-run -- --zone_id $CLOUDFLARE_ZONE_ID --start_minutes_ago 5

# And alternatively, only output the logs
task docker-run -- --zone_id $CLOUDFLARE_ZONE_ID --start_minutes_ago 5 2>/dev/null

# Do not specify a start time, relying on a starting timestamp stored in the database
# If a timestamp is not found in the database, or specified with --start_minutes_ago, the downloader will start 5 minutes prior
# This functionality makes it easy to run waf-downloader as a cron job
# NOTE: specifying --start_minutes_ago will always override the timestamp stored in the database, causing potential gaps in the data
task docker-run -- --zone_id $CLOUDFLARE_ZONE_ID

Publishing to PyPI

GitHub-based version publishing

The simplest way to publish a new version (if you have committer rights) is to tag a commit and push it to the repo:

# At a certain commit, ideally after merging a PR to main
git tag v0.1.x
git push origin v0.1.x

A GitHub Action will run, build the library and publish it to the PyPI repositories.

Manual publish

These steps can also be performed locally. For these commands to work, you will need to export two environment variables (or define them in .env):

export TESTPYPI_PASSWORD=... # token for https://test.pypi.org/legacy/
export PYPI_PASSWORD=... # token for https://upload.pypi.org/legacy/

First, publish to the test repo and inspect the package:

task publish-test

If correct, distribute the wheel to the PyPI index:

task publish

Verify the distributed code

task publish-verify

Cloudflare WAF documentation

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

waf_downloader-0.1.22.tar.gz (225.9 kB view details)

Uploaded Oct 20, 2024 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

waf_downloader-0.1.22-py3-none-any.whl (20.8 kB view details)

Uploaded Oct 20, 2024 Python 3

File details

Details for the file waf_downloader-0.1.22.tar.gz.

File metadata

Download URL: waf_downloader-0.1.22.tar.gz
Upload date: Oct 20, 2024
Size: 225.9 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.1.1 CPython/3.13.0

File hashes

Hashes for waf_downloader-0.1.22.tar.gz
Algorithm	Hash digest
SHA256	`e144c3c6d2aad4618eead86a5d3d94f936f4a44476a012a507af1c9a36f846ea`
MD5	`deea4b9affa30f660a99f342427cb382`
BLAKE2b-256	`6b150c1f001e16de33cccafde4381b0e916b0a23d2a233842ce17a2244929fc0`

See more details on using hashes here.

File details

Details for the file waf_downloader-0.1.22-py3-none-any.whl.

File metadata

Download URL: waf_downloader-0.1.22-py3-none-any.whl
Upload date: Oct 20, 2024
Size: 20.8 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.1.1 CPython/3.13.0

File hashes

Hashes for waf_downloader-0.1.22-py3-none-any.whl
Algorithm	Hash digest
SHA256	`a565ec573250e7d95f024c4141d663540592f88b4d17a1ed5b87f22a5f456de8`
MD5	`be795f814ac421d68483e4a9cbeebc7a`
BLAKE2b-256	`f984c00fde252e9ff28d3ef783a83ef8497f14556871f869c2091cfd67f99e5c`