cnpg-drill

Prove that a CloudNativePG backup can restore. cnpg-drill creates a disposable CloudNativePG cluster from the source cluster's Barman Cloud object store, waits for PostgreSQL to become ready, runs read-only SQL assertions, emits a JSON result, and deletes the drill cluster. It can run locally or on a schedule as a Kubernetes CronJob.

The core is free and works without an account or external service. It does not make backups or copy database contents to a vendor. The proposed hosted service in RESEARCH.md is separate and has not been built.

Status

Early alpha. Unit tests cover manifest safety and execution state. A disposable local integration run on v0.1.0 proved full restore, PITR, read-only archive access, denied-WAL reporting, failed-check reporting, and Cluster/PVC cleanup on one version matrix. The v0.1.1 release archive and public chart 0.1.2 also restored a fresh backup and checked a row in an application database. A separate single-node pgEdge chart restore passed SQL checks in its app database. Validate against your operator, PostgreSQL image, storage, object store, and Barman plugin versions before treating a pass as disaster recovery assurance.

The v0.1.3 release accepts CloudNativePG storage.pvcTemplate capacity as well as storage.size and avoids binding drill PVCs to source volumes or snapshots. It includes the MIT-licensed kubectl plugin archive and checksum. The default Krew index submission packages v0.1.2, while the custom Krew index submission packages v0.1.0; both remain under review. kubetools is reviewing a Backup Tools listing.

The Helm chart is listed on Artifact Hub as a Verified Publisher package. Chart version 0.1.4 uses the v0.1.2 application image and includes the publisher's website.

The Python CLI is also published on PyPI. Its wheel and source distribution are built from the release tag with GitHub trusted publishing.

What it supports

• One source CloudNativePG cluster with one enabled Barman Cloud plugin and a named ObjectStore in the same namespace.
• An optional separate recoveryObjectStore with credentials limited to archive reads. Preflight checks that its destination and endpoint match the source store before creating a Cluster.
• Full restore from the latest completed Barman plugin Backup resource, or PITR to an explicit RFC3339 targetTime. The selected backup ID is pinned and included in the report; a stale backup fails preflight.
• Single-instance drill cluster using the source's PostgreSQL image, storage.size or storage.pvcTemplate capacity and class, optional WAL storage, and resource requests. Existing volume bindings and snapshot data sources are removed from the drill PVC template.
• Copies literal AWS checksum compatibility settings from the source Cluster for S3-compatible archive restores; it does not copy writer credentials or arbitrary environment variables.
• Read-only SQL checks in postgres or a named application database. Exact expected values make reports easy to audit.
• Optional timestamp freshness assertions (maxAgeSeconds) for an application-defined recovery point. This measures age of that application's data, not the database WAL RPO.
• A JSON report on stdout and optionally in a local file. Query outputs are hashed rather than logged. Exit code 0 means all checks, Cluster deletion, and PVC cleanup passed; 1 means the drill failed; 2 means invalid input or a failed preflight.
• Scheduled execution with the Helm CronJob in deploy/helm/cnpg-drill.

The CLI refuses source clusters with tablespaces or recovery bootstrap because a naive clone could produce an incomplete or misleading pass. Volume snapshots, cross-namespace restores, cross-region recovery, custom Postgres extension image changes, and multi-cluster fleet management are future work.

Install and use

For a first run on your own cluster, follow the first-run guide.

Requirements: Python 3.10+, kubectl in PATH, CloudNativePG and the Barman Cloud plugin installed in the target cluster, and Kubernetes access to read the source Cluster and ObjectStore, create/get/delete a drill Cluster, and exec into its PostgreSQL pod.

uv tool install cnpg-drill==0.1.3
cnpg-drill plan --config examples/drill.json
cnpg-drill run --config examples/drill.json --report reports/latest.json

pipx install cnpg-drill==0.1.3 is another way to install the CLI in its own environment. For source development, clone this repository and run python3 -m pip install -e ..

plan reads the source configuration and prints the exact Cluster manifest it would create. Review that manifest before running the first drill. The example's table-count assertion is illustrative; replace it with an invariant from your own application. A minimal check is SELECT 1, but it only proves connection to the recovered server, not useful application data.

Example drill.json:

{
  "namespace": "production",
  "cluster": "app-db",
  "recoveryObjectStore": "app-db-recovery-readonly",
  "timeoutSeconds": 1800,
  "maxBackupAgeSeconds": 691200,
  "checks": [
    {"name": "postgres-ready", "query": "SELECT 1", "expected": "1"},
    {"name": "orders-exist", "database": "app", "query": "SELECT count(*) > 0 FROM public.orders", "expected": "t"},
    {"name": "recent-order", "database": "app", "query": "SELECT max(created_at)::timestamptz FROM public.orders", "maxAgeSeconds": 86400}
  ]
}

For PITR, add "targetTime": "2026-09-29T10:00:00Z". Pick a time inside the backup and WAL retention window. A passing PITR check proves only that target; use a second drill for the latest recovery path.

Scheduled drill

The Helm chart creates a namespace-scoped ServiceAccount, Role, RoleBinding, ConfigMap, and CronJob. Set cluster, checks, and the other settings in a values file and install it in the same namespace as the source Cluster and ObjectStore. The chart defaults to suspension so installation does not immediately start a restore.

The example image build pins kubectl 1.36.4, suitable for Kubernetes 1.35–1.37 under the project's version skew policy. Set KUBECTL_VERSION at image build time for another supported cluster version.

helm install cnpg-drill oci://ghcr.io/danielgaskins/charts/cnpg-drill \
  --version 0.1.4 -n production -f drill-values.yaml

See the first-run guide and chart values for a suspended first run with a separate read-only recovery ObjectStore. The chart pins a public multi-architecture image digest. The CronJob uses concurrencyPolicy: Forbid and a bounded job deadline. Its logs contain the JSON result; failed runs have nonzero exit status. Optional PVC reports and Prometheus rules preserve results and flag missed or failed runs. Fleet policy remains future work.

Safety boundary

• The generated recovery Cluster never copies spec.plugins, so it is not configured to archive WAL back to the source bucket. Set recoveryObjectStore to a separately named ObjectStore with read-only archive credentials. The tool compares its destination and endpoint with the source store; it cannot verify credential permissions, so test that writes are denied before relying on this boundary. Omitting this field uses the source ObjectStore for compatibility.
• A recovery timeout includes a machine-readable failureReason when the recovery Pod logs identify archive access denial or unavailable WAL. It does not copy raw logs into the report; inspect Pod and operator logs for detail.
• The drill uses the same namespace because the plugin's ObjectStore and its credentials are namespace-scoped. The source Cluster object is never modified.
• SQL checks run in a BEGIN READ ONLY transaction. Queries are limited to one SELECT or WITH statement without semicolons. Only provide trusted SQL; this is a guard against mistakes, not a security sandbox.
• By default, the tool deletes the drill Cluster even when recovery or a check fails. retainOnFailure leaves it for investigation and can incur storage costs. Cleanup failures turn the result red.
• The tool checks that PVCs labeled for the drill cluster disappear after Cluster deletion and fails the report if they remain. Kubernetes PersistentVolume cleanup depends on the storage class; inspect cloud volumes after a first drill.
• The planned SaaS boundary is metadata only. This repository contains no telemetry upload or hosted service.

Contributing and upstream distribution

Run PYTHONPATH=src python3 -m unittest discover -s tests -v. See CONTRIBUTING.md for how to add a recovery path, the live integration test gate, and the Krew package plan. We will submit only a tested, released package that stands on its own. Upstream acceptance is not assumed.

UPSTREAM.md records the specific repositories, proposed contribution value, and submission gates.

Recovery references

• CloudNativePG backup guidance
• CloudNativePG recovery documentation
• Barman Cloud recovery example