pulumi-journaling

An independent fork of Pulumi, maintained by E Brake ebrake123@icloud.com, with hardened, recoverable journaling for every DIY state storage backend: local files, Amazon S3 and compatible stores, Google Cloud Storage, Azure Blob Storage, and PostgreSQL. Based on upstream v3.274.1 at commit 8f735e662d805b7a5bda70931bd627c69fbf38b8, the latest stable release when this fork was prepared.

This project is independently maintained and is not affiliated with, sponsored by, or endorsed by Pulumi Corporation. Pulumi is a trademark of Pulumi Corporation.

Install

python -m pip install pulumi-journaling==3.274.1.post1
pulumi-journaling version

The Python distribution provides both pulumi-journaling and pulumi commands, plus the matching upstream Python SDK, so existing import pulumi programs continue to work. On first invocation, the launcher downloads the matching CLI and language hosts from this fork's GitHub releases, verifies the SHA-256 checksum embedded in the Python package, and caches the bundle. That first invocation requires network access. Set PULUMI_JOURNALING_CACHE to choose the cache directory. Linux and macOS on x86-64 and ARM64 are supported; Windows packages are not currently published.

The Python launcher enables DIY journaling by default. Existing Pulumi commands and backend URLs continue to work:

pulumi login 's3://my-state-bucket'
pulumi up

Set PULUMI_DIY_BACKEND_JOURNALING=false to stop creating new journal epochs. Recovery of an unfinished epoch still runs before a mutating operation. Standalone GitHub bundles require PULUMI_DIY_BACKEND_JOURNALING=true to opt in. Pulumi Cloud retains its upstream service-managed persistence and journaling behavior.

Rapid GCS and storage latency

Google Cloud Storage Rapid buckets are supported with an explicit backend option:

pulumi login 'gs://my-rapid-state-bucket?gcs_api=rapid'
PULUMI_DIY_BACKEND_JOURNALING=true pulumi up

Choose low-latency state storage close to the machine running Pulumi. For Rapid, place CI runners in the bucket's exact zone. The fork uses the Google Cloud Storage Go SDK's gRPC transport and keeps one append stream open for the update, flushing required journal entries durably and finalizing once at close. Set PULUMI_GCS_DIRECT_PATH=required on colocated runners to reject connections that cannot use direct connectivity. Remote operator CLIs can leave it unset.

Rapid is zonal storage and does not provide object versioning or soft delete. Keep verified checkpoint archives outside the bucket and exclude current state and active journals from lifecycle deletion. See the Rapid transport guide. Other DIY backends retain their existing journal and recovery paths.

Basis and hardening

DIY journaling is based on Jason Rowsell's prototype in Pulumi PR #22074. E Brake hardened and extended that basis with:

• Encrypted immutable entries, bounded asynchronous writes, and acknowledgements that wait for the entire submitted prefix to become durable
• Epoch identity, checkpoint hashes and required-feature guards that prevent stock clients from consuming an unfinished journal
• Native stack locking, crash-safe consolidation, idempotent recovery, and preservation of unexpected or corrupt state
• Consistent read-only access to the last clean checkpoint, provider-free recovery, and explicit evidence-preserving discard
• Checkpoint backups, retention and compression support, integrity reporting, and protection against completing a replaced epoch
• Conditional checkpoint replacement for GCS generations, S3 ETags Azure ETags, and exact PostgreSQL row versions, plus atomic replacement and fsync for local files
• Fault-injection, cancellation, durability-ordering, concurrent-reader, recovery and storage-driver tests; version-pinned CLI bundles and a checksummed Python launcher

See the journaling and recovery guide for operational details and tradeoffs. S3-compatible services must implement conditional writes and strongly consistent reads and listings. An endpoint that rejects required preconditions fails the operation; the CLI does not fall back to unconditional checkpoint replacement.

Recovery

pulumi state journal recover --stack organization/project/stack

Recovery invokes no resource providers. If the previous process crashed, first verify it has stopped, then remove its stale native lock with pulumi cancel. If recovery refuses corrupt or unexplained state, preserve the evidence and consult the recovery guide before using pulumi state journal discard --yes.

Complete recovery before changing to an upstream CLI or another fork. Clean snapshots and provider packages retain upstream contracts.

Development and releases

The release configuration pins the upstream commit and archive checksums. The validation workflow runs regression tests, linters, lifecycle tests and the actual file, GCS, S3, Azure and PostgreSQL storage drivers against isolated local emulators. The release workflow builds all four platform bundles, exercises the CLI, packages the launcher with the resulting checksums, and can publish to GitHub and PyPI. When hosted runners are unavailable, its prebuilt option accepts a reviewed draft release, verifies all platform and Python-package checksums and common source provenance, then publishes through the same trusted publisher.

The upstream contributing guide covers the Go monorepo. Report fork issues in this repository. See authors, notices and the Apache 2.0 license.