Skip to main content

Manifest Builder

Generate materialized Kubernetes manifests from various types of configuration

Installation

To install or upgrade to the latest version:

uv pip install --upgrade manifest-builder

Large CRDs and Argo CD

Generated CustomResourceDefinitions automatically receive argocd.argoproj.io/sync-options: ServerSideApply=true when their estimated client-side apply annotation usage reaches 240 KiB (245760 bytes). This applies to both rendered charts and copied manifests. Other sync options are preserved; an existing ServerSideApply=false is replaced for large CRDs.

Client-side apply stores the resource as JSON in the kubectl.kubernetes.io/last-applied-configuration annotation. Kubernetes limits the combined annotation keys and values to 256 KiB (262144 bytes), so the check counts both the serialized resource and existing annotations, with conservative JSON escaping. The 16 KiB margin allows for metadata added during deployment. YAML file size alone is not a reliable measure of this limit.

Development

This project is using uv for development. YAML is loaded and dumped by a native extension written in Rust (rust/), so building it also needs rustup; rust-toolchain.toml selects the toolchain. To set up your dev environment, run uv sync, which compiles the extension. uv run recompiles it after a change to the Rust sources. Tests and checks can be run with the following commands:

  • uv run ruff check
  • uv run ruff format --check
  • uv run ty check
  • uv run pytest
  • cargo fmt --manifest-path rust/Cargo.toml --check
  • cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings

Set MATURIN_PEP517_ARGS="--profile dev" for faster, unoptimised builds of the extension while developing.

The extension produces the same documents and the same bytes as PyYAML with libyaml, which the manifest-id annotation relies on, and hands anything it does not cover back to PyYAML. Its loader follows YAML 1.2 rather than PyYAML's YAML 1.1 in a few whitespace corner cases that manifests do not use: NEL, LS and PS are content rather than line breaks, and some malformed block scalars are accepted. tests/test_fastyaml.py compares both directions against PyYAML.

Requirements

  • Python 3.14+
  • Helm 3.x (must be installed and available in PATH)
  • Git (required for --create-commit feature)

Python API

Use manifest_builder.generate to generate manifests from Python:

from pathlib import Path

from manifest_builder import generate

written_paths = generate(Path("conf"), Path("output"))

Extra template variables can be supplied directly from Python. They are merged with the [variables] table in config.toml just like values loaded with --vars-from:

generate(Path("conf"), Path("output"), vars={"domain": "example.com"})

Every generated object carries a noa.re/manifest-id annotation: a hash of the object's generated content, and of nothing else. Generating the same objects writes the same bytes whichever manifest-builder version does it, and an object that did not change is left exactly as it was. When the output is a git checkout, the result's manifest_ids gives the manifest-id of each created or modified object, so a caller can wait for the cluster to hold that content.

manifest_builder.get_version() returns the version string of the running manifest-builder, for calling code that wants to log which version it used:

from manifest_builder import get_version

logger.info(f"Using manifest-builder {get_version()}")

Targets and sections

A configuration directory comes in one of two layouts, told apart by the version field of its top-level config.toml.

By default, or with version = 1, config.toml declares the config blocks directly.

With version = 2, config.toml declares targets instead. A target names the sections it is built from, and carries the variables those sections are rendered with. This lets one configuration directory describe several deployments of the same sections:

version = 2

[[target]]
name = "platform-dev"
sections = ["base", "platform"]
[target.vars]
cluster_name = "platform-dev"
vanity_domain = "portswigger.com"

[[target]]
name = "platform-prod"
sections = ["base", "platform"]
[target.vars]
cluster_name = "platform-prod"
vanity_domain = "portswigger.net"

A section is a subdirectory of the configuration directory holding a section.toml of blocks — the same content a version = 1 config.toml would hold:

conf/
├── config.toml          # targets only
├── base/
│   ├── section.toml     # [[helm]], [[simple]], … blocks
│   └── argocd/
│       └── values.yaml
├── platform/
│   ├── section.toml
│   └── idcat/
│       └── idcat.toml
├── images.toml
├── releases.yaml
├── owners/
└── plugins/

Which target to generate is selected with --target, and is required for a version = 2 directory:

manifest-builder --config-dir conf --output-dir output --target platform-dev

From Python it is the target argument of generate:

generate(Path("conf"), Path("output"), target="platform-dev")

Notes:

  • A section's blocks are read from that section's own section.toml, so the paths they reference resolve inside the section directory. In the layout above, base/section.toml refers to its values file as argocd/values.yaml. Two sections can therefore use the same relative path without colliding.
  • A section file may also be named config.toml or manifest-builder.toml, so moving an existing top-level config file into a section directory works unchanged. section.toml wins if more than one is present.
  • A section may add a [variables] table of its own. It is merged with the target's vars, as are variables from --vars-from and generate(vars=...). A variable defined by more than one of these is an error rather than one silently winning.
  • images.toml, releases.yaml, owners/, and plugins/ stay at the top of the configuration directory and are shared by every target.
  • Only the sections the selected target names are loaded.

Asking which targets a configuration declares

A caller that generates a configuration into several places has somewhere to put each target, and needs to know which targets there are. declared_targets is that question:

from manifest_builder import declared_targets

declared_targets(Path("conf"))  # ("platform-dev", "platform-prod")

It returns the names in the order the file declares them, or None for a configuration directory that declares config blocks directly — which has no targets to choose between, and is generated with variables instead — and for one holding no config file at all, which generate() reports on better than a version check would. A malformed target fails here the same way generating it would.

Ask rather than reading config.toml: a caller that parses the file itself has to keep its own idea of what a target is, and can come to a different conclusion about a configuration than generate() does.

Image template variables

Shared container image definitions can be placed in images.toml in the configuration directory:

[git]
repo = "alpine/git"
version = "2.47.2"

[static-web-server]
repo = "ghcr.io/static-web-server/static-web-server"
version = "2.36.1"

Each entry is made available to Mustache templates as both the full image reference and the version. Dashes in image names are converted to underscores:

  • {{git_image}} renders as alpine/git:2.47.2
  • {{git_version}} renders as 2.47.2
  • {{static_web_server_image}} renders as ghcr.io/static-web-server/static-web-server:2.36.1
  • {{static_web_server_version}} renders as 2.36.1

Config block plugins

[[copy]] is the one config block shipped here, in a module under manifest_builder/blocks/, discovered at startup. Every other block belongs to the configuration directory that uses it, dropped into a plugins/ subdirectory:

conf/
├── config.toml
└── plugins/
    ├── helm.py                         # defines HelmBlock
    ├── simple.py                       # defines SimpleBlock
    ├── public_repo.py                  # defines PublicRepoBlock
    ├── templates/
    │   ├── simple/                     # namespaced per block
    │   └── public_repo/
    │       └── repository.yaml
    └── tests/
        └── test_public_repo.py

Blocks live with their configuration because that is where they change: a deployment's idea of what a [[simple]] app needs is the deployment's business, and two configuration directories are free to disagree. This package keeps the toolkit they are built from — chart pulling, releases.yaml parsing, Kubernetes naming, ConfigMap construction, YAML output routing — so a block stays small.

Every module in plugins/ is imported, and any concrete ConfigBlock subclass it defines is registered under the top-level TOML key its top_level_config_name() returns. A plugin block is used exactly like the built-in one:

[[public-repo]]
name = "idcat"

A plugin module implements the same interface as a built-in block:

from manifest_builder.blocks import ConfigBlock, GenerationContext
from manifest_builder.output import write_documents


class PublicRepoBlock(ConfigBlock[PublicRepoConfig]):
    def top_level_config_name(self) -> str:
        return "public-repo"

    ...

Notes:

  • Modules whose names start with _ or . are skipped, as are test modules (test_*.py, *_test.py, conftest.py) and tests/ directories, so a plugin directory can keep its own tests beside the code.
  • Plugins are imported under the manifest_builder_plugins package rather than onto sys.path, so a plugin named json.py cannot shadow an installed module. Sibling modules are reachable with a relative import (from .helpers import ...).
  • Blocks are registered in a stable order, sorted by their top-level key, so a run does not depend on filesystem order. Two blocks claiming the same key is an error.
  • Bundled templates should be resolved relative to the plugin module, for example Path(__file__).parent / "templates" / "public_repo".

Plugin modules are imported from the configuration directory, so that directory is trusted to the same degree as the manifest-builder installation itself.

Plugins from outside the configuration directory

Configuration is sometimes read from a repository that does not carry the plugins needed to parse it. generate() then takes a second plugins directory, used in addition to conf/plugins:

from manifest_builder import ExternalPlugins, generate

generate(
    Path("conf"),
    Path("output"),
    plugins=ExternalPlugins(
        path=Path("plugins-checkout/plugins"),
        source="https://example.com/plugins.git@def456",
    ),
)

path is the plugins directory itself, resolved relative to repo_root if it is not absolute, and must exist. source records where those plugins came from, and is written as a Plugins from: line in a commit created with create_commit=True. A module name offered by both directories is an error rather than one silently shadowing the other.

On the command line the same directory is given with --extra-plugins, and its source is recorded as provided on command line:

manifest-builder --config-dir conf --extra-plugins plugins-checkout/plugins

Plugins in a long-running process

A process that calls generate() repeatedly may be pointed at a configuration directory that has been checked out again between calls, possibly at the same path. Every call re-reads the plugins rather than reusing what it imported before, so no cache invalidation is needed from the caller:

  • Plugin modules are dropped from sys.modules and imported afresh on each call, so replaced module source takes effect.
  • Import-system directory caches are invalidated, so plugins added or removed since the last call are seen.
  • No __pycache__ is written for plugin modules. This keeps the configuration checkout clean, and avoids the one case where a re-import could still pick up stale code, since bytecode validity is judged on source mtime truncated to the second plus file size.
  • Templates should be read at generation time rather than cached at import, as in the example above; combined with the module reload, they then always come from the current checkout.

Plugin loading is serialized with a lock, so concurrent generation from two different configuration directories cannot interleave one load with another. manifest_builder.discovery.forget_plugin_modules() is available for a caller that wants to release imported plugin modules itself, but calling it is not required.

Externally-owned output roots

When the output repository is shared with other services or pipelines that make their own commits, manifest-builder can be told which top-level output directories it does not own. Files in those directories are left alone during cleanup, and generation fails fast if any output would land in one of them.

To declare ownership, add an owners/ directory to your config directory and drop one or more TOML files into it. Each file may set:

# A single output root owned by another pipeline:
owned = "team-a"

# Or a list of output roots:
owned = ["cluster", "monitoring", "logging"]

Entries from all owners/*.toml files are merged into a single set of externally-owned output roots.

License

MIT

Metadata

Release files for manifest-builder 0.9.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for manifest-builder 0.9.0
File Size Uploaded Provenance Actions
manifest_builder-0.9.0.tar.gz 55.1 kB Without provenance Details for manifest_builder-0.9.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for manifest-builder 0.9.0
File Interpreter ABI Platform Actions
manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_x86_64.whl CPython 3.14 abi3 Linux glibc 2.28+ x86-64 Details for manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_x86_64.whl
manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_aarch64.whl CPython 3.14 abi3 Linux glibc 2.28+ ARM64 Details for manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_aarch64.whl
manifest_builder-0.9.0-cp314-abi3-macosx_11_0_arm64.whl CPython 3.14 abi3 macOS 11.0+ ARM64 Details for manifest_builder-0.9.0-cp314-abi3-macosx_11_0_arm64.whl

Total release size: 2.7 MB

Release files / manifest_builder-0.9.0.tar.gz

Download URL manifest_builder-0.9.0.tar.gz
Size 55.1 kB
Tags Source
SHA-256 checksum
How to use checksums
50f3552c5931d2b3131a777559f1093ec815e098617de8806716c24a105bded6
BLAKE2b-256 checksum
How to use checksums
c78c00318a532ceb1eff2eb5233a04a528d94aacecebad9ac06353965869e3f9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_x86_64.whl

Download URL manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_x86_64.whl
Size 936.0 kB
Tags CPython 3.14 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
26fb6540635082eb28bc97d204fdb57eee0b0e8c2d9722a382ad9e2919f0cfd3
BLAKE2b-256 checksum
How to use checksums
08302fb16616f57d2ac780ebfa6c45dd870d437ef2f50d9c73d4be9702b510f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_aarch64.whl

Download URL manifest_builder-0.9.0-cp314-abi3-manylinux_2_28_aarch64.whl
Size 905.6 kB
Tags CPython 3.14 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
4b2e554693204262338b51c7377f900a1ae77f1b719ae47fcb716867d756d902
BLAKE2b-256 checksum
How to use checksums
82130b9a8b3f7aa0ee041e33af9d0eda012ba0027297bf68d735df0a6d30f872
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / manifest_builder-0.9.0-cp314-abi3-macosx_11_0_arm64.whl

Download URL manifest_builder-0.9.0-cp314-abi3-macosx_11_0_arm64.whl
Size 829.5 kB
Tags CPython 3.14 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b6de88dae2783339c83e603ae50f14bf7cb4ca3591ca6b59fc6c5b3f91e51864
BLAKE2b-256 checksum
How to use checksums
09f355465d65ec8f12ef4e47ee04794b775a330d67f4762d9b552706de93b77a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for manifest-builder 0.9.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page