Manifest Builder
Generate materialized Kubernetes manifests from various types of configuration
Installation
To install or upgrade to the latest version:
uv pip install --upgrade manifest-builder
Large CRDs and Argo CD
Generated CustomResourceDefinitions automatically receive
argocd.argoproj.io/sync-options: ServerSideApply=true when their estimated
client-side apply annotation usage reaches 240 KiB (245760 bytes). This applies
to both rendered charts and copied manifests. Other sync options are preserved;
an existing ServerSideApply=false is replaced for large CRDs.
Client-side apply stores the resource as JSON in the
kubectl.kubernetes.io/last-applied-configuration annotation. Kubernetes limits
the combined annotation keys and values to 256 KiB (262144 bytes), so the check
counts both the serialized resource and existing annotations, with conservative
JSON escaping. The 16 KiB margin allows for metadata added during deployment.
YAML file size alone is not a reliable measure of this limit.
Development
This project is using uv for development. YAML is
loaded and dumped by a native extension written in Rust (rust/), so building
it also needs rustup; rust-toolchain.toml selects the
toolchain. To set up your dev environment, run uv sync, which compiles the
extension. uv run recompiles it after a change to the Rust sources. Tests and
checks can be run with the following commands:
uv run ruff checkuv run ruff format --checkuv run ty checkuv run pytestcargo fmt --manifest-path rust/Cargo.toml --checkcargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
Set MATURIN_PEP517_ARGS="--profile dev" for faster, unoptimised builds of
the extension while developing.
The extension produces the same documents and the same bytes as PyYAML with
libyaml, which the manifest-id annotation relies on, and hands anything it
does not cover back to PyYAML. Its loader follows YAML 1.2 rather than PyYAML's
YAML 1.1 in a few whitespace corner cases that manifests do not use: NEL, LS
and PS are content rather than line breaks, and some malformed block scalars
are accepted. tests/test_fastyaml.py compares both directions against PyYAML.
Written manifests also use literal blocks for multiline strings with trailing
spaces, so embedded configuration stays readable without changing its contents.
Manifest IDs use the original serialization to remain stable across these
presentation changes.
Requirements
- Python 3.14+
- Helm 3.x (must be installed and available in PATH)
- Git (required for
--create-commitfeature)
Python API
Use manifest_builder.generate to generate manifests from Python:
from pathlib import Path
from manifest_builder import generate
written_paths = generate(Path("conf"), Path("output"))
Extra template variables can be supplied directly from Python. They are merged
with the [variables] table in config.toml just like values loaded with
--vars-from:
generate(Path("conf"), Path("output"), vars={"domain": "example.com"})
Every generated object carries a noa.re/manifest-id annotation: a hash of the
object's generated content, and of nothing else. Generating the same objects
produces the same manifest IDs whichever manifest-builder version does it,
even when output formatting changes. When the output is a git
checkout, the result's manifest_ids gives the manifest-id of each created or
modified object, so a caller can wait for the cluster to hold that content.
manifest_builder.get_version() returns the version string of the running
manifest-builder, for calling code that wants to log which version it used:
from manifest_builder import get_version
logger.info(f"Using manifest-builder {get_version()}")
Targets and sections
A configuration directory comes in one of two layouts, told apart by the
version field of its top-level config.toml.
By default, or with version = 1, config.toml declares the config blocks
directly.
With version = 2, config.toml declares targets instead. A target names
the sections it is built from, and carries the variables those sections are
rendered with. This lets one configuration directory describe several
deployments of the same sections:
version = 2
[[target]]
name = "platform-dev"
sections = ["base", "platform"]
[target.vars]
cluster_name = "platform-dev"
vanity_domain = "portswigger.com"
[[target]]
name = "platform-prod"
sections = ["base", "platform"]
[target.vars]
cluster_name = "platform-prod"
vanity_domain = "portswigger.net"
A section is a subdirectory of the configuration directory holding a
section.toml of blocks — the same content a version = 1 config.toml would
hold:
conf/
├── config.toml # targets only
├── base/
│ ├── section.toml # [[helm]], [[simple]], … blocks
│ └── argocd/
│ └── values.yaml
├── platform/
│ ├── section.toml
│ └── idcat/
│ └── idcat.toml
├── images.toml
├── releases.yaml
├── owners/
└── plugins/
Which target to generate is selected with --target, and is required for a
version = 2 directory:
manifest-builder --config-dir conf --output-dir output --target platform-dev
From Python it is the target argument of generate:
generate(Path("conf"), Path("output"), target="platform-dev")
Notes:
- A section's blocks are read from that section's own
section.toml, so the paths they reference resolve inside the section directory. In the layout above,base/section.tomlrefers to its values file asargocd/values.yaml. Two sections can therefore use the same relative path without colliding. - A section file may also be named
config.tomlormanifest-builder.toml, so moving an existing top-level config file into a section directory works unchanged.section.tomlwins if more than one is present. - A section may add a
[variables]table of its own. It is merged with the target'svars, as are variables from--vars-fromandgenerate(vars=...). A variable defined by more than one of these is an error rather than one silently winning. images.toml,releases.yaml,owners/, andplugins/stay at the top of the configuration directory and are shared by every target.- Only the sections the selected target names are loaded.
Asking which targets a configuration declares
A caller that generates a configuration into several places has somewhere to put
each target, and needs to know which targets there are. declared_targets is
that question:
from manifest_builder import declared_targets
declared_targets(Path("conf")) # ("platform-dev", "platform-prod")
It returns the names in the order the file declares them, or None for a
configuration directory that declares config blocks directly — which has no
targets to choose between, and is generated with variables instead — and for one
holding no config file at all, which generate() reports on better than a
version check would. A malformed target fails here the same way generating it
would.
Ask rather than reading config.toml: a caller that parses the file itself has
to keep its own idea of what a target is, and can come to a different conclusion
about a configuration than generate() does.
Image template variables
Shared container image definitions can be placed in images.toml in the
configuration directory:
[git]
repo = "alpine/git"
version = "2.47.2"
[static-web-server]
repo = "ghcr.io/static-web-server/static-web-server"
version = "2.36.1"
Each entry is made available to Mustache templates as both the full image reference and the version. Dashes in image names are converted to underscores:
{{git_image}}renders asalpine/git:2.47.2{{git_version}}renders as2.47.2{{static_web_server_image}}renders asghcr.io/static-web-server/static-web-server:2.36.1{{static_web_server_version}}renders as2.36.1
Config block plugins
[[copy]] is the one config block shipped here, in a module under
manifest_builder/blocks/, discovered at startup. Every other block belongs to
the configuration directory that uses it, dropped into a plugins/
subdirectory:
conf/
├── config.toml
└── plugins/
├── helm.py # defines HelmBlock
├── simple.py # defines SimpleBlock
├── public_repo.py # defines PublicRepoBlock
├── templates/
│ ├── simple/ # namespaced per block
│ └── public_repo/
│ └── repository.yaml
└── tests/
└── test_public_repo.py
Blocks live with their configuration because that is where they change: a
deployment's idea of what a [[simple]] app needs is the deployment's business,
and two configuration directories are free to disagree. This package keeps the
toolkit they are built from — chart pulling, releases.yaml parsing,
Kubernetes naming, ConfigMap construction, YAML output routing — so a block
stays small.
Every module in plugins/ is imported, and any concrete ConfigBlock
subclass it defines is registered under the top-level TOML key its
top_level_config_name() returns. A plugin block is used exactly like the
built-in one:
[[public-repo]]
name = "idcat"
A plugin module implements the same interface as a built-in block:
from manifest_builder.blocks import ConfigBlock, GenerationContext
from manifest_builder.output import write_documents
class PublicRepoBlock(ConfigBlock[PublicRepoConfig]):
def top_level_config_name(self) -> str:
return "public-repo"
...
Notes:
- Modules whose names start with
_or.are skipped, as are test modules (test_*.py,*_test.py,conftest.py) andtests/directories, so a plugin directory can keep its own tests beside the code. - Plugins are imported under the
manifest_builder_pluginspackage rather than ontosys.path, so a plugin namedjson.pycannot shadow an installed module. Sibling modules are reachable with a relative import (from .helpers import ...). - Blocks are registered in a stable order, sorted by their top-level key, so a run does not depend on filesystem order. Two blocks claiming the same key is an error.
- Bundled templates should be resolved relative to the plugin module, for
example
Path(__file__).parent / "templates" / "public_repo".
Plugin modules are imported from the configuration directory, so that directory is trusted to the same degree as the manifest-builder installation itself.
Plugins from outside the configuration directory
Configuration is sometimes read from a repository that does not carry the
plugins needed to parse it. generate() then takes a second plugins directory,
used in addition to conf/plugins:
from manifest_builder import ExternalPlugins, generate
generate(
Path("conf"),
Path("output"),
plugins=ExternalPlugins(
path=Path("plugins-checkout/plugins"),
source="https://example.com/plugins.git@def456",
),
)
path is the plugins directory itself, resolved relative to repo_root if it
is not absolute, and must exist. source records where those plugins came
from, and is written as a Plugins from: line in a commit created with
create_commit=True. A module name offered by both directories is an error
rather than one silently shadowing the other.
On the command line the same directory is given with --extra-plugins, and its
source is recorded as provided on command line:
manifest-builder --config-dir conf --extra-plugins plugins-checkout/plugins
Plugins in a long-running process
A process that calls generate() repeatedly may be pointed at a configuration
directory that has been checked out again between calls, possibly at the same
path. Every call re-reads the plugins rather than reusing what it imported
before, so no cache invalidation is needed from the caller:
- Plugin modules are dropped from
sys.modulesand imported afresh on each call, so replaced module source takes effect. - Import-system directory caches are invalidated, so plugins added or removed since the last call are seen.
- No
__pycache__is written for plugin modules. This keeps the configuration checkout clean, and avoids the one case where a re-import could still pick up stale code, since bytecode validity is judged on source mtime truncated to the second plus file size. - Templates should be read at generation time rather than cached at import, as in the example above; combined with the module reload, they then always come from the current checkout.
Plugin loading is serialized with a lock, so concurrent generation from two
different configuration directories cannot interleave one load with another.
manifest_builder.discovery.forget_plugin_modules() is available for a caller
that wants to release imported plugin modules itself, but calling it is not
required.
Externally-owned output roots
When the output repository is shared with other services or pipelines that make their own commits, manifest-builder can be told which top-level output directories it does not own. Files in those directories are left alone during cleanup, and generation fails fast if any output would land in one of them.
To declare ownership, add an owners/ directory to your config directory and
drop one or more TOML files into it. Each file may set:
# A single output root owned by another pipeline:
owned = "team-a"
# Or a list of output roots:
owned = ["cluster", "monitoring", "logging"]
Entries from all owners/*.toml files are merged into a single set of
externally-owned output roots.
License
MIT
Metadata
Release files for manifest-builder 0.9.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| manifest_builder-0.9.1.tar.gz | 56.2 kB | Without provenance | Details for manifest_builder-0.9.1.tar.gz |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_x86_64.whl | CPython 3.14 | abi3 | Linux glibc 2.28+ x86-64 | Details for manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_x86_64.whl |
| manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_aarch64.whl | CPython 3.14 | abi3 | Linux glibc 2.28+ ARM64 | Details for manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_aarch64.whl |
| manifest_builder-0.9.1-cp314-abi3-macosx_11_0_arm64.whl | CPython 3.14 | abi3 | macOS 11.0+ ARM64 | Details for manifest_builder-0.9.1-cp314-abi3-macosx_11_0_arm64.whl |
Total release size: 2.7 MB
Release files / manifest_builder-0.9.1.tar.gz
| Download URL | manifest_builder-0.9.1.tar.gz |
|---|---|
| Size | 56.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4fce1a2ea09c646251439787722a02a81eaf8540dc352707f7a8e9bd53fd86a6
|
|
BLAKE2b-256 checksum How to use checksums |
94f91b658d16322871f4d8c9633aaa404f8d4ead2202c4ffa0ce7324cbced9e9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_x86_64.whl
| Download URL | manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 939.0 kB |
| Tags | CPython 3.14 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
69e2f219a171b74974564d2fce89eb16f92d7bd165d85c72f9acb78972e84930
|
|
BLAKE2b-256 checksum How to use checksums |
b54ae52c61f782ca629228b29f64baeb0347e08290624635575b1e547dc5880a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_aarch64.whl
| Download URL | manifest_builder-0.9.1-cp314-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 909.3 kB |
| Tags | CPython 3.14 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
d4dff0ab856d5be6569a4975fa041e64157599da5e391bdc619b3a56884ab684
|
|
BLAKE2b-256 checksum How to use checksums |
1b53b30ec54b5b47c67632c9cf8da60d63e1f206d255b0e56193b2c7691a8cd4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / manifest_builder-0.9.1-cp314-abi3-macosx_11_0_arm64.whl
| Download URL | manifest_builder-0.9.1-cp314-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 831.4 kB |
| Tags | CPython 3.14 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
3b0b46fe98cb5ed01a52bf291365af59069f0a059bb49aa9a95724253e1f38ad
|
|
BLAKE2b-256 checksum How to use checksums |
dff82b33344bfad96afaf467240ba7f7676dd5a3be268dab8d2bc4874be27216
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for manifest-builder 0.9.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.