data-product-forge-custom-scaffold

The custom-scaffold engine for data-product-forge. Install it alongside the CLI, plug in any scaffold bundle, and generate a complete project from your contract.

pip install data-product-forge data-product-forge-custom-scaffold

This pulls data-product-forge-sdk transitively (import path: fluid_sdk). Requires Python >=3.10.

Then in any fluid contract:

extensions:
  customScaffold:
    libraries:
      - id: ci
        source: { kind: git, url: "https://github.com/example/my-bundle", ref: "v1.0" }
    patterns:
      - use: ci:basic

fluid custom-scaffold
# ✓ 3 files written, 0 failed
#   README.md
#   .gitlab-ci.yml
#   docs/runbook.md

Deterministic, idempotent, atomic.

What this engine does

This is the runtime for data-product-forge's custom-scaffold feature:

1. Discovers itself with the CLI via Python entry-points (just pip install).
2. Resolves bundles from path (local), git (clone into cache), or entrypoint (installed Python plugin).
3. Validates the extensions.customScaffold block in your contract.
4. Renders each pattern through Jinja2.
5. Writes the generated files atomically with path-traversal protection.
6. Copies the bundle's static/ directory verbatim alongside rendered templates (for binary fixtures, sample data, pre-rendered files).

What's in the contract

extensions:
  customScaffold:
    libraries:
      - id: ci                                          # local alias
        source:
          kind: git                                     # path | git
          url: "https://github.com/example/ci-bundle"
          ref: "v1.0"
          subdir: "scaffold"                            # optional
          auth: { secret_ref: GITHUB_TOKEN }            # optional
    patterns:
      - use: ci:gitlab-ci                               # <library-id>:<pattern-name>
        variables:
          parentCiTemplateRef: "my-org/ci-templates@main"

Source kinds

Auth via auth.secret_ref (env-var name, never persisted).

Note: explicit npm and pypi source kinds (auto-fetch from registry) are not in v0. For pip-installable Python plugins, use kind: entrypoint after pip install-ing the plugin package. Git covers most YAML/Jinja distribution today. File an issue if you need direct-from-registry npm/pypi fetch.

Bundle authoring

Two paths — pick whichever fits:

A. YAML + Jinja bundle (no Python)

Drop a directory like this:

my-bundle/
├── fluid-scaffold.yaml         ← manifest
├── templates/
│   ├── README.md.j2            ← Jinja templates
│   └── .gitlab-ci.yml.j2
└── static/                      ← optional — copied verbatim
    └── docs/
        └── runbook.md

The engine's built-in TemplatedCustomScaffold reads the manifest, renders the templates, and copies static/ verbatim.

→ See docs/walkthrough/build-a-yaml-bundle.md for the step-by-step.

B. Python plugin bundle

Subclass fluid_sdk.CustomScaffold directly. Full programmatic control.

→ See the SDK walkthrough.

CLI surface

fluid custom-scaffold [OPTIONS]

  -c, --contract PATH    Path to contract.fluid.yaml (default: ./contract.fluid.yaml)
  -o, --output PATH      Output root (default: cwd)
      --dry-run          Plan only — print the file list, write nothing.
      --pattern USE      Restrict to specific patterns (repeatable)
      --lib ID           Restrict to specific library ids (repeatable)
      --pin              Reproducible re-run: resolve git sources to the commit
                         recorded in fluid-scaffold.lock (not the floating ref).
      --update           Update an existing output to the evolved template —
                         3-way merge that preserves your edits (see below).
      --target REF       With --update: the git ref/commit to update to.
      --json             Emit JSON instead of human output

Exit codes:

Reproducibility & updates

Every successful generation writes a fluid-scaffold.lock at the output root (commit it alongside the generated files). It records, per resolved library, the exact commit it resolved to, plus the patterns and variables used — the same model as copier's .copier-answers.yml.

# Generate — writes the output + fluid-scaffold.lock
fluid custom-scaffold -c contract.fluid.yaml -o ./my-project

# Reproducible re-run — resolve git sources to the LOCKED commit, not the
# moving ref. (npm-ci / poetry --frozen semantics; non-git sources can't be
# reproducibly pinned and the engine says so.)
fluid custom-scaffold -c contract.fluid.yaml -o ./my-project --pin

# Update — the template evolved? Re-render at the new ref and 3-way-merge it
# onto your working tree, preserving your edits. Non-overlapping changes merge
# cleanly; overlaps get Git-style conflict markers (exit 4).
fluid custom-scaffold -c contract.fluid.yaml -o ./my-project --update

--update renders the template at the locked commit (the baseline you started from) and at the new ref, then merges with git merge-file (base = old render, ours = your file, theirs = new render). On success the lock advances. Full walkthrough: docs/walkthrough/reproducible-updates.md.

Documentation

License

Apache-2.0.