dde-cargo

Stimela cabs for DDFacet/Killms on both arm64 and x86-64 systems.

DDFacet is a direction-dependent radio-interferometric imager: it grids visibilities and deconvolves wide fields in facets, applying per-direction Jones terms and primary beams. killMS is its companion calibration engine, solving per-antenna, per-direction complex gains against a sky model. Together they form the imaging and direction-dependent calibration halves of the DDE loop. Release history is in CHANGELOG.md.

Cabs and images

Full parameter reference: docs/cabs.md.

All images are published for amd64 and arm64.

Install

pip install dde-cargo

Requires Python >= 3.10,< 3.15 (the same window as skarabina-cargo) and installs stimela>=2.1.4 — every cab runs in its container, so the host needs nothing else.

Usage

In a stimela 2 recipe, load the cabs from the installed package with configuratt's package-include syntax. The cultcargo include must come first: ddfacet.yml carries an image-only override for cult-cargo's bdsf.catalog cab (the upstream image is amd64-only; these cabs redirect it to the multi-arch ddfacet image), and stimela builds every cab at load time, so bdsf.yml must already be loaded when ddfacet.yml merges in:

_include:
  - (cultcargo):
      - bdsf.yml
  - (dde_cargo):
      - ddfacet.yml
      - pick-reference-ant.yml
      - ddfacet-memory-plan.yml
      - killms-memory-plan.yml

then reference them from steps:

steps:
  ddfacet-target:
      cab: ddfacet
      params:
          Data.MS: mydata.ms
          Output.Name: target0
  killms-target:
      cab: killms
      params:
          MSName: mydata.ms
          InCol: CORRECTED_DATA

Only ddfacet.yml is essential; the memory-plan files are optional extras (wire them ahead of a step as shown in their header comments — e.g. ddfacet-memory-plan suggests Parallel.NCPU, Data.ChunkHours and Selection.ChanStep for the imaging step that follows). They are self-contained: the calibrated model and its tessellation code are inlined in the cab's python-code command, so they need no Lib/ModelFile input and no meerkat_imaging checkout (tests/test_sync.py keeps it that way).

Notes carried over from the upstream cab comments:

• The autofacet cab imports auto_facet_tool/auto_facet.py from the working directory, and pick-reference-ant defaults its lib input to tools/pick_reference_ant.py, so both need a checkout of meerkat_imaging beside the recipe (stimela binds the working directory into the container).
• pick-reference-ant.yml must be included after ddfacet.yml: its image resolves vars.tim-molteno.images from ddfacet.yml's vars block, so it always runs on the same DDFacet pin as the imaging cab.
• The memory-plan cabs pin their image "to whatever the recipe uses": a recipe with its own image var should override image: on the cab instead of editing the package.
• The DDFacet cab wraps every parset namespace (Data, Image, Facets, Deconv, ...) as dashed CLI options (--Data-MS, --Image-NPix, ...); the killMS cab passes its flat parameter set through kMS.py.

Publishing

Trusted publishing (OIDC) — there is no API token. The GitHub Actions workflow .github/workflows/publish_pypi.yaml fires on a vX.Y.Z tag, builds with uv, and publishes from the pypi environment.

Automated release

invoke release drives the tag, PyPI publish and GitHub release end to end. See Release automation under Development for the full task reference and prerequisites.

Manual release

1. Register the pending publisher on PyPI (once): pypi.org/manage/account/publishing with owner tmolteno, repository dde-cargo, workflow publish_pypi.yaml, environment pypi. The workflow filename must match exactly.
2. Bump version in pyproject.toml.
3. git commit -m "..." && git tag -a vX.Y.Z -m "..." && git push origin main --follow-tags

The workflow run appears under the repo's Actions with the pypi environment; the publish is live at pypi.org/project/dde-cargo.

Development

uv sync --all-groups
uv run pytest          # schema + generated-docs contracts
uv build               # wheel + sdist into dist/
invoke --list          # tasks: version, changelog, docs, release
invoke docs            # regenerate docs/cabs.md
invoke docs --check    # verify docs/cabs.md is current

An end-to-end smoke check (loads the cabs through stimela's real spec loader, as a recipe would):

uv run stimela run /tmp/dde-cargo-smoke.yml dde-cargo-smoke -d

with /tmp/dde-cargo-smoke.yml any recipe that _includes the package as shown above and runs one or two of the cabs.

Where the cabs are edited: invoke sync

The cab YAMLs and schemas in dde_cargo/ are not edited in this repository. Their source is meerkat_imaging's cabs/ directory (where the pipeline runs and tests them):

invoke sync                    # copy the six files from ../meerkat_imaging (MEERKAT_REPO to override)
invoke sync --check            # exit 1 if dde_cargo/ differs

sync refuses when those files have uncommitted changes in meerkat_imaging (or its generated cabs are stale), copies them, and records the meerkat_imaging commit and the files' hashes in dde_cargo/meerkat-sync.json (tests/test_sync.py fails if a packaged file is edited by hand). release runs sync first.

Release automation

tasks.py is an invoke + plumbum script that automates a release from the sibling ../DDFacet, ../killMS and ../meerkat_imaging repositories. It syncs the cabs from meerkat_imaging, resolves the latest upstream release tags, verifies that the synced cabs pin exactly those tags (it refuses, with a message, instead of rewriting a pin), and generates the changelog from the upstream commits between the old and new tags plus meerkat_imaging's commits to the cab files since the previously synced commit.

Prerequisites:

• the dev dependency group — uv sync --all-groups installs invoke and plumbum;
• uv and an authenticated gh (the PyPI workflow uses OIDC, so no API token is involved);
• ../DDFacet and ../killMS checkouts containing their vX.Y.Z release tags, and a ../meerkat_imaging checkout with the cabs committed (and their pins at those tags) — override the paths with DDFACET_REPO, KILLMS_REPO and MEERKAT_REPO;
• a clean main branch, with the package version not yet tagged or published.

Available tasks:

invoke --list                  # show the available release tasks
invoke version                 # package version and pinned/latest image tags
invoke changelog               # print the next release's changelog section
invoke changelog --allow-same  # ... even when no image tag has moved
invoke docs                    # regenerate docs/cabs.md
invoke docs --check            # verify docs/cabs.md is current
invoke sync                    # copy the cabs from meerkat_imaging
invoke release --dry-run       # show the complete plan; change nothing
invoke release --part patch    # the real release
invoke release --part minor    # ... when a cab interface changed (0.1.1 -> 0.2.0)

Use --part minor whenever a cab's inputs or outputs changed (or an input was removed, as the memory-plan cabs' Lib/ModelFile were for 0.2.0); patch (the default) for pin bumps and internal fixes. --allow-same releases when nothing changed (no new pins, cabs or commits).

The release order

Normally nobody runs this by hand: meerkat_imaging's invoke release does, after DDFacet and killMS are tagged and their images are live:

1. meerkat_imaging commits its updated cab pins (and regenerated memory-plan cabs);
2. it runs invoke release --part <patch|minor> --yes here (path: DDE_CARGO_REPO);
3. this release syncs the cabs, checks the pins, publishes to PyPI and creates the GitHub release, and exits non-zero unless the version is live on PyPI;
4. meerkat_imaging bumps its dde-cargo==X.Y.Z pin and makes one pins: ddfacet vA, killms vB, dde-cargo vC commit.

invoke version reports the package version, the DDFacet/killMS tags currently pinned by the cabs, and the latest tags in the upstream checkouts.

invoke changelog prints the section that would be inserted into CHANGELOG.md. It groups the upstream commits per release tag (for example v0.16.2 … v0.16.6) and notes killMS releases that are rebuilds on a new DDFacet image.

invoke docs regenerates the cab reference from the packaged YAMLs and docs/cab_notes.yml; invoke docs --check reports whether the checked-in docs/cabs.md is current.

invoke release performs the complete chain:

1. generates and inserts the changelog section, keeping any [Unreleased] notes;
2. bumps version in pyproject.toml, updates the DDFacet/killMS pins in the cab YAMLs and the image table above, and regenerates docs/cabs.md;
3. refreshes uv.lock, runs the test suite and uv build;
4. commits the changes, creates an annotated vX.Y.Z tag, and pushes main and the tag;
5. waits for the tag's publish_pypi.yaml workflow to succeed and verifies that the version is live on PyPI;
6. creates the GitHub release with the generated notes and the built wheel and sdist attached.

Options:

• --part patch|minor|major — which part of the package version to bump (default patch);
• --ddfacet-tag / --killms-tag — override the upstream target tags (default: the latest release tag in each checkout);
• --allow-same — release packaging-only changes when both images are already at the pinned tags;
• --fetch / --no-fetch — fetch upstream tags first (default: fetch, falling back to the local tags with a warning if the remotes are unreachable);
• --yes — skip the confirmation prompt;
• --dry-run — print the plan and generated changelog without changing, committing or publishing anything.

The script refuses to run on a dirty tree, from a branch other than main, or when the target tag or PyPI version already exists. It never force-pushes.