beampipe-palette 0.5.2

beampipe-palette provides the two project-neutral native DALiuGE applications that connect a portable graph to Beampipe Core: one injects Core's immutable execution manifest at the graph boundary, and one verifies durable outputs at the terminal boundary.

What it does

• Manifest ingress: beampipe-ingest receives inline JSON injected by Core, canonicalizes it, and emits it once through a path-backed FileDROP.

• Durable publication: beampipe-publish independently discovers every required output, publishes each object create-only, reads it back, and verifies its SHA-256.

• Trusted completion evidence: the publisher emits beampipe-output-inventory/v1 to DALiuGE and writes the byte-identical receipt to an attempt-scoped session handoff.

• Callback-free remote runs: Core retrieves the receipt over its existing SSH/SFTP control connection after Slurm completion. Graphs contain no Core URL, bearer token, storage credential, or project-specific topology.

Native DALiuGE components

beampipe-ingest and beampipe-publish are native BarrierAppDROP applications—not PyFunc snippets. Their graph-facing contracts stay deliberately small; Core and the selected deployment profile supply execution details outside the logical graph.

Manifest entering beampipe-ingest

{
  "inputs": {},
  "sources": [{
    "source_identifier": "source-1",
    "groups": [{
      "group_key": "group-1",
      "records": [{
        "record_id": "record-1",
        "access_url": "https://data.example/record-1"
      }]
    }]
  }],
  "_beampipe": {
    "selection": [{
      "source_identifier": "source-1",
      "group_key": "group-1",
      "record_ids": ["record-1"]
    }]
  }
}

Inventory emitted by beampipe-publish

{
  "schema": "beampipe-output-inventory/v1",
  "execution_id":
    "607fd31f-db2a-4978-a64f-ef8ce412483a",
  "execution_attempt": 0,
  "patterns": ["**/result.bin"],
  "pattern_counts": {"**/result.bin": 1},
  "products": [{
    "path": "products/result.bin",
    "bytes": 1048576,
    "sha256":
      "a40f3e3d82fd741d64da75530ccb205588b8fbb5a5fbd7d52f18a77ef85d5e60"
  }],
  "inventory_sha256":
    "5efa576ea4393a413c121862a75efcb5086cffa4b651ec77178dfa1c8caa07ce",
  "durable_destination_uri":
    "file:///srv/products/executions/607fd31f-db2a-4978-a64f-ef8ce412483a/attempt-0",
  "publication": {
    "acknowledged": true,
    "publisher": "beampipe-publish",
    "receipt_id":
      "beampipe-publish/v1:5efa576ea4393a413c121862a75efcb5086cffa4b651ec77178dfa1c8caa07ce",
    "published_at": "2026-08-27T10:32:16Z"
  }
}

The examples show the stable project-neutral fields. Real manifests may add project-owned record metadata, while real inventories contain one entry for every selected product and the exact hashes observed after durable read-back.

Where the nodes go

• Put beampipe-ingest at the graph boundary. Core patches its manifest_path setting with the immutable execution manifest. Connect its single manifest_bytes FileDROP to the first project application that needs those bytes.

• Put beampipe-publish after the project-owned completion barrier. The barrier must become complete only after every required output is closed and stable. Connect the publisher's inventory output to one terminal FileDROP.

• Do not draw a callback edge to Core. On remote Slurm deployments the publisher writes the canonical handoff inside the session; Core pulls and verifies it after the scheduler reports completion.

First-time setup

Python 3.10 or newer is required. Install the validated public release into the same Python environment used by every DALiuGE execution node:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install \
  https://github.com/jbwod/beampipe-palette/releases/download/v0.5.2/beampipe_palette-0.5.2-py3-none-any.whl
beampipe-publish --version

The released wheel is normally installed into an existing DALiuGE runtime. A source checkout can create a self-contained DALiuGE 6.x environment with python -m pip install -e '.[daliuge]'.

Install the bounded S3 transport only on runtimes that select an S3 destination:

python -m pip install \
  'beampipe-palette[s3] @ https://github.com/jbwod/beampipe-palette/archive/refs/tags/v0.5.2.tar.gz'

Download or load the checked-in EAGLE palette:

https://raw.githubusercontent.com/jbwod/beampipe-palette/v0.5.2/daliuge/palettes/beampipe.palette

EAGLE graph contract

Load daliuge/palettes/beampipe.palette in EAGLE. The wheel also installs it at share/beampipe-palette/daliuge/palettes/beampipe.palette under the active Python prefix.

Component	Field	EAGLE usage	Contract
beampipe-ingest	manifest_path	NoPort	inline JSON from Core or an absolute manifest path
beampipe-ingest	manifest_bytes	OutputPort	one path-backed FileDROP, pickle encoding
beampipe-publish	completion	InputPort	one project-owned completion FileDROP
beampipe-publish	inventory	OutputPort	one path-backed canonical inventory FileDROP
beampipe-publish	expected_patterns_json	NoPort	project-owned JSON string array of required output globs

The Python classes are:

beampipe_palette.apps.BeampipeIngestApp
beampipe_palette.apps.BeampipePublishApp

The ingest output intentionally uses DALiuGE's pickle encoding. It wraps the canonical bytes with DALiuGE's own serialized-data envelope before the single pickle port write. This is the common named-port contract that returns exact bytes on both DALiuGE 6.6 and 6.7. The publisher uses path-backed FileDROPs directly and performs no second DROP write.

Projects must set expected_patterns_json explicitly. Matching is bounded and Core-compatible: * and ? stay within one path component, while ** is allowed only as a complete component. Character classes, brace expansion, embedded double-stars, traversal, backslashes, controls, and duplicates are rejected. The output tree is scanned once regardless of pattern count.

Runtime contract

Runtime details are environment variables, never graph fields:

Environment variable	Meaning
BEAMPIPE_OUTPUT_ROOT	source directory containing completed outputs
BEAMPIPE_OUTPUT_DESTINATION_URI	base file:// or s3:// destination
BEAMPIPE_EXECUTION_ID	canonical execution UUID
BEAMPIPE_EXECUTION_ATTEMPT	zero-based locked Core retry_count
BEAMPIPE_OUTPUT_INVENTORY_HANDOFF_PATH	required attempt-scoped receipt path prepared by the session transport

The durable destination is always namespaced as:

<base>/executions/<execution_uuid>/attempt-<retry_count>

The handoff path must be absolute, non-root, outside BEAMPIPE_OUTPUT_ROOT, free of symbolic-link components, and have an existing private parent. No Core URL, Core credential, storage secret, project identifier, or output pattern is embedded in the component.

Publication guarantees

• Create-only writes: files, inventories, and handoffs are never overwritten. An existing byte-identical object is reused; conflicting evidence fails the run.

• Read-back verification: every durable file or S3 object is retrieved in full and hashed before it is acknowledged.

• Exact retries: a retry of the same execution attempt reuses the same canonical inventory bytes and publication timestamp. A later attempt has a separate immutable namespace.

• Fail-closed evidence: missing patterns, empty outputs, symlinks, unsafe paths, oversized receipts, changed files, partial uploads, and mismatched prior evidence all fail publication.

Filesystem destinations use absolute, non-overlapping file:// roots. S3 destinations require beampipe-palette[s3], boto3 1.38 or newer, conditional single-object/multipart creation, and full object read-back. NGAS is not claimed until an adapter can provide authoritative file ID/version assignment and retrieval verification.

Development

The DALiuGE 6.7 Python component guide uses dlg_paletteGen with inline Doxygen metadata. Install the two system tools once on Debian/Ubuntu:

sudo apt-get install doxygen xsltproc

make virtualenv
make lint
make test
make palette-check
make docs
make build

Palette generation reads the EAGLE_START component descriptions in beampipe_palette/apps.py through pinned dlg_paletteGen 0.6.7, then canonicalizes IDs and timestamps so the checked EAGLE palette is deterministic.

The offline suite covers both native applications, canonical JSON, path and port policy, one-pass discovery, filesystem/S3 create-only publication, read-back hashes, exact retries, attempt isolation, session handoff, palette signature drift, and project-neutral defaults. CI additionally runs the native DALiuGE boundary regression against the released 6.6 engine and the pinned 6.7 source contract used by the linked 6.7 developer documentation.

Documentation

Task	Page
Download the current wheel and source archive	v0.5.2 release
Review the publication sequence	Publisher guide
Import the components into EAGLE	Checked-in palette
Contribute and run the local gates	Contributing guide
Review template provenance	Template notes

This repository was created from ICRAR's daliuge-component-template and retains its license.