NoPayloadDB Python client

Typed Python client for the NoPayloadDB v5.1 metadata API and payload-file service.

The normal use case is one experiment backed by one NoPayloadDB server. An optional second independently managed experiment source can also be queried when both experiments use common GlobalTag names and IOV coordinates.

The normal analysis workflow is:

1. select a common GlobalTag and analysis IOV;
2. query the experiment's payload metadata;
3. download and verify the selected payload files;
4. work with a payload_type -> local Path mapping.

GlobalTags correlate independently managed subsystem conditions at one shared IOV coordinate system. In HEP-style workflows this is commonly a two-level key:

• major_iov for long intervals (for example ATLAS RunNumber, typically hours);
• minor_iov for short intervals inside the major interval (for example LumiBlockNumber, typically about one minute).

In production there are usually many IOV slices per payload type. Querying the same GlobalTag at different (major_iov, minor_iov) points selects the revision valid at each point.

Downloaded files use a shared checksum-addressed cache. A valid cached file is used before the remote file service.

Installation

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install nopayloaddb-client

For local development in this repository:

python -m pip install --editable ".[test]"

Local test stack (optional quickstart)

For a self-contained demo, deploy two local NoPayloadDB instances (requires a Kubernetes cluster plus kubectl and helm):

make deploy-test-server

For local tutorials, use the Local tutorial value entries from the Configuration table below.

Configuration

Recommended pattern: set environment variables and let the client read them.

from nopayloaddb_client import NoPayloadDBClient

client = NoPayloadDBClient.from_env()

Configuration mapping:

Token validation is entirely server-side; the client only forwards the bearer token.

CLI

The package installs a simple CLI binary:

nopayloaddb --help

Supported commands:

nopayloaddb list-global-tags
nopayloaddb get-global-tag <name> [--light]
nopayloaddb get-global-tag-payload-lists <global_tag>
nopayloaddb query-payloads <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
nopayloaddb get-payload-files <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
nopayloaddb-populate-test-data [--source-profile single|experiment-a|experiment-b]

Required configuration:

• all commands require NOPAYLOADDB_URL (or --base-url);
• get-payload-files also requires NOPAYLOADDB_FILES_URL (or --files-url).
• nopayloaddb-populate-test-data also needs NOPAYLOADDB_TOKEN for uploads.

If required values are missing, the CLI exits with an explicit configuration error.

nopayloaddb-populate-test-data is a stable bootstrap helper for creating tutorial content on an already-running NoPayloadDB server. Its supported public arguments are:

nopayloaddb-populate-test-data \
  [--base-url URL] \
  [--files-url URL] \
  [--global-tag NAME] \
  [--author NAME] \
  [--source-profile single|experiment-a|experiment-b]

Single-experiment analysis

Query a GlobalTag at IOV (0, 50) and retrieve its files:

payload_metadata = client.query_payloads("AnalysisGT", 0, 50)
payload_files = client.get_payload_files("AnalysisGT", 0, 50)

beam = payload_files["BeamCalibration"]
geometry = payload_files["DetectorGeometry"]

Scan multiple IOV points in one run-like interval:

for minor_iov in (0, 50, 100, 150, 199):
    selected = client.query_payloads("AnalysisGT", 0, minor_iov)
    print(minor_iov, [item["payload_url"] for item in selected])

The returned mapping is directly keyed by payload type. Files are streamed to a temporary path, checksum-verified and moved atomically into the cache. Repeating the request returns the cached path without contacting the file service.

Optional multi-experiment analysis

Use MultiRIReadClient when the analysis needs compatible data from two experiments. RI 1/RI 2 configuration surfaces are mapped in the table above:

from nopayloaddb_client import MultiRIReadClient

client = MultiRIReadClient.from_env()

Query both experiments at IOV (0, 50):

payload_metadata = client.query_payloads("CommonAnalysisGT", 0, 50)

This returns a MultiRIReadResponse:

responses = payload_metadata.by_source()
experiment_a = responses["experiment-a"]
experiment_b = responses["experiment-b"]

print(experiment_a.server_url)
print(experiment_a.unwrap())
print(experiment_b.unwrap())

Each SourceResponse contains:

• source_name;
• server_url;
• value, when the request succeeded;
• error, when that source failed;
• unwrap(), which returns the value or raises its source-specific error.

One unavailable experiment does not discard successful results from the other. Responses are not merged because payload types with the same name can have different experiment-specific meaning.

Resolve every selected payload:

payload_files = client.get_payload_files("CommonAnalysisGT", 0, 50)

The result is grouped by source:

responses = payload_files.by_source()
beam = responses["experiment-a"].unwrap()["BeamCalibration"]
geometry = responses["experiment-b"].unwrap()["DetectorGeometry"]

Retrieve one use case without assuming which experiment owns it:

alignment_files = client.get_payload_files(
    "CommonAnalysisGT",
    0,
    50,
    payload_type="DetectorAlignment",
)

Sources that do not define the requested payload type return an empty mapping.

Federated CLI:

multiRI-nopayloaddb list-global-tags
multiRI-nopayloaddb get-global-tag <name> [--light]
multiRI-nopayloaddb get-global-tag-payload-lists <global_tag>
multiRI-nopayloaddb query-payloads <global_tag> <major_iov> <minor_iov> [--payload-type <name>]
multiRI-nopayloaddb get-payload-files <global_tag> <major_iov> <minor_iov> [--payload-type <name>]

To inspect metadata before downloading:

payloads = client.query_payloads("CommonAnalysisGT", 0, 50)

for source_name, response in payloads.by_source().items():
    for payload in response.unwrap():
        print(source_name, payload["payload_type_name"], payload["payload_url"])

The two experiments share the checksum-addressed cache.

Expert publication

Writes are always single-instance. NoPayloadDBClient never mirrors POST, PUT, upload or DELETE requests across experiments.

An expert publishing to experiment B should construct a client with experiment B as its first and only server:

export NOPAYLOADDB_URL=https://conditions-b.example
export NOPAYLOADDB_FILES_URL=https://files-b.example
export NOPAYLOADDB_TOKEN=your-token

experiment_b_writer = NoPayloadDBClient()

Upload opaque subsystem formats independently:

uploaded = experiment_b_writer.upload_payload(
    "detector-alignment-v7.xml",
    payload_url="alignment/detector-alignment-v7.xml",
)

experiment_b_writer.create_and_attach_payload_iov(
    payload_list["name"],
    payload_url=uploaded.payload_url,
    checksum=uploaded.checksum,
    major_iov=0,
    minor_iov=100,
    major_iov_end=0,
    minor_iov_end=200,
)

Uploads use authenticated PUT /upload/<payload_url>. Downloads use GET /dbstore/<payload_url>.

Local tutorials

This section assumes the local stack from Local test stack (optional quickstart) is already running and your environment variables are already exported.

Single-server tutorial

The primary tutorial uses only experiment A and preserves the standard single-server return types.

make populate-test-data
make tutorial

It creates ClientTutorialGT with JSON, ROOT, XML, and DataQualitySummary payloads and walks through metadata lookup, IOV selection, file resolution, cache reuse, and payload-type filtering.

Multi-server tutorial

The additional tutorial seeds a common GlobalTag on both servers:

make populate-multi-source-test-data
make multi-source-tutorial

The servers receive the common GlobalTag MultiSourceTutorialGT, but different experiment-owned payloads:

The tutorial demonstrates per-source metadata/results, source-grouped local file resolution, cache reuse, and querying one payload type without assuming which experiment owns it.

DataQualityTool example

The helper scripts/data_quality_tool.py demonstrates event-loop usage for the DataQualitySummary payload returned from ClientTutorialGT.

python scripts/data_quality_tool.py

It loads DataQualitySummary from the tutorial GlobalTag and calls assess_data_quality(event) for 20 generated events.

Tests

make test
make test-integration

The unit suite covers the original single-source return types and multi-source attribution, failure isolation and file routing. The integration suite performs a single-server query/download and then queries and downloads distinct payloads from both servers using one common temporary GlobalTag.

Remove both test environments and their persistent data:

make delete-test-server

Package build

python -m pip install build
python -m build

Release publishing

Production PyPI publishing is tag-driven:

1. push a v* tag;
2. Release checks runs tests, type-check, license/audit checks, version-tag verification, and build;
3. on success, Publish to PyPI publishes the checked artifacts.

TestPyPI publishing remains manual via the workflow Publish to TestPyPI (manual) and can target any ref.

Third-party license inventory

The tracked THIRD_PARTY_LICENSES.md file is the runtime dependency license inventory and is verified during publish CI.

make third-party-licenses