Skip to main content

AIND Metadata Reader

Read historical AIND metadata, inspect incomplete records, and record explicit corrections while preserving the source. Readers support selected representations from aind-data-schema and biodata-schema without requiring either package at runtime. Python 3.10 and newer are supported.

This library is experimental. Its API may evolve before version 1.0. The examples below run from this checkout using the included historical fixtures.

What the library does

Metadata changes in several ways: fields move, vocabulary grows, and concepts change meaning. A processing input recorded as a file path conveys different information from an input recorded as an asset reference. The reader exposes those distinctions through small views of code, references, and coordinate frames.

Reading preserves the original values and unknown fields. Schema-invalid metadata remains inspectable, and a malformed component does not hide usable siblings. You can separately request a validation report or derive a corrected interpretation. Neither operation silently replaces the source.

flowchart LR
    source["JSON file or mapping"] --> capture["Preserved snapshot"]
    capture --> views["Views, available values, and diagnostics"]
    capture --> validation["Optional validation report"]
    views --> changes["Explicit correction or inference"]
    changes --> derived["Derived snapshot and change manifest"]

You choose the source. The library accepts local files and caller-supplied data; it does not query DocDB, reconcile stores, or decide which record is authoritative. Image loading, registration, and transform application stay with their existing libraries.

Get started with a historical record

Prepare the checkout environment:

uv sync

Read the included processing example from aind-data-schema v1.4.0:

from aind_metadata_reader import ReadStatus, read_json, read_processing

record = read_json("tests/fixtures/history/v1.4.0-processing.json")
processing = read_processing(record)

print(record.schema_version)  # 1.1.4: the core schema version, not the package version
for process in processing.processes:
    print(process.process_type, process.code_version)

The example contains three pipeline steps and two separately recorded analyses. Their views retain the original ordering within each group and their actual source pointers. The reader does not construct a dependency graph from that ordering.

With your own data, replace the fixture path with a processing file or an aggregate such as metadata.nd.json. read_processing selects the processing core from an aggregate; read_acquisition does the same for acquisition metadata.

The main objects have different jobs:

Object What it holds
Snapshot The complete preserved payload, source provenance, and original bytes when supplied.
Document An object selected within a snapshot. Children retain the same root snapshot.
Processing, Process, Acquisition, CoordinateFrame Views of recognized concepts in a historical representation.
ReadValue A requested interpretation, its availability, diagnostics, and raw source field.
Interpretation Original and derived views, applied changes, retained rule decisions, and a manifest.

For data fetched by your own client, use Document(mapping, provenance=Source(...)). A Source can retain an S3 locator or a DocDB host, API version, database, collection, and record ID. Supply known conversion evidence explicitly.

Inspect recorded content directly with record.raw or record.field("/processing_pipeline/pipeline_version"). raw and field values return copies. A field's state distinguishes present, null, and absent. Pointers passed to a document are relative to that document; returned field pointers identify the location in the complete snapshot.

Check what an interpretation can establish

Selecting a process by type can produce several matches. find() returns them all; select() reports whether exactly one matches:

selected = processing.select("Image tile fusing")
if selected.status == ReadStatus.AVAILABLE:
    fusion = selected.value
    print(fusion.document.pointer)  # /processing_pipeline/data_processes/0
    print(fusion.code.version.value)  # 0.1

    inputs = fusion.inputs
    if inputs.status == ReadStatus.AVAILABLE:
        for reference in inputs.value:
            print(reference.kind, reference.location.value)
else:
    print(selected.status, selected.diagnostics)

assert processing.select("Analysis").status == ReadStatus.AMBIGUOUS

Code version and pipeline version are separate concepts. process.code exposes independent version, url, parameters, and inputs results. Input/output references distinguish file, asset, asset_group, and unknown; asset groups retain their members. An asset URL does not establish a file-level input.

A ReadValue gives you four things:

Attribute Meaning
status Whether the requested interpretation is available, and why it may be unavailable.
value The interpreted value when available; otherwise None.
field The actual recorded field, including its raw value, pointer, and snapshot.
diagnostics Structured explanations with codes and source locations.

Availability describes the requested interpretation:

Status What it tells you
available The reader established a value.
absent The source field was not recorded.
null The source explicitly recorded JSON null.
invalid The recorded content does not fit this requested interpretation.
unsupported The reader has no established interpretation for this representation or meaning.
ambiguous Several candidates exist and none has been selected.

invalid describes one attempted interpretation; it does not declare the entire metadata record invalid.

Convenience accessors such as code_version, parameters, processes, and array_axes make common cases shorter. Use the corresponding component results when missing information matters. An empty processes tuple, for example, could mean an empty recorded collection or an unsupported representation: process_entries.status distinguishes those situations.

selected.require(), processing.one(type), and acquisition.one_frame() explicitly assert a requirement and can raise. Ordinary inspection uses status-bearing results.

Incomplete metadata stays inspectable

This illustrative record has a numeric code version and a malformed second entry:

from aind_metadata_reader import Document

incomplete = Document(
    {
        "schema_version": "2.0.0",
        "data_processes": [
            {
                "process_type": "An unfamiliar processing term",
                "name": "example step",
                "code": {"version": 42, "parameters": {"threshold": 12}},
                "output_path": "/results/image.zarr",
            },
            "unfinished entry",
        ],
    }
)
inspected = read_processing(incomplete)
first, second = inspected.process_entries.value
step = first.value

assert step.code.version.status == ReadStatus.INVALID
assert step.code.version.field.value == 42
assert step.code.parameters.value == {"threshold": 12}
assert step.output_location == "/results/image.zarr"
assert second.status == ReadStatus.INVALID
assert second.field.value == "unfinished entry"
assert inspected.raw == incomplete.raw

Unfamiliar vocabulary remains recorded text. An untested schema version produces an unverified_version diagnostic while recognized components remain readable. Schema identity, recorded layout, and known historical boundaries select the reader; each component then reports its own availability. An unfamiliar direction, for example, withholds anatomical orientation while preserving the frame, its raw axes, and other readable frames. This describes what the reader can interpret; validation separately describes whether the record satisfies a schema.

Raw capture requires a finite JSON object. Malformed JSON, duplicate object keys, and nonfinite numbers are parsing errors; schema-invalid content within a parsed record is retained.

Inspect coordinate frames separately from array dimensions

Acquisition views keep local and global coordinate systems separate. Select a scope or a source pointer when several frames exist:

from aind_metadata_reader import read_acquisition

acquisition = read_acquisition(read_json("tests/fixtures/v2.9.0-exaspim_acquisition.json"))
selected_frame = acquisition.select_frame(scope="local")
if selected_frame.status == ReadStatus.AVAILABLE:
    frame = selected_frame.value
    print(frame.name)
    print([(axis.name, axis.position, axis.dimension) for axis in frame.axes])
    print(frame.orientation.status, frame.orientation.value)

Axis.position follows source coordinate order. Axis.dimension is an image-array dimension only where recorded. array_mapping reports whether a mapping is established; mapping it to a particular Zarr array still requires evidence about that array's dimensions. A frame can have a known anatomical orientation while its array mapping remains unavailable.

Configured metadata supports stream and device selection. Legacy imaging metadata exposes scope legacy and reports unavailable stream/device selectors. Recorded image transforms remain accessible through frame.configuration; the reader does not compose them or substitute a local frame for a global one.

Request validation as a separate report

Install the optional validator in the checkout with uv sync --extra validation. Supply the historical schema explicitly:

from aind_metadata_reader import validate

schema = read_json("tests/fixtures/history/v1.4.0-processing-schema.json")
report = validate(
    record,
    schema,
    dialect="https://json-schema.org/draft/2020-12/schema",
)
assert report.status == "valid"
assert processing.validation.status == "not_checked"

validate() returns a separate report and leaves the record and its views unchanged. Reports distinguish not_checked, valid, invalid, and unavailable. An unusable schema, missing validator, or unresolved external reference means unavailable validation. Each report identifies the input, schema, and chosen dialect. A dialect must be supplied when the schema does not declare one.

This checks JSON Schema assertions using jsonschema. It does not insert defaults, coerce values, fetch remote schemas, run custom upstream Pydantic validators, or check format annotations. A successful report establishes conformance to those assertions; scientific accuracy and source authority remain separate questions. Reading has no mandatory validator or upstream-schema dependency.

Derive an attributed correction

A correction produces a new interpretation. The following example assumes a calibration note establishes a revised spacing:

from aind_metadata_reader import Change, apply_changes

calibration_record = Document({"voxel_spacing_um": 2.0})
change = Change.propose(
    calibration_record,
    "/voxel_spacing_um",
    1.8,
    operation_id="spacing-calibration",
    rule_id="manual.voxel-spacing",
    rule_revision="1",
    implementation="calibration-note/1",
    kind="correction",
    author="experimenter",
    reason="Updated calibration measurement",
    evidence=("calibration-record-123",),
    scope="physical spacing in micrometers",
)
result = apply_changes(calibration_record, [change], applied_at="2026-11-10T12:00:00+00:00")
assert result.original.raw == {"voxel_spacing_um": 2.0}
assert result.derived.raw == {"voxel_spacing_um": 1.8}

The attribution arguments become the audit trail. Operation kinds distinguish correction, inference (requires assumptions), and processing_frame (the conventions of a historical computation). Physical corrections and historical processing frames retain distinct identities even if their resulting values match.

Application checks the exact input capture, original field value, selected document scope, and conflicting edits. It does not require unrelated metadata to validate. Original snapshots stay intact, and no-op operations remain recorded.

Domain rules propose changes through RuleEvaluation, whose outcome is applicable, not_applicable, or unknown. For example, smartspim.evaluate_pyramid_translation_correction checks pyramid geometry and caller-supplied association and block-averaging evidence. Reading and evaluation never apply its proposals. Evidenced defect signatures can qualify even when the producer version is unverified; no first-fixed fusion version is assumed.

Pass decisions to apply_changes(..., evaluations=(evaluation,)) to retain their inputs, reader provenance, and links to applied operations. Unknown decisions and proposals deliberately left unapplied can also be retained. Archive supporting snapshots along with the target snapshot. The API guide describes the rule and saved-record contracts in more detail.

Save the inputs and replay recorded changes

Save the source archive and interpretation manifest with your analysis:

import json
from pathlib import Path
from aind_metadata_reader import Snapshot, replay

output = Path("analysis-provenance")
output.mkdir(exist_ok=True)
(output / "source.json").write_text(json.dumps(calibration_record.snapshot.archive(), indent=2))
(output / "interpretation.json").write_text(json.dumps(result.manifest(), indent=2))

restored = Document.from_snapshot(Snapshot.from_archive(read_json(output / "source.json").raw))
again = replay(restored, read_json(output / "interpretation.json").raw)
assert again.derived.snapshot.snapshot_id == result.derived.snapshot.snapshot_id

Replay applies recorded replacements to matching archived inputs. It does not rerun the latest reader or rule. Reevaluation is a separate computation using a selected implementation and its evidence.

content_digest identifies parsed values; original_digest identifies original bytes when retained. snapshot_id also incorporates source provenance, including retrieval evidence. Derived snapshots identify their parent snapshot and interpretation. Restore archives for exact replay: rereading a live file creates a new capture even if its bytes are unchanged. These hashes detect mismatches; they do not authenticate evidence.

Archives and manifests have independent format versions. New records use /2, and explicit readers retain /1 support through fixed historical fixtures. Persisted-record loaders and replay check their contracts strictly; the records can always be inspected separately as JSON.

Layout compatibility and test coverage

The reader selects a known layout from the recorded schema namespace and core type, structural markers, and historical layout boundaries. AIND acquisition v1 uses root axes with array dimensions; configured acquisition uses coordinate frames. Legacy processing stores processes under processing_pipeline with name and flat code fields; modern processing uses process_type and nested code. Conflicting layout markers or declarations withhold the ambiguous interpretation and retain the complete raw record.

The versions below identify test coverage, independently of Python package, pipeline, and component-code versions. Format.versions records these exercised core versions. An unlisted version can use the same evidenced layout, with an unverified_version diagnostic tied to its original version field. This does not certify compatibility for fields the reader does not interpret.

Representation Core versions exercised Concepts exposed
AIND legacy acquisition 1.0.0, 1.0.3, 1.0.4 Recorded axes and image dimensions
AIND configured acquisition 2.0.0, 2.5.1, 2.5.3, 2.5.4 Frames and configuration references
biodata acquisition 3.0.0 Frames and configuration references
AIND legacy processing 1.1.0, 1.1.4 Processes, analyses, and file inputs
AIND flat processing 2.0.0, 2.2.5, 2.3.1, 2.3.2, 2.3.3 Processes, pipeline links, and asset/group inputs
biodata processing 3.0.0 Processes, pipeline links, and asset/group inputs

When a record omits its namespace, historical major-version conventions can identify AIND v1/v2 or biodata v3. Otherwise the layout must identify one reader uniquely, or the caller must supply a format_hint. For example, an unversioned configured record with a recognized namespace needs no hint; the same layout without a namespace can match both families and needs an explicit choice. Hints cannot override a contradictory namespace, core type, or known layout boundary. Missing or malformed versions remain recorded and produce diagnostics; they do not prevent reading components when identity and layout are established.

Reader selection records retain the original declared version and the evidence used to choose a layout, including any untested-version qualification. Reading does not convert metadata or replace its declared schema version. The library's own snapshot, interpretation, and replay formats retain strict version checks; their saved-contract tests remain independent of current reader dispatch.

Tests include tagged historical examples/schemas, public production metadata, and paired v1/v2 DocDB representations. Constructed cases exercise documented historical changes such as output cardinality. V3 support is provisional against upstream examples until production outputs are available. See fixture provenance and historical evolution evidence.

Parsed integers retain Python precision; fractional values use binary64. Original bytes preserve source number spellings. Content hashing uses the documented Python JSON convention, not RFC 8785 canonicalization.

Development

See CONTRIBUTING.md for setup and checks, the design proposal for rationale, and implementation status for remaining work. The project was rendered from aind-python-copier-template commit 5dc04dcba3b3c16b8ee9c4b7c1c2314c08567bfc. Repository setup and CI are configured. Tagged releases publish source and wheel artifacts through the release workflow. Licensed under MIT.

SmartSPIM Registration Inputs

aind_metadata_reader.smartspim.registration_input(process) returns an inspectable ReadValue of file references for an Image importing process. It preserves legacy file inputs and recognizes the observed aind-smartspim-ccf-registration 0.0.37 convention that stores the input under output_parameters.input_location. Each reference retains its source pointer. Other producer versions need independent evidence before this convention applies; an asset-level input is not converted into a file path. This helper reads provenance and does not apply spatial corrections.

Pyramid Translation Corrections

aind_metadata_reader.smartspim.evaluate_pyramid_translation_correction diagnoses the observed missing-translation and voxel-count translation signatures in NGFF 0.4 fused pyramids. Callers must provide evidence linking the artifact to the selected fusion process and establishing block averaging. These requirements also apply when the producer version is missing or unverified; no first-fixed version is assumed. Correct physical translations yield not_applicable, while unrelated geometry yields unknown with no proposed changes.

Evaluation proposes changes without applying them. Pass its changes and evaluation to apply_changes to retain the source snapshot and an auditable interpretation. Previously recorded rule identities remain replayable without rerunning evaluation. Historical registration frames remain outside this physical correction's scope.

Metadata

Release files for aind-metadata-reader 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aind-metadata-reader 0.3.0
File Size Uploaded Provenance Actions
aind_metadata_reader-0.3.0.tar.gz 224.1 kB Without provenance Details for aind_metadata_reader-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aind-metadata-reader 0.3.0
File Interpreter ABI Platform Actions
aind_metadata_reader-0.3.0-py3-none-any.whl Python 3 none any Details for aind_metadata_reader-0.3.0-py3-none-any.whl

Total release size: 267.3 kB

Release files / aind_metadata_reader-0.3.0.tar.gz

Download URL aind_metadata_reader-0.3.0.tar.gz
Size 224.1 kB
Tags Source
SHA-256 checksum
How to use checksums
1cfe546d7423269c7ac9117fee6ab33ac17448b2236918c42068873c3f79e154
BLAKE2b-256 checksum
How to use checksums
eee1dc95805bd5404f4510f778a69d18e8ed85efc7cea5906bb8cc0c1effeb1d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / aind_metadata_reader-0.3.0-py3-none-any.whl

Download URL aind_metadata_reader-0.3.0-py3-none-any.whl
Size 43.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b3a678bd61000ba222cd4a914c99d362dc1536ecae429d60140b41a05849b2bb
BLAKE2b-256 checksum
How to use checksums
9cde0ae29f297e88f1a0cfc5ed463a1f97046be06b0524f3923e5cd8e22987b7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for aind-metadata-reader 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page