Skip to main content

AIND Metadata Reader

Read historical AIND metadata, inspect incomplete records, and record explicit corrections while preserving the source. Readers support selected representations from aind-data-schema and biodata-schema without requiring either package at runtime. Python 3.10 and newer are supported.

This library is experimental. Its API may evolve before version 1.0. The examples below run from this checkout using the included historical fixtures.

What the library does

Metadata changes in several ways: fields move, vocabulary grows, and concepts change meaning. A processing input recorded as a file path conveys different information from an input recorded as an asset reference. The reader exposes those distinctions through small views of code, references, and coordinate frames.

Reading preserves the original values and unknown fields. Schema-invalid metadata remains inspectable, and a malformed component does not hide usable siblings. You can separately request a validation report or derive a corrected interpretation. Neither operation silently replaces the source.

flowchart LR
    source["JSON file or mapping"] --> capture["Preserved snapshot"]
    capture --> views["Views, available values, and diagnostics"]
    capture --> validation["Optional validation report"]
    views --> changes["Explicit correction or inference"]
    changes --> derived["Derived snapshot and change manifest"]

You choose the source. The library accepts local files and caller-supplied data; it does not query DocDB, reconcile stores, or decide which record is authoritative. Image loading, registration, and transform application stay with their existing libraries.

Get started with a historical record

Prepare the checkout environment:

uv sync

Read the included processing example from aind-data-schema v1.4.0:

from aind_metadata_reader import ReadStatus, read_json, read_processing

record = read_json("tests/fixtures/history/v1.4.0-processing.json")
processing = read_processing(record)

print(record.schema_version)  # 1.1.4: the core schema version, not the package version
for process in processing.processes:
    print(process.process_type, process.code_version)

The example contains three pipeline steps and two separately recorded analyses. Their views retain the original ordering within each group and their actual source pointers. The reader does not construct a dependency graph from that ordering.

With your own data, replace the fixture path with a processing file or an aggregate such as metadata.nd.json. read_processing selects the processing core from an aggregate; read_acquisition does the same for acquisition metadata.

The main objects have different jobs:

Object What it holds
Snapshot The complete preserved payload, source provenance, and original bytes when supplied.
Document An object selected within a snapshot. Children retain the same root snapshot.
Processing, Process, Acquisition, CoordinateFrame Views of recognized concepts in a historical representation.
ReadValue A requested interpretation, its availability, diagnostics, and raw source field.
Interpretation Original and derived views, applied changes, retained rule decisions, and a manifest.

For data fetched by your own client, use Document(mapping, provenance=Source(...)). A Source can retain an S3 locator or a DocDB host, API version, database, collection, and record ID. Supply known conversion evidence explicitly.

Inspect recorded content directly with record.raw or record.field("/processing_pipeline/pipeline_version"). raw and field values return copies. A field's state distinguishes present, null, and absent. Pointers passed to a document are relative to that document; returned field pointers identify the location in the complete snapshot.

Check what an interpretation can establish

Selecting a process by type can produce several matches. find() returns them all; select() reports whether exactly one matches:

selected = processing.select("Image tile fusing")
if selected.status == ReadStatus.AVAILABLE:
    fusion = selected.value
    print(fusion.document.pointer)  # /processing_pipeline/data_processes/0
    print(fusion.code.version.value)  # 0.1

    inputs = fusion.inputs
    if inputs.status == ReadStatus.AVAILABLE:
        for reference in inputs.value:
            print(reference.kind, reference.location.value)
else:
    print(selected.status, selected.diagnostics)

assert processing.select("Analysis").status == ReadStatus.AMBIGUOUS

Code version and pipeline version are separate concepts. process.code exposes independent version, url, parameters, and inputs results. Input/output references distinguish file, asset, asset_group, and unknown; asset groups retain their members. An asset URL does not establish a file-level input.

A ReadValue gives you four things:

Attribute Meaning
status Whether the requested interpretation is available, and why it may be unavailable.
value The interpreted value when available; otherwise None.
field The actual recorded field, including its raw value, pointer, and snapshot.
diagnostics Structured explanations with codes and source locations.

Availability describes the requested interpretation:

Status What it tells you
available The reader established a value.
absent The source field was not recorded.
null The source explicitly recorded JSON null.
invalid The recorded content does not fit this requested interpretation.
unsupported The reader has no established interpretation for this representation or meaning.
ambiguous Several candidates exist and none has been selected.

invalid describes one attempted interpretation; it does not declare the entire metadata record invalid.

Convenience accessors such as code_version, parameters, processes, and array_axes make common cases shorter. Use the corresponding component results when missing information matters. An empty processes tuple, for example, could mean an empty recorded collection or an unsupported representation: process_entries.status distinguishes those situations.

selected.require(), processing.one(type), and acquisition.one_frame() explicitly assert a requirement and can raise. Ordinary inspection uses status-bearing results.

Incomplete metadata stays inspectable

This illustrative record has a numeric code version and a malformed second entry:

from aind_metadata_reader import Document

incomplete = Document(
    {
        "schema_version": "2.0.0",
        "data_processes": [
            {
                "process_type": "An unfamiliar processing term",
                "name": "example step",
                "code": {"version": 42, "parameters": {"threshold": 12}},
                "output_path": "/results/image.zarr",
            },
            "unfinished entry",
        ],
    }
)
inspected = read_processing(incomplete)
first, second = inspected.process_entries.value
step = first.value

assert step.code.version.status == ReadStatus.INVALID
assert step.code.version.field.value == 42
assert step.code.parameters.value == {"threshold": 12}
assert step.output_location == "/results/image.zarr"
assert second.status == ReadStatus.INVALID
assert second.field.value == "unfinished entry"
assert inspected.raw == incomplete.raw

Unfamiliar vocabulary remains recorded text. Unknown schema versions produce unsupported interpretations with the complete raw core still accessible. The reader does not assume compatibility from a major version or interpret a new layout by guessing from familiar field names.

Raw capture requires a finite JSON object. Malformed JSON, duplicate object keys, and nonfinite numbers are parsing errors; schema-invalid content within a parsed record is retained.

Inspect coordinate frames separately from array dimensions

Acquisition views keep local and global coordinate systems separate. Select a scope or a source pointer when several frames exist:

from aind_metadata_reader import read_acquisition

acquisition = read_acquisition(read_json("tests/fixtures/v2.9.0-exaspim_acquisition.json"))
selected_frame = acquisition.select_frame(scope="local")
if selected_frame.status == ReadStatus.AVAILABLE:
    frame = selected_frame.value
    print(frame.name)
    print([(axis.name, axis.position, axis.dimension) for axis in frame.axes])
    print(frame.orientation.status, frame.orientation.value)

Axis.position follows source coordinate order. Axis.dimension is an image-array dimension only where recorded. array_mapping reports whether a mapping is established; mapping it to a particular Zarr array still requires evidence about that array's dimensions. A frame can have a known anatomical orientation while its array mapping remains unavailable.

Configured metadata supports stream and device selection. Legacy imaging metadata exposes scope legacy and reports unavailable stream/device selectors. Recorded image transforms remain accessible through frame.configuration; the reader does not compose them or substitute a local frame for a global one.

Request validation as a separate report

Install the optional validator in the checkout with uv sync --extra validation. Supply the historical schema explicitly:

from aind_metadata_reader import validate

schema = read_json("tests/fixtures/history/v1.4.0-processing-schema.json")
report = validate(
    record,
    schema,
    dialect="https://json-schema.org/draft/2020-12/schema",
)
assert report.status == "valid"
assert processing.validation.status == "not_checked"

validate() returns a separate report and leaves the record and its views unchanged. Reports distinguish not_checked, valid, invalid, and unavailable. An unusable schema, missing validator, or unresolved external reference means unavailable validation. Each report identifies the input, schema, and chosen dialect. A dialect must be supplied when the schema does not declare one.

This checks JSON Schema assertions using jsonschema. It does not insert defaults, coerce values, fetch remote schemas, run custom upstream Pydantic validators, or check format annotations. A successful report establishes conformance to those assertions; scientific accuracy and source authority remain separate questions. Reading has no mandatory validator or upstream-schema dependency.

Derive an attributed correction

A correction produces a new interpretation. The following example assumes a calibration note establishes a revised spacing:

from aind_metadata_reader import Change, apply_changes

calibration_record = Document({"voxel_spacing_um": 2.0})
change = Change.propose(
    calibration_record,
    "/voxel_spacing_um",
    1.8,
    operation_id="spacing-calibration",
    rule_id="manual.voxel-spacing",
    rule_revision="1",
    implementation="calibration-note/1",
    kind="correction",
    author="experimenter",
    reason="Updated calibration measurement",
    evidence=("calibration-record-123",),
    scope="physical spacing in micrometers",
)
result = apply_changes(calibration_record, [change], applied_at="2026-11-10T12:00:00+00:00")
assert result.original.raw == {"voxel_spacing_um": 2.0}
assert result.derived.raw == {"voxel_spacing_um": 1.8}

The attribution arguments become the audit trail. Operation kinds distinguish correction, inference (requires assumptions), and processing_frame (the conventions of a historical computation). Physical corrections and historical processing frames retain distinct identities even if their resulting values match.

Application checks the exact input capture, original field value, selected document scope, and conflicting edits. It does not require unrelated metadata to validate. Original snapshots stay intact, and no-op operations remain recorded.

Domain rules propose changes through RuleEvaluation, whose outcome is applicable, not_applicable, or unknown. For example, smartspim.evaluate_ls17 checks known fusion versions, pyramid geometry, and caller-supplied association and block-averaging evidence. Reading and evaluation never apply its proposals. Unknown producer versions do not activate the correction; no first-fixed fusion version has been established.

Pass decisions to apply_changes(..., evaluations=(evaluation,)) to retain their inputs, reader provenance, and links to applied operations. Unknown decisions and proposals deliberately left unapplied can also be retained. Archive supporting snapshots along with the target snapshot. The API guide describes the rule and saved-record contracts in more detail.

Save the inputs and replay recorded changes

Save the source archive and interpretation manifest with your analysis:

import json
from pathlib import Path
from aind_metadata_reader import Snapshot, replay

output = Path("analysis-provenance")
output.mkdir(exist_ok=True)
(output / "source.json").write_text(json.dumps(calibration_record.snapshot.archive(), indent=2))
(output / "interpretation.json").write_text(json.dumps(result.manifest(), indent=2))

restored = Document.from_snapshot(Snapshot.from_archive(read_json(output / "source.json").raw))
again = replay(restored, read_json(output / "interpretation.json").raw)
assert again.derived.snapshot.snapshot_id == result.derived.snapshot.snapshot_id

Replay applies recorded replacements to matching archived inputs. It does not rerun the latest reader or rule. Reevaluation is a separate computation using a selected implementation and its evidence.

content_digest identifies parsed values; original_digest identifies original bytes when retained. snapshot_id also incorporates source provenance, including retrieval evidence. Derived snapshots identify their parent snapshot and interpretation. Restore archives for exact replay: rereading a live file creates a new capture even if its bytes are unchanged. These hashes detect mismatches; they do not authenticate evidence.

Archives and manifests have independent format versions. New records use /2, and explicit readers retain /1 support through fixed historical fixtures. Persisted-record loaders and replay check their contracts strictly; the records can always be inspected separately as JSON.

Supported representations and current limits

These are core schema versions, independent of Python package, pipeline, and component-code versions. A listed format describes tested interpretation capabilities; individual records can still contain unavailable components.

Representation Core versions exercised Concepts exposed
AIND legacy acquisition 1.0.0, 1.0.3, 1.0.4 Recorded axes and image dimensions
AIND configured acquisition 2.0.0, 2.5.3, 2.5.4 Frames and configuration references
biodata acquisition 3.0.0 Frames and configuration references
AIND legacy processing 1.1.0, 1.1.4 Processes, analyses, and file inputs
AIND flat processing 2.0.0, 2.2.5, 2.3.1, 2.3.2, 2.3.3 Processes, pipeline links, and asset/group inputs
biodata processing 3.0.0 Processes, pipeline links, and asset/group inputs

Unversioned legacy structures are recognized explicitly. Unversioned configured records need a recorded caller-supplied format_hint; hints cannot override contradictory declarations. Unknown versions retain their raw records and diagnostics.

Tests include tagged historical examples/schemas, public production metadata, and paired v1/v2 DocDB representations. Constructed cases exercise documented historical changes such as output cardinality. V3 support is provisional against upstream examples until production outputs are available. See fixture provenance and historical evolution evidence.

Parsed integers retain Python precision; fractional values use binary64. Original bytes preserve source number spellings. Content hashing uses the documented Python JSON convention, not RFC 8785 canonicalization.

Development

See CONTRIBUTING.md for setup and checks, the design proposal for rationale, and implementation status for remaining work. The project was rendered from aind-python-copier-template commit 5dc04dcba3b3c16b8ee9c4b7c1c2314c08567bfc. Repository setup and CI are configured. Tagged releases publish source and wheel artifacts through the release workflow. Licensed under MIT.

SmartSPIM Registration Inputs

aind_metadata_reader.smartspim.registration_input(process) returns an inspectable ReadValue of file references for an Image importing process. It preserves legacy file inputs and recognizes the observed aind-smartspim-ccf-registration 0.0.37 convention that stores the input under output_parameters.input_location. Each reference retains its source pointer. Other producer versions need independent evidence before this convention applies; an asset-level input is not converted into a file path. This helper reads provenance and does not apply spatial corrections.

Metadata

Release files for aind-metadata-reader 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aind-metadata-reader 0.2.0
File Size Uploaded Provenance Actions
aind_metadata_reader-0.2.0.tar.gz 218.2 kB Without provenance Details for aind_metadata_reader-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aind-metadata-reader 0.2.0
File Interpreter ABI Platform Actions
aind_metadata_reader-0.2.0-py3-none-any.whl Python 3 none any Details for aind_metadata_reader-0.2.0-py3-none-any.whl

Total release size: 259.5 kB

Release files / aind_metadata_reader-0.2.0.tar.gz

Download URL aind_metadata_reader-0.2.0.tar.gz
Size 218.2 kB
Tags Source
SHA-256 checksum
How to use checksums
d3a01035c58f41af2a21def00ece6f3dadbf336297dbd543fbd5f666c3565e70
BLAKE2b-256 checksum
How to use checksums
f6c9e4d1480b8e1470ad05b557d843da1abc5414478d8297df7cc0ed67df71f2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / aind_metadata_reader-0.2.0-py3-none-any.whl

Download URL aind_metadata_reader-0.2.0-py3-none-any.whl
Size 41.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f37d3ac46e8b6477d2f44502e3ccf4baf6796582ff56441a4bcd5a7121f368e5
BLAKE2b-256 checksum
How to use checksums
3ef934ba2522eeb3953fe3fcd83522859bda78313d5542ab73af87f6044bab06
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for aind-metadata-reader 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page