AIND Metadata Reader
Read historical AIND metadata, inspect incomplete records, and record explicit
corrections while preserving the source. Readers support selected representations
from aind-data-schema and biodata-schema without requiring either package at
runtime. Python 3.10 and newer are supported.
This library is experimental. Its API may evolve before version 1.0. The examples below run from this checkout using the included historical fixtures.
What the library does
Metadata changes in several ways: fields move, vocabulary grows, and concepts change meaning. A processing input recorded as a file path conveys different information from an input recorded as an asset reference. The reader exposes those distinctions through small views of code, references, and coordinate frames.
Reading preserves the original values and unknown fields. Schema-invalid metadata remains inspectable, and a malformed component does not hide usable siblings. You can separately request a validation report or derive a corrected interpretation. Neither operation silently replaces the source.
flowchart LR
source["JSON file or mapping"] --> capture["Preserved snapshot"]
capture --> views["Views, available values, and diagnostics"]
capture --> validation["Optional validation report"]
views --> changes["Explicit correction or inference"]
changes --> derived["Derived snapshot and change manifest"]
You choose the source. The library accepts local files and caller-supplied data; it does not query DocDB, reconcile stores, or decide which record is authoritative. Image loading, registration, and transform application stay with their existing libraries.
Get started with a historical record
Prepare the checkout environment:
uv sync
Read the included processing example from aind-data-schema v1.4.0:
from aind_metadata_reader import ReadStatus, read_json, read_processing
record = read_json("tests/fixtures/history/v1.4.0-processing.json")
processing = read_processing(record)
print(record.schema_version) # 1.1.4: the core schema version, not the package version
for process in processing.processes:
print(process.process_type, process.code_version)
The example contains three pipeline steps and two separately recorded analyses. Their views retain the original ordering within each group and their actual source pointers. The reader does not construct a dependency graph from that ordering.
With your own data, replace the fixture path with a processing file or an aggregate
such as metadata.nd.json. read_processing selects the processing core from an
aggregate; read_acquisition does the same for acquisition metadata.
The main objects have different jobs:
| Object | What it holds |
|---|---|
Snapshot |
The complete preserved payload, source provenance, and original bytes when supplied. |
Document |
An object selected within a snapshot. Children retain the same root snapshot. |
Processing, Process, Acquisition, CoordinateFrame |
Views of recognized concepts in a historical representation. |
ReadValue |
A requested interpretation, its availability, diagnostics, and raw source field. |
Interpretation |
Original and derived views, applied changes, retained rule decisions, and a manifest. |
For data fetched by your own client, use Document(mapping, provenance=Source(...)).
A Source can retain an S3 locator or a DocDB host, API version, database,
collection, and record ID. Supply known conversion evidence explicitly.
Inspect recorded content directly with record.raw or
record.field("/processing_pipeline/pipeline_version"). raw and field values
return copies. A field's state distinguishes present, null, and absent.
Pointers passed to a document are relative to that document; returned field pointers
identify the location in the complete snapshot.
Check what an interpretation can establish
Selecting a process by type can produce several matches. find() returns them
all; select() reports whether exactly one matches:
selected = processing.select("Image tile fusing")
if selected.status == ReadStatus.AVAILABLE:
fusion = selected.value
print(fusion.document.pointer) # /processing_pipeline/data_processes/0
print(fusion.code.version.value) # 0.1
inputs = fusion.inputs
if inputs.status == ReadStatus.AVAILABLE:
for reference in inputs.value:
print(reference.kind, reference.location.value)
else:
print(selected.status, selected.diagnostics)
assert processing.select("Analysis").status == ReadStatus.AMBIGUOUS
Code version and pipeline version are separate concepts. process.code exposes
independent version, url, parameters, and inputs results. Input/output
references distinguish file, asset, asset_group, and unknown; asset groups
retain their members. An asset URL does not establish a file-level input.
A ReadValue gives you four things:
| Attribute | Meaning |
|---|---|
status |
Whether the requested interpretation is available, and why it may be unavailable. |
value |
The interpreted value when available; otherwise None. |
field |
The actual recorded field, including its raw value, pointer, and snapshot. |
diagnostics |
Structured explanations with codes and source locations. |
Availability describes the requested interpretation:
| Status | What it tells you |
|---|---|
available |
The reader established a value. |
absent |
The source field was not recorded. |
null |
The source explicitly recorded JSON null. |
invalid |
The recorded content does not fit this requested interpretation. |
unsupported |
The reader has no established interpretation for this representation or meaning. |
ambiguous |
Several candidates exist and none has been selected. |
invalid describes one attempted interpretation; it does not declare the entire
metadata record invalid.
Convenience accessors such as code_version, parameters, processes, and
array_axes make common cases shorter. Use the corresponding component results
when missing information matters. An empty processes tuple, for example, could
mean an empty recorded collection or an unsupported representation:
process_entries.status distinguishes those situations.
selected.require(), processing.one(type), and acquisition.one_frame() explicitly
assert a requirement and can raise. Ordinary inspection uses status-bearing results.
Incomplete metadata stays inspectable
This illustrative record has a numeric code version and a malformed second entry:
from aind_metadata_reader import Document
incomplete = Document(
{
"schema_version": "2.0.0",
"data_processes": [
{
"process_type": "An unfamiliar processing term",
"name": "example step",
"code": {"version": 42, "parameters": {"threshold": 12}},
"output_path": "/results/image.zarr",
},
"unfinished entry",
],
}
)
inspected = read_processing(incomplete)
first, second = inspected.process_entries.value
step = first.value
assert step.code.version.status == ReadStatus.INVALID
assert step.code.version.field.value == 42
assert step.code.parameters.value == {"threshold": 12}
assert step.output_location == "/results/image.zarr"
assert second.status == ReadStatus.INVALID
assert second.field.value == "unfinished entry"
assert inspected.raw == incomplete.raw
Unfamiliar vocabulary remains recorded text. An untested schema version produces
an unverified_version diagnostic while recognized components remain readable.
Schema identity, recorded layout, and known historical boundaries select the
reader; each component then reports its own availability. An unfamiliar direction,
for example, withholds anatomical orientation while preserving the frame, its raw
axes, and other readable frames. This describes what the reader can interpret;
validation separately describes whether the record satisfies a schema.
Raw capture requires a finite JSON object. Malformed JSON, duplicate object keys, and nonfinite numbers are parsing errors; schema-invalid content within a parsed record is retained.
Inspect coordinate frames separately from array dimensions
Acquisition views keep local and global coordinate systems separate. Select a scope or a source pointer when several frames exist:
from aind_metadata_reader import read_acquisition
acquisition = read_acquisition(read_json("tests/fixtures/v2.9.0-exaspim_acquisition.json"))
selected_frame = acquisition.select_frame(scope="local")
if selected_frame.status == ReadStatus.AVAILABLE:
frame = selected_frame.value
print(frame.name)
print([(axis.name, axis.position, axis.dimension) for axis in frame.axes])
print(frame.orientation.status, frame.orientation.value)
Axis.position follows source coordinate order. Axis.dimension is an image-array
dimension only where recorded. array_mapping reports whether a mapping is
established; mapping it to a particular Zarr array still requires evidence about
that array's dimensions. A frame can have a known anatomical orientation while
its array mapping remains unavailable.
Configured metadata supports stream and device selection. Legacy imaging metadata
exposes scope legacy and reports unavailable stream/device selectors. Recorded
image transforms remain accessible through frame.configuration; the reader does
not compose them or substitute a local frame for a global one.
Request validation as a separate report
Install the optional validator in the checkout with uv sync --extra validation.
Supply the historical schema explicitly:
from aind_metadata_reader import validate
schema = read_json("tests/fixtures/history/v1.4.0-processing-schema.json")
report = validate(
record,
schema,
dialect="https://json-schema.org/draft/2020-12/schema",
)
assert report.status == "valid"
assert processing.validation.status == "not_checked"
validate() returns a separate report and leaves the record and its views unchanged.
Reports distinguish not_checked, valid, invalid, and unavailable. An unusable
schema, missing validator, or unresolved external reference means unavailable
validation. Each report identifies the input, schema, and chosen dialect. A dialect
must be supplied when the schema does not declare one.
This checks JSON Schema assertions using jsonschema. It does not insert defaults,
coerce values, fetch remote schemas, run custom upstream Pydantic validators, or
check format annotations. A successful report establishes conformance to those
assertions; scientific accuracy and source authority remain separate questions.
Reading has no mandatory validator or upstream-schema dependency.
Derive an attributed correction
A correction produces a new interpretation. The following example assumes a calibration note establishes a revised spacing:
from aind_metadata_reader import Change, apply_changes
calibration_record = Document({"voxel_spacing_um": 2.0})
change = Change.propose(
calibration_record,
"/voxel_spacing_um",
1.8,
operation_id="spacing-calibration",
rule_id="manual.voxel-spacing",
rule_revision="1",
implementation="calibration-note/1",
kind="correction",
author="experimenter",
reason="Updated calibration measurement",
evidence=("calibration-record-123",),
scope="physical spacing in micrometers",
)
result = apply_changes(calibration_record, [change], applied_at="2026-11-10T12:00:00+00:00")
assert result.original.raw == {"voxel_spacing_um": 2.0}
assert result.derived.raw == {"voxel_spacing_um": 1.8}
The attribution arguments become the audit trail. Operation kinds distinguish
correction, inference (requires assumptions), and processing_frame
(the conventions of a historical computation). Physical corrections and historical
processing frames retain distinct identities even if their resulting values match.
Application checks the exact input capture, original field value, selected document scope, and conflicting edits. It does not require unrelated metadata to validate. Original snapshots stay intact, and no-op operations remain recorded.
Domain rules propose changes through RuleEvaluation, whose outcome is
applicable, not_applicable, or unknown. For example,
smartspim.evaluate_pyramid_translation_correction checks pyramid geometry and
caller-supplied association and block-averaging evidence. Reading and evaluation
never apply its proposals. Evidenced defect signatures can qualify even when the
producer version is unverified; no first-fixed fusion version is assumed.
Pass decisions to apply_changes(..., evaluations=(evaluation,)) to retain their
inputs, reader provenance, and links to applied operations. Unknown decisions and
proposals deliberately left unapplied can also be retained. Archive supporting
snapshots along with the target snapshot. The API guide describes
the rule and saved-record contracts in more detail.
Save the inputs and replay recorded changes
Save the source archive and interpretation manifest with your analysis:
import json
from pathlib import Path
from aind_metadata_reader import Snapshot, replay
output = Path("analysis-provenance")
output.mkdir(exist_ok=True)
(output / "source.json").write_text(json.dumps(calibration_record.snapshot.archive(), indent=2))
(output / "interpretation.json").write_text(json.dumps(result.manifest(), indent=2))
restored = Document.from_snapshot(Snapshot.from_archive(read_json(output / "source.json").raw))
again = replay(restored, read_json(output / "interpretation.json").raw)
assert again.derived.snapshot.snapshot_id == result.derived.snapshot.snapshot_id
Replay applies recorded replacements to matching archived inputs. It does not rerun the latest reader or rule. Reevaluation is a separate computation using a selected implementation and its evidence.
content_digest identifies parsed values; original_digest identifies original
bytes when retained. snapshot_id also incorporates source provenance, including
retrieval evidence. Derived snapshots identify their parent snapshot and
interpretation. Restore archives for exact replay: rereading a live file creates
a new capture even if its bytes are unchanged. These hashes detect mismatches;
they do not authenticate evidence.
Archives and manifests have independent format versions. New records use /2,
and explicit readers retain /1 support through fixed historical fixtures.
Persisted-record loaders and replay check their contracts strictly; the records
can always be inspected separately as JSON.
Layout compatibility and test coverage
The reader selects a known layout from the recorded schema namespace and core
type, structural markers, and historical layout boundaries. AIND acquisition v1
uses root axes with array dimensions; configured acquisition uses coordinate
frames. Legacy processing stores processes under processing_pipeline with
name and flat code fields; modern processing uses process_type and nested
code. Conflicting layout markers or declarations withhold the ambiguous
interpretation and retain the complete raw record.
The versions below identify test coverage, independently of Python package,
pipeline, and component-code versions. Format.versions records these exercised
core versions. An unlisted version can use the same evidenced layout, with an
unverified_version diagnostic tied to its original version field. This does
not certify compatibility for fields the reader does not interpret.
| Representation | Core versions exercised | Concepts exposed |
|---|---|---|
| AIND legacy acquisition | 1.0.0, 1.0.3, 1.0.4 | Recorded axes and image dimensions |
| AIND configured acquisition | 2.0.0, 2.5.1, 2.5.3, 2.5.4 | Frames and configuration references |
| biodata acquisition | 3.0.0 | Frames and configuration references |
| AIND legacy processing | 1.1.0, 1.1.4 | Processes, analyses, and file inputs |
| AIND flat processing | 2.0.0, 2.2.5, 2.3.1, 2.3.2, 2.3.3 | Processes, pipeline links, and asset/group inputs |
| biodata processing | 3.0.0 | Processes, pipeline links, and asset/group inputs |
When a record omits its namespace, historical major-version conventions can
identify AIND v1/v2 or biodata v3. Otherwise the layout must identify one reader
uniquely, or the caller must supply a format_hint. For example, an unversioned
configured record with a recognized namespace needs no hint; the same layout
without a namespace can match both families and needs an explicit choice.
Hints cannot override a contradictory namespace, core type, or known layout
boundary. Missing or malformed versions remain recorded and produce diagnostics;
they do not prevent reading components when identity and layout are established.
Reader selection records retain the original declared version and the evidence used to choose a layout, including any untested-version qualification. Reading does not convert metadata or replace its declared schema version. The library's own snapshot, interpretation, and replay formats retain strict version checks; their saved-contract tests remain independent of current reader dispatch.
Tests include tagged historical examples/schemas, public production metadata, and paired v1/v2 DocDB representations. Constructed cases exercise documented historical changes such as output cardinality. V3 support is provisional against upstream examples until production outputs are available. See fixture provenance and historical evolution evidence.
Parsed integers retain Python precision; fractional values use binary64. Original bytes preserve source number spellings. Content hashing uses the documented Python JSON convention, not RFC 8785 canonicalization.
Development
See CONTRIBUTING.md for setup and checks, the
design proposal for rationale, and
implementation status for remaining work.
The project was rendered from aind-python-copier-template commit
5dc04dcba3b3c16b8ee9c4b7c1c2314c08567bfc. Repository setup and CI are configured. Tagged releases publish source and wheel
artifacts through the release workflow. Licensed under MIT.
SmartSPIM Registration Inputs
aind_metadata_reader.smartspim.registration_input(process) returns an inspectable
ReadValue of file references for an Image importing process. It preserves
legacy file inputs and recognizes the observed
aind-smartspim-ccf-registration 0.0.37 convention that stores the input under
output_parameters.input_location. Each reference retains its source pointer.
Other producer versions need independent evidence before this convention applies;
an asset-level input is not converted into a file path. This helper reads
provenance and does not apply spatial corrections.
Pyramid Translation Corrections
aind_metadata_reader.smartspim.evaluate_pyramid_translation_correction diagnoses the observed
missing-translation and voxel-count translation signatures in NGFF 0.4 fused
pyramids. Callers must provide evidence linking the artifact to the selected
fusion process and establishing block averaging. These requirements also apply
when the producer version is missing or unverified; no first-fixed version is
assumed. Correct physical translations yield not_applicable, while unrelated
geometry yields unknown with no proposed changes.
Evaluation proposes changes without applying them. Pass its changes and evaluation
to apply_changes to retain the source snapshot and an auditable interpretation.
Previously recorded rule identities remain replayable without rerunning evaluation.
Historical registration frames remain outside this physical correction's scope.
Metadata
Release files for aind-metadata-reader 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aind_metadata_reader-0.3.0.tar.gz | 224.1 kB | Without provenance | Details for aind_metadata_reader-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aind_metadata_reader-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for aind_metadata_reader-0.3.0-py3-none-any.whl |
Total release size: 267.3 kB
Release files / aind_metadata_reader-0.3.0.tar.gz
| Download URL | aind_metadata_reader-0.3.0.tar.gz |
|---|---|
| Size | 224.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1cfe546d7423269c7ac9117fee6ab33ac17448b2236918c42068873c3f79e154
|
|
BLAKE2b-256 checksum How to use checksums |
eee1dc95805bd5404f4510f778a69d18e8ed85efc7cea5906bb8cc0c1effeb1d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / aind_metadata_reader-0.3.0-py3-none-any.whl
| Download URL | aind_metadata_reader-0.3.0-py3-none-any.whl |
|---|---|
| Size | 43.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b3a678bd61000ba222cd4a914c99d362dc1536ecae429d60140b41a05849b2bb
|
|
BLAKE2b-256 checksum How to use checksums |
9cde0ae29f297e88f1a0cfc5ed463a1f97046be06b0524f3923e5cd8e22987b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for aind-metadata-reader 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.