aind-metadata-mapper

Code Style Interrogate Coverage Python

Repository to contain code that will parse source files into aind-data-schema models.

Usage

The GatherMetadataJob is used to create the data_description.json and pull the subject.json and procedures.json from aind-metadata-service. Users are expected to provide the instrument.json and the acquisition.json as well as optional processing.json, quality_control.json and model.json. If a user provides procedures.json, it will be merged with the procedures fetched from the service (subject and specimen procedures are deduplicated). After all core files are assembled, the job runs an enrichment step that fills optional acquisition fields from the metadata service (currently subject weights from Waterlog and ethics_review_id from LabTracks). See Enrichment below. The job will attempt to validate all of the metadata files, displaying errors, and then will save all metadata fields into the selected folder.

Using the GatherMetadataJob

The following are the minimum required settings:

output_dir (str): Location where metadata files will be saved. If a metadata_dir is not provided, this will also be the location that the job searches for metadata files.
data_description_settings:
- project_name (str): Project name used to fetch funding and investigator information.
- modalities (List[Modality]): List of data modalities for this dataset.

from aind_data_schema_models.modalities import Modality
from aind_metadata_mapper.gather_metadata import GatherMetadataJob
from aind_metadata_mapper.models import JobSettings, DataDescriptionSettings

job_settings = JobSettings(
  output_dir="/path/to/output",
  subject_id="123456",
  data_description_settings=DataDescriptionSettings(
    project_name="<project-name>",
    modalities=[Modality.ECEPHYS],
  )
)

job = GatherMetadataJob(job_settings=job_settings)
job.run_job()

Default behavior

The GatherMetadataJob attempts to find all of the core metadata files (instrument.json, acquisition.json, etc) and then validates them as a full Metadata object.

The job will always prioritize an exact match for a core file when it finds one in the metadata_dir. An exact-match acquisition.json is still passed through the enrichment step, which only fills fields that are empty.

If no exact match exists, it will construct, fetch, merge or run mappers to generate the appropriate metadata, if it is available.

File	Method 1	Method 2	Method 3
data_description.json	Exact match in input directory	Construct from settings / fetch from metadata-service
subject.json	Exact match in input directory	Fetch from metadata-service (requires subject_id)	Constructed locally when subject_id is "calibration"
procedures.json	Fetch from metadata-service (requires subject_id)	Merge with user-provided procedures but default to user-provided if there are any duplicates	Constructed locally (empty) when subject_id is "calibration"
acquisition.json	Exact match in input directory	Run mappers on `<mapper>.json` files (and merge)	Merge all `acquisition*.json` files
instrument.json	Exact match in input directory	Fetch from metadata-service (requires instrument_id)	Merge all `instrument*.json` files
processing.json	Exact match in input directory
quality_control.json	Exact match in input directory	Merge all `quality_control*.json` files
model.json	Exact match in input directory

Automated mappers

When mappers are developed from the BaseMapper class and registered in mapper_registry.py they can be automatically run by the GatherMetadataJob. A file matching the mapper name <mapper>.json will be turned into a file acquisition_<mapper>.json and then merged with any other acquisition files.

Enrichment

After all core files are assembled, the job passes the acquisition through aind_metadata_mapper.enrichment.enrich_acquisition, which fills optional acquisition fields with data that exists at upload time but was not supplied by the rig. This runs on every acquisition, including an exact-match acquisition.json, and is skipped when subject_id is "calibration", since there is nothing to look up. The enriched acquisition is what gets written to acquisition.json and validated with the rest of the metadata.

Every enricher must satisfy all of these:

Fills only fields that are currently None or absent. Never overwrites unless replace_existing=True.
If a value is already present and differs from the fetched value, logs a warning and keeps the existing value (unless replace_existing=True).
Touches only schema-optional fields. The rig's acquisition.json already validated, so nothing an enricher adds can be required.
Fetches only through aind-metadata-service endpoints. No direct Dataverse or LabTracks table access.
A failed or empty lookup is a warning, never an exception.
Idempotent. Running twice produces the same result, so the same function can back-fill records later.
Not platform or task specific.

Field	Source	Matching rule	On conflict
`acquisition.subject_details.animal_weight_prior`	metadata-service `/api/v2/dataverse/mouse_weight_records/{subject_id}` (Waterlog)	Latest same-day record at or before `acquisition_start_time`	Keep existing, warn
`acquisition.subject_details.animal_weight_post`	same	Earliest same-day record after `acquisition_start_time`	Keep existing, warn
`acquisition.ethics_review_id`	metadata-service `/api/v2/labtracks/subject?subject_id=` (LabTracks `protocol_number`)	Subject's current protocol	Keep existing, warn

The step is controlled by enrichment_settings (see Optional settings). scripts/demo_enrichment.py takes a data asset name, pulls that record's acquisition and subject from DocDB, runs the same function, and prints the before and after values. With --output-dir it also writes the enriched acquisition.json there through Acquisition.write_standard_file. Nothing is written back to DocDB. That is the pattern a back-fill migration would follow.

Optional settings

metadata_dir (str, optional): Location of existing metadata files, if different from the output_dir. If a file is found here, it will be used instead of constructing/fetching it. acquisition.json is additionally passed through the enrichment step.
subject_id (str): Subject ID used to fetch metadata from the service (subject.json, procedures.json). This setting should only be used when an acquisition.json is not available.
acquisition_start_time (datetime, optional): Acquisition start time in ISO 8601 format. This setting should only be used when an acquisition.json is not available.
subject_settings (optional): Settings for subject metadata. Only used when subject_id is "calibration".
- calibration_object (CalibrationObject, optional): A CalibrationObject from aind_data_schema.components.subjects. When subject_id is "calibration", the metadata service is not contacted — instead a Subject is constructed locally using this object and an empty Procedures (no subject or specimen procedures). If omitted, a default empty CalibrationObject is used.
instrument_settings:
- instrument_id (str): ID for the instrument used in data collection. When set, the instrument.json will attempt to be fetched from the metadata-service and saved as instrument_<modality-abbreviation(s)>.json. If multiple instrument*.json files exist after fetching they will be merged.
enrichment_settings (optional): Settings for the enrichment step (see Enrichment above).
- enabled (bool, default=True): Run the enrichment step after assembling core metadata.
- replace_existing (bool, default=False): If True, enrichers overwrite values already present in acquisition.json. The default keeps existing values and warns on mismatch.
data_description_settings: See DataDescription for details.
- tags (list[str], optional)
- group (str, optional)
- restrictions (str, optional)
- data_summary (str, optional)

from datetime import datetime
from aind_data_schema_models.modalities import Modality
from aind_data_schema_models.data_name_patterns import Group
from aind_metadata_mapper.gather_metadata import GatherMetadataJob
from aind_metadata_mapper.models import JobSettings, DataDescriptionSettings, InstrumentSettings

job_settings = JobSettings(
    metadata_dir="/path/to/input/",
    output_dir="/path/to/output",
    subject_id="828422",
    acquisition_start_time=datetime.fromisoformat("2025-11-13T17:38:37.079861+00:00"),
    data_description_settings=DataDescriptionSettings(
        project_name="Cognitive flexibility in patch foraging",
        modalities=[Modality.BEHAVIOR, Modality.BEHAVIOR_VIDEOS, Modality.FIB],
        tags=["foraging"],
        group=Group.BEHAVIOR,
        restrictions="Internal use only",
        data_summary="VR foraging task with fiber photometry recording",
    ),
    instrument_settings=InstrumentSettings(
        instrument_id="13A",
    ),
    raise_if_invalid=True,
    raise_if_mapper_errors=True,
    metadata_service_url="http://aind-metadata-service",
)

job = GatherMetadataJob(job_settings=job_settings)
job.run_job()

Calibration sessions

When collecting data with a calibration object rather than a live subject, set subject_id to "calibration". The job will skip the metadata service entirely and construct subject.json and procedures.json locally.

from aind_data_schema.components.subjects import CalibrationObject
from aind_data_schema_models.modalities import Modality
from aind_metadata_mapper.gather_metadata import GatherMetadataJob
from aind_metadata_mapper.models import JobSettings, DataDescriptionSettings, SubjectSettings

job_settings = JobSettings(
    output_dir="/path/to/output",
    subject_id="calibration",
    data_description_settings=DataDescriptionSettings(
        project_name="<project-name>",
        modalities=[Modality.ECEPHYS],
    ),
    subject_settings=SubjectSettings(
        calibration_object=CalibrationObject(
            description="Neuropixels dummy probe",
            empty=False,
        )
    ),
)

job = GatherMetadataJob(job_settings=job_settings)
job.run_job()

Validation settings

raise_if_invalid (bool, default=False): Controls validation behavior:
- True: Raises an exception if any fetched metadata is invalid.
- False: Logs a warning or error and continues when validation errors occur.
raise_if_mapper_errors (bool, default=True): Controls mapper execution behavior:
- True: Raises an error if any automated mapper (e.g., for instrument-specific formats) fails.
- False: Logs a warning and continues without that mapper's output.

Metadata service settings

You probably shouldn't be modifying these.

metadata_service_url (str, default=http://aind-metadata-service): Base URL of the metadata service.
metadata_service_*_endpoint (str): API endpoints for specific metadata types:
- metadata_service_subject_endpoint (default="/api/v2/subject")
- metadata_service_procedures_endpoint (default="/api/v2/procedures")
- metadata_service_instrument_endpoint (default="/api/v2/instrument")

Instrument CLI

The aind-instrument command lets you upload and retrieve instruments from the metadata service without writing code.

# Upload an instrument
aind-instrument upload instrument.json

# Upload and overwrite an existing record
aind-instrument upload instrument.json --replace

# Keep the modification date from the file instead of updating to today
aind-instrument upload instrument.json --no-update-modification-date

# Get the latest record for an instrument
aind-instrument get 422_MESO2_20241017

# Get a specific version by modification date
aind-instrument get 422_MESO2_20241017 --modification-date 2024-10-28

# Save to a file instead of printing to stdout
aind-instrument get 422_MESO2_20241017 --output-directory ./output

Developing Mappers

Each MapperJob class should inherit from BaseMapper in base.py. The only parameter should be the MapperJobSettings from base.py. You cannot add additional parameters to your job or it will not be possible for it to be run automatically on the data-transfer-service. GatherMetadataJob will then run your mappers automatically when it detects the extracted metadata output.

Mappers should not look up platform-agnostic fields such as ethics_review_id. Those are filled by the enrichment step after acquisitions are merged.

Writing the output file

In your run_job() function the final step should be to use the write_standard_file() function and pass it the parameters from the job settings. This ensures that any changes we make to how writing files happens in the future will be preserved in your mapper.

acquisition.write_standard_file(output_directory=job_settings.output_directory, filename_suffix=filename_suffix)

Individual mappers

FIP (Fiber photometry)

Metadata

Release files for aind-metadata-mapper 1.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aind-metadata-mapper 1.4.0
File	Size	Uploaded	Provenance
aind_metadata_mapper-1.4.0.tar.gz	66.2 kB	Oct 20, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for aind-metadata-mapper 1.4.0
File	Interpreter	ABI	Platform	Reset
aind_metadata_mapper-1.4.0-py3-none-any.whl 42.6 kB Oct 20, 2026 Without provenance	Python 3	none	any	Details

Total release size: 108.8 kB

Release files / aind_metadata_mapper-1.4.0.tar.gz

Download URL	aind_metadata_mapper-1.4.0.tar.gz
Size	66.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`220bb5b6782ba8c16ed10431502f7745824ed706cccfe075f07dee56f61b34fc`
BLAKE2b-256 checksum How to use checksums	`fffba2c77c5eebd8f96c6a120edbd23f825a8c0ee4666653514ddbb3a3875fcd`
Upload date	Oct 20, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.12.8`

Release files / aind_metadata_mapper-1.4.0-py3-none-any.whl

Download URL	aind_metadata_mapper-1.4.0-py3-none-any.whl
Size	42.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`9b27a70f171a79ba1f22d4e7062b25c195d07fd42de9a46125eeecad458ae989`
BLAKE2b-256 checksum How to use checksums	`24b0e703d34af41cc5048de7adc4512ff3df152cee72d059df80ad8c4a61c322`
Upload date	Oct 20, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.12.8`

Security for aind-metadata-mapper 1.4.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.