aind-nwb-utils
Usage
This package is intended to simplify and standardize the process of creating and interacting with NWB files. It is still in the early development stages, and the API and naming conventions are subject to change at any time.
Combining NWB files with NWBCombineIO
NWBCombineIO merges one or more sub NWB files into a main NWB file and, optionally, exports the result to disk. Data is merged in memory — the input files are opened read-only and are never modified.
The main file is the one that carries device information or units data; files carrying trials data are typically passed as sub files. This split is a current limitation.
Basic usage
As a context manager, which yields the merged NWBFile and the main IO handle:
from pathlib import Path
from aind_nwb_utils import NWBCombineIO
main_path = Path("/data/ecephys.nwb")
sub_paths = [Path("/data/behavior.nwb")]
with NWBCombineIO(main_path, sub_paths) as (nwb_file, main_io):
print(nwb_file.trials.to_dataframe())
As a standalone object, when you need the merged file to outlive a with block:
combiner = NWBCombineIO(main_path, sub_paths)
nwb_file = combiner.read()
# ... work with nwb_file ...
combiner.close()
The IO handles stay open until close() (or context-manager exit) so that lazily loaded datasets remain readable. Accessing data from a merged file after closing will fail.
Writing the combined file
with NWBCombineIO(main_path, sub_paths) as (nwb_file, main_io):
...
combiner = NWBCombineIO(main_path, sub_paths)
combiner.write(Path("/results/combined.nwb.zarr"), format="zarr")
combiner.close()
format accepts "zarr" (default) or "hdf5"; anything else raises ValueError. write() opens the inputs itself if read() has not already been called. The export uses link_data=False, so datasets are copied into the output rather than linked back to the source files — the output stands on its own.
Input formats
Input paths may be HDF5 files or Zarr stores, in any combination; the correct IO class is chosen per file by determine_io, which checks in order:
- File structure — the HDF5 magic signature, or a Zarr root key (
.zgroup,.zmetadata,zarr.json). This is the strongest signal because it ignores naming. - Extension —
.nwbmeans HDF5,.zarrmeans Zarr. - Fallback — a directory is treated as a Zarr store, anything else as HDF5.
s3:// paths are supported. Structure detection over S3 requires fsspec/s3fs; if those are unavailable or the probe fails, detection falls back to the extension and any genuine access problem surfaces at read time.
Extension namespaces
Each NWB file caches the namespaces it was written with, so reading files independently would give each IO handle a TypeMap that only knows that one file's extensions. Because the export runs through the main file's build manager, a container from a sub file whose extension the main file does not know would be downgraded to the nearest base type it does know — an ndx-hed HedLabMetaData would silently become a plain LabMetaData, dropping every extension-specific field.
NWBCombineIO avoids this by loading the cached namespaces of all inputs into a single TypeMap and sharing it across every handle (each handle still gets its own BuildManager, since those cache builder/container pairs per file). Extension types, their attributes, and their cached specs survive both the merge and the export. A file with no cached specs logs a warning and is still read using the namespaces already loaded.
Merge rules
Each field of each sub file is merged into the main file according to its type:
| Field type | Behavior |
|---|---|
Scalars and metadata (str, datetime, list, Subject, tuples, numpy/zarr arrays) |
The main file's value wins. If the main file has no value, the sub file's is copied in, so metadata present in only one input (institution, experimenter, was_generated_by, subject, ...) still reaches the output. A mismatch between two present values is logged as a warning. |
TimeIntervals (intervals, trials, epochs, invalid_times) |
Added under /intervals/<name>, since all four NWBFile fields live there. |
EventsTable |
If a table of the same name already exists, columns are merged into it — new columns are added, and columns present in both have the sub file's data appended. Otherwise the whole table is added. |
Dict-like fields (acquisition, processing, devices, stimulus, ...) |
Each entry is added by name through the corresponding public NWBFile adder (add_acquisition, add_device, ...). |
| Anything else | Raises TypeError. |
Two behaviors are worth knowing about:
- Name collisions favor the main file. If a sub file contains an object whose name already exists in the same field of the main file, the sub file's copy is skipped and the event is logged. Merging is not order-independent when names overlap.
- 64-bit data is downcast. Plain
TimeSeriesandVectorDataholdingfloat64/int64are rebuilt asfloat32/int32to avoid dtype conflicts between files.TimeSeriessubclasses are passed through untouched, because rebuilding them as a plainTimeSerieswould discard their neurodata type.
Installation
To use the software, in the root directory, run
pip install -e .
To develop the code, run
pip install -e .[dev]
Contributing
Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the coverage library, which will run the tests and log a coverage report:
coverage run -m unittest discover && coverage report
- Use interrogate to check that modules, methods, etc. have been documented thoroughly:
interrogate .
- Use flake8 to check that code is up to standards (no unused imports, etc.):
flake8 .
- Use black to automatically format the code into PEP standards:
black .
- Use isort to automatically sort import statements:
isort .
Pull requests
For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:
<type>(<scope>): <short summary>
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
- ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- docs: Documentation only changes
- feat: A new feature
- fix: A bugfix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- test: Adding missing tests or correcting existing tests
Semantic Release
The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):
| Commit message | Release type |
|---|---|
fix(pencil): stop graphite breaking when too much pressure applied |
|
feat(pencil): add 'graphiteWidth' option |
|
perf(pencil): remove graphiteWidth optionBREAKING CHANGE: The graphiteWidth option has been removed.The default graphite width of 10mm is always used for performance reasons. |
(Note that the BREAKING CHANGE: token must be in the footer of the commit) |
Documentation
To generate the rst files source files for documentation, run
sphinx-apidoc -o doc_template/source/ src
Then to create the documentation HTML files, run
sphinx-build -b html doc_template/source/ doc_template/build/html
More info on sphinx installation can be found here.
Metadata
Release files for aind-nwb-utils 0.2.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aind_nwb_utils-0.2.10.tar.gz | 39.2 MB | Without provenance | Details for aind_nwb_utils-0.2.10.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aind_nwb_utils-0.2.10-py3-none-any.whl | Python 3 | none | any | Details for aind_nwb_utils-0.2.10-py3-none-any.whl |
Total release size: 39.3 MB
Release files / aind_nwb_utils-0.2.10.tar.gz
| Download URL | aind_nwb_utils-0.2.10.tar.gz |
|---|---|
| Size | 39.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9d5133069d8f4506e01d2872c68e2be76d3fa081c844ba6822117ba11b4f33f1
|
|
BLAKE2b-256 checksum How to use checksums |
039e6bddf9d810d634462aa5c35a76034250ab5802a60e5cbb6446c9ce999dca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / aind_nwb_utils-0.2.10-py3-none-any.whl
| Download URL | aind_nwb_utils-0.2.10-py3-none-any.whl |
|---|---|
| Size | 19.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1d2be259eb7ee2493bb2a6368ed59494a8d68a777c90585f4895aba491b2e792
|
|
BLAKE2b-256 checksum How to use checksums |
88299e72abd84d7972797ac3863a859189083acff3bcf6b26b33d0da5f22fa99
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for aind-nwb-utils 0.2.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.