Installation

Install the neurobank Python package and its dependencies:

pip install neurobank

Archive setup

First, initialize an archive:

nbank [-a username:password] [-r registry-url] init [-n name] [-u umask] [-g group] [--shared] my-archive-path

my-archive-path must be a directory on a locally-accessible filesystem (which could be an NFS or SSHFS mount).

registry-url specifies the registry to use, which can be any service that implements the API defined in django-neurobank. If not supplied, the script will try to use the value of the environment variable NBANK_REGISTRY. The registry URL also determines the domain for the resource identifiers. For example, if the registry is at http://melizalab.org/neurobank/resources/ and you deposit a resource with the identifier st32_1_2_1, the full identifier is http://melizalab.org/neurobank/resources/st32_1_2_1/. st32_1_2_1 is guaranteed to be unique within this domain.

If your registry requires authentication, this must be supplied with the -a flag, or in your netrc file.

The script will attempt to contact the registry service through the supplied URL and add the archive. By default, the archive is named after the basename of the archive path. For example, /home/data/intracellular would have the name intracellular. You can can override this behavior with the -n flag; however, the registry may only allow you to have one archive name for each path to avoid confusion. Then the script will create and initialize the archive under my-archive-path. You’ll get an error if the target directory already contains an nbank.json, README.md, or .gitignore file, or if the registry already has an archive with the same name.

By default, the archive is owned by you and your primary group, and its umask is 002. Use -g to give the archive a different group, and -u to set a different umask (for example, 027 to keep out users who aren’t in the group). If several users will deposit into the archive under their own accounts, add --shared. See Shared archives for how to set up and maintain shared archives.

Registering and storing resources

Before you start an experiment, register all the resources you plan to use. For example, let’s say you’re presenting a set of acoustic stimuli to an animal while recording neural responses. To register the stimuli:

nbank [-a user:pass] deposit [options] my-archive-path stimfile-1 stimfile-2 ...

Each stimulus will be given an identifier and moved to the archive. The command will output a JSON-encoded list of the resources that were deposited, including a mapping from the identifiers to the old filenames, if automatic identifiers were used.

The deposit command takes several options:

• -d, --dtype: specify the datatype for the deposited resources. Your registry may require this.

• -k: specify a metadata key-value pair. You can use this flag multiple times to set multiple fields.

• -H, --hash: if set, nbank will calculate a SHA1 hash of each file and store it in the registry. Use this if you expect the contents of the file to be unique.

• -A, --auto-id: if set, nbank will ask the registry to assign each file an automatically generated identifier, overriding the auto_identifiers policy if it is set to false.

• -j, --json-out: if set, the script will output info about each deposited file as line-deliminated JSON

Now run your experiment, making sure to record the identifiers of the stimuli. The short identifier suffices in most cases, but make sure you record the registry URL somewhere, too. If you’re running the experiment on machine that doesn’t have direct access to the archive, you can use the nbank fetch command on the experiment machine to retrieve the resources.

After the experiment, deposit the data files into the archive using the same command. If you deposit containers or directories, you’re responsible for organizing the contents and assigning any internal identifiers.

nbank [-r registry-url] dtype list

nbank [-a user:pass] [-r registry-url] dtype add dtype-name content-type

Accessing archived resources

The deposit command moves resource files to the archive under the resources directory, so you can always manually locate your files based on the identifier. Resources are sorted into subdirectories using the first two characters of the identifier to avoid having too many files in one directory. For example, if the identifier is edd0ccae-c34c-48cb-b515-a5e6f9ed91bc, you’ll find the file under resources/ed.

nbank also acts as a command-line interface to the registry. You can perform the following operations:

• nbank locate [options] id-1 [id-2 [id-3] ...]: look up the location(s) of the resources associated with each identifier. You can supply full URL-based identifiers, or short ids. If short ids are used, the default registry (specified with -r argument or NBANK_REGISTRY environment variable) is used to resolve the full URL. Use the -L flag to create symbolic links or the -0 flag to pipe the paths to another program. This should be your primary way of accessing resources. Don’t make unnecessary copies!

• nbank info id: returns the registry information on the resource in json format.

• nbank search [options] query: searches the database for resources that match query. The default is to search by identifier, but you can also search by hash, dtype, archive, or any metadata fields. The default is to return only the identifiers of the resources, but you can use the -j flag to output json instead, which is useful if you want to extract the registry records for further processing.

• nbank verify [options] files: computes a hash for each file and searches the registry for a match. Running this before starting an experiment is a good idea if you’re using copies (for example, on a data-collection machine), as you’ll be able to tell if any of your stimulus files have changed. It’s also useful if the same identifier is used in more than one domain or if you have a data file that was inadvertently renamed.

• nbank modify [-k key=value] id: update the metadata for id. Multiple -k flags can be used. You can’t change the hash or any other fields outside the metadata.

• nbank export [options] out id-1 [id-2 ...]: copy resources from the archives on this host to a tar file (out.tar), a zip file (out.zip), or a directory (anything else), for example to share them or put them in cold storage. Don’t use this for local copies (use nbank locate -L to create symbolic links instead). Each resource is checked against its registered hash as it’s copied, and any that don’t match or can’t be read are reported and left out. Use -f to read identifiers from a file, -a to read only from one archive, and --compress to compress the members of a zip file (they’re stored as-is by default, since most data files don’t compress well). Add --manifest to include the registry records of the exported resources (identifier, path, hash, datatype, metadata, and when and by whom they were created) as manifest.json.

Managing archives

You can check whether an archive contains all the files it’s supposed to by running nbank archive check <path_to_archive>. This command will compare each resource in the archive to its record in the registry and provide a summary of any resources missing from the archive and files that don’t have matches in the registry (which might indicate corrupted data).

To copy resources into another archive, run nbank copy <path_of_archive> id-1 [id-2 ...] on a host that can read both archives. Each resource is checked against its registered hash as it’s stored, and the new copy is added to the resource’s locations in the registry. Resources the registry already lists in the destination are skipped, so the same list can be run again after a failure. Use -f to read identifiers from a file, -a to read only from one archive, and -y to check the resources without copying them. To move resources, copy them and then remove them from the old archive with nbank archive prune.

Some resources, like raw extracellular data, can be moved to cold storage when they are no longer needed. The Meliza lab uses tape for this because of its long shelf life, low cost, and low environmental impact (no need for power). Moving resources to cold storage is a multi-step process:

• Identify the resources to archive, using lists of identifiers from project directories or nbank search.

• Copy the resources to a tar file with nbank export -f <list_of_identifiers> <name_of_tar_file>. Each resource is checked against its registered hash as it’s read, and any that don’t match or can’t be read are reported and left out, so the tar file only holds verified copies. Add -a <archive_name> to read only from that archive.

• Write the tar file to tape (or some other media), for example with dd if=<name_of_tar_file> of=/dev/nst0 bs=256k. Keep two copies, one offsite, for important data.

• Optional: check what was written with nbank check tar <tape_device_or_tar_file>, which reads every registered resource back and compares it to its registered hash. This command works on tar files, straight from the tape drive (e.g. /dev/nst0 after positioning the tape with mt fsf), or from standard input (-).

• Register what’s in the tape archive with neurobank by running nbank archive register-tar -n <name_of_archive> <name_of_tape> <tape_index> <tar_file>. This will create a record for the tape archive and update the records for the resources in the tar file. To avoid having to re-read the entire tape, this command works on the tar file.

• To remove the tape-archived resources from live storage, run nbank archive prune <live_archive_name> <list_of_identifiers>. This command will delete files from the local filesystem archive and update records for the resources. It will only do this for resources that have another location registered.

• To copy data back to live storage, run nbank archive import-tar <tar_file> <path_of_archive>. The tar file can be read straight from the tape drive (e.g. /dev/nst0 after positioning the tape with mt fsf) or from standard input (-), so it doesn’t need to be extracted first. Each resource is checked against its registered hash before it’s stored. To check a tape without importing anything, add -y. To import only some of the resources, list their identifiers after the archive path, or put them in a file and pass it with -f. This mode works much faster when working from a seekable file.

It’s a good idea to set up a cron job to check archive integrity every month or so with nbank check all. Data on tapes should be checked every year or two using nbank check tar. Add --archive <name_of_archive> to also confirm that every resource the registry thinks is on that tape is actually there. If the tape can’t be read past some point (for example, a damaged stretch), the check stops. Check the kernel log (dmesg) to see whether the drive reported a problem with the tape (Medium Error) or with itself (Hardware Error).

Development

Use uv for development.

Run uv sync to create a virtual environment and install dependencies. uv sync --no-dev --frozen for deployment.

Testing: uv run pytest runs the unit tests against a mock registry.

Integration tests run against a live registry (django-neurobank) that the tests start themselves, using a temporary sqlite database. They are skipped by default, and you need to install the integration dependency group, which uv sync doesn’t do by default:

uv run --group integration pytest -m integration

The integration tests also run on GitHub against Postgres (see .github/workflows/integration_tests.yml). To use Postgres locally, add --group postgres and set NBANK_TEST_DB=postgres and, if the defaults don’t apply, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST and POSTGRES_PORT. To use an existing registry, set NBANK_TEST_REGISTRY to its base URL and NBANK_TEST_AUTH to user:password for an account that can write.

Python interface

To be written

Best Practices

See docs/examples for some additional notes on how the Meliza Lab uses neurobank.

License

neurobank is licensed under the GNU Public License, version 2. That means you are free to use the code for anything you want, including a commerical work, but you have to provide the source code, including any modifications you make. You still own your data files and any associated metadata. See COPYING for more details.