dandi 0.7.0

Command line client for interaction with DANDI archive elements

DANDI Client

This project is under heavy development. Beware of hidden and disclosed issues and Work-in-Progress (WiP) (again might be hidden or public).

Installation

At the moment DANDI client releases are available from PyPI. You could install them in your Python (native, virtualenv, or conda) environment via

pip install dandi

dandi tool

This package provides a dandi command line utility with a basic interface which should assist you in preparing and uploading your data to and/or obtaining data from the http://dandiarchive.org:

$> dandi
Usage: dandi [OPTIONS] COMMAND [ARGS]...

  A client to support interactions with DANDI archive
  (http://dandiarchive.org).

  To see help for a specific command, run

      dandi COMMAND --help

  e.g. dandi upload --help

Options:
  --version
  -l, --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level name  [default: INFO]
  --pdb                           Fall into pdb if errors out
  --help                          Show this message and exit.

Commands:
  download  Download a file or entire folder from DANDI
  ls        List .nwb files and dandisets metadata.
  organize  (Re)organize files according to the metadata.
  register  Register a new dandiset in the DANDI archive
  upload    Upload dandiset (files) to DANDI archive.
  validate  Validate files for NWB (and DANDI) compliance.

Each of the commands has a set of options to alter their behavior. Please run dandi COMMAND --help to get more information, e.g.

$> dandi ls --help
Usage: dandi ls [OPTIONS] [PATHS]...

  List .nwb files metadata

Options:
  -F, --fields TEXT               Comma-separated list of fields to display.
                                  An empty value to trigger a list of
                                  available fields to be printed out
  -f, --format [auto|pyout|json|json_pp|yaml]
                                  Choose the format/frontend for output. If
                                  'auto', 'pyout' will be used in case of
                                  multiple files, and 'yaml' for a single
                                  file.
  --help                          Show this message and exit.

Preparing and uploading a dandiset to dandiarchive.org

Although some functionality and final interface is still Work-in-Progress (WiP), overall tentative target workflow will be:

1. Collect or convert your data files to NWB format. Files should have .nwb file extension.

2. Use dandi validate to verify that files conform NWB schema and DANDI requirements (WiP) on contained in them metadata. If necessary, adjust your conversion scripts or use helper utilities to address concerns identified by dandi validate command.

3. Use dandi organize to

   • re-layout (move, rename) your files into a consistent naming convention
   • generate a template dataset.yaml with some fields pre-populated from metadata extracted from the .nwb files.

   1. If file names for some files could not be disambiguated, possibly see in providing additional metadata within .nwb files so they could be named without collisions, or file an issue describing your case. dandi ls command could come useful to quickly view metadata we consider.

   2. Fill out missing fields marked REQUIRED in the dataset.yaml, remove templated RECOMMENDED or OPTIONAL.

   Result of the reorganization is a dandiset -- a dataset with consistent layout, and dataset level metadata.

4. Rerun dandi validate on the entire dandiset to assure that everything is correct.

5. Use dandi register to register a new dataset ID on DANDI archive. If you run it within a dandiset, its dandiset.yaml will be automatically updated to contain new dandiset identifier.

6. Use dandi upload to upload your dandiset to the archive "drafts" collection.

If you change anything in your dandiset and decide to update its state in the archive, just use dandi upload again.

You could also visit doc/demos/basic-workflow1.sh for an example script which does all above actions (assuming no changes to files are necessary).

Downloading dandiset from the archive

dandi download can be used to download full dandisets or individual files or folders from the archive.

Using --existing refresh option available for dandi upload and dandi download it is possible to avoid transfer if files are identical locally and in the archive.

Warning: There is no version control tracking beyond checking correspondence of file size and modification time. So in collaborative setting it is possible to "refresh" a file which was modified locally with a version from the archive, or vise versa.

Development/contributing

Please see DEVELOPMENT.md file.

3rd party components included

dandi/support/generatorify.py

From https://github.com/eric-wieser/generatorify, as of 7bd759ecf88f836ece6cdbcf7ce1074260c0c5ef Copyright (c) 2019 Eric Wieser, MIT/Expat licensed.

dandi/tests/skip.py

From https://github.com/ReproNim/reproman, as of v0.2.1-40-gf4f026d Copyright (c) 2016-2020 ReproMan Team