morphology-workflows 0.4.2

Workflows used for morphology processing.

Morphology Workflows

This project contains several workflows for processing morphologies:

• Fetch: download morphologies from online database (like Allen, NeuroMorpho or MouseLight).
• Placeholders: compute the place holders for a given region and mtype set.
• Curate: from raw morphologies, ensures that morphologies can be used with the rest of BBP codes.
• Annotate: create various annotations on morphologies needed by specific BBP codes.
• Repair: process morphologies to correct for artifacts of in-vitro reconstruction.

In a nutshell, the user provides a list of morphologies in a .csv file, with their names and paths and a luigi.cfg configuration file. Each workflow is run independently and creates an output folder, with one subfolder per task. In each, there will be a report.csv and a data folder containing the output files of the task if any. In the report.csv file, columns contain paths to these files, additional information, error messages if the task failed on that morphologies, as well as a flag is_valid, used in subsequent tasks to filter valid morphologies. At the end of each workflow, another report.csv file is created, with the main output columns of each tasks, and a report.pdf containing a human readable summary of the result of the workflow.

Usually, the user should run the Curate workflow, then the Annotate workflow and finally the Repair workflow.

The complete documentation can be found here:

• stable: https://morphology-workflows.readthedocs.io/en/stable/
• latest: https://morphology-workflows.readthedocs.io/en/latest/

Installation

This should be installed using pip:

pip install morphology-workflows

Usage

Create inputs for the Fetch workflow

This workflow helps fetching morphologies from online databases. This workflow only needs a configuration file, which depends on the source from which the morphologies are fetched.

The possible sources are:

• NeuroMorpho
• MouseLight
• Allen

For each of them, the configuration file should be a JSON file containing a list of objects like the following examples:

• NeuroMorpho:

  [
      {
          "species": "mouse",
          "brain_region": "neocortex",
          "cell_type": "interneuron",
          "nb_morphologies": 10
      }
  ]
  

• MouseLight:

  [
      {
          "brain_region": "neocortex",
          "nb_morphologies": 10,
          "seed": 0
      }
  ]
  

• Allen:

  [
      {
          "species": "Mus musculus",
          "brain_region": "VISli",
          "cell_type": "pyramidal",
          "nb_morphologies": 10,
          "seed": 0
      }
  ]
  

In these examples, the seed attribute is optional and is only used to sample which morphologies are fetched among those which pass the filter.

Each JSON object in the list will give a set of morphologies to fetch, depending on the given filters. Note that all attributes are optional, so it's possible to pass an empty object to fetch all the morphologies from the database, though it is not recommended.

Create inputs for the Curate, Annotate and Repair workflow

The Annotate and Repair workflows should usually be run after the Curate workflow since their inputs should be the outputs of the Curate workflow. But it is still possible to run them on arbitrary inputs (though the morphologies must be valid, as the ones processed by the Curate workflow).

The inputs should consist in:

• a directory containing the input morphologies.
• a CSV file with the following columns:
  1. morph_path: the path to the morphology file.
  2. morph_name: the name of the morphology.
  3. any other column is kept into the results but not used in the workflows.
• a luigi.cfg file containing the configuration for all the tasks of the workflow.
• an optional logging.conf file containing the logging configuration. If you prefer default logging behavior, remove this file and comment line in logging_conf_file = logging.conf in luigi.cfg.

The examples folder contains examples for the luigi.cfg and logging.conf files.

Run the workflows

These workflows are based on the luigi library but can be run via the command line interface. For example, you can run the Curate workflow with the following command:

morphology_workflows Curate

NOTE: This command must be executed from a directory containing a luigi.cfg file. An example of such file is given in the examples directory.

By default, a local scheduler is used but it is also possible to use a Luigi's master scheduler using the -m / --master-scheduler trigger:

morphology_workflows -m Curate

More details can be found in the command line interface section of the documentation or by running the command:

morphology_workflows <workflow> --help

Examples

The examples folder contains a simple example that will fetch and process a set of morphologies. A dataset.csv file is provided which is taken as input for the workflows. A luigi.cfg file is also provided to give a default configuration for the workflows. This example can simply be run using the following command:

./run_curation.sh

This script will create a new directory out_curated which will contain the report and all the results.

Funding & Acknowledgment

The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.

For license and authors, see LICENSE.txt and AUTHORS.md respectively.

Copyright © 2021-2022 Blue Brain Project/EPFL