chunkflow 0.2.3

Large Scale 3d Convolution Net Inference

chunkflow

Chunk operations for large scale 3D image dataset processing

Introduction

3D image dataset could be too large to be processed in a single computer, and distributed processing was required. In most cases, the image dataset could be choped to chunks and distributed to computers for processing. This package provide a framework to perform distributed chunk processing.

Features

• Decoupled frontend and backend. The computational heavy backend could be any computer with internet connection and Amazon Web Services (AWS) authentication.
• Composable Commandline interface. The chunk operators could be freely composed in commandline for flexible usage. This is also super useful for tests and experiments.

Usage

Installation

This package was registered in PyPi, just run a simple command to install:

pip install chunkflow

Get Help

chunkflow --help

get help for commands: chunkflow command --help

Examples

The commands could be composed and used flexiblly. The first command should be a generator though.

chunkflow create-chunk view
chunkflow create-chunk 

Some Typical Operators

• Convolutional Network Inference. Currently, we support PyTorch and pznet
• Task Generator. Fetch task from AWS SQS.
• Cutout service. Cutout chunk from datasets formatted as neuroglancer precomputed using cloudvolume
• Save. Save chunk to neuroglancer precomputed.
• Real File. Read image from hdf5 and tiff files.
• Upload Log. upload log information to storage.
• View. View chunk using cloudvolume viewer.
• Mask. Mask out the chunk using a precomputed dataset.
• Cloud Watch. Realtime speedometer using AWS CloudWatch.

Produce tasks to AWS SQS queue

in bin,

python produce_tasks.py --help

Terminology

• patch: the input/output 3D/4D array for convnet with typical size like 32x256x256.
• chunk: the input/output 3D/4D array after blending in each machine with typical size like 116x1216x1216.
• block: the final main output array of each machine which should be aligned with storage backend such as neuroglancer precomputed. The typical size is like 112x1152x1152.

Use specific GPU device

We can simply set an environment variable to use specific GPU device.

CUDA_VISIBLE_DEVICES=2 python consume_tasks.py

Development

Create a new release in PyPi

python setup.py bdist_wheel --universal
twine upload dist/my-new-wheel

Add a new operator

To be added.