Skip to main content

A small wrapper around quilt3 to make distributing manifest style datasets even easier.

Project description


Build Status Documentation Code Coverage DOI

dataset packaging and distribution

People commonly work with tabular datasets, people want to share their data, this makes that easier through Quilt3.


  • Automatically determines which files to upload based off CSV headers. (Explicit override available)
  • Simple interface for attaching metadata to each file based off the manifest contents.
  • Groups metadata for files that are referenced multiple times.
  • Validates and runs basic cleaning operations on your dataset manifest CSV.
  • Optionally add license details and usage instructions to your dataset README.
  • Parses README for any referenced files and packages them up as well.
  • Support for adding extra files not contained in the manifest.
  • Constructs an "associates" map that is placed into each files metadata for quick navigation around the package.
  • Enforces that the metadata attached to each file is standardized across the package for each file column.

Quick Start

Construct a csv (or pandas dataframe) dataset manifest (Example):

CellId Structure 2dReadPath 3dReadPath
1 lysosome 2d/1.png 3d/1.tiff
2 laminb1 2d/2.png 3d/2.tiff
3 golgi 2d/3.png 3d/3.tiff
4 myosin 2d/4.png 3d/4.tiff
from quilt3distribute import Dataset

# Create the dataset
ds = Dataset(

# Optionally add common additional requirements

# Optionally indicate column values to use for file metadata
ds.set_metadata_columns(["CellId", "Structure"])

# Optionally rename the columns on the package level
    "2dReadPath": "images_2d",
    "3dReadPath": "images_3d"

# Distribute
pkg = ds.distribute(push_uri="s3://quilt-jacksonb", message="Initial dataset example")


(remote Package)

Example Metadata:

    "CellId": 1,
    "Structure": "lysosome",
    "associates": {
        "images_2d": "images_2d/03cdf019_1.png",
        "images_3d": "images_3d/a0ce6e01_1.tiff"


Stable Release: pip install quilt3distribute
Development Head: pip install git+


This package was created with Cookiecutter. Original repository

Free software: Allen Institute Software License

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

quilt3distribute-0.1.4.tar.gz (24.7 kB view hashes)

Uploaded source

Built Distribution

quilt3distribute-0.1.4-py2.py3-none-any.whl (21.8 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page