Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

A small wrapper around quilt3 to make distributing manifest style datasets even easier.

Project description


Build Status Documentation Code Coverage
dataset packaging and distribution

People commonly work with tabular datasets, people want to share their data, this makes that easier through Quilt3.


  • Automatically determines which files to upload based off CSV headers. (Explicit override available)
  • Simple interface for attaching metadata to each file based off the manifest contents.
  • Groups metadata for files that are referenced multiple times.
  • Validates and runs basic cleaning operations on your dataset manifest CSV.
  • Optionally add license details and usage instructions to your dataset README.
  • Parses README for any referenced files and packages them up as well.
  • Support for adding extra files not contained in the manifest.
  • Constructs an "associates" map that is placed into each files metadata for quick navigation around the package.
  • Enforces that the metadata attached to each file is standardized across the package for each file column.

Quick Start

Construct a csv (or pandas dataframe) dataset manifest (Example):

CellId Structure 2dReadPath 3dReadPath
1 lysosome 2d/1.png 3d/1.tiff
2 laminb1 2d/2.png 3d/2.tiff
3 golgi 2d/3.png 3d/3.tiff
4 myosin 2d/4.png 3d/4.tiff
from quilt3distribute import Dataset

# Create the dataset
ds = Dataset(

# Optionally add common additional requirements

# Optionally indicate column values to use for file metadata
ds.set_metadata_columns(["CellId", "Structure"])

# Optionally rename the columns on the package level
    "2dReadPath": "images_2d",
    "3dReadPath": "images_3d"

# Distribute
pkg = ds.distribute(push_uri="s3://quilt-jacksonb", message="Initial dataset example")


(remote Package)

Example Metadata:

    "CellId": 1,
    "Structure": "lysosome",
    "associates": {
        "images_2d": "images_2d/03cdf019_1.png",
        "images_3d": "images_3d/a0ce6e01_1.tiff"


Stable Release: pip install quilt3distribute
Development Head: pip install git+


This package was created with Cookiecutter. Original repository

Free software: Allen Institute Software License

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for quilt3distribute, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size quilt3distribute-0.1.2-py2.py3-none-any.whl (21.8 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size quilt3distribute-0.1.2.tar.gz (24.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page