Skip to main content

"Dataset management command line and API"

Project description

PyPI version pre-commit DOI


Full documentation can be found at

This projects aims at grouping utilities to deal with the numerous and heterogenous datasets present on the Web. It aims at being

  1. a reference for available resources, listing datasets
  2. a tool to automatically download and process resources (when freely available)
  3. integration with the experimaestro experiment manager.
  4. (planned) a tool that allows to copy data from one computer to another

Each datasets is uniquely identified by a qualified name such as com.lecun.mnist, which is usually the inversed path to the domain name of the website associated with the dataset.

The main repository only deals with very generic processing (downloading, basic pre-processing and data types). Plugins can then be registered that provide access to domain specific datasets.

List of repositories

Command line interface (CLI)

The command line interface allows to interact with the datasets. The commands are listed below, help can be found by typing datamaestro COMMAND --help:

  • search search dataset by name, tags and/or tasks
  • download download files (if accessible on Internet) or ask for download path otherwise
  • prepare download dataset files and outputs a JSON containing path and other dataset information
  • repositories list the available repositories
  • orphans list data directories that do no correspond to any registered dataset (and allows to clean them up)
  • create-dataset creates a dataset definition

Example (CLI)

Retrieve and download

The commmand line interface allows to download automatically the different resources. Datamaestro extensions can provide additional processing tools.

$ datamaestro search tag:image
[image] com.lecun.mnist

$ datamaestro prepare com.lecun.mnist
INFO:root:Downloading into /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-labels-idx1-ubyte
INFO:root:Transforming file
INFO:root:Created file /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-labels-idx1-ubyte
INFO:root:Downloading into /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-images-idx3-ubyte
INFO:root:Transforming file
INFO:root:Created file /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/t10k-images-idx3-ubyte
INFO:root:Downloading into /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-labels-idx1-ubyte
Downloading 32.8kB [00:00, 92.1kB/s]                                                            INFO:root:Transforming file
INFO:root:Created file /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-labels-idx1-ubyte
INFO:root:Downloading into /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-images-idx3-ubyte
Downloading 9.92MB [00:00, 10.6MB/s]
INFO:root:Transforming file
INFO:root:Created file /home/bpiwowar/datamaestro/data/image/com/lecun/mnist/train-images-idx3-ubyte

The previous command also returns a JSON on standard output

  "train": {
    "images": {
      "path": ".../data/image/com/lecun/mnist/train_images.idx"
    "labels": {
      "path": ".../data/image/com/lecun/mnist/train_labels.idx"
  "test": {
    "images": {
      "path": ".../data/image/com/lecun/mnist/test_images.idx"
    "labels": {
      "path": ".../data/image/com/lecun/mnist/test_labels.idx"
  "id": "com.lecun.mnist"

For those using Python, this is even better since the IDX format is supported

In [1]: from datamaestro import prepare_dataset
In [2]: ds = prepare_dataset("com.lecun.mnist")
In [3]:,
Out[3]: (dtype('uint8'), (60000, 28, 28))

Python definition of datasets

Each dataset (or a set of related datasets) is described in Python using a mix of declarative and imperative statements. This allows to quickly define how to download dataset using the datamaestro declarative API; the imperative part is used when creating the JSON output, and is integrated with experimaestro.

Its syntax is described in the documentation.

For MNIST, this corresponds to.

from import ImageClassification, LabelledImages, Base, IDXImage
from import filedownloader
from datamaestro.definitions import  argument, datatasks, datatags, dataset
from import IDX

@filedownloader("train_images.idx", "")
@filedownloader("train_labels.idx", "")
@filedownloader("test_images.idx", "")
@filedownloader("test_labels.idx", "")
def MNIST(train_images, train_labels, test_images, test_labels):
  """The MNIST database

  The MNIST database of handwritten digits, available from this page, has a
  training set of 60,000 examples, and a test set of 10,000 examples. It is a
  subset of a larger set available from NIST. The digits have been
  size-normalized and centered in a fixed-size image.
  return {
    "train": LabelledImages(
    "test": LabelledImages(


  • Integration with other repositories: abstracting away the notion of dataset
  • Repository prefix
  • Set sub-datasets IDs automatically


  • Updates for new experimaestro (0.8.5)
  • Search types with "type:..."


  • Allow remote access through rpyc


version command

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datamaestro-0.8.16.tar.gz (56.5 kB view hashes)

Uploaded source

Built Distribution

datamaestro-0.8.16-py3-none-any.whl (56.5 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page