PyPI version pre-commit DOI

Introduction

Full documentation can be found at http://datamaestro.rtfd.io

This projects aims at grouping utilities to deal with the numerous and heterogenous datasets present on the Web. It aims at being

  1. a reference for available resources, listing datasets
  2. a tool to automatically download and process resources (when freely available)
  3. integration with the experimaestro experiment manager.
  4. (planned) a tool that allows to copy data from one computer to another

Each datasets is uniquely identified by a qualified name such as com.lecun.mnist, which is usually the inversed path to the domain name of the website associated with the dataset.

The main repository only deals with very generic processing (downloading, basic pre-processing and data types). Plugins can then be registered that provide access to domain specific datasets.

List of repositories

Command line interface (CLI)

The command line interface allows to interact with the datasets. The commands are listed below, help can be found by typing datamaestro COMMAND --help:

  • search search dataset by name, tags and/or tasks
  • download download files (if accessible on Internet) or ask for download path otherwise
  • prepare download dataset files and outputs a JSON containing path and other dataset information
  • repositories list the available repositories
  • orphans list data directories that do no correspond to any registered dataset (and allows to clean them up)
  • create-dataset creates a dataset definition

Example (CLI)

Retrieve and download

The commmand line interface allows to download automatically the different resources. Datamaestro extensions can provide additional processing tools.

$ datamaestro search tag:image
[image] com.lecun.mnist

$ datamaestro prepare com.lecun.mnist
INFO:root:Materializing 4 resources
INFO:root:Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz into .../datamaestro/store/com/lecun/train_images.idx
INFO:root:Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz into .../datamaestro/store/com/lecun/test_images.idx
INFO:root:Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz into .../datamaestro/store/com/lecun/test_labels.idx

The previous command also returns a JSON on standard output

{
  "train": {
    "images": {
      "path": ".../data/image/com/lecun/mnist/train_images.idx"
    },
    "labels": {
      "path": ".../data/image/com/lecun/mnist/train_labels.idx"
    }
  },
  "test": {
    "images": {
      "path": ".../data/image/com/lecun/mnist/test_images.idx"
    },
    "labels": {
      "path": ".../data/image/com/lecun/mnist/test_labels.idx"
    }
  },
  "id": "com.lecun.mnist"
}

For those using Python, this is even better since the IDX format is supported

In [1]: from datamaestro import prepare_dataset
In [2]: ds = prepare_dataset("com.lecun.mnist")
In [3]: ds.train.images.data().dtype, ds.train.images.data().shape
Out[3]: (dtype('uint8'), (60000, 28, 28))

Python definition of datasets

Datasets are defined as Python classes with resource attributes that describe how to download and process data. The framework automatically builds a dependency graph and handles downloads with two-path safety and state tracking.

from datamaestro_image.data import ImageClassification, LabelledImages
from datamaestro.data.tensor import IDX
from datamaestro.download.single import FileDownloader
from datamaestro.definitions import Dataset, dataset


@dataset(url="http://yann.lecun.com/exdb/mnist/")
class MNIST(Dataset):
    """The MNIST database of handwritten digits."""

    TRAIN_IMAGES = FileDownloader(
        "train_images.idx",
        "http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz",
    )
    TRAIN_LABELS = FileDownloader(
        "train_labels.idx",
        "http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz",
    )
    TEST_IMAGES = FileDownloader(
        "test_images.idx",
        "http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz",
    )
    TEST_LABELS = FileDownloader(
        "test_labels.idx",
        "http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz",
    )

    def config(self) -> ImageClassification:
        return ImageClassification.C(
            train=LabelledImages(
                images=IDX(path=self.TRAIN_IMAGES.path),
                labels=IDX(path=self.TRAIN_LABELS.path),
            ),
            test=LabelledImages(
                images=IDX(path=self.TEST_IMAGES.path),
                labels=IDX(path=self.TEST_LABELS.path),
            ),
        )

Its syntax is described in the documentation.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for bpiwowar from gravatar.com bpiwowar

Credits

Author: Benjamin Piwowarski

License

GNU General Public License v3 (GPLv3)

Requires

Python >=3.10

Provides Extra

docs

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datamaestro-1.15.2.tar.gz (303.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

datamaestro-1.15.2-py3-none-any.whl (144.9 kB view details)

Uploaded Python 3

File details

Details for the file datamaestro-1.15.2.tar.gz.

File metadata

  • Download URL: datamaestro-1.15.2.tar.gz
  • Upload date:
  • Size: 303.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for datamaestro-1.15.2.tar.gz
Algorithm Hash digest
SHA256 2f43fe1a2e268d7b4daf79b0c5cef5a4379457f6fda78c069dbaf009144b5baa
MD5 32d6bf11748101e84542ac66657d7833
BLAKE2b-256 a67c51723d8a2dff520e3c29252a037a3776f635f41c8476fddff54175d7b5c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for datamaestro-1.15.2.tar.gz:

Publisher: python-publish.yml on experimaestro/datamaestro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file datamaestro-1.15.2-py3-none-any.whl.

File metadata

  • Download URL: datamaestro-1.15.2-py3-none-any.whl
  • Upload date:
  • Size: 144.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for datamaestro-1.15.2-py3-none-any.whl
Algorithm Hash digest
SHA256 ade4856e2b77298b1c1f796668074405182a5a769718d6fb3c770820fdcc8032
MD5 eb4fde972871ff1c6936a4e4fb403cfc
BLAKE2b-256 8e016b16679bf39dcc8b4f0a467d0da879ec812bf6a0a0d6c69886db545c70a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for datamaestro-1.15.2-py3-none-any.whl:

Publisher: python-publish.yml on experimaestro/datamaestro

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.15.2

1.15.1

1.15.0

1.14.0

1.13.0

1.12.0

1.11.3

1.11.2

1.11.0

1.10.4

1.10.3

1.10.2

1.10.1

1.10.0

1.9.10

1.9.9

1.9.8

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.2

1.6.1

1.6.0

1.5.2

1.5.1

1.5.0

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.2

1.3.1

1.3.0

1.2.1

1.2.0

1.1.0

1.0.6

1.0.5

1.0.4

1.0.3

1.0.2

1.0.1

1.0.0

0.8.16

0.8.15

0.8.14

0.8.13

0.8.12

0.8.11

0.8.10

0.8.9

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.1

0.8.0

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.24

0.6.23

0.6.22

0.6.21

0.6.20

0.6.19

0.6.18

0.6.17

0.6.16

0.6.15

0.6.13

0.6.12

0.6.11

0.6.10

0.6.9

0.6.8

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.8

0.5.7

0.5.2

0.5.0

0.2.8

0.2.5

0.2.4

0.2.3

0.2.2

0.2.1

0.2

0.1

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page