Skip to main content

Data Pipeline 101

https://img.shields.io/pypi/v/pipelime.svg https://img.shields.io/travis/eyecan-ai/pipelime.svg Documentation Status Updates

Installation

pip install pipelime

Basic Usage

Underfolder Format

The Underfolder format is one of the pipelime dataset formats: i.e. a flexible way to (model and) store a generic dataset through filesystem.

underfolder structure

An Underfolder dataset is a collection of samples. A sample is a collection of items. An item is a unitary block of data, it can be a generic tensor (e.g. a multi-channel image, or a plain matrix), a dictionary. or more.

Underfolder datasets must contain a subfolder named data that will actually contain the samples and items. Optionally you can store the items in the root folder directly, they will act as “global” items injected into each sample.

naming convention

Items are named using the following naming convention:

$ID_$ITEM.$EXT

Where:

  • $ID is the sample identifier, must be a unique string for each sample.

  • ITEM is the item name.

  • EXT is the item extension. Currently supported extensions are:

    • The most common image formats like PNG, JPEG, BMP, and many others…

    • YAML and JSON for dictionary-like objects.

    • TXT for numpy 2D matrix notation.

    • NPY and NPZ for numpy arrays.

    • PKL for generic pickable python objects.

Root files follow the same convention but they lack the sample identifier part:

$ITEM.$EXT

Reading an Underfolder Dataset

Pipelime provides an intuitive interface to read, manipulate and write Underfolder Datasets. You don’t have to memorize complex signatures, instantiate weird object iterators, or write tens of lines of boilerplate code. It all boils down to a reader, a writer and objects that behave like built-in python types such as lists and dictionaries.

from pipelime.sequences.readers.filesystem import UnderfolderReader

# Read an underfolder dataset with a single line of code
dataset = UnderfolderReader('tests/sample_data/datasets/underfolder_minimnist')

# A dataset behaves like a Sequence
len(dataset) # The number of samples (20)
sample = dataset[4] # Get a Sample from the dataset

# A Sample is a MutableMapping
len(sample) # The number of items (10)
set(sample.keys()) # The set of all the item names {'cfg', 'image', 'image_mask', ...}
item = sample['image'] # Get an item from the sample

# An item can be any python object, depending on which extension is used to store it.
type(item) # numpy.ndarray
item.shape # (28, 28, 3)

Writing an Underfolder Dataset

You can write a dataset by simply creating and running a writer object.

from pipelime.sequences.writers.filesystem import UnderfolderWriter

# Create the writer object from a destination path
writer = UnderfolderWriter('/tmp/my_output_dataset')
# Write the dataset to file system
writer(dataset)

By default, UnderfolderWriter saves every sample with the extension it was originally read with. If for any reason it is unable to retrieve the original extension, it will use pickle to serialize the object.

If you don’t want to use pickle, you can choose a custom extension for each item name. You can also choose which items are going to be saved as root files (if the contained data is the same for all samples).

from pipelime.sequences.writers.filesystem import UnderfolderWriter

# These items are going to be saved as root files
root_files = ['cfg', 'numbers', 'pose']

# Associate a custom extension to each item name
extensions = {
        'image': 'jpg',
        'image_mask': 'png',
        'image_maskinv': 'png',
        'label': 'txt',
        'metadata': 'json',
        'metadatay': 'yml',
        'points': 'txt',
        'numbers': 'txt',
        'pose': 'txt',
        'cfg': 'yml'
}

# Create a customized writer object
writer = UnderfolderWriter(
        '/tmp/my_output_dataset',
        root_files_keys=root_files,
        extensions_map=extensions,
)
# Write the dataset to file system
writer(dataset)

Metadata

Release files for pipelime 0.1.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pipelime 0.1.8
File Size Uploaded Provenance Actions
pipelime-0.1.8.tar.gz 237.7 kB Without provenance Details for pipelime-0.1.8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pipelime 0.1.8
File Interpreter ABI Platform Actions
pipelime-0.1.8-py2.py3-none-any.whl Python 3, Python 2 none any Details for pipelime-0.1.8-py2.py3-none-any.whl

Total release size: 363.5 kB

Release files / pipelime-0.1.8.tar.gz

Download URL pipelime-0.1.8.tar.gz
Size 237.7 kB
Tags Source
SHA-256 checksum
How to use checksums
fe7feb1df3634869f09436f51335f24d462b59bda68a6cf6a6db779d84e08a14
BLAKE2b-256 checksum
How to use checksums
6cbd27f12472c12fa3a21d3fb3d028f6abde6476580dbf092f692e2f7a1d5add
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.0 CPython/3.9.13

Release files / pipelime-0.1.8-py2.py3-none-any.whl

Download URL pipelime-0.1.8-py2.py3-none-any.whl
Size 125.8 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
22cbab7a5bf88f03031e6de56b97302a8cc8a3de71a69b45cf5ba3f1cb36739b
BLAKE2b-256 checksum
How to use checksums
7fbc4d02204b667f52c2ae32fb0de2b592d0dcfb32ea3db0ab41a1ff07e57701
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.0 CPython/3.9.13

Security for pipelime 0.1.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page