melon 0.0.6

Lightweight package meant to simplify data processing for Deep Learning

Melon

Melon is a lightweight package meant to simplify data processing for Deep Learning.

It removes the need for boilerplate code to pre-process the data prior to (model) training, testing and inference.

It aims at standardizing data serialization and manipulation approaches.

The default formats align with the requirements by frameworks such as Tensorflow / PyTorch.

The tool also provides various level of customizations depending on the use-case.

Installation

Install and update using pip:

$ pip install melon

Supported in Python >=3.4.0

Examples

Images

With default options:

from melon import ImageReader

def train():
    source_dir = "resources/images"
    reader = ImageReader(source_dir)
    X, Y = reader.read()
    ...
    with tf.Session() as s:
        s.run(..., feed_dict = {X_placeholder: X, Y_placeholder: Y})

source_dir directory should contain images that need to be read. See tests/resources/images/sample for a sample directory. In the sample directory there is an optional labels.txt file that is described in Labeling.

---

Since number of images may be too large to fit into memory the tool supports batch-processing.

from melon import ImageReader

def train():
    source_dir = "resources/images"
    options = { "batch_size": 32 }
    reader = ImageReader(source_dir, options)
    while reader.has_next():
        X, Y = reader.read()
        ...

This reads images in the batches of 32 until all images are read. If batch_size is not specified then reader.read() will read all images.

---

With custom options:

from melon import ImageReader

def train():
    source_dir = "resources/images"
    options = { "data_format": "channels_last", "normalize": False }
    reader = ImageReader(source_dir, options)
    ...

This changes format of data to channels-last (each sample will be Height x Width x Channel) and doesn’t normalize the data. See options for available options.

Options

Images

width

Width of the output (pixels). default: 255

height

Height of the output (pixels). default: 255

batch_size

Batch size of each read. default: All images in a directory

data_format

Format of the images data

channels_first - Channel x Height x Width (default)

channels_last - Height x Width x Channel

label_format

Format of the labels data

one_hot - as a matrix, with one-hot vector per image (default)

label - as a vector, with a single label per image

normalize

Normalize data. default: True

num_threads - number of threads for parallel processing

default: Number of cores of the machine

Labeling

In supervised learning each image needs to be mapped to a label.

While the tool supports reading images without labels (e.g. for inference) it also provides a way to label them.

---

Generating labels file

To generate labels file use the following command:

$ melon generate
> Source dir:

After providing source directory the tool will generate labels file in that directory with blank labels.

Final step is to add a label to each row in the generated file.

For reference see tests/resources/images/sample/labels.txt:

#legend
pedestrian:0
cat:1
parrot:2
car:3
apple tree:4

#map
img275.jpg:1
img324.jpg:2
img551.jpg:3
img928.jpg:1
img999.png:0
img736.png:4

#legend section is optional but #map section is required to map a label to an image.

---

Format of the labels

Label’s format can be specified in Custom options. It defaults to one-hot format.

Roadmap

• Support for video data

• Support for textual data