tf-datachain 0.1.0

A local dataset loader based on tf.data input pipeline

tf-datachain

tf-datachain is a local dataset loader based on tf.data input pipeline. It handles the job of reading and encoding data direct in your disk and simplify the processing by providing several predefined methods.

Object Detection

from tf_datachain import ObjectDetection as od

Before using ObjectDetection functions, you have to define some basic information, like folder path and class name list.

od.imageFolder = "data/images"

# hard-code class names
od.classNames = ["class1", "class2", "class3"]
# or read them from csv file
import pandas as pd
od.classNames = pd.read_csv("class.csv", header=None).iloc[:,0].values.tolist()

Then, a ready-to-use tf.data input pipeline can be built within 3 steps:

• Preparation: prepare the list to process without reading content.
• Data Loading: load data from prepared list via tf.data.
• Augmentation: shuffle, batch, and resize.

The best practices to load dataset with different format are shown below.

Pascal VOC XML Format

from tf_datachain.utils import split

BATCH_SIZE = 4
# read .xml file within data/annotaions folder
# then split them with the ratio of 6:2:2
trainDataset, validationDataset, testDataset = split(od.prepareAnnotation("data/annotations", ".xml"), 6, 2, 2)

trainDataset = tf.data.Dataset.from_tensor_slices(trainDataset)
trainDataset = trainDataset.map(lambda data: od.loadData(data, "Pascal VOC XML", "xyxy"), num_parallel_calls=tf.data.AUTOTUNE)
# shuffle, ragged batch, and jittered resize
trainDataset = od.datasetProcessing(trainDataset, BATCH_SIZE, "Jittered Resize", (960, 960), "xyxy")
trainDataset = trainDataset.prefetch(tf.data.AUTOTUNE)

Visualize Dataset

# visualize single data
for data in dataset.take(1):
  visualizeData(data, "xyxy")

# visualize dataset shown in 2x2 grid
visualizeDataset(dataset, "xyxy", rows=2, cols=2)