getdaft 0.0.17

A Distributed DataFrame library for large scale complex data processing.

Website • Docs • Installation • 10-minute tour of Daft • Community and Support

Daft: the distributed Python dataframe for media data

Daft is a fast, Pythonic and scalable open-source dataframe library built for Python and Machine Learning workloads.

Daft is currently in its Alpha release phase - please expect bugs and rapid improvements to the project. We welcome user feedback/feature requests in our Discussions forums

Table of Contents

• About Daft

• Getting Started

• License

About Daft

The Daft dataframe is a table of data with rows and columns. Columns can contain any Python objects, which allows Daft to support rich media data types such as images, audio, video and more.

1. Any Data: Columns can contain any Python objects, which means that the Python libraries you already use for running machine learning or custom data processing will work natively with Daft!

2. Notebook Computing: Daft is built for the interactive developer experience on a notebook - intelligent caching/query optimizations accelerates your experimentation and data exploration.

3. Distributed Computing: Rich media formats such as images can quickly outgrow your local laptop’s computational resources - Daft integrates natively with Ray for running dataframes on large clusters of machines with thousands of CPUs/GPUs.

Getting Started

Installation

Install Daft with pip install getdaft.

Quickstart

Check out our full quickstart tutorial!

Load a dataframe - in this example we load the MNIST dataset from a JSON file, but Daft also supports many other formats such as CSV, Parquet and folders/buckets of files.

from daft import DataFrame

URL = "https://github.com/Eventual-Inc/mnist-json/raw/master/mnist_handwritten_test.json.gz"

df = DataFrame.from_json(URL)
df.show(4)

Filter the dataframe for rows where the "label" column is equal to 5

df = df.where(df["label"] == 5)
df.show(4)

Run any function on the dataframe (here we convert a list of pixels into an image using Numpy and the Pillow libraries)

import numpy as np
from PIL import Image

def image_from_pixel_list(pixels: list) -> Image.Image:
    arr = np.array(pixels).astype(np.uint8)
    return Image.fromarray(arr.reshape(28, 28))

df = df.with_column(
    "image_pil",
    df["image"].apply(image_from_pixel_list),
)
df.show(4)

More Resources

• 10-minute tour of Daft - learn more about Daft’s full range of capabilities including dataloading from URLs, joins, user-defined functions (UDF), groupby, aggregations and more.

• User Guide - take a deep-dive into each topic within Daft

• API Reference - API reference for public classes/functions of Daft

License

Daft has an Apache 2.0 license - please see the LICENSE file.