Meerkat is building new data abstractions to make machine learning easier.
Project description
Meerkat provides fast and flexible data structures for working with complex machine learning datasets.
Getting Started | What is Meerkat? | Docs | Contributing | Blogpost | About
⚡️ Quickstart
pip install meerkat-ml
Optional: some parts of Meerkat rely on optional dependencies. If you know which optional dependencies you'd like to install, you can do so using something like
pip install meerkat-ml[dev,text]
instead. Seesetup.py
for a full list of optional dependencies.
Installing from dev:
pip install "meerkat-ml[text] @ git+https://github.com/robustness-gym/meerkat@dev"
Load a dataset into a DataPanel
and get going!
import meerkat as mk
from meerkat.contrib.imagenette import download_imagenette
download_imagenette(".")
dp = mk.DataPanel.from_csv("imagenette2-160/imagenette.csv")
dp["img"] = mk.ImageColumn.from_filepaths(dp["img_path"])
dp[["label", "split", "img"]].lz[:3]
To learn more, continue following along in our tutorial:
💡 What is Meerkat?
Meerkat makes it easier for ML practitioners to interact with high-dimensional, multi-modal data. It provides simple abstractions for data inspection, model evaluation and model training supported by efficient and robust IO under the hood.
Meerkat's core contribution is the DataPanel
, a simple columnar data abstraction. The Meerkat DataPanel
can house columns of arbitrary type – from integers and strings to complex, high-dimensional objects like videos, images, medical volumes and graphs.
DataPanel
loads high-dimensional data lazily. A full high-dimensional dataset won't typically fit in memory. Behind the scenes, DataPanel
handles this by only materializing these objects when they are needed.
import meerkat as mk
# Images are NOT read from disk at DataPanel creation...
dp = mk.DataPanel({
'text': ['The quick brown fox.', 'Jumped over.', 'The lazy dog.'],
'image': mk.ImageColumn.from_filepaths(['fox.png', 'jump.png', 'dog.png']),
'label': [0, 1, 0]
})
# ...only at this point is "fox.png" read from disk
dp["image"][0]
DataPanel
supports advanced indexing. Using indexing patterns similar to those of Pandas and NumPy, we can access a subset of a DataPanel
's rows and columns.
import meerkat as mk
dp = ... # create DataPanel
# Pull a column out of the DataPanel
new_col: mk.ImageColumn = dp["image"]
# Create a new DataPanel from a subset of the columns in an existing one
new_dp: mk.DataPanel = dp[["image", "label"]]
# Create a new DataPanel from a subset of the rows in an existing one
new_dp: mk.DataPanel = dp[10:20]
new_dp: mk.DataPanel = dp[np.array([0,2,4,8])]
# Pull a column out of the DataPanel and get a subset of its rows
new_col: mk.ImageColumn = dp["image"][10:20]
DataPanel
supports map
, update
and filter
operations. When training and evaluating our models, we often perform operations on each example in our dataset (e.g. compute a model's prediction on each example, tokenize each sentence, compute a model's embedding for each example) and store them . The DataPanel
makes it easy to perform these operations and produce new columns (via DataPanel.map
), store the columns alongside the original data (via DataPanel.update
), and extract an important subset of the datset (via DataPanel.filter
). Under the hood, dataloading is multiprocessed so that costly I/O doesn't bottleneck our computation. Consider the example below where we use update a DataPanel
with two new columns holding model predictions and probabilities.
# A simple evaluation loop using Meerkat
dp: DataPanel = ... # get DataPanel
model: nn.Module = ... # get the model
model.to(0).eval() # prepare the model for evaluation
@torch.no_grad()
def predict(batch: dict):
probs = torch.softmax(model(batch["input"].to(0)), dim=-1)
return {"probs": probs.cpu(), "pred": probs.cpu().argmax(dim=-1)}
# updated_dp has two new `TensorColumn`s: 1 for probabilities and one
# for predictions
updated_dp: mk.DataPanel = dp.update(function=predict, batch_size=128, is_batched_fn=True)
✉️ About
Meerkat is being developed at Stanford's Hazy Research Lab. Please reach out to kgoel [at] cs [dot] stanford [dot] edu
if you would like to use or contribute to Meerkat.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for meerkat_ml-0.2.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78610f105ac8226eed2fe125de186a2c593af348c9fc0dfb57d3dd5e7adcf743 |
|
MD5 | f21464a7ba02a7fa23e9a8d568b3d51c |
|
BLAKE2b-256 | 05b8e6a75c99d74db4aba3ea2dc1915d75f08b0b1993249aef88e99ff0e01e41 |