DataShift

DataShift is a dataset versioning and migration framework designed for Machine Learning workflows. Think of it as "Git for Data", allowing you to track changes, compare versions, and manage dataset lifecycles with ease.

Key Features

• Dataset Versioning: Snapshot datasets (CSV, Parquet) and track their evolution over time.
• Diffing: Compare two versions of a dataset to see added/removed rows and schema changes.
• Drift Detection: Guardrails to check for data drift between versions (e.g., row count changes, null distribution).
• Tags & Channels: specific versions with tags (e.g., #latest) or moving channels (e.g., :prod).
• Python API & CLI: Flexible usage through a command-line interface or directly within your Python code.
• Experiment Tracking: Link datasets to experiments for reproducibility.

Installation

From Source

pip install .

With Optional Dependencies

For PyTorch integration:

pip install .[torch]

For Parquet support:

pip install .[parquet]

For everything (including dev tools):

pip install .[all]

For Development

pip install -e .[dev]

Quick Start

CLI Usage

1. Initialize DataShift in your project directory:

   datashift init
   

2. Snapshot a Dataset:

   # Create a version of your customers data
   datashift snapshot ./data/customers.csv --name customers
   

3. List Datasets:

   datashift list
   

4. Show Dataset Details:

   datashift show customers
   

5. Compare Versions:

   # Compare version 1 and version 2
   datashift diff customers@v1 customers@v2
   

6. Checkout a Version:

   # Restore a specific version to a file
   datashift checkout customers@v1 ./restored_customers.csv
   

7. Drift Check (Guardrails):

   # Check if the new version deviates too much from the baseline
   datashift check customers@v2 --baseline customers@v1 --max-row-change 0.1
   

Python API Usage

import pandas as pd
from datashift import snapshot_dataset, load, diff_datasets, format_diff_summary

# 1. Snapshot a dataset
result = snapshot_dataset(dataset_name="metrics", source_path="metrics.csv")
print(f"Created version: {result.version}")

# 2. Load a specific version into a DataFrame
df = load("metrics#latest")
print(df.head())

# 3. Diff two versions
diff = diff_datasets("metrics@v1", "metrics@v2")
print(format_diff_summary(diff))

Development

1. Clone the repository.
2. Install dependencies:

   pip install -e .[dev]
   

3. Run tests:

   pytest
   

License

MIT License