datamole 0.1.0

Dataset versioning and management for ML projects.

datamole

Simple data versioning for ML projects. Track, version, and share your datasets with minimal overhead.

Features

• 🚀 Simple CLI interface (dtm command)
• 📦 Version datasets with automatic hashing
• 🏷️ Tag versions for easy reference
• 🔍 Smart lookup: pull by hash, prefix, or tag
• 💾 Multiple storage backends (local, GCS, S3, Azure)
• 🔒 Transaction-safe uploads
• 🤝 Collaboration-friendly with shared storage

Installation

pip install datamole

After installation, the dtm command will be available globally.

Quick Start

# Configure storage backend (one-time setup)
dtm config --backend local --remote-uri /path/to/shared/storage

# Initialize in your project
cd my-ml-project
dtm init

# Add your data and create a version
dtm add-version -m "Initial dataset" -t v1.0

# List versions
dtm list-versions

# Pull a specific version (by tag, hash, or prefix)
dtm pull v1.0
dtm pull abc123  # by hash prefix
dtm pull latest  # pull current version

CLI Commands

Setup & Configuration

# Configure storage backend
dtm config --backend local --remote-uri /path/to/storage

# Initialize project
dtm init [--data-dir data] [--backend local] [--no-pull]

Version Management

# Create a new version
dtm add-version [-m "message"] [-t tag-name]

# Pull a version
dtm pull [version] [-f]

# List all versions
dtm list-versions

# Show current version
dtm current-version

Python API

from datamole.core import DataMole

# Initialize
dtm = DataMole()
dtm.init(data_dir="data", backend="local")

# Create versions
dtm.add_version(message="Initial dataset", tag="v1.0")

# Pull versions
dtm.pull("v1.0")
dtm.pull("abc123")  # by hash prefix
dtm.pull()  # pull current version

Storage Backends

• local: Local filesystem storage
• gcs: Google Cloud Storage (coming soon)
• s3: AWS S3 (coming soon)
• azure: Azure Blob Storage (coming soon)

Development

# Clone repository
git clone https://github.com/yourusername/datamole.git
cd datamole

# Install in development mode
uv pip install -e ".[dev]"

# Run tests
pytest

License

MIT