silver-data

Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.

The base install uses only the Python standard library for records, JSON, JSONL, and CSV. Add pandas only when you need DataFrame conversion:

pip install 'silver-data[pandas]'

Installation

pip install silver-data

Quick Start

from silver_data import Dataset

# Load from CSV file
dataset = Dataset.from_csv("my_data", "path/to/data.csv")

# Optional: load from pandas
import pandas as pd
df = pd.read_csv("path/to/data.csv")
dataset = Dataset.from_pandas("my_data", df)

# Inspect dataset
report = dataset.inspect()
print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
for col in report.columns:
    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")

# Validate dataset
validation = dataset.validate()
if not validation.valid:
    print("Errors:", validation.errors)
if validation.warnings:
    print("Warnings:", validation.warnings)

# Split dataset for ML workflows
train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")

Features

• Multiple Data Sources: Load from CSV, JSON, JSONL, and pandas DataFrames
• Dataset Inspection: Get detailed column statistics and metadata
• Data Validation: Automatic detection of missing values, inconsistent columns, and data quality issues
• Deterministic Fingerprinting: Generate unique identifiers for datasets to ensure reproducibility
• Smart Splitting: Train/validation/test splitting with customizable ratios
• Immutable Design: Safe data handling with copy-on-write semantics
• Type Safety: Full type hints for better IDE support and fewer bugs

Use Cases

ML Pipeline Integration

from silver_data import Dataset
import pandas as pd

# Load and validate training data
df = pd.read_csv("train.csv")
dataset = Dataset.from_pandas("training", df)

# Ensure data quality before training
validation = dataset.validate()
if not validation.valid:
    raise ValueError(f"Dataset validation failed: {validation.errors}")

# Split for cross-validation
train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)

# Use fingerprints for caching
cache_key = dataset.fingerprint()
print(f"Dataset fingerprint: {cache_key}")

Data Quality Monitoring

from silver_data import Dataset

# Monitor data drift over time
dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")

if dataset_v1.fingerprint() != dataset_v2.fingerprint():
    print("Dataset has changed - retrain models")

# Check for new data quality issues
report_v2 = dataset_v2.inspect()
for col in report_v2.columns:
    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
        print(f"Warning: {col.name} has high missing rate: {col.missing}")

Experiment Reproducibility

from silver_data import Dataset

# Ensure exact same data across experiments
dataset = Dataset.from_csv("experiment", "data.csv")
experiment_id = f"exp_{dataset.fingerprint()}"

# Log for reproducibility
print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")

Advanced Usage

Custom Data Loading

from silver_data import Dataset
import json

# Load from custom JSON format
with open("custom_data.json") as f:
    data = json.load(f)
dataset = Dataset.from_json("custom", data)

# Load from streaming JSONL
with open("streaming_data.jsonl") as f:
    dataset = Dataset.from_jsonl("streaming", f.read())

Data Type Analysis

from silver_data import Dataset

dataset = Dataset.from_csv("analysis", "mixed_data.csv")
report = dataset.inspect()

# Analyze column types
string_cols = [c.name for c in report.columns if c.value_type == "string"]
numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]

print(f"String columns: {string_cols}")
print(f"Numeric columns: {numeric_cols}")
print(f"Mixed type columns: {mixed_cols}")

Requirements

• Python 3.8+
• pandas 1.0+

Development

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=silver_data --cov-report=html

# Run linting
flake8 src/ tests/
mypy src/

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

Apache-2.0 - see LICENSE file for details.

Related Packages

• silver-run - Training lifecycle management
• silver-diagnostics - ML diagnostics
• silver-adapters - Framework adapters