silver-data

Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.

The base install uses only the Python standard library for records, JSON, JSONL, and CSV. Add pandas only when you need DataFrame conversion:

pip install 'silver-data[pandas]'

Installation

pip install silver-data

Quick Start

from silver_data import Dataset

# Load from CSV file
dataset = Dataset.from_csv("my_data", "path/to/data.csv")

# Optional: load from pandas
import pandas as pd
df = pd.read_csv("path/to/data.csv")
dataset = Dataset.from_pandas("my_data", df)

# Inspect dataset
report = dataset.inspect()
print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
for col in report.columns:
    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")

# Validate dataset
validation = dataset.validate()
if not validation.valid:
    print("Errors:", validation.errors)
if validation.warnings:
    print("Warnings:", validation.warnings)

# Split dataset for ML workflows
train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")

Architecture-aware visualizations

Silver can select only the visualizations justified by the packages and data signals present in an architecture. Omitted visualizations explain what is missing, so dashboards do not claim to show metrics that were never produced.

from silver_data import design_visualization_plan

plan = design_visualization_plan({"architecture": {
    "uses": ["data", "diagnostics", "run", "torch"],
    "signals": ["dataset", "missingness", "labels", "predictions", "metrics", "history", "run", "events", "features", "pipeline"],
}})
for visualization in plan.selected:
    print(visualization.key, visualization.reason)

Call plan.to_dict() to pass the explainable plan to a UI or report builder, and use max_items when a surface has limited space.

Features

• Multiple Data Sources: Load from CSV, JSON, JSONL, and pandas DataFrames
• Dataset Inspection: Get detailed column statistics and metadata
• Data Validation: Automatic detection of missing values, inconsistent columns, and data quality issues
• Deterministic Fingerprinting: Generate unique identifiers for datasets to ensure reproducibility
• Smart Splitting: Train/validation/test splitting with customizable ratios
• Immutable Design: Safe data handling with copy-on-write semantics
• Type Safety: Full type hints for better IDE support and fewer bugs

Use Cases

ML Pipeline Integration

from silver_data import Dataset
import pandas as pd

# Load and validate training data
df = pd.read_csv("train.csv")
dataset = Dataset.from_pandas("training", df)

# Ensure data quality before training
validation = dataset.validate()
if not validation.valid:
    raise ValueError(f"Dataset validation failed: {validation.errors}")

# Split for cross-validation
train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)

# Use fingerprints for caching
cache_key = dataset.fingerprint()
print(f"Dataset fingerprint: {cache_key}")

Data Quality Monitoring

from silver_data import Dataset

# Monitor data drift over time
dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")

if dataset_v1.fingerprint() != dataset_v2.fingerprint():
    print("Dataset has changed - retrain models")

# Check for new data quality issues
report_v2 = dataset_v2.inspect()
for col in report_v2.columns:
    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
        print(f"Warning: {col.name} has high missing rate: {col.missing}")

Experiment Reproducibility

from silver_data import Dataset

# Ensure exact same data across experiments
dataset = Dataset.from_csv("experiment", "data.csv")
experiment_id = f"exp_{dataset.fingerprint()}"

# Log for reproducibility
print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")

Advanced Usage

Custom Data Loading

from silver_data import Dataset
import json

# Load from custom JSON format
with open("custom_data.json") as f:
    data = json.load(f)
dataset = Dataset.from_json("custom", data)

# Load from streaming JSONL
with open("streaming_data.jsonl") as f:
    dataset = Dataset.from_jsonl("streaming", f.read())

Data Type Analysis

from silver_data import Dataset

dataset = Dataset.from_csv("analysis", "mixed_data.csv")
report = dataset.inspect()

# Analyze column types
string_cols = [c.name for c in report.columns if c.value_type == "string"]
numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]

print(f"String columns: {string_cols}")
print(f"Numeric columns: {numeric_cols}")
print(f"Mixed type columns: {mixed_cols}")

Requirements

• Python 3.8+
• pandas 1.0+

Development

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=silver_data --cov-report=html

# Run linting
flake8 src/ tests/
mypy src/

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

Apache-2.0 - see LICENSE file for details.

Related Packages

• silver-run - Training lifecycle management
• silver-diagnostics - ML diagnostics
• silver-adapters - Framework adapters