datalineage-thedebuggers 1.0.2

Git-like versioning system for ML datasets — immutable, reproducible, auditable.

DataLineage: Minimal Dataset Versioning System for Text Data

Overview

DataLineage is a minimal, reproducible dataset versioning system designed for text data with strong emphasis on immutability, transparency, and auditability. It treats datasets like code: immutable, versioned, measurable, and explainable.

This system addresses critical failures in large-scale ML pipelines arising from poorly managed, ad-hoc data preprocessing without explicit tracking of transformations and their impact.

---

Core Features

1. Immutable Dataset Versions

• Once created, dataset versions cannot be modified
• Any change in raw data or preprocessing creates a new version
• Ensures reproducibility, traceability, and auditability
• All versions stored in .mydata/versions/{version_id}/

2. Cryptographic Version Identification

• Each version has a unique SHA256 hash
• Hash formula: version_hash = SHA256(SHA256(raw_input_bytes) + ':' + SHA256(config_json))
• Hash encodes both: the raw data identity AND the preprocessing configuration
• Two versions with same data but different configs will have different version IDs
• Two versions with different data but same config will also have different version IDs
• Graph-based parent/child linkage maintains version history

3. Configurable Preprocessing Pipeline

• Basic Cleaning:

  • Smart null handling (median for numeric, mode for categorical)
  • Duplicate removal
  • Text normalization (unicode, lowercasing, punctuation removal)
  • Unwanted value filtering

• Advanced Transformations:

  • Column selection/dropping
  • Row filtering (numeric ranges, value inclusion/exclusion)
  • Value mapping (categorical encoding, replacements)
  • Column renaming

• Deterministic & Reproducible:

  • Same input + same config → identical output every time
  • No randomness, no ML models, CPU-only execution
  • Full preprocessing config stored with each version

4. Version Comparison

• Side-by-side analysis of two dataset versions
• Metrics compared: row count, column differences, data changes
• Detailed diff reports stored as JSON
• Clear explanations of differences

5. Filesystem-Based Storage (No Database)

• Lightweight, portable, Git-compatible
• .mydata/ directory structure:

  .mydata/
  ├── HEAD                          # Current version pointer
  ├── logs.jsonl                    # All version commit history (JSON Lines format)
  ├── repo_meta.json               # Repository metadata
  ├── versions/
  │   ├── {version_hash}/
  │   │   ├── raw_snapshot.csv     # Original input data
  │   │   ├── processed.csv        # Preprocessed output
  │   │   └── metadata.json        # Version metadata & config
  │   └── ...
  └── raw_data/                    # Raw input archive
      └── {hash}__{filename}
  

---

System Architecture

Version Creation Modes

Mode 1: Raw Dataset Input (Option 3)

Raw Dataset → Default Preprocessing (smart_fill, text cleanup, dedup) → Version

• User chooses: apply preprocessing or store raw as-is
• Default preprocessing: deterministic, data-preserving (smart_fill strategy)
• Suitable for: initial data ingestion, baseline versioning

Mode 2: HEAD + Configuration (Option 2)

Current HEAD Version → User Provided Config (transformations) → New Version

• Takes already-processed data and applies custom preprocessing/transformations
• Supports: column selection, filtering, encoding, renaming
• Suitable for: feature engineering, comparative analysis, incremental pipelines

Mode 3: Deduplication

New Config Applied → Same Output as Existing Version → Mark as Duplicate

• Automatically detects when different configs produce identical output
• Reuses existing version (storage efficient)
• Logged as "dedupe_hit" event

---

Preprocessing Configuration

Configuration Structure

{
  "drop_nulls": true,
  "null_strategy": "smart_fill",
  "null_fill_text": "",
  "null_fill_numeric": 0,
  
  "cleanup_text": true,
  "lowercase_text": true,
  "remove_punctuation": true,
  "collapse_spaces": true,
  "normalize_unicode": true,
  
  "drop_duplicates": true,
  "sort_rows": true,
  
  "select_columns": ["col1", "col2"],
  "drop_columns": ["col3"],
  
  "filter_numeric": {
    "age": {"min": 18, "max": 65}
  },
  "filter_include": {
    "pclass": [1, 2]
  },
  
  "value_mappings": {
    "sex": {"male": "m", "female": "f"}
  },
  
  "rename_columns": {
    "old_name": "new_name"
  }
}

Null Handling Strategies

• drop_any: Drop rows with ANY null values (aggressive)
• drop_all: Drop only rows where ALL values are null
• fill: Fill nulls with specified defaults
• smart_fill: Intelligent filling (numeric → median, categorical → mode)
• keep: Keep nulls as-is

---

Reproducibility Guarantees

1. Deterministic Preprocessing

• No randomness in any operation
• Median/mode used for null filling (stable statistics)
• Unicode normalization deterministic
• Sorting stable and reproducible

2. Version Hash Encodes Input and Config

• Same raw data + same config → same version hash
• Version hash is computed from the raw input bytes and the config JSON, not from processed output
• Bit-for-bit reproducible across machines

3. Full Auditability

• All preprocessing steps logged
• Config stored with each version
• Parent/child version tracking
• Commit messages document intent

4. Immutability

• Versions never modified (read-only after creation)
• Changes require creating new version
• Complete history preserved

5. Reproduce Command (Option 13)

• Re-runs preprocessing from stored raw_snapshot + config_snapshot
• Recomputation of version_hash, verifies it matches stored hash — live proof of reproducibility

---

Usage Guide

6. Reproduce Version (Verify Reproducibility)

# Choose: 13) Reproduce Version (Verify Reproducibility)
# Enter version ID to reproduce and verify

1. Initialize Repository

python app.py
# Choose: 1) Initialize / Show Repository Status

2. Commit Raw Dataset

# Choose: 3) Input of Raw Dataset
# Dataset path: path/to/dataset.csv
# Apply preprocessing: yes (smart_fill) or no (raw as-is)

3. Apply Configuration & Create Version

# Choose: 2) Commit from Current HEAD + Config
# Config path: path/to/config.json

4. List All Versions

# Choose: 4) List Versions

5. Compare Two Versions

# Choose: 7) Compare Two Versions

---

Data Formats Supported

• CSV - Comma-separated values
• TSV - Tab-separated values
• JSON - JSON arrays of objects
• JSONL - Line-delimited JSON
• TXT - One text sample per line (raw text)

---

Constraints Met

✅ CPU-Only Execution
✅ Python + Pandas + NumPy Only
✅ Filesystem-Based Storage
✅ Supports CSV, TSV, JSON, JSONL, and TXT input formats
✅ Deterministic & Reproducible

---

Quick Start

cd DataLineage
python app.py
# Follow prompts to create versions and compare them

---

CLI Menu Reference

When you run python app.py, you will see the following menu options:

  1) Init/Status
     - Initializes project structure or shows current status
  2) Commit from HEAD + Config
     - Takes config file, creates new version from HEAD
  3) Commit from Raw Dataset
     - Takes raw dataset path + message, optionally applies default preprocessing
  4) List Versions
     - Shows committed versions with parent linkage and summary
  5) Checkout Version
     - Shows versions, then lets you select directly or view one first
  6) View Specific Version Logs
     - Shows detailed metadata for only the version you request
  7) Compare Two Versions
     - Shows concise differences and stores detailed diff report
  8) Verify Version Integrity
     - Checks integrity of a specific version by its ID
  9) Train Bag-of-Words Classifier
     - Trains a NumPy-only Naive Bayes classifier on a text column for a selected version
 10) Evaluate Version
     - Run BoW evaluation and store metrics for a version
 11) Show Metric Trend
     - Show F1 progression across version history
 12) Recommend Best Version
     - Scan all evaluated versions and rank by F1 score
 13) Exit

---

Example Workflow

1. Initialize the repository:

   python app.py
   # Choose: 1) Init/Status
   

2. Commit a raw dataset:

   # Choose: 3) Commit from Raw Dataset
   # Provide dataset path and commit message
   # Optionally apply default preprocessing
   

3. Apply a config to HEAD:

   # Choose: 2) Commit from HEAD + Config
   # Provide config file path
   

4. List and compare versions:

   # Choose: 4) List Versions
   # Choose: 7) Compare Two Versions
   

5. Verify version integrity:

   # Choose: 8) Verify Version Integrity
   # Enter version ID to check
   

6. Train and evaluate classifier:

   # Choose: 9) Train Bag-of-Words Classifier
   # Choose: 10) Evaluate Version
   

7. Show metric trend and best version:

   # Choose: 11) Show Metric Trend
   # Choose: 12) Recommend Best Version
   

---

How to Verify Integrity

To check if a version is uncorrupted and exactly as originally committed:

• Use menu option 8) Verify Version Integrity
• Enter the version ID you want to check
• The system will recompute the hash of the processed data and compare it to the stored hash
• If the hashes match, the version is valid and unmodified

---

Reproducibility Note

• All versioning and preprocessing is deterministic and fully auditable.
• The integrity check guarantees that your data and config have not changed since commit.
• If you ever need to prove a version is bit-for-bit identical to its original state, use the integrity check.

---

Support

For questions or issues, contact the hackathon team or open an issue in your project repository.