marco-dvcs 0.1.67

A minimal dataset versioning system for text data with a focus on reproducibility.

Marco Dataset Versioning System (DVCS)

A minimal dataset versioning system for text data with a strong focus on reproducibility and transparency. Treat your text datasets like code — immutable, versioned, reproducible, and explainable.

Marco acts as a lightweight Python CLI tool that you initialize inside any ML project folder to safely version and preprocess your datasets without altering your original files.

---

🚀 Installation

# Recommended: install inside a virtual environment
python -m venv venv
source venv/bin/activate        # Linux / macOS
venv\Scripts\activate           # Windows

pip install marco-dvcs

---

🗂️ Quick Command Reference

Command	Description
marco init	Initialize a Marco repository in the current directory
marco upload <file>	Create a versioned, preprocessed dataset snapshot
marco list	List all tracked versions
marco lineage	Print ASCII lineage tree
marco diff <v1> <v2>	Compare two versions (unified diff or metrics)
marco restore <version>	Restore processed data to your workspace
marco attach-results <version> <file>	Attach a training results file to an existing version
marco export <version> <dest>	Export a version to .tar.gz
marco import <tarball>	Import a version from a .tar.gz
marco delete <version>	Delete a version and heal the lineage
marco token-analytics <v1> <v2>	KL-divergence token shift analysis
marco evaluate <version>	Dataset structural health report
marco drift <v1> <v2>	Detect model performance degradation
marco verify <version>	Cryptographic reproducibility proof
marco verify-all	Audit every version in the repo
marco generate-web	Launch the interactive web UI

---

🛠️ Usage Guide

1. Initialize a Repository

Initialize Marco tracking in your current project directory. Creates a .marco/ folder with registry, chain, and lock files.

marco init

---

2. Create an Immutable Version (upload)

Upload a raw .txt, .csv, or .tsv dataset. Marco runs the preprocessing pipeline, computes a SHA-256 hash, and creates an immutable snapshot.

Interactive Mode — no config file needed; Marco prompts you step by step:

marco upload my_dataset.csv -t v1-raw

Config Mode — supply a config.json to skip the prompts:

marco upload my_dataset.csv -c config.json -t v1-processed

With Training Results — bundle a results file at upload time:

marco upload my_dataset.csv -t v1-trained -r ./results/training_results.txt

Note: During upload, Marco auto-generates raw_stats.json (pre-pipeline statistics) and post_stats.json (post-pipeline reduction statistics) inside the version directory.

The config file format for a preprocessing pipeline (config.json):

{
  "dag": {
    "step_1_normalize_newlines": { "func": "normalize_newlines", "params": {}, "depends_on": [] },
    "step_2_lowercase":          { "func": "lowercase",          "params": { "enabled": true }, "depends_on": ["step_1_normalize_newlines"] },
    "step_3_remove_stopwords":   { "func": "remove_stopwords",   "params": { "language": "english" }, "depends_on": ["step_2_lowercase"] },
    "step_4_filter_length":      { "func": "filter_length",      "params": { "min_tokens": 5, "max_tokens": 200 }, "depends_on": ["step_3_remove_stopwords"] },
    "step_5_deduplicate":        { "func": "deduplicate",        "params": { "method": "exact" }, "depends_on": ["step_4_filter_length"] }
  }
}

Available pipeline functions:

Function	Params	Description
normalize_newlines	—	Collapse \r\n and \r to \n
lowercase	enabled: bool	Convert all text to lowercase
tokenize	method: "whitespace"|"word"|"char"	Tokenize text (adds n_tokens column)
remove_stopwords	language: "english"|...	Remove common stopwords
filter_length	min_tokens: int, max_tokens: int	Drop docs outside token range
deduplicate	method: "exact"	Remove duplicate rows

---

3. Attach Training Results (Decoupled Workflow)

If you train after uploading, attach results to the existing version later:

# Step 1 — restore preprocessed data for training
marco restore v1-raw -o ./my_training_data.tsv

# Step 2 — train your model, then attach the results
marco attach-results v1-raw training_results.txt
# alias:
marco attach v1-raw training_results.txt

---

4. List Versions

marco list

Prints a table of all versions with short hash, timestamp, user, and tags.

---

5. View Lineage Tree

ASCII-art visualisation of the full project history up to the Root Node:

marco lineage

---

6. Restore / Checkout Data

Extract the processed dataset back into your workspace for training:

marco restore v1-processed -o ./training_data.tsv
# alias:
marco checkout v1-processed -o ./training_data.tsv

---

7. Export / Import Versions

Share dataset versions with teammates as portable .tar.gz archives:

# Export
marco export v1-raw ./exports/

# Import
marco import ./exports/marco_version_e5e0b767.tar.gz

---

8. Delete a Version

Deletes the version files and heals the lineage so descendant version links remain valid:

marco delete v1-raw
# alias:
marco rm e5e0b767

---

9. Diff Versions

Compare two versions with a human-readable metric summary and unified line-by-line diff:

# Diff raw input (default)
marco diff v1.0 v2.0

# Diff preprocessed output
marco diff v1.0 v2.0 --target preprocessed

# Diff DAG config
marco diff v1.0 v2.0 --target config

# Control context lines (default 3)
marco diff v1.0 v2.0 --context 10

# Save a Markdown metrics report instead of printing to terminal
marco diff v1.0 v2.0 --save ./reports

Example output:

📊 Summary: v1.0 → v2.0
  ► N Documents: 2.00% increase (50 → 51)
  ► N Tokens: 15.30% increase (1000 → 1153)
  ► Vocab Size: 5.10% increase (300 → 315)
─────────────────────────────────────────────────────────────────

--- v1.0 (abc12345)/raw.txt
+++ v2.0 (def67890)/raw.txt
@@ -1,5 +1,6 @@
  positive  Great product, love it!
-negative  Terrible experience.
+negative  Poor experience, not recommended.
+negative  Worst purchase I ever made.

---

10. Token Analytics (KL Divergence)

Measure how token probability distributions shift between two versions using Kullback-Leibler divergence:

marco token-analytics v1-raw v2-processed

Returns a JSON report with vocabulary overlap, token frequency shifts, and top divergent terms.

---

11. Evaluate Dataset Health

Get a comprehensive structural health report for a single version — pipeline determinism grade, data volatility index, and reproducibility confidence score:

marco evaluate v1-processed

Example output:

📊 Evaluating Structural Health of Dataset 'v1-processed'...

  ► Pipeline Determinism... [A Grade]
  ► Computing Data Volatility (vs. Raw)... [12% Volatile]
  ► Running Reproducibility Proofs... [High Confident]

✅ Evaluation Complete! Overall Health: EXCELLENT
📄 Generating full report: .marco/versions/<id>/evaluation_report.json

---

12. Model Drift Detection

Detect performance degradation when training data changes. Marco extracts metrics from your attached training_results.txt (regex-based) and runs an internal NaiveBayes model against the new data in-memory:

marco drift v1-trained v2-processed

Example output:

MODEL DRIFT ANALYSIS
════════════════════════════════════════
Trained on:   v1-trained (fe4403)
Evaluated on: v2-processed (a3f9c1)
════════════════════════════════════════
Metric       Baseline    New Data    Drop     Severity
────────────────────────────────────────────────────
Accuracy     0.700       0.612      -12.6%   CRITICAL
F1           0.547       0.485      -11.3%   CRITICAL
════════════════════════════════════════
Overall: RETRAIN

---

13. Reproducibility Verification

Verify a single version

Re-runs the pipeline from scratch and compares the hash against the stored manifest:

marco verify v1-raw
# also accepts full hash
marco verify fe44032c

Example output:

🔬 Verifying version: fe44032c70164a71...

  Pipeline Step Verification:
    ✅  step_1 normalize_newlines
    ✅  step_2 lowercase

  Stored  output_hash: ed610672ea28e065...
  Recomputed   hash:   ed610672ea28e065...

  ✅ VERIFIED — output hash matches. Version is reproducible.

Verify all versions at once

marco verify-all

Returns exit code 1 if any version fails — CI/CD compatible.

---

14. Web UI (generate-web)

Launch a local interactive web UI to visually explore all your tracked versions, their DAG pipelines, preprocessed data, training results, and health statistics.

marco generate-web

Opens http://localhost:7654 in your browser automatically. Features include:

View	Description
DAG	Interactive pipeline graph — click any node to inspect step metrics and parameters
Code	Pipeline steps rendered as readable code blocks
Data	Browse raw input, preprocessed output, training results, and config per version
Sidebar	Git-style lineage tree grouped by chain with ref labels (v1-raw, v2-updated, etc.)
Summary Bar	Key metrics (docs, tokens, vocab) with +/- delta when comparing two versions

Press Ctrl+C to stop the server.

---

🔬 Reproducibility Engine — 7-Layer Architecture

#	Guarantee	How
1	Step-by-step hash verification	Hashes output after every DAG step; pinpoints the exact step that introduced non-determinism
2	Byte-exact canonical enforcer	Serializes data to a fixed TSV format (Unix LF, fixed columns, UTF-8) before hashing
3	Row-level delta on FAIL	Computes exactly which rows differ and saves them in verification_report.json
4	Permanent audit trail	Every run appended to .marco/verification_log.jsonl (tamper-evident)
5	Environment fingerprinting	Stores Python / NumPy / Pandas versions; warns on re-run if environment changed
6	Stochastic operation detector	Warns if pipeline contains shuffle / random_sample without a seed
7	Manifest integrity update	Sets "verified": true/false in manifest.json after every run

---

🧠 Architecture Overview

marco/
├── cli/
│   ├── main.py          — CLI entry point, all command routing
│   ├── interactive.py   — Interactive pipeline builder (prompts)
│   └── web_serve.py     — HTTP server for the React web UI
├── core/
│   ├── repository.py    — Version CRUD, refs.json tagging, file immutability
│   ├── preprocessor.py  — DAG-based preprocessing engine (topological sort)
│   ├── comparator.py    — Unified diff and Markdown metric reports
│   └── evaluate.py      — Structural health evaluation (DVI, RCS grades)
├── token_analytics/
│   └── distribution.py  — KL divergence & token frequency shift analysis
├── model_drift/         — In-memory NaiveBayes drift detection
├── reproducibility_proof/
│   ├── verify.py        — End-to-end reproducibility verification
│   ├── canonicalizer.py — Byte-exact canonical serialization
│   └── audit_log.py     — Append-only verification log
└── ui/                  — React + Vite web UI source
    └── dist/            — Built UI assets served by generate-web

---

Have fun building safer machine learning pipelines! 🚀