autolineage 0.1.0

Automatic ML data lineage tracking with zero manual logging

AutoLineage

Automatic ML Data Lineage Tracking

Track your data lineage automatically - from raw data to trained models - without manual logging.

Quick Start

pip install autolineage

import autolineage.auto
import pandas as pd

# Your normal code - everything tracked automatically!
df = pd.read_csv('data.csv')
df_clean = df.dropna()
df_clean.to_csv('clean.csv')

# That's it! Lineage tracked automatically 

Features

• Zero Manual Logging - Track lineage automatically with zero code changes
• Visual Graphs - Beautiful interactive and static lineage visualizations
• EU AI Act Compliant - Generate compliance reports instantly
• Jupyter Support - Magic commands for notebooks
• Multi-Environment - Works in Jupyter, Python scripts, CLI
• Lightweight - SQLite backend, no complex setup
• Cryptographic Verification - SHA-256 hashes for data integrity

Three Ways to Use

1️⃣ Automatic (Recommended)

import autolineage.auto

# Just write normal pandas/numpy code
# Everything is tracked automatically!

2️⃣ CLI

lineage track my_pipeline.py
lineage show --format html
lineage report

3️⃣ Jupyter Magic

%load_ext autolineage
%lineage_start

# Your code...

%lineage_show
%lineage_report

What Gets Tracked

AutoLineage automatically hooks into:

Library	Functions
pandas	read_csv, to_csv, read_parquet, to_parquet, read_json, to_json, read_excel, to_excel, read_pickle, to_pickle
numpy	load, save, loadtxt, savetxt
pickle	dump, load
joblib	dump, load

Plus: Automatic lineage relationships between files!

Visualizations

Generate beautiful lineage graphs:

# Interactive HTML
lineage show --format html --output graph.html

# Static PNG
lineage show --format png --output graph.png

Features:

• Color-coded by file type
• Hover for details
• Click to explore
• Export for presentations

EU AI Act Compliance

Generate compliance reports with one command:

lineage report --format markdown

Includes:

• Complete data inventory with SHA-256 hashes
• All transformation operations documented
• Full lineage graph with verification
• Reproducibility instructions
• Regulatory compliance statement

Perfect for:

• EU AI Act Article 10 requirements
• Model governance and auditing
• Research reproducibility
• Team collaboration

CLI Reference

lineage track SCRIPT      # Track a Python script
lineage show              # Visualize lineage graph
lineage summary           # Show statistics
lineage report            # Generate compliance report
lineage clear             # Delete database

See docs/cli.md for complete reference.

Jupyter Notebook

%load_ext autolineage
%lineage_start

import pandas as pd
df = pd.read_csv('data.csv')
df.to_csv('output.csv')

%lineage_summary          # Show stats
%lineage_show             # Display graph
%lineage_report           # Generate report

See examples/jupyter_demo.ipynb for complete demo.

Documentation

• QuickStart Guide - Get started in 5 minutes
• CLI Reference - Complete command-line guide
• Compliance Guide - EU AI Act reporting
• Examples - Working code samples

Use Cases

Research Reproducibility

Track every step from raw data to published results. Never wonder "which dataset did I use?" again.

ML Model Governance

Automatic compliance documentation for regulated industries. EU AI Act ready.

Team Collaboration

Share complete data provenance with your team. Everyone knows exactly what transformations were applied.

Debugging

Trace model issues back to data sources instantly. Full audit trail included.

Architecture

Raw Data → [Transformation 1] → Intermediate → [Transformation 2] → Model
   ↓              ↓                  ↓                ↓              ↓
Tracked    Logged & Hashed    Tracked     Logged & Hashed    Tracked

• SQLite Database - Portable, zero-config storage
• Function Hooking - Automatic tracking via monkey-patching
• Cryptographic Hashing - SHA-256 for data integrity
• Graph Generation - NetworkX for lineage DAG

Contributing

This is a research project being developed for a PhD in AI.

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new features
4. Submit a pull request

License

MIT License - see LICENSE for details.

Author

Built by Kishan as part of PhD research on ML reproducibility and data governance.

• GitHub: @kishanraj41

• Email: [e-mail]

Star History

If you find AutoLineage useful, please star the repository!

Citation

If you use AutoLineage in your research, please cite:

@software{autolineage2025,
  author = Kishan Raj Vandhavasi Goutham Kumar,
  title = {AutoLineage: Automatic ML Data Lineage Tracking},
  year = {2025},
  url = {https://github.com/kishanraj41/autolineage}
}

Roadmap

• Automatic pandas/numpy tracking
• Visual lineage graphs
• CLI interface
• EU AI Act compliance reports
• Jupyter magic commands
• MLflow integration
• Git integration
• Column-level lineage
• Data drift detection
• Team collaboration features
• Cloud storage support

FAQ

Q: Does this slow down my code? A: Minimal overhead - just file I/O tracking. Typically <1% performance impact.

Q: Do I need to change my code? A: No! Just import autolineage.auto at the top. Everything else is automatic.

Q: What Python versions are supported? A: Python 3.8+

Q: Can I use this in production? A: Yes! It's lightweight and has minimal dependencies.

Q: How is this different from MLflow? A: AutoLineage focuses on automatic data lineage (zero manual logging), while MLflow is a complete MLOps platform. They complement each other!

---

Made for the ML community

**Save:** `Ctrl+S`

---

## **4.3: Create LICENSE File**

**Create file: `LICENSE`**

MIT License

Copyright (c) 2025 Kishan

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

**Save:** `Ctrl+S`

---

## **4.4: Create .gitignore (if not exists)**

**Update: `.gitignore`**

Python

pycache/ *.py[cod] *$py.class *.so .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ pip-wheel-metadata/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST

Virtual environment

venv/ env/ ENV/ .venv

IDE

.vscode/ .idea/ *.swp *.swo *~

Database

*.db *.sqlite *.sqlite3

OS

.DS_Store .DS_Store? ._* .Spotlight-V100 .Trashes ehthumbs.db Thumbs.db

Testing

.pytest_cache/ .coverage htmlcov/ .tox/

Documentation

docs/_build/ site/

Jupyter

.ipynb_checkpoints/ *.ipynb_checkpoints

Generated files

*.csv *.parquet *.json *.png *.html .md !README.md !docs/.md !LICENSE

Logs

*.log