litelineage 0.1.2

A decorator-based data lineage tracker.

LiteLineage 🕸️

Zero-Infrastructure Data Lineage for Python.

LiteLineage is a lightweight, decorator-based library that tracks data dependencies in your Python pipelines. It stores lineage metadata in a local SQLite file and generates professional Mermaid.js graphs instantly.

Perfect for: Solo Data Engineers, Local ETL scripts, and POCs where setting up DataHub or Amundsen is overkill.

🚀 Key Features

• Zero Setup: No servers to run. Just install and import.
• Decorator Driven: Add @tracker.track to your existing functions.
• Visual Graphs: Auto-generates an HTML interactive graph of your data flow.
• Standard Tech: Uses SQLite for storage and Mermaid.js for rendering.

📦 Installation

pip install litelineage

⚡ Quick Start

1. Track Your Functions

Import the tracker and use the decorator. You simply declare what the function reads (inputs) and what it creates (outputs).

import time
from litelineage import tracker

# Example: Ingesting data
@tracker.track(inputs=["s3://raw-bucket/users.csv"], outputs=["local/users_clean.parquet"])
def clean_users():
    print("Cleaning user data...")
    time.sleep(1)

# Example: Creating a report
@tracker.track(inputs=["local/users_clean.parquet"], outputs=["reports/daily_users.pdf"])
def generate_report():
    print("Generating PDF...")
    time.sleep(1)

if __name__ == "__main__":
    clean_users()
    generate_report()
    print("Pipeline finished!")

2. View the Lineage

After running your script, a lineage.db file is created automatically. To see the graph, run the CLI command:

litelineage-show

This will generate lineage.html. Open it in your browser to see your data flow diagram.

Open your browser :

python -m http.server 8000

🛠️ CLI Reference

The package includes a command-line tool to visualize your database.

# Default (looks for lineage.db in current folder)
litelineage-show

# Specify custom file paths (Python usage)
# python -m litelineage.visualizer --db my_custom.db --out my_graph.html

📖 How It Works

1. Capture: When a decorated function finishes successfully, LiteLineage logs a row to a local SQLite file (lineage.db).
2. Store: It records the timestamp, function_name, input_asset, and output_asset.
3. Visualize: The visualizer reads the SQLite rows and converts them into Mermaid.js syntax (standard "Flowchart" logic), creating a standalone HTML file you can share with stakeholders.

🛡️ FAQ

Q: Does it affect performance? A: Negligible. It performs one quick SQLite write after your function finishes.

Q: Can I use it in production? A: Yes. Since it uses SQLite, it works perfectly on single-node Airflow, cron jobs, or containerized scripts. For distributed systems (Spark/Databricks), a centralized DB would be needed (roadmap feature).

---

License

MIT License. Free to use for personal and commercial projects.