datadoc-cli 0.3.0

 ____    _  _____  _    ____   ___   ____
|  _ \  / \|_   _|/ \  |  _ \ / _ \ / ___|
| | | |/ _ \ | | / _ \ | | | | | | | |
| |_| / ___ \| |/ ___ \| |_| | |_| | |___
|____/_/   \_\_/_/   \_\____/ \___/ \____|
  

The Open Source Operating System for Dataset Engineering.

📖 View Official Documentation Website

Install • Why DATADOC? • Quick Start • CLI Commands • Architecture

🚀 What is DATADOC?

DATADOC is an intelligent, blazing-fast Command Line Interface (CLI) and Python Library designed to completely automate the most tedious part of Machine Learning: Dataset Engineering and Data Cleaning.

Powered by a high-performance Polars backend, DATADOC analyzes your raw CSV files, diagnoses missing values, outliers, and schema issues, and automatically engineers a machine-learning-ready dataset in milliseconds.

DATADOC is NOT just another EDA (Exploratory Data Analysis) tool. It doesn't just show you charts. It fixes your data and hands you a portable, deterministic Python script to replicate the pipeline anywhere.

⚡ The Impact: Why Industry Professionals Use DATADOC

Data Scientists and ML Engineers spend 80% of their time cleaning data and only 20% training models. DATADOC eliminates the 80%.

• Save Hundreds of Hours: Stop writing boilerplate code to impute nulls, one-hot encode categorical variables, or clip outliers. DATADOC does it in one command.
• Zero Black-Box AI: Every transformation is strictly mathematical (IQR, medians, mode). It is 100% deterministic, explainable, and safe for enterprise production environments.
• Lightning Fast: By utilizing polars (written in Rust) instead of pandas, DATADOC processes millions of rows with minimal memory overhead.
• Agentic AI Integration: DATADOC features a built-in AI Planner and an interactive Chat Assistant that can autonomously analyze your dataset, generate engineering plans, and execute plugins using tool-calling!
• Avoid Data Leakage: Built-in safeguards ensure that data scaling and imputation are handled correctly.

📦 Installation

DATADOC is published on PyPI. You can install it globally via pip or uv:

pip install datadoc-cli

(Requires Python 3.9+)

🛠️ Quick Start (CLI)

You don't need to write a single line of Python to clean your data. Just use the CLI.

# 1. Analyze your dataset's health (shows a beautiful terminal report)
datadoc analyze raw_data.csv

# 2. Get recommendations (DATADOC tells you exactly what is wrong)
datadoc recommend raw_data.csv

# 3. AUTO-ENGINEER! (Fixes everything and saves clean_raw_data.csv)
datadoc engineer raw_data.csv

# 4. Compare the before vs. after visually in your terminal
datadoc compare raw_data.csv clean_raw_data.csv

# 5. Export a standalone Python script to automate this in the future
datadoc pipeline raw_data.csv

# 6. Have an interactive AI session where the LLM engineers your data via chat!
datadoc chat raw_data.csv

*(Pro Tip: Add `--ai` to `analyze`, `recommend`, or `engineer` for AI-driven insights and orchestration!)*

🐍 Python SDK (Library Usage)

DATADOC is also a powerful Python library. You can import the engine directly into your Jupyter Notebooks or backend servers:

from datadoc.core.engine import DATADOC

# Initialize the blazing-fast Polars engine
doc = DATADOC("raw_data.csv")

# Generate a diagnostic report
report = doc.analyze()
print(report)

# Automatically engineer the dataset
clean_df = doc.engineer()
clean_df.write_csv("clean_data.csv")

# Export the generated pipeline script
with open("my_pipeline.py", "w") as f:
    f.write(doc.pipeline())

💻 CLI Commands Reference

(Note: Use the --ai flag on analyze, recommend, or engineer for LLM-powered execution!)

🧩 Architecture & Plugins

DATADOC operates as an orchestrator. It passes your dataset through an isolated chain of plugins in a strict priority order.

Every plugin implements a strict BasePlugin interface ensuring it can analyze(), apply(), rollback(), and generate_code().

Want to build your own? See CONTRIBUTING.md to learn how to create and register custom plugins!

🗺️ Roadmap

• Core Engine with plugin orchestration
• 5 Built-in deterministic plugins
• Stunning Rich Terminal UI
• Pipeline export capability
• Polars Backend Migration (100x Performance Boost)
• PyPI Release (pip install datadoc-cli)
• Phase 2: Agentic AI Planner (LLM Orchestration)
• Interactive AI Chat with Tool-Calling capabilities
• Export targets for dbt and Apache Airflow
• REST API (FastAPI) wrapper

⚖️ License

This project is licensed under the MIT License. See LICENSE for details.

🤝 Contributing

We welcome contributions from the community! If you'd like to add a new plugin or improve the core engine, please see CONTRIBUTING.md.