octolearn 0.5.2

Structured AutoML Pipeline with Intelligent Dataset Profiling

Octolearn 🐙

Structured AutoML Pipeline with Intelligent Dataset Profiling

Octolearn generates professional-grade intelligence dossiers on your datasets automatically. In under 1 second, you get:

• 📊 Risk Score (0-100 data quality assessment)
• 📈 Feature Importance (baseline model + SHAP analysis)
• 🔧 Preprocessing Strategy (imputation, encoding, scaling recommendations)
• 📉 Visual Analysis (distributions, correlations, SHAP plots)
• 💡 Strategic Recommendations (machine learning insights)
• 📄 Professional PDF Report (ready for stakeholders)

No code. No notebooks. Just intelligence.

---

⚡ Quick Start

Installation

pip install octolearn

Basic Usage

from octolearn import AutoML
from sklearn.datasets import load_iris

# Load your dataset
data = load_iris(as_frame=True)
X, y = data.data, data.target

# User-driven preprocessing: override imputer, encoder, scaler, and ID columns
user_params = {
   'imputer_strategy': {'numeric': 'median', 'categorical': 'mode'},
   'encoder_strategy': {'ordinal_cols': ['petal length (cm)'], 'bool_cols': ['sex']},
   'scaler': 'standard',
   'id_columns': ['sample_id']
}

# Initialize and fit with user params
automl = AutoML(**user_params)
automl.fit(X, y)

# Generate comprehensive report (modern black/red PDF)
pdf_file = automl.generate_report()
print(f"Report saved: {pdf_file}")
# Output: octolearn_report_<hash>.pdf

# All trained models are saved in trained_models/ and tracked in the registry

Access Individual Analyses

# Get dataset risk score
risk = automl.get_risk_score()
print(f"Risk Score: {risk['score']}/100")
print(f"Category: {risk['category']}")

# Get feature importance
importance = automl.get_feature_importance()
for feature, score in list(importance.items())[:5]:
    print(f"{feature}: {score:.4f}")

# Get preprocessing suggestions
suggestions = automl.get_preprocessing_suggestions()
for strategy, recs in suggestions.items():
    print(f"{strategy}: {recs}")

---

🎯 Features

1. Dataset Intelligence (16 Metrics)

• Row/column count
• Feature type detection (numeric, categorical, datetime)
• Missing value analysis
• Duplicate detection
• Cardinality assessment
• Skewness detection
• Task type auto-detection

2. Risk Scoring (0-100)

Comprehensive data quality assessment including:

• Missing data impact (0-20 points)
• Duplicate rows (0-15 points)
• Class imbalance (0-15 points)
• Feature skewness (0-10 points)
• Constant columns (0-10 points)
• High cardinality features (0-10 points)
• Feature-to-sample ratio (0-10 points)
• Sample size penalty (0-5 points)

Risk Categories:

• ✅ 0-30: Low Risk
• ⚠️ 31-60: Moderate Risk
• ❌ 61-100: High Risk

3. Preprocessing Recommendations

Auto-generated strategy for:

• Missing Values: Mean, median, KNN, or iterative imputation
• Categorical Encoding: One-Hot, Ordinal, or Target Encoding
• Feature Scaling: StandardScaler vs RobustScaler vs MinMaxScaler
• Feature Engineering: Polynomial features, interactions, temporal features
• Column Actions: Removal of constants, cardinality handling

4. Feature Importance

• Baseline Model: Fast Random Forest (~150ms)
• SHAP Analysis: TreeExplainer Shapley values
• Ranking: Top features identified automatically

5. Visual Analysis

6 professional visualizations embedded in PDF:

• Feature distributions (histograms with KDE)
• Correlation heatmap
• SHAP feature importance bar chart

6. Professional Reports

177KB PDF with:

• Color-coded risk assessment
• Executive summary
• Detailed diagnostics
• Visual analysis
• Actionable recommendations
• Professional formatting

---

📊 Example Output

DATASET PROFILE:
   Rows: 150
   Columns: 4
   Task Type: classification
   Hash: 8145f023f195

RISK SCORE: 10/100 (Low Risk)
   Risk Factors:
   • 0.7% duplicate rows
   • Features/samples ratio: 0.03

FEATURE IMPORTANCE (Top 3):
   1. petal length (cm): 0.4442
   2. petal width (cm): 0.4181
   3. sepal length (cm): 0.1099

PREPROCESSING SUGGESTIONS:
   Missing Values: No missing values detected
   Categorical Encoding: petal length (cm): Ordinal Encoding, sex: Label Encoding (0/1), others: One-Hot Encoding
   Scaling: StandardScaler applied
   Feature Engineering: Consider polynomial features

All model files are saved in trained_models/ and tracked in the registry.
PDF and all plots use a modern black background with red accents and modern font.

---

🔬 API Reference

AutoML()

Main orchestrator class.

Methods

fit(X, y)

Train profiler on dataset.

• Parameters: X (DataFrame), y (Series or array)
• Returns: self

generate_report()

Create comprehensive PDF report.

• Returns: filename (str)
• Output: Saves PDF with hash-based name

get_risk_score()

Get data quality assessment without full report.

• Returns: {"score": int, "category": str, "factors": dict}

get_preprocessing_suggestions()

Get preprocessing strategy without full report.

• Returns: dict with 5 recommendation categories

get_feature_importance()

Get baseline feature importance without full report.

• Returns: dict of {feature: score}

report()

Get raw profile dataclass.

• Returns: DatasetProfile

---

🏗 Architecture

Octolearn 🐙 — The Ultimate User-Driven AutoML & Data Intelligence Suite

Production-Ready, Fully User-Param Driven, Transparent, and Extensible AutoML for Real-World Data Science

Octolearn generates professional-grade intelligence dossiers on your datasets automatically. In under 1 second, you get:

• Full user control at every step — All preprocessing, modeling, and evaluation is user-param driven, with sensible defaults and full override capability.
• Professional, modern PDF reports — Black background, red accents, modern font, and a model benchmarks/results table with all metrics and parameters.
• Robust model registry — Every trained model is saved, versioned, and tracked with full metadata in trained_models/ and the registry.
• Transparent, modular architecture — Every phase is accessible, extensible, and documented. You can access, override, or extend any step.
• Lightning-fast, production-grade outputs — Full pipeline (profiling, cleaning, modeling, reporting) in under 1 second for most datasets.
• Zero configuration, but infinite customization — Works out of the box, but you can control every detail.

No code. No notebooks. Just intelligence.

🚀 What is Octolearn?

Octolearn is a next-generation AutoML and data intelligence library that gives you:

• Full user control at every step — All preprocessing, modeling, and evaluation is user-param driven, with sensible defaults and full override capability.
• Professional, modern PDF reports — Black background, red accents, modern font, and a model benchmarks/results table with all metrics and parameters.
• Robust model registry — Every trained model is saved, versioned, and tracked with full metadata in trained_models/ and the registry.
• Transparent, modular architecture — Every phase is accessible, extensible, and documented. You can access, override, or extend any step.
• Lightning-fast, production-grade outputs — Full pipeline (profiling, cleaning, modeling, reporting) in under 1 second for most datasets.
• Zero configuration, but infinite customization — Works out of the box, but you can control every detail.

---

💡 Why Octolearn?

• User-param driven: You control imputation, encoding, scaling, ID columns, model selection, evaluation metric, and more.
• No data leakage: Specify ID columns and they are never used in modeling or predictions.
• Model registry: All models are saved, versioned, and tracked for reproducibility and deployment.
• Professional reporting: PDF reports are modern, beautiful, and include a full model benchmarks/results table.
• API-first, notebook-friendly: Use as a script, in notebooks, or as a backend for your own tools.
• Extensible: Add your own models, metrics, or reporting sections easily.
• Transparent: Access every intermediate result, log, and artifact.
• Production-ready: Used in real-world projects, with robust error handling and logging.

---

⚡ Quick Start

Installation

pip install octolearn

Minimal Usage

from octolearn import AutoML
import pandas as pd

df = pd.read_csv('your_data.csv')
X = df.drop('target', axis=1)
y = df['target']

# All defaults, full pipeline
automl = AutoML()
automl.fit(X, y)
pdf = automl.generate_report()
print(f"Report: {pdf}")

---

🛠️ Full User-Param Control

automl = AutoML(
    imputer_strategy={"numeric": "median", "categorical": "mode"},
    encoder_strategy={"ordinal_cols": ["grade"], "default": "ohe"},
    scaler="minmax",
    id_columns=["id", "customer_id"],
    train_models=True,
    use_registry=True,
    n_models=3,
    evaluation_metric="f1",
    parallel_processing=True
)
automl.fit(X, y)
results = automl.train_auto_models()
pdf = automl.generate_report()

• Dataset Profiling: profiling/data_profiler.py performs automatic type detection (numeric, categorical, datetime), missing-value summaries, duplicate detection, skewness and constant-column checks, high-cardinality detection, and produces a stable dataset hash.

---

🔥 Key Features & Power

• User-param driven preprocessing: Imputer, encoder, scaler, and ID columns are all user-overridable at any point.
• ID column handling: Any columns listed in id_columns are never used for modeling or predictions, but are preserved for merging and reference.
• Model registry: Every trained model is saved in trained_models/ and tracked in the registry with full metadata (parameters, metrics, version, hash).
• Model benchmarks/results table: PDF report includes a table of all trained models, their parameters, and all metrics, with the best model at the top (by your chosen metric).
• User-param driven evaluation metric: Select the metric (e.g., F1, RMSE, accuracy, etc.) for best model selection and benchmarking.
• Professional PDF reporting: Black background, red accents, modern font, and full transparency of all results.
• Access at every point: Access cleaned data, profile, outlier analysis, feature interactions, cleaning log, all trained models, best model, and model benchmarks at any time.
• Extensible and modular: Add your own models, metrics, or reporting sections with minimal code changes.
• API-first: All phases and results are accessible via the API.
• Production-grade error handling and logging: Robust, clear, and debug-friendly.

---

📊 Example Output

DATASET PROFILE:
   Rows: 150
   Columns: 4
   Task Type: classification
   Hash: 8145f023f195

RISK SCORE: 10/100 (Low Risk)
   Risk Factors:
   • 0.7% duplicate rows
   • Features/samples ratio: 0.03

FEATURE IMPORTANCE (Top 3):
   1. petal length (cm): 0.4442
   2. petal width (cm): 0.4181
   3. sepal length (cm): 0.1099

PREPROCESSING SUGGESTIONS:
   Missing Values: No missing values detected
   Categorical Encoding: petal length (cm): Ordinal Encoding, sex: Label Encoding (0/1), others: One-Hot Encoding
   Scaling: StandardScaler applied
   Feature Engineering: Consider polynomial features

All model files are saved in trained_models/ and tracked in the registry.
PDF and all plots use a modern black background with red accents and modern font.

---

🧑‍💻 Advanced Usage & API

Data Access at Any Point

• automl.X_ — Cleaned features (IDs dropped for modeling, but available for merging)
• automl.y_ — Cleaned target
• automl.profile_ — Full dataset profile (metrics, types, etc.)
• automl.outlier_results_ — Outlier analysis
• automl.interaction_results_ — Feature interaction analysis
• automl.cleaning_log_ — Step-by-step cleaning log
• automl.trained_models_ — All trained model objects
• automl.best_model_ — Best model object
• automl.model_benchmarks_ — List of all model results/benchmarks

API Reference

• fit(X, y, ...) — Run the full pipeline with user-param overrides
• generate_report() — Create a professional PDF report
• train_auto_models(evaluation_metric=...) — Train all models and select best by your metric
• get_risk_score() — Get risk score and factors
• get_preprocessing_suggestions() — Get preprocessing plan
• get_feature_importance() — Get feature ranking
• get_trained_models() — Dict of all trained models
• get_best_model() — Best model object
• report() — Get the full dataset profile

---

🏗 Architecture & Extensibility

octolearn/
├── core.py                      ← Main AutoML class
├── config.py                    ← Configuration
├── profiling/
│     └── data_profiler.py        ← Dataset analysis (16 metrics)
├── experiments/
│   ├── risk_scorer.py          ← Risk assessment (0-100)
│   ├── preprocessing_suggester.py  ← Preprocessing strategy
│   ├── baseline_importance.py   ← Feature ranking
│   ├── plot_generator.py        ← Visualization + SHAP
│   ├── recommendation_engine.py ← Strategic insights
│   └── report_generator.py      ← PDF factory
├── feature/                    ← Feature engineering & selection
├── models/                     ← Model training, registry, selection
├── optimization/               ← Optimization & distributed support
├── evaluation/                 ← Metrics & evaluation
├── utils/                      ← Logging, helpers, error handling

---

🌟 Power Features & Philosophy

• User-param driven at every step: Override any phase, any time.
• No data leakage: ID columns are never used for modeling or predictions.
• Model registry: All models are saved, versioned, and tracked.
• Professional, modern PDF reporting: Black background, red accents, modern font, and a model benchmarks/results table.
• API-first, notebook-friendly, and scriptable: Use anywhere.
• Extensible and modular: Add your own models, metrics, or reporting sections.
• Transparent and debuggable: Access every intermediate result, log, and artifact.
• Production-ready: Robust error handling, logging, and reproducibility.

---

⚡ Performance

Task	Time
Profile dataset	~50ms
Calculate risk score	~30ms
Generate suggestions	~20ms
Train baseline model	~150ms
Create visualizations	~200ms
Generate PDF	~100ms
TOTAL	~550ms

---

📋 Requirements

• Python 3.8+
• pandas ≥ 1.0.0
• numpy ≥ 1.19.0
• scikit-learn ≥ 0.24.0
• reportlab ≥ 3.6.0
• matplotlib ≥ 3.3.0
• seaborn ≥ 0.11.0
• shap ≥ 0.40.0

---

🚀 Roadmap

• User-param driven preprocessing (imputer, encoder, scaler, ID columns)
• Robust model saving and registry
• Modern black/red PDF/visuals
• Model benchmarks/results table in PDF
• User-param driven evaluation metric
• Full API and data access at every point
• Concise, up-to-date documentation
• Outlier detection & visualization (coming soon)
• Feature interaction analysis (coming soon)
• Ensemble model selection (coming soon)
• MLflow integration (coming soon)

---

🧪 Testing

Run Tests

# Basic test
python test_octolearn.py

# Validation test
python validation.py

# Interactive notebook
jupyter notebook octolearn_demo.ipynb

Example Notebooks

• octolearn_demo.ipynb: Interactive feature showcase
• notebooks/test_octolearn_full.ipynb: Full pipeline test

---

📄 License

MIT License - see LICENSE file for details.

---

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Commit changes
4. Push to branch
5. Open a Pull Request

---

📞 Support

Issues or Questions?

• Check documentation in ARCHITECTURE.md
• Review octolearn_demo.ipynb
• Open an issue on GitHub

---

🐙 Philosophy

"Build the skeleton first. Then make it breathe."

Octolearn follows modular, automated, and professional principles:

• ✅ Modular Architecture - Each component independent
• ✅ Automation First - Recommendations, not questions
• ✅ Professional Quality - Production-ready outputs
• ✅ Reproducible Results - Hash-based naming
• ✅ Zero Configuration - Works out of the box
• ✅ User-Param Driven - Full control at every step

---

Made with ☕ and 🐙 Logic.

• Risk Scoring: experiments/risk_scorer.py computes a 0–100 data quality risk score and returns contributing factors and a categorical label (Low / Moderate / High).
• Preprocessing Suggestions: experiments/preprocessing_suggester.py generates actionable strategies for imputation, encoding, scaling, cardinality handling, and column-specific actions.
• Baseline Feature Importance: experiments/baseline_importance.py trains a fast Random Forest baseline to produce feature importances.
• Explainability (SHAP): experiments/plot_generator.py integrates SHAP TreeExplainer to produce SHAP summary plots and an importance bar chart for tree-based models.
• Visual Diagnostics: experiments/plot_generator.py creates per-feature distribution plots, correlation heatmaps, and saves PNG artifacts (non-interactive Agg backend supported).
• Automated PDF Report: experiments/report_generator.py composes a professional PDF dossier that embeds executive summary, risk banner, preprocessing suggestions, baseline importance, and visual diagnostics.
• Recommendation Engine: experiments/recommendation_engine.py synthesizes profiling and model signals into concise, prioritized recommendations.
• Core API / Orchestration: core.py exposes the high-level AutoML class with methods fit(X, y), generate_report(), get_risk_score(), get_preprocessing_suggestions(), get_feature_importance(), and report() (raw profile).
• Model & Optimization Utilities: models/ (selector & registry) and optimization/optimizer.py provide foundations for model selection and future HPO workflows.
• Preprocessing Pipeline Builder: preprocessing/pipeline_builder.py offers utilities to construct reproducible sklearn-compatible pipelines from suggested strategies.
• Feature Engineering & Selection: feature/feature_engineer.py and feature/feature_selector.py include initial helpers for transformations and automatic selection heuristics.
• Evaluation & Metrics: evaluation/metrics.py contains common evaluation helpers used by experiments and baseline checks.
• Tracking & Utilities: experiments/tracker.py plus utils/helpers.py provide lightweight run-tracking, reproducibility helpers, and I/O utilities.
• Fonts & Assets: fonts/ contains font assets used when rendering the PDF reports for consistent professional typography.

These components are designed to work together end-to-end: run AutoML().fit(X,y) to create a DatasetProfile, inspect suggestions and risks via the API, and call generate_report() to produce a stakeholder-ready PDF report and visual artifacts.

⚡ Performance

Task	Time
Profile dataset	~50ms
Calculate risk score	~30ms
Generate suggestions	~20ms
Train baseline model	~150ms
Create visualizations	~200ms
Generate PDF	~100ms
TOTAL	~550ms

---

📋 Requirements

• Python 3.8+
• pandas ≥ 1.0.0
• numpy ≥ 1.19.0
• scikit-learn ≥ 0.24.0
• reportlab ≥ 3.6.0
• matplotlib ≥ 3.3.0
• seaborn ≥ 0.11.0
• shap ≥ 0.40.0

---

🧪 Testing

Run Tests

# Basic test
python test_octolearn.py

# Validation test
python validation.py

# Interactive notebook
jupyter notebook octolearn_demo.ipynb

Example Notebooks

• octolearn_demo.ipynb: Interactive feature showcase

---

📄 License

MIT License - see LICENSE file for details.

---

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Commit changes
4. Push to branch
5. Open a Pull Request

---

📞 Support

Issues or Questions?

• Check documentation in ARCHITECTURE.md
• Review octolearn_demo.ipynb
• Open an issue on GitHub

---

🐙 Philosophy

"Build the skeleton first. Then make it breathe."

Octolearn follows modular, automated principles:

• ✅ Modular Architecture - Each component independent
• ✅ Automation First - Recommendations, not questions
• ✅ Professional Quality - Production-ready outputs
• ✅ Reproducible Results - Hash-based naming
• ✅ Zero Configuration - Works out of the box

---

Made with ☕ and 🐙 Logic.

Octolearn v0.5.2 — Intelligent AutoML for Everyone

Optional distributed execution: if you need scalable runs with large datasets or parallel Optuna trials, install the distributed extras:

pip install .[distributed]

Benchmarks and reproducible examples are available in the benchmarks/ and notebooks/ folders. See benchmarks/README.md for details.