pythios 0.1.0

The oracle for your data — low-code ML for everyone.

Pythios — The Oracle for Your Data

Low-code machine learning for everyone — from beginners to professionals.

Domain: pythios.xyz Package: pip install pythios Pronunciation: PITH-ee-os Tagline: Named after Apollo Pythios — the oracular epithet of Apollo at Delphi. The Pythia received a question and returned a prophecy. That is exactly what a machine learning model does.

---

IMPORTANT — READ THIS FIRST

This README is the single source of truth for building Pythios from scratch. Read it entirely before writing a single line of code. Every design decision, every method signature, every edge case is documented here. The goal is that this project builds correctly on the first attempt with no ambiguity.

---

Overview

Pythios is a low-code Python ML library that wraps scikit-learn, PyTorch, TensorFlow, XGBoost, LightGBM, Optuna, SHAP, UMAP, sentence-transformers, and Faiss behind a unified, beginner-friendly interface — while remaining powerful enough for professionals.

The philosophy:

• One object (Data) flows through the entire pipeline
• Every class works with zero arguments (sensible defaults everywhere)
• Heavy dependencies are lazy-imported (only loaded when actually needed)
• Task type (binary / multiclass / regression) is auto-detected and propagated
• Output is always human-readable — no raw numpy arrays printed to console
• Never mutate data in place — transformations return new Data objects

---

Complete Project Structure

Build EXACTLY this structure. Do not deviate.

pythios/ ← outer project root ├── pythios/ ← the actual Python package │ ├── init.py │ ├── _utils.py │ ├── data.py │ ├── preprocess.py │ ├── visualize.py │ ├── models.py │ ├── evaluate.py │ ├── unsupervised.py │ ├── embeddings.py │ └── tools.py ├── tests/ │ ├── init.py ← empty file │ ├── test_data.py │ ├── test_preprocess.py │ ├── test_visualize.py │ ├── test_models.py │ ├── test_evaluate.py │ ├── test_unsupervised.py │ ├── test_embeddings.py │ └── test_tools.py ├── demo/ │ └── demo_all.py ← full runnable demo (see section below) ├── pyproject.toml └── README.md

---

## pyproject.toml — Copy This Exactly

```toml
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "pythios"
version = "0.1.0"
description = "The oracle for your data — low-code ML for everyone."
readme = "README.md"
requires-python = ">=3.9"
license = { text = "MIT" }
authors = [{ name = "Advaith", email = "[e-mail]" }]
dependencies = [
    "pandas>=2.0",
    "numpy>=1.24",
    "scikit-learn>=1.3",
    "matplotlib>=3.7",
    "seaborn>=0.12",
    "requests>=2.28",
]

[project.optional-dependencies]
full = [
    "xgboost>=2.0",
    "lightgbm>=4.0",
    "torch>=2.0",
    "torchvision>=0.15",
    "tensorflow>=2.13",
    "optuna>=3.0",
    "shap>=0.43",
    "umap-learn>=0.5",
    "plotly>=5.0",
    "sentence-transformers>=2.2",
    "faiss-cpu>=1.7",
    "Pillow>=9.0",
    "imbalanced-learn>=0.11",
]

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]

[tool.setuptools]
packages = ["pythios"]

---

Installation

# Clone or create the project folder
cd pythios

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate       # Mac/Linux

# Install in editable mode with base dependencies
pip install -e .

# Install ALL optional dependencies (recommended for development)
pip install -e ".[full]"

# Install pytest
pip install pytest

---

__init__.py — The Package Entry Point

"""
Pythios — The oracle for your data.

Low-code machine learning for everyone.
pip install pythios | pythios.xyz

Classes:
    Data         — load, inspect, split datasets
    Preprocess   — scale, encode, impute, select features
    Visualize    — distributions, correlations, missing values
    Models       — compare, train, tune, save ML models
    Evaluate     — metrics, SHAP, ROC curves, cross-validation
    Unsupervised — clustering, dimensionality reduction
    Embeddings   — text and image vectors, similarity search
    Tools        — utilities, LLM coding assistant, profiling
"""

from .data import Data
from .tools import Tools

try:
    from .preprocess import Preprocess
except Exception:
    pass

try:
    from .visualize import Visualize
except Exception:
    pass

try:
    from .models import Models
except Exception:
    pass

try:
    from .evaluate import Evaluate
except Exception:
    pass

try:
    from .unsupervised import Unsupervised
except Exception:
    pass

try:
    from .embeddings import Embeddings
except Exception:
    pass

__version__ = "0.1.0"
__author__ = "Advaith"
__all__ = [
    "Data",
    "Preprocess",
    "Visualize",
    "Models",
    "Evaluate",
    "Unsupervised",
    "Embeddings",
    "Tools",
]

---

Build Order

Build files in this exact order. Each one depends on the previous.

1. _utils.py — shared helpers (lazy import, task detection, formatting)
2. data.py — Data class (everything else depends on this)
3. tools.py — Tools class (standalone utilities + LLM)
4. preprocess.py — depends on Data
5. models.py — depends on Data
6. evaluate.py — depends on Models and Data
7. visualize.py — depends on Data
8. unsupervised.py — depends on Data
9. embeddings.py — standalone but can accept Data

---

FILE 1: _utils.py

This file contains three things: the lazy import helper, the task type detector, and the table formatter. These are used by every other file.

lazy_import(module_name, install_name=None)

Imports a module only when called. If the module is not installed, raises an ImportError with a clear pip install instruction. This is the mechanism that keeps the base Pythios install lightweight — torch, tensorflow, shap etc. are never imported at the top of any file.

def lazy_import(module_name: str, install_name: str = None):
    try:
        return importlib.import_module(module_name)
    except ImportError:
        pkg = install_name or module_name
        raise ImportError(
            f"'{module_name}' is required for this feature.\n"
            f"Install it with: pip install {pkg}\n"
            f"Or install everything: pip install pythios[full]"
        ) from None

detect_task_type(target_series) -> str

Auto-detects task type from the target column. This is the most important utility function — its output propagates through the entire pipeline.

Rules (apply in this order):

1. If dtype is object or category → classification
   • 2 unique values → "binary"
   • 3+ unique values → "multiclass"
2. If dtype is float → "regression" (always)
3. If dtype is integer:
   • Find unique values, sort them
   • Check if they form a CONTIGUOUS range (e.g. [0,1] or [0,1,2,3])
   • AND max value is <= 100 AND n_unique <= 20
   • If contiguous + small → classification
     • n_unique == 2 → "binary"
     • n_unique > 2 → "multiclass"
   • Otherwise → "regression"

The contiguous range check is the key insight. Class labels like [0,1,2,3] are contiguous. Salary values like [50000,60000,70000] are not contiguous even though they have few unique values in a small dataset.

Implementation:

def detect_task_type(target_series) -> str:
    if target_series is None or len(target_series) == 0:
        return None
    n_unique = target_series.nunique()
    is_categorical = (
        target_series.dtype == "object"
        or target_series.dtype.name == "category"
    )
    is_float = target_series.dtype.kind == "f"

    if is_categorical:
        return "binary" if n_unique == 2 else "multiclass"
    elif is_float:
        return "regression"
    else:
        unique_vals = sorted(target_series.dropna().unique())
        min_val = unique_vals[0]
        max_val = unique_vals[-1]
        is_contiguous = (max_val == min_val + n_unique - 1)
        is_small = (n_unique <= 20 and max_val <= 100)
        if is_contiguous and is_small:
            return "binary" if n_unique == 2 else "multiclass"
        else:
            return "regression"

format_table(headers, rows, title=None)

Pretty-prints a formatted table. Calculates column widths automatically. Used throughout Pythios for readable console output.

---

FILE 2: data.py

The Data class is the universal container. Everything else takes a Data object as input.

Internal State

self._df        # pd.DataFrame — full original data, never mutated
self.target     # str — name of target column (None if unsupervised)
self.task_type  # str — "binary", "multiclass", "regression", or None
self.X_train    # pd.DataFrame — features, training set (set by .split())
self.X_test     # pd.DataFrame — features, test set
self.y_train    # pd.Series — target, training set
self.y_test     # pd.Series — target, test set
self._is_split  # bool — whether .split() has been called

Constructors (all class methods)

Data.from_csv(path, target, sep=",", encoding="utf-8")

Loads a CSV file using pd.read_csv. Returns a Data object.

Data.from_dataframe(df, target)

Wraps an existing pandas DataFrame. Does df.copy() internally.

Data.from_url(url, target, sep=",")

Downloads a CSV from a URL. Uses lazy_import("requests"). Uses requests.get(), then pd.read_csv(StringIO(response.text)).

Data.from_excel(path, target, sheet=0)

Loads an Excel file using pd.read_excel(path, sheet_name=sheet).

Data.from_sklearn(name, target=None)

Loads a built-in sklearn dataset by name. No file download needed. This is for quick prototyping and testing.

Supported datasets and their loaders:

{
    "iris":               datasets.load_iris,
    "wine":               datasets.load_wine,
    "breast_cancer":      datasets.load_breast_cancer,
    "digits":             datasets.load_digits,
    "diabetes":           datasets.load_diabetes,
    "california_housing": datasets.fetch_california_housing,
    "linnerud":           datasets.load_linnerud,
}

Implementation details:

• Import sklearn.datasets at the top of this method (not lazy — sklearn is a required dependency)
• Build DataFrame from dataset.data using dataset.feature_names if available, otherwise name columns "feature_0", "feature_1", etc.
• Target column name defaults to "target" if not provided
• For classification datasets (those with target_names attribute), convert integer targets to string class names using: pd.Categorical.from_codes(dataset.target, dataset.target_names) BUT only if len(unique targets) <= 20 (to avoid treating regression as classification)
• Print confirmation: ✓ Loaded sklearn dataset 'iris' — 150 rows, 5 cols
• Raise ValueError with list of available names if unknown name provided

Data.from_tab(path, target)

Loads an Orange .tab file. These have a specific 3-row header format:

• Row 1: column names (tab-separated)
• Row 2: column types (continuous, discrete, string)
• Row 3: column roles (class, meta, ignore, or blank)

Implementation:

1. Open file, read first 3 lines to get col_names, col_types, col_roles
2. Use pd.read_csv(path, sep="\t", skiprows=3, header=None, names=col_names)
3. Drop columns where role is "meta" or "ignore" (unless it's the target)
4. Cast columns where type is "continuous" to float using pd.to_numeric(errors="coerce")
5. Print dropped columns if any
6. Print confirmation

Data.from_sklearn — Available Datasets Table

Name	Task	Rows	Features
iris	multiclass	150	4
wine	multiclass	178	13
breast_cancer	binary	569	30
digits	multiclass	1797	64
diabetes	regression	442	10
california_housing	regression	20640	8
linnerud	regression	20	3

Inspection Methods

data.describe()

Prints an Orange-style dataset summary. No return value (returns None).

Format:

Pythios · Data Summary
────────────────────────────────────────────────────────────────────────────────
 Rows: 891   Columns: 12
 Target: Survived (binary)
 Missing values: 3 column(s) affected

 Column          Type        Missing    Unique    Notes
────────────────────────────────────────────────────────────────────────────────
 Age             float64     19.9%      88
 Cabin           object      77.1%      147       high cardinality
 Embarked        object      0.2%       3
 Fare            float64     0.0%       248
 Survived        int64       0.0%       2         ← TARGET

Notes logic (check in this order):

• If col == self.target → "← TARGET"
• Elif pct_missing > 50 → "mostly missing"
• Elif pct_missing > 20 → "high missing"
• Elif n_unique == 1 → "constant value"
• Elif n_unique > 100 and dtype == "object" → "high cardinality"
• Else → ""

data.missing_report() -> pd.DataFrame

Returns a DataFrame with columns: Column, Missing_Count, Missing_Percent. Only includes columns that actually have missing values. Sorted by Missing_Count descending.

Split Method

data.split(test=0.2, seed=42, stratify=True)

Splits data into train/test sets using sklearn's train_test_split.

Critical rules:

• If already split → raise ValueError (not a warning, a hard error)
• stratify only applies when task_type in ["binary", "multiclass"] AND y is not None
• Reset index on all four output DataFrames/Series (drop=True)
• Print: ✓ Split 891 rows → 712 train (79.9%), 179 test (20.1%)
• Store results in self.X_train, self.X_test, self.y_train, self.y_test
• Set self._is_split = True

Properties

@property shape -> tuple          # self._df.shape
@property dtypes -> pd.Series     # self._df.dtypes
@property columns -> list         # self._df.columns.tolist()

Utility Methods

All of these return NEW Data objects — never mutate self.

data.head(n=5) -> pd.DataFrame

Returns self._df.head(n). Note: returns DataFrame not Data.

data.tail(n=5) -> pd.DataFrame

Returns self._df.tail(n).

data.copy() -> Data

Deep copies _df, target, task_type. If split exists, copies all four split DataFrames too. Returns new Data object.

data.drop_columns(cols: list) -> Data

Drops specified columns. Uses errors="ignore" so missing columns don't crash. New target = self.target (unchanged — target is never dropped by this method).

data.select_columns(cols: list) -> Data

Keeps only specified columns. If self.target not in cols → new target = None.

data.sample(n=1000, seed=42) -> Data

Random sample of n rows (capped at len). Returns new Data object.

__repr__ and __str__

def __repr__(self):
    split_str = " [split]" if self._is_split else ""
    return f"Data(shape={self.shape}, target='{self.target}', task={self.task_type}){split_str}"

def __str__(self):
    return f"Pythios Data — {self.shape[0]:,} rows × {self.shape[1]} cols"

---

FILE 3: tools.py

Tools contains two categories of functionality:

1. Data utilities (profile, suggest, detect_outliers, balance, validate, export)
2. LLM-powered coding assistant (llm, coder, explain, review, fix, ask_about_data)

ALL methods are static or classmethods. Tools is NEVER instantiated.

The CodeResult class (defined ABOVE the Tools class in tools.py)

This is returned by Tools.coder() and Tools.fix().

class CodeResult:
    def __init__(self, code: str, description: str):
        self.code = code
        self.description = description
        self.namespace = {}
        self._ran = False

    def show(self) -> "CodeResult":
        # Print code with line numbers
        # Return self for chaining

    def run(self, verbose=True) -> "CodeResult":
        # Execute code with exec() in self.namespace
        # Inject variables into caller's frame using:
        #   frame = inspect.currentframe().f_back
        #   frame.f_locals.update({k: v for k, v in self.namespace.items() if not k.startswith("__")})
        #   ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0))
        # Print confirmation with list of created variables
        # Return self for chaining

The frame injection trick is what makes coder.run(); print(n) work — variables created inside the exec'd code are injected into the caller's actual Python frame so they appear as local variables.

Required imports at top of tools.py:

import inspect
import ctypes
import re
import os
import time
import pandas as pd
import numpy as np
from ._utils import lazy_import, format_table

LLM Configuration (class variables on Tools)

_llm_model    = "llama3"
_llm_base_url = "http://localhost:11434"

These are defaults. Users change them with Tools.configure_llm().

Tools._call_llm(prompt) -> str (classmethod, private)

Calls the Ollama REST API at {_llm_base_url}/api/generate. Uses requests.post with json={"model": cls._llm_model, "prompt": prompt, "stream": False}. Timeout: 120 seconds. On any exception, raise ConnectionError with instructions to install Ollama and run ollama pull {model_name}.

Data Utility Methods

Tools.profile(data) -> pd.DataFrame

Computes per-column stats: dtype, missing%, mean, std, min, 25%, 50%, 75%, max, skewness. Numeric columns get all stats. Object columns get "—" for numeric stats. Prints full summary including memory usage. Returns DataFrame.

Tools.suggest(data)

Analyses data and prints actionable suggestions. Checks for:

1. Columns with >20% missing → suggest imputation
2. Class imbalance (ratio > 3:1) → suggest Tools.balance()
3. Constant columns (nunique <= 1) → suggest dropping
4. High cardinality categoricals (nunique > 50, not target) → suggest encoding
5. Duplicate rows → suggest deduplication
6. Skewed features (|skew| > 2) → suggest robust scaling

Format each issue as:

 ⚠  {issue description}
     → {suggested action}

If no issues: print ✓ No obvious issues detected. Looks like a clean dataset!

Tools.detect_outliers(data, method="iqr", threshold=1.5) -> pd.DataFrame

Checks all numeric columns.

• "iqr": flag values outside [Q1 - thresholdIQR, Q3 + thresholdIQR]
• "zscore": flag values where |z-score| > threshold Returns DataFrame with columns: Column, Outlier_Count, Outlier_%, Bounds. Sorted by Outlier_Count descending.

Tools.balance(data, method="oversample", seed=42) -> Data

Fixes class imbalance. Requires target to be set and task_type to be binary or multiclass. Raises ValueError otherwise.

• "oversample": duplicate minority class rows with replacement
• "undersample": randomly remove majority class rows
• "smote": use imbalanced-learn SMOTE (lazy import "imblearn.over_sampling")

Print before/after class distribution. Return new Data object.

Tools.validate(data) -> bool

Runs sanity checks. Returns True if all pass, False if any fail. Checks: empty dataset, missing target, all-missing columns, infinite values, single-class target. Print ✓ for passes, ✗ for failures.

Tools.export(data, path, split="full")

Save to file. Support .csv, .xlsx, .json, .parquet based on extension. split can be "full", "train", or "test" (train/test require .split() called). Raise ValueError for unsupported extensions. Print: ✓ Exported 891 rows to 'output.csv'

Tools.memory_usage(data) -> str

Return human-readable memory string: "4.2 MB", "800 KB", "1.2 GB". Use data._df.memory_usage(deep=True).sum() for byte count.

Tools.timer (context manager class nested inside Tools)

class timer:
    def __init__(self, label="operation"):
        self.label = label
    def __enter__(self):
        self._start = time.time()
        return self
    def __exit__(self, *args):
        elapsed = time.time() - self._start
        print(f"⏱  {self.label} took {elapsed:.2f}s")

Tools.set_seed(seed=42)

Sets seed for: random, numpy, and torch (lazy import — only if installed). Print: ✓ Random seed set to 42

Tools.compare_datasets(data1, data2) -> pd.DataFrame

Side-by-side comparison of two Data objects. Compare: rows, columns, missing values, memory, numeric cols, categorical cols. Useful for checking before vs after preprocessing.

LLM Methods

Tools.configure_llm(model=None, base_url=None)

Updates cls._llm_model and/or cls._llm_base_url. Print confirmation.

Tools.llm(prompt, print_response=True) -> str

Direct LLM prompt. Prints header, calls _call_llm, prints response. Returns response string.

Tools.coder(description, show=True) -> CodeResult

Generates Python code from natural language.

System prompt must:

1. Ask for ONLY raw Python code, no markdown, no backticks, no explanation
2. Ask for no import statements unless absolutely necessary
3. Ask to store results in clearly named variables

After getting response, strip markdown fences with regex:

code = re.sub(r"```(?:python)?", "", raw)
code = re.sub(r"```", "", code).strip()

Return CodeResult(code=code, description=description). If show=True, call result.show() before returning.

Tools.explain(code) -> str

Sends code to LLM asking for plain English explanation. Prints explanation. Returns string.

Tools.review(code) -> str

Sends code to LLM asking for bug review, performance issues, PEP8 style. Asks LLM to respond as a bullet list. Prints review. Returns string.

Tools.fix(code, error) -> CodeResult

Sends broken code + error message to LLM, gets fixed code back. Strip markdown fences from response. Call result.show() before returning.

Tools.ask_about_data(data, question) -> str

Builds a dataset summary (NOT raw rows — only statistics) and sends it to the LLM with the user's question. The summary includes:

• Row and column counts
• Target and task type
• Column dtypes
• Missing percentages
• 5 sample values per column (from .unique())

IMPORTANT: Raw data rows are never sent to the LLM. Only statistics. This preserves privacy and keeps prompts short.

---

FILE 4: preprocess.py

Class Behaviour

Preprocess wraps sklearn's ColumnTransformer and Pipeline. It accepts a Data object and builds a transformation pipeline that is fitted on X_train and applied to both X_train and X_test (preventing data leakage).

Constructor

def __init__(self, data: Data):
    # Store data reference
    # Detect numeric and categorical columns (excluding target)
    # Initialize _transformer to None
    # Initialize _steps list (records what the user configured)

Column detection:

numeric_cols     = df.select_dtypes(include="number").columns.tolist()
categorical_cols = df.select_dtypes(include="object").columns.tolist()
# Remove target from both lists

Configuration Methods (all return self for chaining)

p.auto()

Automatically configure the full pipeline. This is the recommended path.

For numeric columns: median imputation + StandardScaler For categorical columns: most_frequent imputation + OneHotEncoder(handle_unknown="ignore", sparse_output=False) For high-cardinality categorical columns (nunique > 15): OrdinalEncoder instead of OneHotEncoder

p.scale(method="standard")

Options: "standard" (StandardScaler), "minmax" (MinMaxScaler), "robust" (RobustScaler), "none" (no scaling). Stores setting in self._scale_method.

p.encode(method="onehot")

Options: "onehot" (OneHotEncoder), "label" (LabelEncoder), "ordinal" (OrdinalEncoder), "target" (TargetEncoder). Stores setting in self._encode_method.

p.impute(method="median", k=5)

Options: "median", "mean", "mode", "knn" (KNNImputer with k neighbors), "drop" (dropna). Stores setting in self._impute_method.

p.select_features(k=10, method="mutual_info")

Options: "mutual_info" (SelectKBest with mutual_info), "variance", "rfe" (with random forest estimator). Stores setting in self._feature_selection.

p.pca(n_components=0.95)

n_components can be int (exact) or float < 1 (variance explained). Stores setting in self._pca_components.

p.apply() -> Data

This is the most important method. It:

1. Builds the sklearn ColumnTransformer from stored settings
2. Fits on X_train ONLY
3. Transforms both X_train and X_test
4. Wraps results back into a new Data object (preserving y_train, y_test, task_type, target)
5. Restores column names (use get_feature_names_out() from ColumnTransformer)
6. Prints summary: ✓ Preprocessing applied — 10 → 47 features (after one-hot encoding)
7. Returns new Data object

CRITICAL: Must fit only on training data. Never fit on test data. The returned Data object must have _is_split = True with the transformed splits.

Properties

@property
def pipeline(self):
    # Returns the underlying sklearn Pipeline/ColumnTransformer
    # Raises RuntimeError if apply() hasn't been called yet

def get_feature_names(self) -> list:
    # Returns output column names after encoding
    # Raises RuntimeError if apply() hasn't been called yet

---

FILE 5: models.py

Class Behaviour

Models takes a Data object (must be split) and provides a unified interface to train scikit-learn, XGBoost, LightGBM, and PyTorch neural networks.

Constructor

def __init__(self, data: Data):
    if not data._is_split:
        raise ValueError(
            "Data must be split before training. Call data.split() first."
        )
    self._data = data
    self.current_model = None
    self.model_name = None
    self.params = {}
    self.compare_results = None

Model Registry

Maintain a dict mapping string names to model constructors. Task type determines which variant to use.

_REGISTRY = {
    "random_forest":      {"binary": RandomForestClassifier,
                           "multiclass": RandomForestClassifier,
                           "regression": RandomForestRegressor},
    "logistic_regression":{"binary": LogisticRegression,
                           "multiclass": LogisticRegression,
                           "regression": None},  # not applicable
    "linear_regression":  {"regression": LinearRegression,
                           "binary": None,
                           "multiclass": None},
    "svm":                {"binary": SVC,
                           "multiclass": SVC,
                           "regression": SVR},
    "knn":                {"binary": KNeighborsClassifier,
                           "multiclass": KNeighborsClassifier,
                           "regression": KNeighborsRegressor},
    "decision_tree":      {"binary": DecisionTreeClassifier,
                           "multiclass": DecisionTreeClassifier,
                           "regression": DecisionTreeRegressor},
    "gradient_boosting":  {"binary": GradientBoostingClassifier,
                           "multiclass": GradientBoostingClassifier,
                           "regression": GradientBoostingRegressor},
    "naive_bayes":        {"binary": GaussianNB,
                           "multiclass": GaussianNB,
                           "regression": None},
    "ridge":              {"regression": Ridge,
                           "binary": None,
                           "multiclass": None},
    "lasso":              {"regression": Lasso,
                           "binary": None,
                           "multiclass": None},
    "xgboost":            {"binary": "xgboost.XGBClassifier",
                           "multiclass": "xgboost.XGBClassifier",
                           "regression": "xgboost.XGBRegressor"},
    "lightgbm":           {"binary": "lightgbm.LGBMClassifier",
                           "multiclass": "lightgbm.LGBMClassifier",
                           "regression": "lightgbm.LGBMRegressor"},
}

For string values (lazy imports), parse the module and class name and use lazy_import to load them on demand.

m.compare(include=None, time_limit=None) -> pd.DataFrame

Train every model in the registry on X_train, score on X_test. Skip models where the variant for the current task is None. If include is provided, only train those models. If time_limit is provided (seconds), stop after that time.

For each model, catch all exceptions and record as "ERROR" in results — compare() should never crash even if one model fails.

Primary metric:

• binary → AUC-ROC
• multiclass → macro F1
• regression → R²

Print ranked table:

Pythios · Model Comparison  (task: binary, metric: AUC-ROC)
────────────────────────────────────────────────────────────────────
 Rank  Model                Score     Time(s)
────────────────────────────────────────────────────────────────────
 1     xgboost              0.891     1.2
 2     random_forest        0.883     0.8
 3     gradient_boosting    0.876     2.1
 ...
────────────────────────────────────────────────────────────────────
 Best: xgboost · run m.train("xgboost") to use it

Store results in self.compare_results (DataFrame). Return the DataFrame.

m.train(model_name, **kwargs) -> self

Look up model in registry for current task type. Raise ValueError if model is None for this task (e.g. "lasso" for binary). Raise ValueError if model_name not in registry.

For lazy-imported models (xgboost, lightgbm), parse the module path string and load with lazy_import.

For "neural_net": see Neural Network Builder section below.

Fit on X_train, y_train. Store in self.current_model, self.model_name, self.params = kwargs. Print: ✓ Trained random_forest on 712 samples

m.tune(model_name, trials=50) -> self

Use Optuna (lazy import) to tune hyperparameters. Define an objective function that:

1. Samples params using trial.suggest_* methods
2. Builds and fits model on X_train
3. Returns score on X_test

After optimization, retrain with best params using self.train(). Print best params and best score.

Default search spaces per model:

• random_forest: n_estimators (50-500), max_depth (3-20), min_samples_split (2-20)
• xgboost: n_estimators (50-500), max_depth (3-10), learning_rate (0.01-0.3), subsample (0.6-1.0)
• lightgbm: n_estimators (50-500), num_leaves (20-300), learning_rate (0.01-0.3)

Suppress Optuna logging (optuna.logging.set_verbosity(optuna.logging.WARNING)).

Neural Network Builder

When model_name == "neural_net":

def _build_and_train_nn(self, layers=[128, 64], epochs=50, lr=0.001, backend="pytorch"):

For PyTorch backend:

1. lazy_import("torch")
2. Convert X_train, X_test, y_train, y_test to tensors
3. Build MLP: Linear layers with ReLU activations between them
4. Final activation: Sigmoid (binary), Softmax (multiclass), none (regression)
5. Loss: BCELoss (binary), CrossEntropyLoss (multiclass), MSELoss (regression)
6. Optimizer: Adam with lr
7. Training loop: epochs iterations over batched data
8. Print progress every 10 epochs: Epoch 10/50 — loss: 0.342
9. Wrap trained model in a sklearn-compatible wrapper with .predict() and .predict_proba()

For TensorFlow backend:

1. lazy_import("tensorflow")
2. Build Sequential model with Dense layers
3. Compile with appropriate optimizer and loss
4. Fit with validation_split=0.1

m.save(path) -> None

Use joblib.dump to save:

{
    "model": self.current_model,
    "model_name": self.model_name,
    "params": self.params,
    "task_type": self._data.task_type,
    "feature_names": self._data.X_train.columns.tolist(),
    "target": self._data.target,
    "saved_at": str(datetime.now()),
}

m.load(path) -> self

Use joblib.load. Restore model and metadata.

m.predict(new_data) -> np.ndarray

Accepts Data object or pd.DataFrame. If Data object, use ._df (drop target column if present). Return predictions from self.current_model.predict().

m.predict_proba(new_data) -> np.ndarray

Same as predict but uses .predict_proba(). Raise ValueError if model doesn't support probability prediction.

---

FILE 6: evaluate.py

Constructor

def __init__(self, model: Models):
    self._model = model
    self._data = model._data

e.metrics(on="test") -> dict

Get predictions then compute task-appropriate metrics.

Binary metrics:

from sklearn.metrics import (
    accuracy_score, f1_score, precision_score, recall_score,
    roc_auc_score, matthews_corrcoef
)
return {
    "accuracy": accuracy_score(y_true, y_pred),
    "f1": f1_score(y_true, y_pred, average="binary"),
    "precision": precision_score(y_true, y_pred),
    "recall": recall_score(y_true, y_pred),
    "auc_roc": roc_auc_score(y_true, y_pred),
    "mcc": matthews_corrcoef(y_true, y_pred),
}

Multiclass metrics:

return {
    "accuracy": accuracy_score(y_true, y_pred),
    "macro_f1": f1_score(y_true, y_pred, average="macro"),
    "weighted_f1": f1_score(y_true, y_pred, average="weighted"),
}

Regression metrics:

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
return {
    "mae": mean_absolute_error(y_true, y_pred),
    "rmse": np.sqrt(mean_squared_error(y_true, y_pred)),
    "r2": r2_score(y_true, y_pred),
}

Print formatted table of metrics. Return dict.

e.overfit_check()

Call metrics(on="train") and metrics(on="test"). Print both side by side. For each metric, if gap > 0.1 print a warning flag.

e.confusion_matrix(normalise=True)

Use sklearn.metrics.confusion_matrix. Plot with matplotlib using imshow. Add text annotations (actual counts). Title: "Confusion Matrix — {model_name}".

e.roc_curve()

Binary: plot ROC curve with AUC in legend. Multiclass: one-vs-rest ROC curve per class. Raise ValueError for regression tasks.

e.precision_recall_curve()

Binary only. Plot precision-recall curve.

e.residuals()

Regression only. Two subplots:

1. Predicted vs actual scatter (with y=x line)
2. Residual distribution histogram

e.shap(sample=None)

Lazy import shap. If sample is not None, use shap.sample(X, sample). Auto-select explainer (see below). Plot shap.summary_plot (beeswarm).

Explainer selection:

tree_types = {RandomForestClassifier, RandomForestRegressor,
              GradientBoostingClassifier, GradientBoostingRegressor,
              DecisionTreeClassifier, DecisionTreeRegressor}
# Also check for XGBClassifier/XGBRegressor by name

if type(model) in tree_types or "XGB" in type(model).__name__ or "LGBM" in type(model).__name__:
    explainer = shap.TreeExplainer(model)
elif type(model) in {LogisticRegression, Ridge, Lasso}:
    explainer = shap.LinearExplainer(model, X)
else:
    explainer = shap.KernelExplainer(model.predict, shap.sample(X, 100))

e.shap_waterfall(index=0)

Single prediction explanation. Use shap.plots.waterfall.

e.shap_dependence(feature)

Dependence plot for one feature. Use shap.dependence_plot.

e.feature_importance() -> pd.DataFrame

Try model.feature_importances_ first (RF, XGB, LGBM, tree models). Fall back to permutation_importance(model, X_test, y_test, n_repeats=10). Plot horizontal bar chart (top 20 features max). Return DataFrame with Feature and Importance columns.

e.cross_validate(k=5, stratify=True) -> dict

Use sklearn cross_val_score on FULL dataset (not just train split). Use StratifiedKFold for classification, KFold for regression. Print: mean ± std per metric. Return dict of scores.

e.report(save=None)

Print complete plain-text summary (no matplotlib):

• Model name and task type
• Test set size
• All metrics
• Top 5 features by importance (if available)
• Any warnings

If save is provided, write to that file path.

---

FILE 7: visualize.py

Constructor

def __init__(self, data: Data):
    self._data = data
    import matplotlib.pyplot as plt
    import seaborn as sns
    plt.style.use("seaborn-v0_8-whitegrid")

All methods accept save=None (path to save instead of showing) and interactive=False (use plotly instead of matplotlib).

Methods

v.distribution(col=None, save=None, interactive=False)

If col is None: histograms for all numeric columns in a grid layout. If col is specified: single histogram with KDE overlay. Auto-size figure based on number of columns.

v.correlation(save=None)

Pearson correlation heatmap for all numeric columns. Use seaborn.heatmap with annot=True, fmt=".2f", cmap="coolwarm".

v.missing(save=None)

Missingness matrix. White = present, black/red = missing. Plot using matplotlib imshow on df.isnull() boolean mask. Show percentage missing in ylabel.

v.pairplot(save=None)

seaborn pairplot. If target is set and task is classification, colour by target using hue=self._data.target.

v.class_balance(save=None)

Bar chart of target value counts. Only for classification tasks. Raise ValueError for regression.

v.target_vs(feature, save=None)

If regression: scatter plot of feature (x) vs target (y). If classification: violin plot, x=target classes, y=feature values.

v.target_vs_all(save=None)

Grid of target_vs for every feature. Auto-size figure.

v.feature_importance(model, save=None)

Horizontal bar chart. Accepts a trained Models object. Uses model.current_model.feature_importances_ if available.

v.learning_curve(model, save=None)

Train vs validation score over increasing training set sizes. Use sklearn.model_selection.learning_curve.

v.residuals(model, save=None)

Regression only. Delegates to Evaluate(model).residuals().

v.pca_explained(save=None)

Scree plot — cumulative variance explained by PCA components. Fit PCA on X_train (or full _df if not split). Plot with a horizontal line at 95%.

v.cluster_plot(labels, save=None)

Scatter of first 2 PCA components, coloured by label array. Used after Unsupervised clustering.

v.summary(save=None)

2x2 grid: distribution (numeric only) + correlation + missing + class balance. All in one figure.

---

FILE 8: unsupervised.py

Constructor

def __init__(self, data: Data):
    self._data = data
    self.labels = None
    self.embeddings = None
    # Extract feature matrix (drop target if present)
    if data.target and data.target in data._df.columns:
        self._X = data._df.drop(columns=[data.target]).select_dtypes(include="number").values
    else:
        self._X = data._df.select_dtypes(include="number").values

u.cluster(method="kmeans", **kwargs) -> self

Methods and their kwargs:

• "kmeans": k (default 5), random_state=42, n_init=10
• "dbscan": eps (default 0.5), min_samples (default 5)
• "agglomerative": k (default 5), linkage (default "ward")
• "gaussian_mixture": k (default 5), random_state=42

Fit the chosen model on self._X. Store predicted labels in self.labels. Print: ✓ KMeans clustering complete — 5 clusters found Return self.

u.find_k(max_k=15) -> int

Run KMeans for k in range(2, max_k+1). Compute inertia and silhouette_score for each k. Plot two subplots side by side: elbow (inertia) and silhouette. Find best k as argmax of silhouette scores. Print: Suggested k = {best_k} (highest silhouette score) Return best_k as int.

u.reduce(method="pca", n=2, **kwargs) -> self

Methods:

• "pca": sklearn.decomposition.PCA(n_components=n)
• "tsne": sklearn.manifold.TSNE(n_components=n, perplexity=kwargs.get("perplexity", 30))
• "umap": lazy_import("umap", "umap-learn"), UMAP(n_components=n, n_neighbors=kwargs.get("n_neighbors", 15))

Fit and transform self._X. Store result in self.embeddings. Print: ✓ Reduced to {n}D using {method} — shape: ({len(X)}, {n}) Return self.

u.plot_clusters(labels=None, save=None)

2D scatter plot. If labels is None, use self.labels. If self.embeddings is not None and has 2 columns, use it for x/y. Otherwise reduce to 2D with PCA first. Colour points by label. Add legend.

u.silhouette_score() -> float

Requires self.labels to be set. Use sklearn.metrics.silhouette_score(self._X, self.labels). Print: Silhouette score: {score:.4f} (range: -1 to 1, higher is better) Return float.

u.davies_bouldin_score() -> float

Use sklearn.metrics.davies_bouldin_score(self._X, self.labels). Print: Davies-Bouldin score: {score:.4f} (lower is better) Return float.

---

FILE 9: embeddings.py

The most standalone class — does not require Data object (but can accept one).

Constructor

def __init__(self):
    self._text_model = None
    self._text_model_name = None
    self._image_model = None
    self._image_model_name = None
    self._index = None
    self.dim = None

emb.text(model_name="all-MiniLM-L6-v2") -> self

Store model_name. Don't load yet (load lazily at encode time). Return self.

emb.encode(texts: list) -> np.ndarray

Lazy load sentence_transformers.SentenceTransformer if not loaded. Call .encode(texts, convert_to_numpy=True, show_progress_bar=len(texts)>100). Store self.dim = result.shape[1]. Return np.ndarray of shape (N, dim).

emb.image(model_name="resnet50") -> self

Store model_name. Return self.

emb.encode_images(paths: list) -> np.ndarray

Lazy import torch, torchvision, PIL.Image. Standard ImageNet preprocessing:

transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

Load model from torchvision.models, remove final classification layer. Run inference in torch.no_grad() context. Return np.ndarray of shape (N, dim).

emb.similarity(vec_a, vec_b) -> float

Cosine similarity between two vectors.

a = vec_a / (np.linalg.norm(vec_a) + 1e-9)
b = vec_b / (np.linalg.norm(vec_b) + 1e-9)
return float(np.dot(a, b))

emb.similarity_matrix(vectors) -> np.ndarray

Pairwise cosine similarity matrix. Shape (N, N). Normalize all vectors first, then return vectors @ vectors.T.

emb.search(query, corpus_vectors, top_k=5) -> list

If query is a string, encode it first. Compute cosine similarity with corpus_vectors. Return list of (index, score) tuples sorted by score descending.

emb.build_index(vectors) -> self

Lazy import faiss (install_name="faiss-cpu").

dim = vectors.shape[1]
self._index = faiss.IndexFlatL2(dim)
self._index.add(vectors.astype("float32"))

Print: ✓ Faiss index built — {len(vectors)} vectors, dim={dim} Return self.

emb.search_index(query_vec, top_k=5) -> list

Requires self._index. Raise RuntimeError if not built. Return list of (index, distance) tuples.

emb.save_index(path) -> None

Lazy import faiss. faiss.write_index(self._index, path).

emb.load_index(path) -> self

faiss.read_index(path). Return self.

---

Tests

Write tests in the tests/ folder. Each test file tests one class.

tests/test_data.py

The following tests must all pass:

import pandas as pd
import pytest
from pythios import Data


def make_sample_df():
    return pd.DataFrame({
        "age":    [25, 30, 35, 40, 45],
        "salary": [50000, 60000, 70000, 80000, 90000],
        "hired":  [1, 0, 1, 0, 1],
    })


def test_from_dataframe():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert d.shape == (5, 3)
    assert d.target == "hired"


def test_task_type_binary():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert d.task_type == "binary"


def test_task_type_regression():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="salary")
    assert d.task_type == "regression"


def test_task_type_multiclass():
    df = pd.DataFrame({
        "x": range(6),
        "y": [0, 1, 2, 0, 1, 2]
    })
    d = Data.from_dataframe(df, target="y")
    assert d.task_type == "multiclass"


def test_task_type_string_binary():
    df = pd.DataFrame({"x": [1, 2, 3], "y": ["cat", "dog", "cat"]})
    d = Data.from_dataframe(df, target="y")
    assert d.task_type == "binary"


def test_split():
    df = pd.DataFrame({"x": range(100), "y": [i % 2 for i in range(100)]})
    d = Data.from_dataframe(df, target="y")
    d.split(test=0.2, seed=42)
    assert len(d.X_train) == 80
    assert len(d.X_test) == 20
    assert d._is_split is True


def test_split_twice_raises_error():
    df = pd.DataFrame({"x": range(50), "y": [i % 2 for i in range(50)]})
    d = Data.from_dataframe(df, target="y")
    d.split(test=0.2)
    with pytest.raises(ValueError):
        d.split(test=0.2)


def test_drop_columns():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    d2 = d.drop_columns(["age"])
    assert "age" not in d2.columns
    assert "salary" in d2.columns


def test_select_columns():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    d2 = d.select_columns(["age", "hired"])
    assert d2.columns == ["age", "hired"]


def test_sample():
    df = pd.DataFrame({"x": range(100), "y": range(100)})
    d = Data.from_dataframe(df, target="y")
    d2 = d.sample(n=10)
    assert len(d2._df) == 10


def test_copy():
    df = make_sample_df()
    d1 = Data.from_dataframe(df, target="hired")
    d2 = d1.copy()
    d2._df.loc[0, "age"] = 999
    assert d1._df.loc[0, "age"] == 25


def test_shape_property():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert d.shape == (5, 3)


def test_columns_property():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert "age" in d.columns
    assert "salary" in d.columns
    assert "hired" in d.columns


def test_invalid_target_raises_error():
    df = make_sample_df()
    with pytest.raises(ValueError):
        Data.from_dataframe(df, target="nonexistent_column")


def test_from_sklearn():
    d = Data.from_sklearn("iris")
    assert d.shape[0] == 150
    assert d.target == "target"


def test_from_sklearn_invalid():
    with pytest.raises(ValueError):
        Data.from_sklearn("does_not_exist")


def test_repr():
    df = make_sample_df()
    d = Data.from_dataframe(df, target="hired")
    assert "Data" in repr(d)


def test_missing_report():
    df = pd.DataFrame({
        "a": [1, None, 3],
        "b": [1, 2, 3],
        "y": [0, 1, 0]
    })
    d = Data.from_dataframe(df, target="y")
    report = d.missing_report()
    assert "a" in report["Column"].values
    assert "b" not in report["Column"].values

tests/test_tools.py

import pandas as pd
import pytest
from pythios import Data, Tools


def make_data():
    df = pd.DataFrame({
        "age":    [25, 30, 35, 40, 45, 50, 55, 60],
        "salary": [50000, 60000, 70000, 80000, 90000, 100000, 110000, 120000],
        "score":  [1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8],
        "hired":  [0, 1, 0, 1, 0, 1, 0, 1],
    })
    return Data.from_dataframe(df, target="hired")


def test_profile_returns_dataframe():
    d = make_data()
    result = Tools.profile(d)
    assert isinstance(result, pd.DataFrame)
    assert "Column" in result.columns


def test_memory_usage_returns_string():
    d = make_data()
    result = Tools.memory_usage(d)
    assert isinstance(result, str)
    assert any(unit in result for unit in ["B", "KB", "MB", "GB"])


def test_detect_outliers_iqr():
    d = make_data()
    result = Tools.detect_outliers(d, method="iqr")
    assert isinstance(result, pd.DataFrame)
    assert "Outlier_Count" in result.columns


def test_detect_outliers_zscore():
    d = make_data()
    result = Tools.detect_outliers(d, method="zscore", threshold=3.0)
    assert isinstance(result, pd.DataFrame)


def test_detect_outliers_invalid_method():
    d = make_data()
    with pytest.raises(ValueError):
        Tools.detect_outliers(d, method="invalid")


def test_validate_clean_data():
    d = make_data()
    result = Tools.validate(d)
    assert result is True


def test_validate_empty_data():
    df = pd.DataFrame({"x": [], "y": []})
    d = Data(df, target="y")
    result = Tools.validate(d)
    assert result is False


def test_export_csv(tmp_path):
    d = make_data()
    path = str(tmp_path / "test.csv")
    Tools.export(d, path)
    import pandas as pd
    loaded = pd.read_csv(path)
    assert len(loaded) == 8


def test_export_json(tmp_path):
    d = make_data()
    path = str(tmp_path / "test.json")
    Tools.export(d, path)
    import json
    with open(path) as f:
        data = json.load(f)
    assert len(data) == 8


def test_export_invalid_extension(tmp_path):
    d = make_data()
    with pytest.raises(ValueError):
        Tools.export(d, str(tmp_path / "test.xyz"))


def test_export_train_split(tmp_path):
    d = make_data()
    d.split(test=0.25, seed=42)
    path = str(tmp_path / "train.csv")
    Tools.export(d, path, split="train")
    import pandas as pd
    loaded = pd.read_csv(path)
    assert len(loaded) == 6


def test_export_no_split_raises(tmp_path):
    d = make_data()
    with pytest.raises(ValueError):
        Tools.export(d, str(tmp_path / "train.csv"), split="train")


def test_timer():
    import time
    with Tools.timer("test operation"):
        time.sleep(0.01)


def test_compare_datasets():
    d1 = make_data()
    d2 = make_data()
    result = Tools.compare_datasets(d1, d2)
    assert isinstance(result, pd.DataFrame)


def test_set_seed():
    Tools.set_seed(42)
    import numpy as np
    val1 = np.random.rand()
    Tools.set_seed(42)
    val2 = np.random.rand()
    assert val1 == val2


def test_balance_oversample():
    df = pd.DataFrame({
        "x": range(15),
        "y": [0]*10 + [1]*5
    })
    d = Data.from_dataframe(df, target="y")
    balanced = Tools.balance(d, method="oversample")
    counts = balanced._df["y"].value_counts()
    assert counts[0] == counts[1]


def test_balance_undersample():
    df = pd.DataFrame({
        "x": range(15),
        "y": [0]*10 + [1]*5
    })
    d = Data.from_dataframe(df, target="y")
    balanced = Tools.balance(d, method="undersample")
    counts = balanced._df["y"].value_counts()
    assert counts[0] == counts[1]


def test_balance_no_target():
    df = pd.DataFrame({"x": range(10)})
    d = Data(df)
    with pytest.raises(ValueError):
        Tools.balance(d)


def test_suggest_runs_without_error():
    d = make_data()
    Tools.suggest(d)


def test_coder_returns_code_result():
    # This test does NOT require Ollama — it tests the CodeResult class directly
    from pythios.tools import CodeResult
    result = CodeResult(code="n = 42", description="test")
    result.run()
    assert result._ran is True
    assert "n" in result.namespace
    assert result.namespace["n"] == 42


def test_code_result_show():
    from pythios.tools import CodeResult
    result = CodeResult(code="x = 1 + 1", description="test")
    result.show()  # should not raise


def test_configure_llm():
    Tools.configure_llm(model="mistral")
    assert Tools._llm_model == "mistral"
    Tools.configure_llm(model="llama3")  # reset

---

demo/demo_all.py — Complete Runnable Demo

This file demonstrates every feature of Pythios. It can be run top to bottom after building the library. It uses only sklearn built-in datasets (no external files needed) so it works immediately.

"""
Pythios — Complete Demo
Run this file to test every feature of the library.
No external data files needed — uses sklearn built-in datasets.

Usage:
    python demo/demo_all.py
"""

import pandas as pd
import numpy as np
from pythios import Data, Tools


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 1: DATA LOADING
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 1: DATA LOADING")
print("="*70)

# Load from sklearn built-in datasets — no file needed
print("\n--- from_sklearn ---")
d_iris        = Data.from_sklearn("iris")
d_wine        = Data.from_sklearn("wine")
d_cancer      = Data.from_sklearn("breast_cancer")
d_digits      = Data.from_sklearn("digits")
d_diabetes    = Data.from_sklearn("diabetes")
d_california  = Data.from_sklearn("california_housing")

# Load from a pandas DataFrame
print("\n--- from_dataframe ---")
df = pd.DataFrame({
    "age":    [25, 30, 35, 40, 45, 50, 55, 60, 65, 70],
    "income": [40000, 50000, 60000, 75000, 85000, 95000, 105000, 115000, 120000, 125000],
    "score":  [1.2, 2.3, 3.4, 4.5, 5.6, 6.7, 7.8, 8.9, 9.1, 9.5],
    "hired":  [0, 0, 1, 1, 0, 1, 1, 1, 0, 1],
})
d_custom = Data.from_dataframe(df, target="hired")
print(d_custom)

# Load from URL (tests requests lazy import)
print("\n--- from_url ---")
d_url = Data.from_url(
    "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv",
    target="Survived"
)
print(d_url)


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 2: DATA INSPECTION
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 2: DATA INSPECTION")
print("="*70)

print("\n--- describe() ---")
d_url.describe()

print("\n--- missing_report() ---")
print(d_url.missing_report())

print("\n--- task types ---")
print(f"iris task type:         {d_iris.task_type}")         # multiclass
print(f"breast_cancer task:     {d_cancer.task_type}")       # binary
print(f"diabetes task:          {d_diabetes.task_type}")     # regression
print(f"california task:        {d_california.task_type}")   # regression
print(f"custom hired task:      {d_custom.task_type}")       # binary

print("\n--- shape, columns, dtypes ---")
print(f"Shape: {d_iris.shape}")
print(f"Columns: {d_iris.columns}")
print(f"Dtypes:\n{d_iris.dtypes}")

print("\n--- head ---")
print(d_iris.head(3))


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 3: DATA MANIPULATION
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 3: DATA MANIPULATION")
print("="*70)

print("\n--- split ---")
d_cancer_work = d_cancer.copy()
d_cancer_work.split(test=0.2, seed=42)
print(f"X_train shape: {d_cancer_work.X_train.shape}")
print(f"X_test shape:  {d_cancer_work.X_test.shape}")

print("\n--- drop_columns ---")
d2 = d_url.drop_columns(["Name", "Ticket", "Cabin"])
print(f"Columns after drop: {d2.columns}")

print("\n--- select_columns ---")
d3 = d_url.select_columns(["Age", "Fare", "Pclass", "Survived"])
print(f"Columns after select: {d3.columns}")

print("\n--- sample ---")
d4 = d_url.sample(n=100)
print(f"Sampled: {d4.shape[0]} rows")

print("\n--- copy ---")
d_original = d_custom.copy()
d_modified = d_original.copy()
d_modified._df.loc[0, "age"] = 999
print(f"Original age[0]: {d_original._df.loc[0, 'age']}")   # still 25
print(f"Modified age[0]: {d_modified._df.loc[0, 'age']}")   # 999


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 4: TOOLS UTILITIES
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 4: TOOLS")
print("="*70)

print("\n--- profile ---")
Tools.profile(d_custom)

print("\n--- suggest ---")
Tools.suggest(d_url)

print("\n--- detect_outliers (IQR) ---")
report = Tools.detect_outliers(d_url, method="iqr")
print(report.head())

print("\n--- detect_outliers (z-score) ---")
report2 = Tools.detect_outliers(d_url, method="zscore", threshold=3.0)
print(report2.head())

print("\n--- validate ---")
is_valid = Tools.validate(d_url)
print(f"Dataset valid: {is_valid}")

print("\n--- memory_usage ---")
print(f"Memory: {Tools.memory_usage(d_url)}")

print("\n--- compare_datasets ---")
d_small = d_url.sample(100)
Tools.compare_datasets(d_url, d_small)

print("\n--- set_seed ---")
Tools.set_seed(42)

print("\n--- timer ---")
import time
with Tools.timer("demo sleep"):
    time.sleep(0.05)

print("\n--- balance (oversample) ---")
d_url_split = d_url.copy()
balanced = Tools.balance(d_url_split, method="oversample", seed=42)
print(f"Balanced target counts:\n{balanced._df['Survived'].value_counts()}")

print("\n--- balance (undersample) ---")
undersampled = Tools.balance(d_url_split, method="undersample", seed=42)
print(f"Undersampled target counts:\n{undersampled._df['Survived'].value_counts()}")

print("\n--- export (CSV) ---")
Tools.export(d_custom, "/tmp/pythios_demo_export.csv")

print("\n--- export (JSON) ---")
Tools.export(d_custom, "/tmp/pythios_demo_export.json")

print("\n--- export train split ---")
d_cancer_work_export = d_cancer.copy()
d_cancer_work_export.split(test=0.2)
Tools.export(d_cancer_work_export, "/tmp/pythios_train.csv", split="train")
Tools.export(d_cancer_work_export, "/tmp/pythios_test.csv", split="test")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 5: PREPROCESSING
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 5: PREPROCESS")
print("="*70)

try:
    from pythios import Preprocess

    d_titanic = d_url.drop_columns(["Name", "Ticket", "Cabin", "PassengerId"])
    d_titanic.split(test=0.2, seed=42)

    print("\n--- auto() ---")
    p = Preprocess(d_titanic)
    p.auto()
    clean = p.apply()
    print(f"Before: {d_titanic.X_train.shape}")
    print(f"After:  {clean.X_train.shape}")

    print("\n--- manual pipeline ---")
    d_titanic2 = d_url.drop_columns(["Name", "Ticket", "Cabin", "PassengerId"])
    d_titanic2.split(test=0.2, seed=42)
    p2 = Preprocess(d_titanic2)
    p2.impute("median")
    p2.scale("robust")
    p2.encode("onehot")
    clean2 = p2.apply()
    print(f"Manual pipeline result shape: {clean2.X_train.shape}")

    Tools.compare_datasets(d_titanic, clean)

except ImportError:
    print("Preprocess not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 6: MODELS
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 6: MODELS")
print("="*70)

try:
    from pythios import Models

    d_cancer_m = d_cancer.copy()
    d_cancer_m.split(test=0.2, seed=42)
    m = Models(d_cancer_m)

    print("\n--- compare() ---")
    results = m.compare()
    print(results.head())

    print("\n--- train random_forest ---")
    m.train("random_forest", n_estimators=100)

    print("\n--- train logistic_regression ---")
    m2 = Models(d_cancer_m)
    m2.train("logistic_regression")

    print("\n--- train xgboost ---")
    try:
        m3 = Models(d_cancer_m)
        m3.train("xgboost")
    except ImportError as e:
        print(f"XGBoost not installed: {e}")

    print("\n--- predict ---")
    preds = m.predict(d_cancer_m)
    print(f"Predictions shape: {preds.shape}")
    print(f"First 5: {preds[:5]}")

    print("\n--- save and load ---")
    m.save("/tmp/pythios_rf_model.pkl")
    m_loaded = Models(d_cancer_m)
    m_loaded.load("/tmp/pythios_rf_model.pkl")
    print(f"Loaded model name: {m_loaded.model_name}")

    print("\n--- tune (quick, 10 trials) ---")
    m_tune = Models(d_cancer_m)
    m_tune.tune("random_forest", trials=10)

    # Regression task
    print("\n--- regression model ---")
    d_diab = d_diabetes.copy()
    d_diab.split(test=0.2, seed=42)
    m_reg = Models(d_diab)
    m_reg.train("random_forest")
    preds_reg = m_reg.predict(d_diab)
    print(f"Regression predictions sample: {preds_reg[:3]}")

except ImportError:
    print("Models not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 7: EVALUATE
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 7: EVALUATE")
print("="*70)

try:
    from pythios import Evaluate, Models

    d_cancer_e = d_cancer.copy()
    d_cancer_e.split(test=0.2, seed=42)
    m_eval = Models(d_cancer_e)
    m_eval.train("random_forest")
    e = Evaluate(m_eval)

    print("\n--- metrics (test) ---")
    metrics = e.metrics(on="test")
    print(metrics)

    print("\n--- metrics (train) ---")
    metrics_train = e.metrics(on="train")
    print(metrics_train)

    print("\n--- overfit_check ---")
    e.overfit_check()

    print("\n--- cross_validate ---")
    cv_results = e.cross_validate(k=5)
    print(cv_results)

    print("\n--- feature_importance ---")
    fi = e.feature_importance()
    print(fi.head())

    print("\n--- report ---")
    e.report()

    print("\n--- report (save) ---")
    e.report(save="/tmp/pythios_eval_report.txt")
    print("Report saved to /tmp/pythios_eval_report.txt")

    print("\n--- confusion_matrix ---")
    e.confusion_matrix()

    print("\n--- roc_curve ---")
    e.roc_curve()

    print("\n--- shap ---")
    try:
        e.shap(sample=50)
    except ImportError:
        print("shap not installed — skipping")

    # Regression evaluate
    print("\n--- regression evaluate ---")
    d_diab_e = d_diabetes.copy()
    d_diab_e.split(test=0.2, seed=42)
    m_diab = Models(d_diab_e)
    m_diab.train("random_forest")
    e_reg = Evaluate(m_diab)
    e_reg.metrics()
    e_reg.residuals()

except ImportError:
    print("Evaluate not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 8: VISUALIZE
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 8: VISUALIZE")
print("="*70)

try:
    from pythios import Visualize

    d_vis = d_url.copy()

    print("\n--- distribution (all columns) ---")
    v = Visualize(d_vis)
    v.distribution(save="/tmp/pythios_distribution.png")

    print("\n--- distribution (single column) ---")
    v.distribution("Age", save="/tmp/pythios_dist_age.png")

    print("\n--- correlation ---")
    v.correlation(save="/tmp/pythios_correlation.png")

    print("\n--- missing ---")
    v.missing(save="/tmp/pythios_missing.png")

    print("\n--- class_balance ---")
    v.class_balance(save="/tmp/pythios_balance.png")

    print("\n--- target_vs ---")
    v.target_vs("Age", save="/tmp/pythios_target_age.png")
    v.target_vs("Fare", save="/tmp/pythios_target_fare.png")

    print("\n--- pca_explained ---")
    d_vis_num = d_url.drop_columns(["Name", "Ticket", "Cabin"])
    d_vis_num.split(test=0.2)
    v2 = Visualize(d_vis_num)
    v2.pca_explained(save="/tmp/pythios_pca.png")

    print("\n--- summary ---")
    v.summary(save="/tmp/pythios_summary.png")

    print("All plots saved to /tmp/")

except ImportError:
    print("Visualize not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 9: UNSUPERVISED
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 9: UNSUPERVISED")
print("="*70)

try:
    from pythios import Unsupervised

    d_clust = d_iris.copy()

    print("\n--- kmeans clustering ---")
    u = Unsupervised(d_clust)
    u.cluster("kmeans", k=3)
    print(f"Labels: {u.labels[:10]}")

    print("\n--- find_k ---")
    best_k = u.find_k(max_k=10)
    print(f"Best k: {best_k}")

    print("\n--- dbscan ---")
    u2 = Unsupervised(d_clust)
    u2.cluster("dbscan", eps=1.0, min_samples=5)
    print(f"DBSCAN labels (unique): {np.unique(u2.labels)}")

    print("\n--- PCA reduction ---")
    u3 = Unsupervised(d_clust)
    u3.reduce("pca", n=2)
    print(f"Embeddings shape: {u3.embeddings.shape}")

    print("\n--- t-SNE reduction ---")
    u4 = Unsupervised(d_clust)
    u4.reduce("tsne", n=2)
    print(f"t-SNE embeddings shape: {u4.embeddings.shape}")

    print("\n--- UMAP reduction ---")
    try:
        u5 = Unsupervised(d_clust)
        u5.reduce("umap", n=2)
        print(f"UMAP embeddings shape: {u5.embeddings.shape}")
    except ImportError:
        print("umap-learn not installed — skipping UMAP")

    print("\n--- cluster_plot ---")
    u6 = Unsupervised(d_clust)
    u6.reduce("pca", n=2)
    u6.cluster("kmeans", k=3)
    u6.plot_clusters(save="/tmp/pythios_clusters.png")

    print("\n--- silhouette_score ---")
    u6.silhouette_score()

    print("\n--- davies_bouldin_score ---")
    u6.davies_bouldin_score()

except ImportError:
    print("Unsupervised not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 10: EMBEDDINGS
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 10: EMBEDDINGS")
print("="*70)

try:
    from pythios import Embeddings

    emb = Embeddings()

    print("\n--- text embeddings ---")
    emb.text("all-MiniLM-L6-v2")
    texts = [
        "Machine learning is fascinating",
        "Deep learning uses neural networks",
        "Python is great for data science",
        "I love pizza and pasta",
        "The weather is sunny today",
    ]
    vectors = emb.encode(texts)
    print(f"Text vectors shape: {vectors.shape}")
    print(f"Embedding dim: {emb.dim}")

    print("\n--- similarity ---")
    score = emb.similarity(vectors[0], vectors[1])
    print(f"Similarity (ML vs DL): {score:.4f}")

    score2 = emb.similarity(vectors[0], vectors[3])
    print(f"Similarity (ML vs pizza): {score2:.4f}")

    print("\n--- similarity_matrix ---")
    matrix = emb.similarity_matrix(vectors)
    print(f"Similarity matrix shape: {matrix.shape}")

    print("\n--- search ---")
    query = "neural networks and AI"
    results = emb.search(query, vectors, top_k=3)
    print(f"Search results for '{query}':")
    for idx, score in results:
        print(f"  [{idx}] score={score:.4f} — {texts[idx]}")

    print("\n--- Faiss index ---")
    try:
        emb.build_index(vectors)
        query_vec = emb.encode(["artificial intelligence"])[0]
        index_results = emb.search_index(query_vec, top_k=3)
        print(f"Faiss search results: {index_results}")

        emb.save_index("/tmp/pythios_demo.faiss")
        emb2 = Embeddings()
        emb2.load_index("/tmp/pythios_demo.faiss")
        print("Faiss index saved and loaded successfully")

    except ImportError:
        print("faiss-cpu not installed — skipping Faiss index demo")

    print("\n--- image embeddings ---")
    print("(Skipping image embeddings — no test images available)")
    print("Usage:")
    print("  emb.image('resnet50')")
    print("  vecs = emb.encode_images(['cat.jpg', 'dog.jpg'])")

except ImportError:
    print("Embeddings not built yet — skipping")


# ─────────────────────────────────────────────────────────────────────────────
# SECTION 11: TOOLS LLM (requires Ollama running locally)
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" SECTION 11: TOOLS LLM")
print("="*70)
print("(LLM features require Ollama running at http://localhost:11434)")
print("Install: https://ollama.com | Then: ollama pull llama3")
print()

# Test CodeResult directly (does NOT require Ollama)
print("--- CodeResult (no LLM needed) ---")
from pythios.tools import CodeResult

result = CodeResult(code="total = sum(range(1, 101))", description="sum 1 to 100")
result.show()
result.run()
print(f"total = {result.namespace['total']}")   # 5050

result2 = CodeResult(
    code="fibs = [0, 1]\nfor i in range(8):\n    fibs.append(fibs[-1] + fibs[-2])",
    description="fibonacci"
)
result2.show()
result2.run()
print(f"fibs = {result2.namespace['fibs']}")

print("\n--- LLM features (skip if Ollama not running) ---")
SKIP_LLM = True   # Set to False if Ollama is running

if not SKIP_LLM:
    print("\n--- Tools.llm ---")
    Tools.llm("In one sentence, what is a random forest?")

    print("\n--- Tools.coder ---")
    coder = Tools.coder("compute the factorial of 10 and store it in result")
    coder.run()
    print(f"factorial result = {coder.namespace.get('result')}")

    print("\n--- Tools.explain ---")
    Tools.explain("result = [x**2 for x in range(10) if x % 2 == 0]")

    print("\n--- Tools.review ---")
    Tools.review("def divide(a, b):\n    return a / b")

    print("\n--- Tools.fix ---")
    fixed = Tools.fix("print(10 / 0)", error="ZeroDivisionError: division by zero")
    fixed.run()

    print("\n--- Tools.ask_about_data ---")
    d_ask = Data.from_sklearn("breast_cancer")
    Tools.ask_about_data(d_ask, "Which features are most likely to be predictive?")

    print("\n--- configure_llm ---")
    Tools.configure_llm(model="codellama")
    Tools.configure_llm(model="llama3")   # reset to default


# ─────────────────────────────────────────────────────────────────────────────
# DONE
# ─────────────────────────────────────────────────────────────────────────────

print("\n" + "="*70)
print(" DEMO COMPLETE")
print("="*70)
print("""
All sections that are not 'not built yet' ran successfully.

Next steps:
  1. Run the test suite:  pytest tests/ -v
  2. Try the LLM tools:   set SKIP_LLM = False and start Ollama
  3. pip install pythios[full] to unlock all optional features
""")

---

Key Design Rules — Never Violate These

1. Never import torch, tensorflow, shap, umap, faiss, xgboost, lightgbm, sentence_transformers, or Pillow at the top of any file. Always use lazy_import() inside the method that needs them.

2. Never mutate a Data object in place. Methods that transform data return a new Data object.

3. apply() in Preprocess always fits on X_train only. Never fit on X_test or the full dataset. This is a data leakage violation.

4. Data.split() raises ValueError if called twice. Not a warning — a hard error. Prevents accidental re-splitting.

5. Task type auto-detection is immutable. Once set in __init__, task_type never changes. It propagates downstream.

6. compare() in Models never crashes. Wrap each model in try/except and record errors. Users should always get a result table even if some models fail.

7. Tools is never instantiated. All methods are @staticmethod or @classmethod.

8. report() in Evaluate is plain text only. No matplotlib. Visual methods are separate.

9. The target column is never transformed by Preprocess. It is separated before transformation and reattached to the output Data object.

10. Column names are always preserved. After OneHotEncoding, use get_feature_names_out() and restore as DataFrame. Never work with raw numpy arrays if column names can be preserved.

---

Running the Full Test Suite

# Run all tests
pytest tests/ -v

# Run specific class tests
pytest tests/test_data.py -v
pytest tests/test_tools.py -v
pytest tests/test_models.py -v

# Run demo
python demo/demo_all.py

# Run with coverage
pip install pytest-cov
pytest tests/ --cov=pythios --cov-report=term-missing

Expected results:

• All tests in test_data.py pass (14 tests)
• All tests in test_tools.py pass (18 tests)
• demo/demo_all.py runs without errors for all built sections
• Sections for unbuilt classes print "not built yet — skipping" gracefully

---

LLM Setup (for Tools.coder and Tools.llm)

Pythios uses Ollama for local LLM inference. No API key needed. All inference is local — no data leaves your machine.

# Install Ollama
# Visit https://ollama.com and download for your platform

# Pull a model (pick one)
ollama pull llama3          # general purpose, recommended default
ollama pull codellama       # better for code generation
ollama pull mistral         # fast and capable
ollama pull phi3            # lightweight, runs on low-RAM machines

# Start the server
ollama serve

# In Python
from pythios import Tools
Tools.configure_llm(model="codellama")   # switch models anytime
Tools.llm("Explain the bias-variance tradeoff")

---

Version History

• 0.1.0 — Initial release. Data, Tools, Preprocess, Models, Evaluate, Visualize, Unsupervised, Embeddings all implemented.