Diag3D — machine-learning behavior, made spatial

Diag3D transforms machine-learning datasets and model outputs into interactive 3D environments for understanding data before training and interrogating prediction, error, failure, and repair after training.

DATA → MODEL → PREDICTION → ERROR → FAILURE → REPAIR

One viewport retains camera orientation, selected observations or regions, and filters as you move through the model. The existing v0.2 research engine supplies advanced failure discovery, null calibration, full-refit stability and nested repair validation. The evidence report remains available separately.

Version 0.2.0 is the first public release: pip install diag3d. This is research software; visual patterns and diagnostic explanations do not establish causality or universal statistical error control.

Enter a dataset or results table

# Install (Python 3.11+).
pip install diag3d

# Pre-run: schema in the terminal, then the interactive environment.
diag3d inspect data.csv
diag3d report data.csv --target y --open

# Post-run: first-class CSV workflow, no serialized model required.
diag3d report results.csv --target y --prediction y_pred --open

# Classification: specify the task and optional probability mapping explicitly.
diag3d report classes.csv --target class --prediction predicted_class \
  --task classification --probabilities '{"0":"p0","1":"p1"}' --open

# Existing manual visualization and static report remain compatible.
diag3d visualize data.csv --x feature_1 --y feature_2 --z feature_3
diag3d report data.csv --target y --legacy --open

report now opens the model-space explorer by default. --model still invokes the existing held-out baseline report. Dataset.report() retains its established API; Dataset.explore() and diag3d.explore() produce the new environment. analyze now writes explorer.html alongside the original research report.html, JSON, and OOF CSV.

The new report default loads all rows. --max-points 12000 caps only rendering; exact descriptive metrics and field aggregation use every loaded row. --sample N explicitly caps analyzed rows and is disclosed. Large HTML payloads still require memory proportional to the loaded dataset.

from diag3d import explore, ModelSpace, SklearnAdapter

space = explore("results.csv", target="y", prediction="y_pred")
space.save("explorer.html")

# model is already fitted in the caller's Python process.
adapter = SklearnAdapter(model, feature_names=["feature_1", "feature_2"])
ModelSpace.from_adapter(dataframe, adapter, target="y").save("model.html")

# Existing research result plus matching original inputs.
ModelSpace.from_research(result, X, y).save("research-model-space.html")

CSV-only fields are piecewise constant means of observed predictions, with other features pooled. They are not invented queries to an unknown model. Direct adapters can generate support-masked, selected-feature model slices with other inputs fixed at medians. Neither representation reconstructs an arbitrary high-dimensional model.

Portable 3D files, without HTML

Export existing predictions as a self-contained .glb scene that researchers can open, rotate, and zoom in a compatible 3D viewer. No model is fitted during export.

pip install "diag3d[glb]"
diag3d export-glb results.csv --x NDMI --y LST --z cover \
  --target richness --prediction y_pred --color error --output error.glb

In Python, use space.export_glb("error.glb", color="error") or space.save("observations.glb"). Scenes contain colored observations, labeled axes, a legend, and embedded original coordinates and row references. They are snapshots; filters and model switching remain in the explorer. Axes are independently normalized, and the default display cap is 12,000 observations. See the GLB guide for model selection, observation IDs, sampling, and interpretation.

Demonstrations

PYTHONPATH=src python examples/model_space_demos.py
# Open gallery/model-space/regression.html and press “28s walkthrough”.

The three offline demonstrations cover the existing interaction regression benchmark, held-out nonlinear class geometry, and real wine chemistry before training. See sources and licenses. Generation reruns the primary 199-null-search / 50-refit / nested-repair experiment; --quick explicitly uses smaller development budgets.

Read the implementation status, model-space architecture, UX specification, validation, audit, and before/after comparison.

Null-validation milestone

The original interaction result is preserved: 220/221 planted observations, IoU 0.995, error lift 4.49×. It exceeds all 199 full-search residual-permutation null maxima (empirical p 0.005, with plug-in-null assumptions). Full-refit bootstrap detects the geometry in 50/50 runs. A repair chosen entirely on training-side validation reduces untouched outer-test MAE from 0.856 to 0.099.

The negative results matter: the initial 99-replicate null banks accepted 25/300 negative-control datasets, and 8/50 in the 50-feature stress test. We retain these results and the 499-replicate sensitivity extension. The larger frozen-bank 50-feature experiment accepted 30/500 (6.0%), with a conditional-bank 95% interval of 4.2%–8.4%. This does not establish universal 5% error control. Read null calibration, independent refits, nested repairs, and visual evidence.

Lifecycle diagnostics: before, during and after training

Enter at any stage; each works on its own (details: docs/lifecycle.md).

• Preflight — data quality and split design, no model training: diag3d preflight data.csv --id sample_id --target y --role split --group site --objective held_out_group
• Training integration — explicit calls from your own loop (TrainingMonitor.log_metrics, record_predictions, a native LightGBM callback); the monitor never owns the loop, folds or seeds.
• Post-hoc only — frozen predictions (CSV/Parquet/NPZ; pickle only when explicitly trusted), no fitting: diag3d posthoc preds.csv --data data.csv --id sample_id --target y --map y_pred=pred --default model=rf,role=oof --split-design kfold (checks that need to know which samples trained each fold run only when the split design is declared or an explicit split manifest is given).

A Run directory (versioned manifest, append-only event log, Parquet predictions, findings) connects the stages and can be reopened to reproduce the diagnostics. Every check yields a finding with evidence, counts, thresholds and a pass/warn/fail/not_evaluable status; missing metadata is never reported as a pass. Runnable examples: examples/lifecycle_1_preflight.py, examples/lifecycle_2_training.py, examples/lifecycle_3_posthoc.py.

Installation

Python 3.11 or newer (CI covers 3.11–3.13 on Linux and 3.12 on Windows).

pip install diag3d
diag3d --version

Optional extras, installed only when you need them:

pip install "diag3d[xgboost]"    # XGBoost baseline
pip install "diag3d[lightgbm]"   # LightGBM baseline and training callback
pip install "diag3d[gallery]"    # Matplotlib scientific figures
pip install "diag3d[glb]"        # text labels in .glb scenes (Pillow)
pip install "diag3d[export]"     # static Plotly export (Kaleido)

macOS users can install the command-line tool through the project's Homebrew tap:

brew tap falhezaimi/tap
brew install diag3d

import diag3d loads only the core dependencies (NumPy, Polars, SciPy, scikit-learn, Typer, Rich); Plotly, Matplotlib, XGBoost and LightGBM are imported by the functions that use them. The supported API is listed in docs/public_api.md.

Contributors

git clone https://github.com/falhezaimi/DIAG3d.git
cd DIAG3d
python -m venv .venv
source .venv/bin/activate  # PowerShell: .venv\Scripts\Activate.ps1
pip install -e ".[dev,gallery]"
diag3d --help

Use python -m diag3d if the console command is not on PATH. See the dependency policy and releasing.md.

import diag3d
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge

print(diag3d.__version__)
X, y = make_regression(n_samples=100, n_features=2, random_state=42)
analyzer = diag3d.FailureAnalyzer(
    models={"linear": Ridge()}, cv=3,
    config=diag3d.AnalysisConfig(seed=42, bootstrap_replicates=0),
)
result = analyzer.fit_analyze(X, y)
result.save("results")

Research outputs record Diag3D/Python/dependency versions, seeds, model parameters, configuration and validation metadata. Save python -m pip freeze with experiments for exact environment reconstruction. Maintainers: follow docs/releasing.md.

Research CLI

diag3d analyze data.csv --target richness --models rf hgb --cv 5 \
  --discover-failures --bootstrap 30 --null-replicates 199 \
  --refit-bootstrap 50 --alpha 0.05 --output results/

# Group identity is excluded from predictors, and groups never cross train/test.
diag3d analyze data.csv --target richness --models rf,hgb \
  --group site --cv 5 --seed 42 --max-feature-pairs 15 \
  --min-region-support 25 --repair --output results/

Outputs: analysis.json, observation/fold/model-aligned oof_predictions.csv, and an offline report.html with model selection, support/disagreement/persistence overlays, exact cell boundaries and region inspection. A report can embed observation indices and diagnostic values; review it before sharing.

Research analysis uses numeric predictors and a finite continuous target. Nonnumeric predictors are listed as omitted. Supply --ignore record_id,post_outcome_measurement for identifiers or inappropriate predictors; no algorithm can determine whether an available feature was actually known at prediction time. Missing predictor values are imputed using training folds only. The legacy commands retain mixed-type descriptive CSV analysis.

Python API

from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
from diag3d import FailureAnalyzer, AnalysisConfig

models = {
    'rf': RandomForestRegressor(n_estimators=100, min_samples_leaf=3),
    'hgb': HistGradientBoostingRegressor(max_iter=100),
}
analyzer = FailureAnalyzer(models=models, cv=5, random_state=42)
result = analyzer.fit_analyze(X, y, groups=site_ids)  # omit groups for random CV
for region in result.regions:
    print(region.region_id, region.metrics.error_lift, region.characterization)
figure = result.visualize()       # Plotly Figure; call .show() in an appropriate environment
result.report('report.html')
result.save('results')

Pass RepeatedKFold(...), GroupKFold(...), or audited train/test index pairs as cv. Identical folds are reused across all models. When groups are supplied, any custom split with group overlap is rejected. Explicit partial holdout requires AnalysisConfig(allow_partial_validation=True); untested observations never become residual evidence.

Advanced controls belong in AnalysisConfig: uniform or quantile grids, support floors, threshold sweep, bootstrap count, feature/pair caps, score weights and characterization rules. A supplied config controls its own seed and job count. Pipelines and arbitrary compatible regression estimators can be passed as named models. Models are cloned, never fitted in place.

Calibrate, refit, and validate repairs

from diag3d import CalibrationConfig, NestedRepairConfig, RepairGuardrail

# Start from an existing result from fit_analyze(X, y).
analyzer.calibrate(X, y, result=result,
    config=CalibrationConfig(replicates=199, strategy="residual_permutation"))
analyzer.refit_bootstrap(X, y, result=result, replicates=50)
result.nested_validation = analyzer.validate_repairs(X, y,
    config=NestedRepairConfig(guardrail=RepairGuardrail(
        minimum_regional_improvement=0.05,
        outside_tolerance=0.02, global_tolerance=0.02)))
result.report("research-evidence.html")

Default residual permutation assumes an adequate mean and exchangeable homogeneous residuals; its estimated reference makes it a plug-in diagnostic. Grouped data requires a domain-appropriate null generator through run_null_bank, rather than row permutation. Full-refit bootstrap currently supports integer CV. The original conditional bootstrap remains labeled and available.

What is implemented

• Typed prediction, field, region, persistence, stability, support, characterization, ranking and repair results.
• KFold, RepeatedKFold and GroupKFold OOF prediction; train-only imputation/scaling and neighbor-support analysis.
• Supported 1D/2D empirical MAE grids, face-connected components, threshold lineages and interpretable cell-union geometry.
• Conditional row/group bootstrap plus identity-isolated full model refits; fold rediscovery; cross-model error and prediction disagreement.
• Complete-search max-statistic null calibration, six negative controls, multiplicity stress tests and detection-rate experiments.
• Nested training-side repair selection with explicit guardrails and untouched outer-test evaluation.
• Twelve requested scientific figures, three publication composites, and additional calibration/bootstrap comparisons in PNG, PDF and SVG.
• Seven transparent diagnostic categories, exposed evidence-score components and nested held-out repair experiments.
• Six deterministic planted-mechanism benchmarks, grid/seed sensitivity and a null-data audit.
• Offline evidence explorer and research report, plus preserved v0.1 diagnostics/visualization commands.

inspect, report, visualize, fit, diagnose, compare, Dataset, Config, inspect_csv, report_csv, and visualize3d remain available. The legacy fit command uses a single holdout; analyze is the research OOF path. See legacy usage.

Reproduce the evidence

python -m pytest -q
python experiments/run_benchmarks.py --output benchmark_outputs --bootstrap 20 --repairs
python experiments/sensitivity.py --output sensitivity_results.json
python experiments/null_validation.py --output results
python experiments/calibration_sensitivity.py --results results --replicates 499 --workers 4
python experiments/search_precision.py --results results --datasets 500 --workers 4
python experiments/plugin_null_pilot.py --results results
python experiments/make_figures.py --results results
python experiments/make_evidence_report.py --results results
python -m build
python -m twine check dist/*

See actual benchmark results, method, architecture, limitations, and prior-art questions.

Scientific limits

OOF prediction avoids training residuals. It does not remove bias from searching and evaluating slices on the same OOF errors. Conditional bootstrap recurrence is not model-refit uncertainty or statistical significance. Full-refit recurrence is also not a confidence region. The max-statistic calibration depends on the specified null; observed false acceptance and Monte Carlo sensitivity remain documented. Shared error does not distinguish irreducible noise from misspecification. Candidate categories and scores prioritize investigation; independent confirmation is required.