honestml

AutoML для табличных данных с прозрачной оценкой качества. honestml разделяет DEV-оценку после выбора модели и оценку на внешнем holdout. Перенос результата на новые данные требует независимой проверки с заранее заданными метрикой и допуском.

It covers binary / multiclass classification and regression behind a clean, extensible core. The honesty is in how it selects: out-of-fold scoring on a shared CV split; a bootstrap equivalence band that, among the statistically indistinguishable best candidates, ships the simplest one; leakage-controlled feature engineering and selection; an optional untouched outer holdout scored exactly once; and reproducible, fingerprinted runs.

from honestml import AutoML

model = AutoML(task="binary").fit(X, y)
proba = model.predict_proba(X_new)
print(model.best_model_id_, model.leaderboard_)

The library is silent by default (a NullHandler on the honestml logger); enable progress with logging.getLogger("honestml").setLevel(logging.INFO) plus logging.basicConfig().

Install

pip install honestml                 # lightweight core (baseline/linear models)
pip install "honestml[boosting]"     # core + catboost, lightgbm, xgboost
pip install "honestml[all]"          # boosting + optuna (HPO), mlflow (tracking), onnx, shap, report and the rest
pip install "honestml[inference]"    # slim serving runtime (load_artifact + predict only)

Requires Python >= 3.10. Heavy dependencies are optional extras and imported lazily — import honestml stays light, and a missing extra fails fast with the exact pip install honestml[...] hint.

The default zoo (models=None) is baseline + linear + catboost + lightgbm. xgboost ships with [boosting] but is opt-in (models=("xgboost", ...)) — it does not earn its CV/HPO budget over catboost/lightgbm on the honesty benchmarks.

What you get

Standalone inference

from honestml import load_artifact

model = load_artifact("artifact_dir/")   # integrity-checked against the sha256 manifest
predictions = model.predict(X_new)

The artifact directory is self-contained — manifest, preprocessing schema and the model body — and loads under the slim honestml[inference] install: no training stack is imported. Boosting models can be saved with structural native bodies (model_format="native"). Trust model: the default body is joblib/pickle — load only artifacts you trust; native bodies contain no pickle (a non-boosting estimator and the optional calibrator still ship as joblib).

Reproducibility

Каждый запуск вычисляет fingerprint по разрешённой конфигурации, данным, каталогу моделей и версиям библиотек. Отчёт сохраняет этот контекст вместе с решениями FS/HPO/ансамбля и выполненной работой. Таймауты HPO и ресурсные ограничения могут изменить объём завершённой работы и итог при том же fingerprint.

Documentation

Documentation lives in docs/ — quickstart, API reference, correctness guide and the plugin contract; build it locally with mkdocs serve. Source and issue tracker: https://github.com/sukhov-is/HonestML.

Development

uv sync --extra dev --extra boosting --extra shap --extra pyarrow --extra mlflow
uv run pytest                 # full suite (onnx export tests also need `--extra onnx`, Python >=3.11)
uv run ruff check src tests; uv run mypy src/honestml; uv run lint-imports

The layered architecture (core ← adapters ← application ← composition) is enforced by import-linter. See docs/releasing.md for the release pipeline and benchmarks/ for the honesty benchmark suite.

License

MIT (see LICENSE).