dataiku2onnx

Convert Dataiku DSS 14 visual ML models to ONNX. A saved-model version goes in; a single self-contained .onnx file comes out, carrying Dataiku's own feature preprocessing inside the model rather than assuming a Dataiku runtime at scoring time.

This is a first beta — 0.0.1b0. The behaviour described here has been measured, but the package is early: the API may change, and it has not yet been used outside the project that built it.

About

You train a model in the DSS visual ML lab. Somewhere else — a scoring engine, an application, a database — you need the same predictions, without a Dataiku runtime. dataiku2onnx reads the saved-model version DSS exports and emits one ONNX file that reproduces the whole pipeline: the feature preprocessing, the estimator, the calibration, and the predicted label.

What that means in practice:

• Preprocessing is reproduced in the graph, not around it. Imputation, rescaling, dummification, categorical encoding, derived features and the rest are ONNX nodes. Nothing has to be re-implemented by whoever scores the model.
• Nothing is silently approximated. A step this package cannot express faithfully is refused by name at conversion time, and the refusal says what to change. A model that converts is a model whose preprocessing is complete.
• Every conversion is validated, and the validation cannot be turned off. There is no validate=False, no environment variable and no separate validator to remember to call.
• Nothing is re-fitted, and no estimator library is installed. Every fitted parameter — for scikit-learn, XGBoost and LightGBM alike — is read out of the JSON DSS writes at train time. The whole runtime dependency set is onnx and numpy.
• Nothing is unpickled. The loaders read DSS's JSON, never the pickles beside it.
• It does not connect to DSS. You fetch the saved-model version with the client you already have; this package converts the archive you hand it.
• It is a Python library used from scripts. There is no command-line interface.

The emitted model targets ONNX opset 16, IR version 8 by default, with weights stored inline in the file. The limitations section below explains why the default is conservative.

Licence

PROPRIETARY — ALL RIGHTS RESERVED. Installing this package does not grant you a licence to use it. No licence, express or implied, is granted to use, copy, modify or distribute it, except under a separate written agreement with the copyright holder. Full terms are in the LICENSE file inside the installed package.

pip install normally implies permission to use what you installed. Here it does not, which is why it is said plainly rather than left in a file inside the package.

Quickstart

From an exported saved-model version

In DSS: Saved model → the version → Actions → Export → Python scoring. You get a zip.

from dataiku2onnx import FilesystemLoader, convert

model = FilesystemLoader("exports/churn_v3.zip").load()
result = convert(model)
result.save("churn_v3.onnx")

print(result.report.describe())

FilesystemLoader takes an unpacked version directory, a model.zip, or the export zip DSS gives you, and works out which it was handed.

Inside a DSS Python recipe

Against a model deployed in your flow, with the client the recipe already holds. Nothing is written to disk:

from pathlib import Path

import dataiku

from dataiku2onnx import InMemoryLoader, convert

SAVED_MODEL_ID = "YOUR_SAVED_MODEL_ID"

project_key = dataiku.default_project_key()
saved_model = dataiku.api_client().get_project(project_key).get_saved_model(SAVED_MODEL_ID)
version_id = saved_model.get_active_version()["id"]

with saved_model.get_version_details(version_id).get_scoring_python_stream() as stream:
    result = convert(
        InMemoryLoader(
            stream.content,
            project_key=project_key,
            saved_model_id=SAVED_MODEL_ID,
            version_id=version_id,
        ).load()
    )

result.save(Path(dataiku.Folder("model_onnx").get_path()) / "churn_v3.onnx")

print(result.report.describe())

The code env needs dataiku2onnx and DSS's core packages; nothing else. Add no error handling — if a model cannot be converted, the exception ends the job and DSS puts the reason in the log, which is what you want.

The three identifiers are optional and worth passing: an export archive records the saved-model id nowhere, and they are what let a deployed .onnx be traced back to its source months later.

What comes back

result.report.class_order   # ('No', 'Yes') -- DSS's ordering, read out of the emitted model
result.report.outputs       # ('prediction', 'proba_No', 'proba_Yes')
result.report.warnings      # what to resolve before deploying
result.report.notes         # behaviours that change numbers without changing anything visible

Installation

pip install dataiku2onnx

That is everything a conversion needs. No model family needs an extra, because no estimator library is imported — see the description above.

You also need:

• Python 3.10 or 3.11.
• A Dataiku DSS 14 saved-model version: an exported folder or zip, or the scoring-python archive fetched with your own DSS client. Verified against DSS 14.6.0; the format has been read successfully from exports going back to 11.0.3.

If you fetch versions from a live DSS instance, install dataiku-api-client as well — note the distribution name, which differs from the dataikuapi import name. dataiku2onnx deliberately does not declare it: the client is yours, and so is its configuration — the proxy settings and the TLS trust store included. Keep certificate verification on.

What is supported

Model types

The set of algorithms DSS 14 can export portably is closed at nine, so this table is complete rather than a snapshot. Twelve model-and-engine combinations have been measured against DSS.

The support key is how a family is identified throughout this project's documentation. The three boosting engines carry their own keys because they do different arithmetic, even though DSS presents all three as gradient boosting.

What stands behind that. Each combination was converted from a real exported model and scored — 28 model runs of 100 rows each — against DSS's own scored output for the same rows. Every predicted label matched DSS exactly, and every probability and regression value landed inside a tolerance fixed in advance.

Every run exercised categorical, floating-point and integer features together in one model. Each run of 100 rows is 81 ordinary rows, 15 carrying a null in at least one input feature — 12 of those in a numeric feature — and 4 carrying a category the model had never been trained on.

What it does not tell you: that is a measurement on those models, not a guarantee about every model of the same kind. A model built from a preprocessing step none of them used has not been measured.

Probability calibration converts for the SIGMOID method. ISOTONIC is refused, because no real exported model has ever carried one.

Preprocessing steps

A DSS visual ML model carries Dataiku's own feature preprocessing, and all of it has to be reproduced inside the ONNX model, because nothing downstream has a Dataiku runtime to fall back on. DSS has 21 preprocessing steps in total, and every one of them has a verdict here.

Nothing is awaiting implementation. Every one of the 21 steps has a verdict above, and every refusal names the step it refused and what to do instead.

Loaders and the convert function

Two loaders, and they produce equal models. Pick the one that matches where the archive is.

Both accept project_key=, saved_model_id= and version_id=, which are stamped into the emitted model. An export archive records the saved-model id nowhere, so passing them is what lets a deployed .onnx be traced back months later. InMemoryLoader writes no temporary file: bytes are copied once, a stream is not copied at all, and a non-seekable stream is refused rather than quietly buffered — a zip is read from its central directory at the end of the file, so it cannot be read forwards.

.load() gives a DataikuModel. Everything else starts from there.

from dataiku2onnx import FilesystemLoader, convert, plan

model = FilesystemLoader("exports/churn_v3.zip").load()

decision = plan(model)          # what would happen, without building anything
result = convert(model)         # the model and its report
path = result.save("churn_v3.onnx")

convert(model, *, opset=16, numeric_dtype="float64", converted_at=None) returns a ConversionResult carrying model — the onnx.ModelProto — and report. opset selects the target ai.onnx opset (16 by default, 17 also accepted). numeric_dtype is the working element type, float64 by default because Dataiku's own feature matrix is float64, so the arithmetic matches without a cast. converted_at exists so a build can be made reproducible — not so anything can be skipped.

plan(model) answers the same question convert() would, without building a graph. It reports every refusal a model carries rather than the first, which makes it the cheap way to survey a directory of exports and find the ones that will not convert.

result.report is the part to actually read. describe() prints a one-screen summary; the fields behind it are steps (one outcome per preprocessing step, in order, so a silently skipped step is detectable rather than merely discouraged), estimator, inputs and input_types, outputs, class_order, column_order, metadata, model_bytes and diagnostics. report.warnings are things to resolve before deploying; report.notes are behaviours that change numbers without changing anything visible — a value treated as NaN, a float32 tree round trip, an applied calibrator, a decision threshold that is not 0.5. Every finding carries a DiagnosticCode, so you can match on it without parsing English.

A refusal is an exception, never a quietly reduced model. convert() raises UnsupportedPreprocessingError, naming every step it cannot express, or UnsupportedModelError for an estimator it does not handle, rather than handing back something that fails later. The validation runs on every call and there is no flag that skips it.

Limitations

Some models cannot be converted, and the reason tells you whether waiting helps.

Why the emitted opset is conservative

Opsets and IR versions are backward compatible: a runtime that accepts opset 20 accepts 16, and one that accepts IR 10 accepts IR 8. Emitting at opset 16 / IR 8 therefore loads on essentially every ONNX runtime from roughly 2022 onward, current ones included, without asking you which one you have.

Raising the default would buy the string operators the second row above needs, and would pay for them by turning a refusal into a load failure on older runtimes. A load failure is the worse of the two. A refusal happens at conversion time, on your machine, in our words, names the step and comes with a workaround. A load failure happens at deployment time, in the runtime's words, and says nothing about which step caused it. The conservative default turns an unknown into a known.

The target opset is a parameter of convert(), so this is a default rather than a wall.

Keras, TensorFlow and PyTorch

DSS has no portable export for these frameworks, so there is no file to convert. The path ends at export, before any question about ONNX arises — Dataiku's own portable scoring library supports the nine algorithms above and no deep-learning framework.

This is a boundary of the DSS export format, not of this library. Neural networks are covered, by MLP_CLASSIFIER / MLP_REGRESSOR, and they are among the twelve combinations measured above.

Also not covered

Custom user-written Python models, clustering, time-series forecasting, models stacked or ensembled out of other models, and partitioned models. All are detected and refused with a clear message — never silently mis-converted.

The evidence is one environment

Everything measured here was measured against DSS 14.6.0 exports. Another DSS version has not been measured, and a model whose preprocessing none of the measured models used has not been measured either. A conversion that merely runs without error proves that the model was built, not that it agrees with DSS.