omle-convert

Convert trained ML models and pipelines from scikit-learn, Spark ML, XGBoost, LightGBM, and CatBoost into the OMLE interchange format (.omle).

Installation

pip install omle-convert

That is the whole installation. The ML frameworks are not dependencies: converters load them dynamically at call time, so you bring the framework that produced your model. Converting an XGBoost model means xgboost is already in your environment by definition; pinning it here would only invite version conflicts.

What does get installed is omle-convert's own machinery — numpy and pandas, plus pyarrow and sqlglot for reading saved Spark ML directories without a SparkSession or JVM.

Unified API (to_omle, export_omle)

A single entry point that auto-detects the framework from the object or file path.

Pipelines convert whole, not just the estimator. The preprocessing fitted alongside a model is part of the prediction, so every stage becomes a node in one graph and the .omle file reproduces the same end-to-end computation:

from omle_convert import to_omle, export_omle

# A fitted sklearn Pipeline — one node per stage
model = to_omle(pipeline, X=X_test)

# A bare estimator works too — it just produces a single-node graph
model = to_omle(estimator, X=X_test)

# A live PySpark PipelineModel
model = to_omle(pipeline_model, dataset=train_df)

# Pass a file path — extension determines the converter
model = to_omle("model.json")       # XGBoost (or CatBoost JSON — auto-detected by content)
model = to_omle("model.cbm")        # CatBoost native binary
model = to_omle("model.txt")        # LightGBM
model = to_omle("pipeline.joblib")  # sklearn — any estimator or Pipeline
model = to_omle("pipeline.pkl")     # sklearn pickle
model = to_omle("saved_pipeline/")  # Spark ML PipelineModel directory

# Convert and save in one step
export_omle(pipeline, "output.omle", X=X_test)

Sample data is named for its framework — X for the array-based converters (scikit-learn's convention) and dataset for PySpark (matching fit(dataset)). to_omle and export_omle accept either, so the unified entry points stay uniform even though the framework-specific functions keep their native spelling:

to_omle(spark_model, X=train_df)   # same as dataset=train_df
to_omle(sklearn_model, dataset=X)     # same as X=X

CLI — framework is always auto-detected; no subcommand needed:

omle-convert model.json     output.omle   # XGBoost or CatBoost JSON
omle-convert model.ubj      output.omle   # XGBoost UBJSON
omle-convert model.txt      output.omle   # LightGBM text (.bin too)
omle-convert model.cbm      output.omle   # CatBoost binary
omle-convert model.joblib   output.omle   # any pickled estimator (joblib)
omle-convert model.pkl      output.omle   # any pickled estimator (pickle)
omle-convert saved_model/   output.omle   # Spark ML directory
omle-convert model.zip      output.omle   # Spark ML archive (.tar.gz, .tgz)

Both XGBoost and CatBoost write .json, so that extension is disambiguated by content rather than by name. A pickle carries whatever estimator was saved — sklearn, xgboost, lightgbm or catboost — and dispatches on the unpickled object.

Supported frameworks

Each converter handles a pipeline as readily as a bare estimator. Preprocessing stages become graph nodes alongside the model, so the artifact reproduces the whole fit-time transform chain rather than just the final predictor.

Counts are source classes with a converter. Follow the reference for the per-class table — what each maps to, the attribute-level notes, and what is explicitly not supported.

XGBoost, LightGBM and CatBoost have no preprocessing of their own — put them inside an sklearn Pipeline or a Spark PipelineModel and the surrounding stages convert through those converters. category_encoders is the mirror image: not an entry point at all, its transformers convert as pipeline stages via from_sklearn.

Common parameters

Most converter entry points accept these keyword arguments:

Auxiliary-data parameters

Accepted by every live-object converter — sklearn, XGBoost, LightGBM, CatBoost and from_spark_live. from_spark reads an artifact off disk with no model to score, so it takes none of them.

Model operator bodies

Model operators (carry a task_type field)

Preprocessing operators

Derive node functions

Derive is the expression node emitted by sklearn FunctionTransformer and Spark SQLTransformer. When the input is a matrix, the expression is evaluated column-by-column and the results are assembled back into a matrix of the same shape — no intermediate TakeSlots/Concat nodes are needed. Supported functions:

Output types

Model-level OutputSpecs follow the task, not the source framework:

y_pred is an INT64 array of class labels for every classifier, binary and multiclass alike. Cases worth knowing:

Individual nodes carry their own TensorType annotations — see output type annotations.

Node naming

Tensor storage (inline vs. tensor entries)

Model-body parameters (coefficients, split thresholds, leaf values, etc.) are stored inline in the TensorValue when the tensor has at most OMLE_INLINE_TENSOR_LIMIT elements (default 100). Larger tensors are promoted to named TensorEntry records and referenced via TensorRef. Set the environment variable to tune the cutoff:

OMLE_INLINE_TENSOR_LIMIT=0   # always use tensor entries (never inline)
OMLE_INLINE_TENSOR_LIMIT=-1  # always inline (no tensor entries)
OMLE_INLINE_TENSOR_LIMIT=500 # inline up to 500 elements

This applies to all converters: sklearn, XGBoost, LightGBM, and Spark ML.

Contributing

See CONTRIBUTING.md for development setup and the checks a change needs to pass, and CODE_OF_CONDUCT.md for community expectations.

License

Apache License 2.0