orbital

Convert SKLearn pipelines and PyTorch neural networks into SQL queries for execution in a database without the need for a Python environment.

See examples directory for example pipelines and Documentation

Warning:

This is a work in progress.
You might encounter bugs or missing features.

Note:

Not all transformations and models can be represented as SQL queries,
so orbital might not be able to implement the specific pipeline you are using.

Getting Started

Install orbital:

$ pip install orbital

Prepare some data:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

COLUMNS = ["sepal.length", "sepal.width", "petal.length", "petal.width"]

iris = load_iris(as_frame=True)
iris_x = iris.data.set_axis(COLUMNS, axis=1)

# SQL and orbital don't like dots in column names, replace them with underscores
iris_x.columns = COLUMNS = [cname.replace(".", "_") for cname in COLUMNS]

X_train, X_test, y_train, y_test = train_test_split(
    iris_x, iris.target, test_size=0.2, random_state=42
)

Define a Scikit-Learn pipeline and train it:

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline(
    [
        (
            "preprocess",
            ColumnTransformer(
                [("scaler", StandardScaler(with_std=False), COLUMNS)],
                remainder="passthrough",
            ),
        ),
        ("linear_regression", LinearRegression()),
    ]
)
pipeline.fit(X_train, y_train)

Convert the pipeline to orbital:

import orbital
import orbital.types

orbital_pipeline = orbital.parse_pipeline(
    pipeline,
    features={
        "sepal_length": orbital.types.DoubleColumnType(),
        "sepal_width": orbital.types.DoubleColumnType(),
        "petal_length": orbital.types.DoubleColumnType(),
        "petal_width": orbital.types.DoubleColumnType(),
    },
)

You can print the pipeline to see the result:

>>> print(orbital_pipeline)

ParsedPipeline(
    features={
        sepal_length: DoubleColumnType()
        sepal_width: DoubleColumnType()
        petal_length: DoubleColumnType()
        petal_width: DoubleColumnType()
    },
    steps=[
        merged_columns=Concat(
            inputs: sepal_length, sepal_width, petal_length, petal_width,
            attributes: 
             axis=1
        )
        variable1=Sub(
            inputs: merged_columns, Su_Subcst=[5.809166666666666, 3.0616666666666665, 3.7266666666666666, 1.18333333...,
            attributes: 
        )
        multiplied=MatMul(
            inputs: variable1, coef=[-0.11633479416518255, -0.05977785171980231, 0.25491374699772246, 0.5475959...,
            attributes: 
        )
        resh=Add(
            inputs: multiplied, intercept=[0.9916666666666668],
            attributes: 
        )
        variable=Reshape(
            inputs: resh, shape_tensor=[-1, 1],
            attributes: 
        )
    ],
)

Now we can generate the SQL from the pipeline:

sql = orbital.export_sql("DATA_TABLE", orbital_pipeline, dialect="duckdb")

And check the resulting query:

>>> print(sql)

SELECT ("t0"."sepal_length" - 5.809166666666666) * -0.11633479416518255 + 0.9916666666666668 +  
       ("t0"."sepal_width" - 3.0616666666666665) * -0.05977785171980231 + 
       ("t0"."petal_length" - 3.7266666666666666) * 0.25491374699772246 + 
       ("t0"."petal_width" - 1.1833333333333333) * 0.5475959809777828 
AS "variable" FROM "DATA_TABLE" AS "t0"

Once the SQL is generate, you can use it to run the pipeline on a database. From here on the SQL can be exported and reused in other places:

>>> print("\nPrediction with SQL")
>>> duckdb.register("DATA_TABLE", X_test)
>>> print(duckdb.sql(sql).df()["variable"][:5].to_numpy())

Prediction with SQL
[ 1.23071715 -0.04010441  2.21970287  1.34966889  1.28429336]

We can verify that the prediction matches the one done by Scikit-Learn by running the scikitlearn pipeline on the same set of data:

>>> print("\nPrediction with SciKit-Learn")
>>> print(pipeline.predict(X_test)[:5])

Prediction with SciKit-Learn
[ 1.23071715 -0.04010441  2.21970287  1.34966889  1.28429336 ]

PyTorch Neural Networks

Neural networks trained with PyTorch can be converted the same way, with orbital.parse_pytorch_model:

$ pip install orbital[pytorch]

import torch
import orbital
import orbital.types

FEATURES = {
    "amount": orbital.types.DoubleColumnType(),
    "hour": orbital.types.DoubleColumnType(),
}

# Train a tiny fraud-detection network: 2 inputs -> 8 hidden (ReLU) -> 1 sigmoid output
X_train = torch.rand(500, 2) * torch.tensor([500.0, 24.0])
y_train = (X_train[:, 0] > 250).float().unsqueeze(1)

model = torch.nn.Sequential(
    torch.nn.Linear(len(FEATURES), 8),
    torch.nn.ReLU(),
    torch.nn.Linear(8, 1),
    torch.nn.Sigmoid(),
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(200):
    optimizer.zero_grad()
    loss = torch.nn.functional.binary_cross_entropy(model(X_train), y_train)
    loss.backward()
    optimizer.step()

Convert it to orbital and generate the SQL:

orbital_pipeline = orbital.parse_pytorch_model(model, FEATURES)
sql = orbital.export_sql("transactions", orbital_pipeline, dialect="duckdb")

Run the SQL in DuckDB:

import duckdb
import pandas as pd

test_data = pd.DataFrame({"amount": [50.0, 500.0], "hour": [1.0, 20.0]})
duckdb.register("transactions", test_data)
print(duckdb.sql(sql).df())

See examples/pytorch_fraud_detector.py for the full runnable version, with realistic synthetic data and a check against PyTorch's own predictions.

Supported Models

orbital currently supports the following models:

• Linear Regression
• Logistic Regression
• Lasso Regression
• Elastic Net
• Decision Tree Regressor
• Decision Tree Classifier
• Random Forest Classifier
• Gradient Boosting Regressor
• Gradient Boosting Classifier
• Multi-Layer Perceptron (scikit-learn MLPClassifier/MLPRegressor)
• Neural Networks (PyTorch, feed-forward architectures)

Neural network support, for both frameworks, is limited to feed-forward architectures: convolutional, recurrent, attention, and embedding layers are not supported.

Contributing

Before contributing make sure you read .github/copilot-instructions.md, those are guidelines that are meaningful both to human developers and agents working on the codebase.

Testing

Setup testing environment:

$ uv sync --no-dev --extra test

Run Tests:

$ uv run pytest -v

Try Examples:

$ uv run examples/pipeline_lineareg.py

Development

Setup a development environment:

$ uv sync