tdstone2 0.1.13.17

A package for Script Table Operator that applies set theory to machine learning in Python.

tdstone2 Package

Overview

tdstone2 operationalizes Python code for machine learning and data analysis on Teradata Vantage using the Script Table Operator (STO). It leverages Teradata's MPP architecture to run hundreds of Python scripts in parallel across hundreds of data partitions — enabling hyper-segmented model deployment with full lineage, versioning, and minimal data movement.

Features

• Hyper-segmented Model Deployment: Train one independent model per data partition in parallel across Teradata AMPs
• Scikit-learn Pipeline Integration: Auto-generate STO scripts from sklearn Pipelines (classifier, regressor, anomaly detection)
• Deep Learning (Keras / PyTorch): Train one neural net (LSTM, sequence nets) per partition with DeepLearningModel — architecture as a build_fn, weights-only serialization, SQL-engineered input (build_lstm_input), and streaming warm-start training on both the STO and OAF backends
• LightGBM (streaming / native): Train one native LightGBM booster per partition with LightGBMModel — mini-batch continued boosting for partitions too large to fit in memory, native categorical features (no one-hot), and warm-start across epochs; scoring reuses the sklearn path
• Prophet (per-partition forecasting): Fit one Facebook Prophet model per series with ProphetModel and emit a forecast horizon directly — a forecast()-only API (no train/score), trend + seasonality + holidays/promotion periods, on both STO and OAF, with a denormalized live forecast view ready for tdfs4ds
• Feature Engineering: Deploy custom or reducer-based feature engineering per partition
• Vector Embeddings: Install HuggingFace/ONNX models and compute embeddings in-database
• Seq2Seq Inference: Deploy summarization and translation models (e.g., flan-t5) via STO
• Model Lineage & Versioning: Temporal tables track every version of every trained model
• Two Execution Backends: Script Table Operator (Vantage Enterprise + Vantage Cloud Enterprise) and Apply (Vantage Cloud Lake / OAF) — same HyperModel API, picked via use_apply=True
• Per-batch Timing Instrumentation: every scored row carries SCRIPT_UUID, TOTAL_TIME, IMPORT_TIME, LOAD_TIME, PROCESS_TIME, PRINT_TIME, BATCH_NO so you can diagnose which phase dominated wall-clock, per partition, with a single SELECT
• Scoring Error Capture: both successful and failed scoring rows are persisted in the scores table; partitions whose model could not be assembled (e.g. too few data rows to deliver all chunks) emit a -1 sentinel row with a descriptive STATUS JSON so failures are visible without log inspection
• HTML Reports: train(report=True) / score(report=True) emit inline reports — partition-duration histograms, top-10 longest/fastest, input/output row-count distributions, model-size histogram

Installation

pip install tdstone2

Requires access to a Teradata Vantage system with the Script Table Operator enabled.

---

Quick Start

1. Generate Test Data

import teradataml as tdml
from tdstone2.dataset_generation import GenerateEquallyDistributedDataSet

tdml.create_context(**Param)  # Param = {'host': ..., 'user': ..., 'password': ...}

# Generate a synthetic partitioned dataset (21.6M rows, 216 partitions)
dataset = GenerateEquallyDistributedDataSet(n_partitions=216, n_rows=100000)
dataset.to_sql('dataset_00', schema_name=Param['database'])

Output schema: Partition_ID, ID, X1–X9 (features), Y1, Y2 (targets), flag, FOLD

2. Initialize the Framework

from tdstone2.tdstone import TDStone

sto = TDStone(schema_name=Param['database'], SEARCHUIFDBPATH=Param['user'])
sto.setup()  # creates repository tables + installs STO files in Vantage

For Vantage Cloud Lake (OpenAF / STO-via-OAF):

sto = TDStone(schema_name=Param['database'], SEARCHUIFDBPATH=Param['user'], oaf_env='my_env')
sto.setup()

For Vantage Cloud Lake via the Apply backend (recommended on Lake — STO is not always available there):

sto = TDStone(
    schema_name    = Param['database'],
    use_apply      = True,                  # route train/score/FE through tdml.Apply
    apply_env_name = 'tdstone2_sklearn',    # OAF user-environment with sklearn installed
    compute_group  = 'CG_BusGrpB_ANL',      # sets QueryBand 'compute=...' for ACC routing
    connect_kwargs = {                      # explicit cluster pinning (avoids the
        'host':     Param['host'],          # multi-cluster trap when several VCL_*_HOST
        'user':     Param['user'],          # env vars are set at once)
        'password': Param['password'],
        'database': Param['database'],
    },
)
sto.setup()  # creates OFS-resident repository tables + installs Apply scripts in the env

The Apply path stores all repository tables with STORAGE = TD_OFSSTORAGE, replaces the PERIOD FOR ValidPeriod temporal column with a plain CREATION_DATE (OFS rejects temporal periods), and runs the analog tds_*_apply.py scripts inside the user environment. The public HyperModel / FeatureEngineering API is identical to the STO path.

---

Hyper-segmented Models

Deploy a Scikit-learn Classifier

from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from tdstone2.tdshypermodel import HyperModel

steps = [
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(max_depth=5, n_estimators=95))
]

model_parameters = {
    "target": 'Y2',
    "column_categorical": ['flag', 'Y2'],
    "column_names_X": ['X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'flag']
}

model = HyperModel(
    tdstone=sto,
    metadata={'project': 'my_project'},
    skl_pipeline_steps=steps,
    model_parameters=model_parameters,
    dataset=tdml.in_schema(Param['database'], 'dataset_00'),
    id_row='ID',
    id_partition='Partition_ID',
    id_fold='FOLD',
    fold_training='train',
    convert_to_onnx=False,   # set True for ONNX export
    store_pickle=True,
)

model.train()   # trains 216 independent models in parallel (one per partition)
model.score()   # scores all data using each partition's model

Deploy a Scikit-learn Regressor

from sklearn.ensemble import RandomForestRegressor

steps = [
    ('scaler', StandardScaler()),
    ('regressor', RandomForestRegressor(max_depth=5, n_estimators=95))
]

model_parameters = {
    "target": 'Y1',
    "column_names_X": ['X1', 'X2', 'X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9']
}

model = HyperModel(
    tdstone=sto,
    metadata={'project': 'regression'},
    skl_pipeline_steps=steps,
    model_parameters=model_parameters,
    dataset=tdml.in_schema(Param['database'], 'dataset_00'),
    id_row='ID',
    id_partition='Partition_ID',
    id_fold='FOLD',
    fold_training='train',
)
model.train()
model.score()

Deploy Anomaly Detection (OneClassSVM)

from sklearn.svm import OneClassSVM

steps = [
    ('scaler', StandardScaler()),
    ('anomaly', OneClassSVM(nu=0.05))
]

model = HyperModel(
    tdstone=sto,
    metadata={'project': 'anomaly_detection'},
    skl_pipeline_steps=steps,
    model_parameters={"column_names_X": ['X1', 'X2', 'X3', 'X4', 'X5']},
    dataset=tdml.in_schema(Param['database'], 'dataset_00'),
    id_row='ID',
    id_partition='Partition_ID',
    id_fold='FOLD',
    fold_training='train',
)
model.train()
model.score()
# Output: anomaly flag (1=inlier, -1=outlier), decision_function, anomaly_score

Deploy LassoLarsCV (Feature Selection + Regression)

from sklearn.linear_model import LassoLarsCV

steps = [
    ('scaler', StandardScaler()),
    ('lasso', LassoLarsCV())
]

model = HyperModel(
    tdstone=sto,
    metadata={'project': 'lasso_regression'},
    skl_pipeline_steps=steps,
    model_parameters={"target": 'Y1', "column_names_X": ['X1', 'X2', 'X3', 'X4', 'X5']},
    dataset=tdml.in_schema(Param['database'], 'dataset_00'),
    id_row='ID',
    id_partition='Partition_ID',
    id_fold='FOLD',
    fold_training='train',
)
model.train()
model.score()

Retrieve Predictions

# Denormalized view: predictions joined with original features
predictions = model.get_model_predictions()

# Raw (normalized) predictions table
predictions_raw = model.get_model_predictions(denormalized_view=False)

# Include per-batch timing columns (SCRIPT_UUID, TOTAL_TIME, IMPORT_TIME, LOAD_TIME,
# PROCESS_TIME, PRINT_TIME, BATCH_NO) — useful for diagnosing per-partition slowness
predictions_with_timing = model.get_model_predictions(include_timing=True)

# Trained model metadata
trained_models = model.get_trained_models()

Inline HTML Reports

Pass report=True to train() or score() to render an inline HTML report with partition-duration histograms, top-10 longest/fastest partitions, input/output row-count distributions, and (for training) a model-size histogram:

model.train(report=True)
model.score(report=True)

Inspect the Underlying SQL

# View the generated Script Table Operator SQL
print(model.mapper_scoring.generate_sto_query())

---

Reload an Existing Hyper-segmented Model

# List all registered hyper-models
sto.list_hyper_models()

# Reload by UUID
existing_model = HyperModel(tdstone=sto)
existing_model.download(id='0286d259-ecde-4cd0-ae4a-bcb3191383d1')

# Retrain and rescore (no code needed — everything is stored in Vantage)
existing_model.train()
existing_model.score()

---

Deep Learning (per-partition Keras / PyTorch)

DeepLearningModel is the deep-learning sibling of HyperModel: it trains one neural net per partition in-database, in Keras/TensorFlow or PyTorch (dl_framework='keras' | 'pytorch'). A compiled Keras model isn't reliably picklable, so tdstone2 keeps the architecture as code — you pass a build_fn (the counterpart of the sklearn pipeline) — and persists only the weights (Keras get_weights(), or torch state_dict→numpy) + scalers into the trained-model repository. The same in-DB scripts drive both frameworks.

The sequence input vector is engineered in SQL via build_lstm_input (LAG() lag columns), so the generated model is framework-generic — it just selects column_names_X; no Python windowing.

from tdstone2.tdstone import TDStone
from tdstone2.tdsdeeplearning import DeepLearningModel, build_lstm_input

# --- the architecture = the build function --------------------------------
# Keras: return a COMPILED keras.Model.  PyTorch: return a plain nn.Module
# (tdstone2 builds loss/optimizer from dl_arguments and runs the step itself).
def build_model(look_back=1, lstm_layer_nb_neurons=8, **kw):
    import torch.nn as nn
    class LSTMNet(nn.Module):
        def __init__(self):
            super().__init__()
            self.lstm = nn.LSTM(input_size=look_back,
                                hidden_size=lstm_layer_nb_neurons, batch_first=True)
            self.fc = nn.Linear(lstm_layer_nb_neurons, 1)
        def forward(self, x):                       # x: (batch, seq=1, look_back)
            out, _ = self.lstm(x)
            return self.fc(out[:, -1, :])
    return LSTMNet()

# --- engineer the windowed input IN SQL -----------------------------------
lag_view, FEATURES, TARGET = build_lstm_input(
    dataset      = tdml.in_schema(sto.schema_name, 'air_passengers'),
    target       = 'Passengers',
    id_partition = 'SERIES_ID',
    order_column = 'TIME_NO_UNIT',
    look_back    = 1,
    id_fold      = 'FOLD',
    schema_name  = sto.schema_name,
    # difference=True -> predict the first difference for a strongly TRENDED series
)

dlm = DeepLearningModel(
    tdstone      = sto,
    dl_build_fn  = build_model,
    dl_framework = 'pytorch',                       # or 'keras'
    dl_arguments = {'look_back': 1, 'order_column': 'TIME_NO_UNIT',
                    'epochs': 5, 'batch_size': 8, 'lstm_layer_nb_neurons': 8,
                    'optimizer': 'adam', 'loss': 'mean_squared_error',
                    'learning_rate': 0.02},
    model_parameters = {'target': TARGET, 'column_names_X': FEATURES},
    dataset      = tdml.in_schema(*lag_view.split('.')),   # the lag view, not raw
    id_row       = 'TIME_NO_UNIT', id_partition = 'SERIES_ID',
    id_fold      = 'FOLD', fold_training = 'train',
    inner_epochs = 20,                              # gradient passes inside each run
)

dlm.train(epochs=5)                                 # warm-start re-runs (see below)
dlm.score()
preds = dlm.get_model_predictions()                 # melted: <target>_prediction rows

Same model in Keras / TensorFlow

Only the build_fn and dl_framework change — everything else (build_lstm_input, dl_arguments, train/score, warm-start) is identical. For Keras the build_fn returns a compiled keras.Model (loss/optimizer are baked into the model at compile(), so tdstone2 doesn't set them from dl_arguments):

# --- the architecture = the build function (Keras / TensorFlow) -----------
# Return a COMPILED keras.Model. Keep windowing/scaling OUT of here — the input
# vector is engineered in SQL by build_lstm_input; the model just reads columns.
def build_model(look_back=1, lstm_layer_nb_neurons=8,
                loss='mean_squared_error', optimizer='adam', **kw):
    from tensorflow.keras.models import Sequential
    from tensorflow.keras.layers import LSTM, Dense
    model = Sequential()
    model.add(LSTM(lstm_layer_nb_neurons, input_shape=(1, look_back)))
    model.add(Dense(1))
    model.compile(loss=loss, optimizer=optimizer)
    return model

dlm = DeepLearningModel(
    tdstone      = sto,
    dl_build_fn  = build_model,
    dl_framework = 'keras',                         # <- the only other change
    dl_arguments = {'look_back': 1, 'order_column': 'TIME_NO_UNIT',
                    'epochs': 5, 'batch_size': 8, 'lstm_layer_nb_neurons': 8,
                    'loss': 'mean_squared_error', 'optimizer': 'adam'},
    model_parameters = {'target': TARGET, 'column_names_X': FEATURES},
    dataset      = tdml.in_schema(*lag_view.split('.')),
    id_row       = 'TIME_NO_UNIT', id_partition = 'SERIES_ID',
    id_fold      = 'FOLD', fold_training = 'train',
    inner_epochs = 20,
)
dlm.train(epochs=5); dlm.score()

Persisted weights are the framework's native form — Keras get_weights() vs torch state_dict→numpy — but the trained-model repository, retrieval, and warm-start machinery are identical for both.

Streaming + warm-start on both backends

train(epochs=N) re-runs the streaming trainer N times and warm-starts each run from the previous run's weights (epoch 1 cold-starts). Multi-epoch convergence comes from both the outer re-runs and the in-DB inner_epochs shuffled passes — so prefer many inner_epochs × few outer epochs (each outer re-run is a full, expensive query).

Backend	How warm-start is fed	Memory
OAF (use_apply=True, VCL)	previous weights arrive model-first as chunks via the UNION ALL on-clause	streams mini-batches; bounded buffer (max_buffer_rows, single-pass fallback over the cap)
STO (VCE)	previous weights (single CLOB) ride the metadata row via the DL training JOIN on-clause	streams stdin in batch_size mini-batches — never a whole-partition DataFrame, no model.fit

The DL path has its own in-DB scripts and on-clauses (tds_dl_*), so the sklearn HyperModel path is completely untouched. Install the framework in the in-DB Python first: sto.install_python_libs(['torch']) or ['tensorflow'] on OAF; on STO the framework must already be in the node tdpython3.

GPU (OAF only): pass compute_group='GPU_Cluster' to route the Apply to GPU nodes; dlm.probe_gpu() reports whether tensorflow/torch actually sees CUDA in the env. GPU is best-effort — the code lights up with no change once the env exposes the CUDA runtime libs.

See demos/notebooks Demo Deep Learning/ for the evaluation notebooks (Keras and PyTorch, on OAF/VCL and STO/VCE) with per-partition RMSE/MAE + predicted-vs-actual plots, and the tdstone2-deep-learning agent skill for the full contract.

ONNX export → native BYOM (ONNXPredict)

Like HyperModel, a trained DL model can be exported to ONNX for SQL-native scoring via Teradata ONNXPredict — no Python at inference. Pass convert_to_onnx=True; pickle stays the default (warm-start needs it), and ONNX is a final snapshot (inference-only). The two MinMaxScalers are folded into the ONNX graph (the DL analog of sklearn's in-pipeline scaler), and the input is one scalar column per lag feature — the shape ONNXPredict expects.

dlm = DeepLearningModel(..., convert_to_onnx=True, store_pickle=True)
dlm.train(epochs=5)                  # each partition writes BOTH a pickle and an onnx row
catalog = dlm.get_byom_catalog()     # V_<repo>_BYOM_CATALOG: (model_id, model, <partition cols>)

-- Score one partition from pure SQL (no Python) via native ONNXPredict on VCL.
-- The model column must be BLOB, and only one model may be broadcast (DIMENSION),
-- so cast + filter the catalog to the partition being scored:
SELECT * FROM TD_MLDB.ONNXPredict(
    ON (SELECT * FROM <lag_view> WHERE PARTITION_ID = 0) AS InputTable
    ON (SELECT model_id, CAST(model AS BLOB(64000)) AS model
        FROM <schema>.V_<repo>_BYOM_CATALOG WHERE PARTITION_ID = 0) AS ModelTable DIMENSION
    USING Accumulate('PARTITION_ID','TIME_NO_UNIT') OverwriteCachedModel('true')
) AS dt;   -- output: json_report = {"output":[[<prediction>]]}

Keras ONNX needs tf2onnx in the in-DB Python (install_python_libs(['tf2onnx'])); PyTorch needs only torch. Opset is 13 (LSTM ops). Notebook 05 - LSTM ONNX export & BYOM scoring validates one partition ONNXPredict (SQL) against dlm.score() (Python) — they match to float precision.

---

LightGBM (streaming per-partition gradient boosting)

LightGBMModel is the gradient-boosting sibling of HyperModel / DeepLearningModel. tdstone2 already runs LightGBM through the sklearn path (LGBMRegressor / LGBMClassifier inside skl_pipeline_steps), but that trains each partition with one in-memory model.fit(). LightGBMModel instead trains the native booster by streaming mini-batches (continued boosting via init_model=), so partitions too large to fit a worker's memory still train, and it warm-starts across train(epochs=N) re-runs.

Native categorical is the "dedicated input" — columns in column_categorical are integer-encoded with a per-partition vocabulary frozen after the first pass (no one-hot). A partition that fits the row buffer trains with one standard lgb.train (normal LightGBM quality); one that overflows falls to mini-batch continued boosting (bounded memory). Scoring reuses the sklearn scripts (LightGBM is row-independent), so only the streaming trainer is new.

from tdstone2.tdstone import TDStone
from tdstone2.tdslightgbm import LightGBMModel

lgbm = LightGBMModel(
    tdstone      = sto,
    lgbm_task    = 'binary',                          # 'regression' | 'binary' | 'multiclass'
    lgbm_params  = {'num_leaves': 31, 'learning_rate': 0.05},   # native LightGBM params
    model_parameters = {
        'target'            : 'churn',
        'column_names_X'    : ['age', 'usage', 'plan', 'region'],
        'column_categorical': ['plan', 'region'],    # native, no one-hot
    },
    dataset      = tdml.in_schema(sto.schema_name, 'customers'),
    id_row       = 'ID', id_partition = 'SEGMENT_ID',
    id_fold      = 'FOLD', fold_training = 'train',
    num_boost_round = 100,                            # trees per continued-boosting call
    batch_size      = 50000,                          # minibatch size (used on buffer overflow)
    max_buffer_rows = 200000,                         # at/under this -> one standard fit
)

lgbm.train(epochs=2)          # epoch 1 cold; epoch 2 warm-starts (appends trees)
lgbm.score()
preds = lgbm.get_model_predictions()                 # <target>_prediction (+ _proba_<class>)

Needs lightgbm in the in-DB Python (TDStone.install_python_libs(['lightgbm']) on OAF; the node tdpython3 on STO). list_models_catalog() tags it MODEL_KIND='lightgbm'. Tuning caveat: total trees grow with epochs and (for oversized partitions) minibatches — keep num_boost_round modest and inner_epochs=1 (each pass adds trees, unlike the DL path). Optional convert_to_onnx=True emits a best-effort ONNX booster (categorical encoding is not folded into the graph — feed ONNXPredict the integer-encoded vector).

---

Prophet (per-partition forecasting)

ProphetModel is the forecasting sibling. It fits one Facebook Prophet model per partition (per series) on that series' history and directly emits a forecast horizon — yhat, yhat_lower, yhat_upper for every future timestamp. Unlike the other siblings it does not fit the train → persist → score pattern: a score() returns one row per input row, but Prophet emits future dates that don't exist in the input (N history rows in → H future rows out), and there's no artifact worth persisting (Prophet refits from scratch — no warm-start). So it exposes a single forecast().

from tdstone2.tdsprophet import ProphetModel

pm = ProphetModel(
    tdstone      = sto,
    model_parameters = {                      # the single source of truth
        'target':      'sales',
        'time_column': 'order_date',          # a DATE/TIMESTAMP column
        'horizon':     30,                    # future periods to forecast
        'include_history': False,             # future-only (True also emits in-sample fit)
        'arguments': {
            'params': {'seasonality_mode': 'multiplicative', 'weekly_seasonality': True},
            'country_holidays': 'US',
            'holidays': [                     # promotion periods as custom holidays
                {'holiday': 'promo', 'ds': ['2023-06-15', '2024-06-15'],
                 'lower_window': 0, 'upper_window': 13},
            ],
        },
    },
    dataset      = tdml.in_schema(Param['database'], 'store_sales'),
    id_partition = 'STORE_ID',                # one Prophet per store
)

pm.forecast()                                 # single per-partition pass (no train/score)
fc = pm.get_forecasts()                       # wide: STORE_ID, ds, yhat, yhat_lower, yhat_upper

# tdfs4ds integration — a denormalized, governable live forecast view:
view = pm.build_live_forecast_view()          # entity = partition, time = ds, features = yhat*

All configuration lives in model_parameters (JSON-serialized into the metadata row); the frequency is inferred from the series' own timestamps when omitted. Needs prophet in the in-DB Python (TDStone.install_python_libs(['prophet']) on OAF; the node tdpython3 on STO). Prophet's Stan backend (cmdstanpy) is chatty on stdout — the generated model silences it and the Apply script adds an os.dup2(2,1) fd-guard (Error 9134 otherwise). list_models_catalog() tags it MODEL_KIND='prophet'.

---

Unified model catalog

Sklearn HyperModels, DeepLearningModels and LightGBMModels all register into the same TDS_HYPER_MODEL_REPOSITORY. setup() creates a V_MODEL_CATALOG view (and list_models_catalog() returns it) that labels each row by kind and framework — purely additive, no schema change:

sto.list_models_catalog()
#   ID   MODEL_KIND      FRAMEWORK   CREATION_DATE ...
#   a1   sklearn         <null>      ...
#   b2   deep_learning   keras       ...
#   c3   deep_learning   pytorch     ...
#   d4   lightgbm        lightgbm    ...
#   e5   prophet         prophet     ...

list_hyper_models() still returns the raw registry unchanged (backward compatible; rows with no type marker are treated as sklearn).

---

Feature Engineering

Dimensionality Reduction (Reducer)

from tdstone2.tdsfeature_engineering import FeatureEngineering

fe = FeatureEngineering(
    tdstone=sto,
    feature_engineering_type='feature engineering reducer',
    dataset=tdml.in_schema(Param['database'], 'dataset_00'),
    id_row='ID',
    id_partition='Partition_ID',
    id_fold='FOLD',
    fold_training='train',
    metadata={'project': 'pca_reduce'},
)
fe.reduce()
reduced = fe.get_reduced_features()

Custom Feature Engineering

Provide a Python script that computes new columns from existing ones:

fe = FeatureEngineering(
    tdstone=sto,
    feature_engineering_type='feature engineering',
    script_path='path/to/Feature_Interactions.py',
    dataset=tdml.in_schema(Param['database'], 'dataset_00'),
    id_row='ID',
    id_partition='Partition_ID',
    metadata={'project': 'interactions'},
)
fe.transform()

# All original + derived features denormalized
features = fe.get_computed_features()

# Raw (normalized) features table
features_raw = fe.get_computed_features(denormalized_view=False)

---

Vector Embeddings

Install a HuggingFace Model (ONNX)

from tdstone2.tdsgenai import (
    install_model_in_vantage_from_name,
    customize_existing_model,
    install_zip_in_vantage,
    list_installed_files_byom,
    get_model_dimension,
)

# Download, convert to ONNX, customize tokenizer, upload
install_model_in_vantage_from_name(
    model_name='intfloat/multilingual-e5-small',
    model_task='sentence-similarity',
    upload=True,
    generate_zip=True,
)

list_installed_files_byom()  # verify installation
get_model_dimension(model_name='intfloat/multilingual-e5-small')  # e.g., 384

For Vantage Cloud Lake (OAF):

from tdstone2.tdsgenai_lake import install_model_in_vantage_from_name
install_model_in_vantage_from_name(model_name='intfloat/multilingual-e5-small', ...)

Compute embeddings on Vantage Cloud Lake (OAF)

from tdstone2.tdsgenai_lake import compute_vector_embedding

# device='cuda' triggers a pre-flight Apply probe that verifies torch.cuda.is_available()
# in the OAF env *before* launching the real call. With cuda_strict=True (default), the
# call raises RuntimeError with an actionable diagnostic instead of silently falling back
# to CPU when torch is built for a different CUDA major than the cluster's NVIDIA driver
# supports (e.g. cu13 wheel vs CUDA 12 driver).
embeddings = compute_vector_embedding(
    model_name         = 'models--BAAI--bge-small-en-v1.5',
    dataset            = text_dataframe,
    text_column        = 'txt',
    accumulate_columns = ['id'],
    hash_columns       = ['id'],
    oaf_env            = 'bq20251218',         # see OAF env setup below
    schema_name        = Param['database'],
    table_name         = 'embeddings_bge',
    device             = 'cuda',
    cuda_strict        = True,   # raise on real CUDA failure; pass-through on Apply throttling
    compute_group      = 'GPU_Cluster',        # routes Apply to the GPU compute group
)

OAF environment setup for Tesla T4 (CUDA 12.x driver)

The OAF package mirror evolves independently of Vantage compute-node drivers. As of 2026, the mirror's default torch resolves to 2.11.0+cu130, which requires CUDA 13.0 — but current Lake GPU nodes run driver 12.7/12.8. Use an env that already carries torch 2.9.1+cu128 (e.g. bq20251218), or create a new one and install the cu128 build before any other torch package touches the env.

Required packages (install in order via tdml.get_env(env_name).install_lib([...])):

Package	Why
torch==2.9.1	cu128 wheel — matches CUDA 12.x driver
sentence-transformers	ST 5.x
pydantic>=2.0	ST 5.x requires pydantic v2; base image ships only v1

Known compatibility patches (already baked into tds_vector_embedding_lake.py):

• huggingface-hub version guard — the mirror pins hub at 1.2.3 but ST 5.x requires >=1.5.0. ST checks via importlib.metadata.version, not __version__, so both must be spoofed before importing ST.
• types.UnionType in hub validator — transformers 5.x uses Python 3.10+ str | None union syntax in its config dataclasses. Hub 1.2.3's dataclasses.py type-validator has no handler for types.UnionType and raises TypeError: Unsupported type for field 'transformers_version': str | None. Fixed by registering types.UnionType in _BASIC_TYPE_VALIDATORS before model load.

Both patches are applied automatically inside the STO script; no user action needed. pydantic v2 must be installed in the env for ST 5.x's model-config schema to load.

Validated configuration (Tesla T4, CUDA driver 12.7, VCL2 GPU_Cluster):

Component	Version
torch	2.9.1+cu128
sentence-transformers	5.4.1
huggingface-hub (mirror)	1.2.3 (spoofed to 1.9.0 at runtime)
transformers	5.7.0
pydantic	2.13.3
BGE model	BAAI/bge-small-en-v1.5 (384-dim)
Throughput	~1 000 rows / 69 s on a single T4

See demos/notebooks Demo OAF - Vector Embedding/ for the full provisioning notebooks.

Compute Vector Embeddings In-Database (BYOM)

from tdstone2.tdsgenai import compute_vector_embedding_byom

embeddings = compute_vector_embedding_byom(
    model='tdstone2_emb_384_intfloat_multilingual_e5_small',
    dataset=text_dataframe,
    text_column='content',
    accumulate_columns=['id'],
    schema_name=Param['database'],
    table_name='embeddings',
    primary_index=['id'],
)

Output: table with id + 384-dimensional embedding vector per row.

---

Seq2Seq Models (Summarization / Translation / Language Detection)

from tdstone2.tdsgenai_seq2seq import install_seq2seq_model, run_seq2seq

# Install model (e.g., flan-t5-small for summarization)
install_seq2seq_model(
    model_name='google/flan-t5-small',
    model_task='summarization',
    upload=True,
)

# Run via Script Table Operator
results = run_seq2seq(
    dataset=text_dataframe,
    text_column='content',
    model='tdstone2_seq2seq_google_flan_t5_small',
    schema_name=Param['database'],
)

---

Lineage & Registry

sto.list_codes()                       # registered Python scripts/classes
sto.list_models()                      # model configs with all arguments
sto.list_mappers()                     # training/scoring/feature-engineering mappers
sto.list_hyper_models()                # hyper-segmented model registrations
sto.list_feature_engineering_models()  # feature engineering pipeline registrations

---

Local Validation / Debugging

Execute the generated model code locally before deploying to Vantage:

code_and_data = model.get_code_and_data(partition_id=1)

exec(code_and_data['code'])
local_model = MyModel(**code_and_data['arguments'])
df_local = code_and_data['data']
df_local['flag'] = df_local['flag'].astype('category')
df_local['Y2'] = df_local['Y2'].astype('category')
local_model.fit(df_local)
local_model.score(df_local)

---

Documentation & Agent Skills

Architecture & comparison guide

A single-file HTML guide is shipped at docs/tdstone2_documentation.html. It contrasts tdstone2 with teradataml.opensource (td_sklearn), walks the five core abstractions (Code → Model → Mapper → HyperModel / FeatureEngineering), prints DDL for the six repository tables, explains the install-once-then-INSERT model, the ONNX → BYOM-ONNXPredict inference path, and the VCE vs VCL backend split. Sticky sidebar nav, scrollspy, sidebar filter, mobile layout, print stylesheet — open it in any browser, no server needed.

Bundled agent skills (Claude Code / LLM agents)

Twelve .md skills with YAML frontmatter live in tdstone2/skills/. Each is scoped to a single task so an agent picks the right one from the user's intent:

Skill	Triggers on
tdstone2	Decision / routing — when to choose tdstone2 vs td_sklearn
tdstone2-setup	First-time setup, VCE vs VCL choice, TDStone.setup()
tdstone2-hypermodel	Per-partition sklearn (classifier / regressor / anomaly)
tdstone2-deep-learning	Per-partition Keras or PyTorch (LSTM / sequence nets) — weights-only, warm-start
tdstone2-lightgbm	Per-partition streaming LightGBM (LightGBMModel) — native categorical, continued-boosting warm-start
tdstone2-prophet	Per-partition Prophet forecasting (ProphetModel) — forecast()-only, holidays/promotions, tdfs4ds live view
tdstone2-byom-onnx	convert_to_onnx=True, native SQL ONNXPredict via V_<repo>_BYOM_CATALOG
tdstone2-feature-engineering	Reducer + custom transform scripts
tdstone2-genai	Embeddings (BYOM on VCE, OAF on VCL) + Seq2Seq (flan-t5)
tdstone2-custom-script	Non-sklearn Python via Code + Model + Mapper directly
tdstone2-oaf-apply	Raw tdml.Apply script on VCL (no tdstone2 registry) — env reuse, stdout contract
tdstone2-troubleshooting	Apply / OFS errors, ONNXPredict failures, slow scoring, hung embeddings

Install directly into Claude Code's skills directory:

from tdstone2 import install_skills

install_skills()                    # project-local: ./.claude/skills/
install_skills(scope='global')      # user-wide:     ~/.claude/skills/
install_skills(scope='global', overwrite=True)
install_skills(names=['tdstone2', 'tdstone2-hypermodel'], dry_run=True)

Or export to any arbitrary directory:

from tdstone2 import export_skills, list_skills, get_skill

export_skills('~/.claude/skills')                  # user-wide
export_skills('.claude/skills', overwrite=True)    # project-local, replace existing
export_skills('skills/', names=['tdstone2', 'tdstone2-hypermodel'])
export_skills('skills/', dry_run=True)             # preview without writing

list_skills()           # ['tdstone2', 'tdstone2-setup', 'tdstone2-hypermodel', ...]
get_skill('tdstone2')   # raw markdown of one skill

Or via CLI:

python -m tdstone2.skills install               # project-local
python -m tdstone2.skills install --global      # user-wide
python -m tdstone2.skills install --overwrite
python -m tdstone2.skills list
python -m tdstone2.skills show tdstone2-hypermodel
python -m tdstone2.skills export ~/.claude/skills
python -m tdstone2.skills export .claude/skills --overwrite
python -m tdstone2.skills export .claude/skills --dry-run --only tdstone2 --only tdstone2-setup

---

Demo Notebooks

The demos/ folder contains end-to-end worked examples:

Series	Location	Content
Core workflow	demos/notebooks/	Data generation → setup → HyperModel → feature engineering → retrieval (01–16)
Scikit-learn models	demos/notebooks Demo Hypermodel Scikit-Learn/	Classifier, Regressor, Anomaly, LassoLarsCV (STO path)
Scikit-learn models on VCL Apply	demos/notebooks Demo Hypermodel Scikit-Learn with OAF/	Same flow on Vantage Cloud Lake via use_apply=True (VCL1 / VCL2 variants)
Deep Learning (Keras / PyTorch)	demos/notebooks Demo Deep Learning/	Per-partition LSTM forecasting via DeepLearningModel — Keras (nb 01 OAF, 02 STO) and PyTorch (nb 03 OAF, 04 STO), SQL-windowed input, streaming warm-start, per-partition RMSE/MAE + predicted-vs-actual; nb 05 = ONNX export + native ONNXPredict BYOM scoring
Prophet forecasting	demos/notebooks Demo Prophet/	Per-partition retail sales forecasting with promotion periods via ProphetModel — forecast()-only, on both VCE/STO (nb 01) and VCL/OAF (nb 02), per-store predicted-vs-actual + the tdfs4ds live forecast view
Script Table Operator	demos/demo script csae/notebooks/	Raw STO usage with anomaly detection
BYOM Vector Embedding	demos/notebooks Demo BYOM - Vector Embedding/	HuggingFace ONNX install + BYOM embedding
OAF Vector Embedding	demos/notebooks Demo OAF - Vector Embedding/	HuggingFace model on OpenAF platform (VCL1 + VCL2 variants, dedicated tdstone2_embeddings env, CUDA pre-flight)
STO Vector Embedding	demos/notebooks Demo STO - Vector Embedding/	Embedding via Script Table Operator
Seq2Seq	demos/notebooks Demo Seq2Seq/	Summarization and language detection
Chat / Semantic Search	demos/demo chat/notebooks/	Chunking, embedding, vector search

---

Repository Tables (Default Names)

Object	Table
Code	TDS_CODE_REPOSITORY
Model	TDS_MODEL_REPOSITORY
Trained Model	TDS_TRAINED_MODEL_REPOSITORY
Mapper	TDS_MAPPER_REPOSITORY
HyperModel	TDS_HYPER_MODEL_REPOSITORY
Feature Engineering	TDS_FEATURE_ENGINEERING_PROCESS_REPOSITORY

All names are overridable via TDStone.__init__() kwargs.