Skyulf Core

Skyulf Core (skyulf-core) is a standalone, installable Python ML library for teams who want sklearn's dependable estimators with a cohesive, Polars-friendly pipeline layer around them. It unifies preprocessing, classification, regression, clustering, text models, hyperparameter tuning, evaluation, and optional SHAP explanations behind one composable API.

Use it when you want to move from a notebook to a repeatable model artifact without assembling a different interface for every transformer and estimator. Skyulf builds on and is validated against scikit-learn rather than replacing it: sklearn remains the modeling foundation while Skyulf provides pipeline configuration, artifacts, metrics, and safe execution conventions.

Website & Documentation

• Project site / docs: https://www.skyulf.com
• Repository: https://github.com/flyingriverhorse/Skyulf
• PyPI package: https://pypi.org/project/skyulf-core

Installation

Skyulf Core currently packages for Python 3.12+.

pip install skyulf-core

# EDA-focused install (core EDA + optional advanced EDA + visualization)
pip install skyulf-core[eda,viz]

# For visualization support (Rich dashboard + Matplotlib plots)
pip install skyulf-core[viz]

# For advanced EDA add-ons (sentiment + causal discovery)
pip install skyulf-core[eda]

# For hyperparameter tuning engines
pip install skyulf-core[tuning]

# For SHAP explainability
pip install skyulf-core[explainability]

# For dense SentenceEmbedder support
pip install skyulf-core[nlp]

# For imbalance-aware preprocessing (e.g., SMOTE)
pip install skyulf-core[preprocessing-imbalanced]

# For XGBoost modeling nodes
pip install skyulf-core[modeling-xgboost]

# For LightGBM modeling nodes
pip install skyulf-core[modeling-lightgbm]

# For geospatial feature engineering (H3 indexing, spatial stats)
pip install skyulf-core[geo]

# For text sentiment features
pip install skyulf-core[text]

# All non-geo optional runtime features
pip install skyulf-core[all]

# Optional experiment tracking
pip install skyulf-core[mlflow]

all intentionally excludes the native geospatial stack; add [geo] only when you need geospatial nodes.

Databricks Bundle template

The source checkout includes a local-engine Databricks Bundle template for bounded pandas or Polars training and scoring with Skyulf Core. From the repository root, initialize an editable project with:

databricks bundle init skyulf-core/templates/databricks --output-dir ./generated

The generated project has train and score jobs. The first score creates its prediction table; see the Bundle guide for configuration and the single-writer requirement.

Compute engines (pandas / Polars)

Skyulf nodes run on a pandas or Polars engine, chosen per call:

• Auto-detection: pass a pandas or polars DataFrame and the matching engine handles it. The quick start below passes a Polars frame, so it runs on the Polars engine.
• Fallback default: when Skyulf must create a frame itself (no input data to inspect), pandas is the default — the safer, better-covered path. Polars is opt-in for the fallback:

from skyulf.engines import EngineRegistry

EngineRegistry.set_active_engine("polars")

Quick start

import polars as pl
from skyulf import SkyulfPipeline

# Synthetic data, built inline so this runs without any files on disk.
customers = pl.DataFrame(
    {
        "age": [43, 38, None, 52, 61, 29, 47, 55],
        "income": [52000, 61000, 48000, None, 73000, 41000, 58000, 66000],
        "city": ["NY", "LDN", "LDN", "PAR", "NY", "PAR", "NY", "LDN"],
        "purchased": [1, 0, 0, 1, 1, 0, 0, 1],
    }
)
pipeline = SkyulfPipeline(
    {
        "preprocessing": [
            {
                "name": "split",
                "transformer": "TrainTestSplitter",
                "params": {"target_column": "purchased", "test_size": 0.2, "random_state": 42},
            },
            {
                "name": "impute_income",
                "transformer": "SimpleImputer",
                "params": {"columns": ["income"], "strategy": "median"},
            },
            {
                "name": "encode_city",
                "transformer": "OneHotEncoder",
                "params": {"columns": ["city"], "drop_original": True, "handle_unknown": "ignore"},
            },
        ],
        "modeling": {
            "type": "logistic_regression",
            "params": {"max_iter": 500, "random_state": 42},
        },
    }
)

pipeline.fit(customers, target_column="purchased")
pipeline.save("customer_model.pkl")
new_customers = pl.DataFrame(
    {
        "age": [35, 48],
        "income": [55000, None],
        "city": ["PAR", "NY"],
    }
)
predictions = SkyulfPipeline.load("customer_model.pkl").predict(new_customers)

See examples/00_quickstart.ipynb for a complete, executed save/load round trip.

How it fits together

flowchart LR
    subgraph Input
        A[Polars DataFrame]
    end
    subgraph SkyulfPipeline
        B[FeatureEngineer<br/>preprocessing steps]
        C[ModelEstimator<br/>sklearn-backed model]
    end
    subgraph Output
        D[Fitted artifact<br/>.pkl]
        E[Metrics report]
        F[Predictions]
    end
    A --> B --> C
    C --> E
    B -.fit once, reuse.-> D
    D --> C
    A2[New data] --> B
    C --> F

    style A fill:#e8f4fd,stroke:#1a73e8
    style A2 fill:#e8f4fd,stroke:#1a73e8
    style D fill:#fef7e0,stroke:#f9ab00
    style E fill:#e6f4ea,stroke:#188038
    style F fill:#e6f4ea,stroke:#188038

SkyulfPipeline wraps two collaborators: a FeatureEngineer that runs the declared preprocessing steps in order, and a ModelEstimator that fits/ applies the configured sklearn-backed model. fit() runs both once and returns a metrics report; predict() re-applies the already-fitted preprocessing artifacts (no re-fitting, no leakage) before calling the model.

Which API? Use SkyulfPipeline (above) by default. The Calculator/ Applier pairs it wraps (e.g. SimpleImputerCalculator/Applier) are lower-level — only use them directly to embed a single step in a custom (e.g. sklearn) pipeline.

Custom evaluation harnesses: if you want to compare raw sklearn/XGBoost/ CatBoost/etc. estimators against the exact preprocessed split SkyulfPipeline would use internally — instead of reimplementing the split/target-extraction/Polars-to-pandas conversion yourself and risking a different split than fit() actually used — call pipeline.get_fitted_split(data, target_column="..."). It runs the same configured preprocessing chain and returns (X_train, y_train, X_test, y_test) as plain pandas objects, raising a clear error if the config doesn't include a TrainTestSplitter step:

X_train, y_train, X_test, y_test = pipeline.get_fitted_split(customers, target_column="purchased")

Threshold tuning: the default decision rule (argmax for multiclass, 0.5 for binary) is rarely optimal for imbalanced classes or a metric you actually care about (F1, MCC, balanced accuracy, ...). pipeline.optimize_thresholds( X_val, y_val, metric=...) searches per-class thresholds against a metric you supply — evaluated on validation data you pass in explicitly (never the pipeline's internal split) — and predict(use_tuned_thresholds=True) then applies them. X_val must be raw: the method runs the pipeline's fitted preprocessing on it itself, exactly once, so that the cutoffs are fitted against the same probabilities predict() later reproduces. get_fitted_split() above is therefore not a source for it — those frames are already preprocessed, and passing them here transforms the holdout a second time. Carve the holdout off the raw data before fit() instead. The same search is available as standalone array-level functions, from skyulf.modeling import optimize_thresholds, apply_thresholds, for use outside a pipeline. See the Threshold Tuning guide for signatures, a full example, and how the binary grid / multiclass Nelder-Mead search works.

Naming: preprocessing names are PascalCase (SimpleImputer, TrainTestSplitter), modeling names are snake_case (logistic_regression). A few preprocessing nodes are snake_case exceptions: feature_target_split, tokenizer, tfidf_vectorizer, count_vectorizer, hashing_vectorizer, sentence_embedder, feature_selection. Unsure of a name? Use NodeRegistry.list_transformers()/.list_models(). "Split" is a deprecated alias for "TrainTestSplitter".

NodeRegistry.list_models() includes both Modeling and Ensemble nodes. Use category="Ensemble" to list Voting/Stacking models only, or category="Modeling" for the other models. list_transformers() excludes both model categories and can filter by a preprocessing category such as category="Text". Unknown categories return an empty list.

Data leakage safety

Split before any data-dependent preprocessing. Fitting an imputer, scaler, learned encoder, feature selector, learned binning step, outlier detector, or Count/TF-IDF vectorizer on rows that include validation/test data contaminates evaluation. Put TrainTestSplitter first in a pipeline, or explicitly create a SplitDataset first and then fit FeatureEngineer/SkyulfPipeline.

The larger Skyulf platform validates DAGs and hard-blocks data-dependent nodes upstream of a train/test split. Skyulf Core is also useful on its own, so the same ordering is an essential API-level contract.

sequenceDiagram
    participant D as Raw data
    participant S as TrainTestSplitter
    participant Tr as Train split
    participant Te as Test split
    participant P as Learned steps<br/>(Imputer/Encoder/Scaler/...)
    participant M as Model

    D->>S: full dataset
    S->>Tr: train rows
    S->>Te: test rows (held out, untouched)
    Tr->>P: fit(train)
    P->>Tr: transform(train)
    Tr->>M: fit(model)
    P->>Te: transform(test)  Note over P,Te: uses train-fitted statistics only
    Te->>M: evaluate (honest, unseen)

Any step that learns a statistic (imputer medians, encoder categories, scaler mean/std, TF-IDF vocabulary, feature selectors, outlier detectors, learned binning) must come after the split in the preprocessing list. Deterministic, row-independent parses (string splitting, unit conversions, date-part extraction) are safe before the split. Read and run examples/01_house_prices_regression.ipynb to see this pattern applied to a real regression dataset, including a deliberate pre-split/post-split split of "safe" vs. "learned" feature steps.

Use the opt-in static check before fitting:

warnings = skyulf.validate_leakage_safety(config)
warnings = SkyulfPipeline(config).validate_leakage_safety()

Polars-native, no hidden pandas

Users can pass polars.DataFrame inputs directly. PolarsEngine and SklearnBridge convert Polars straight to NumPy at the sklearn boundary—there is no user-facing pandas round trip. Arrow integrations use the explicit to_arrow() path (and therefore pyarrow). Every notebook in examples/ uses Polars + NumPy only — no pandas import appears anywhere in the example code.

Examples

Runnable Jupyter notebooks in examples/, covering the full feature set end-to-end on real datasets. Open with jupyter lab or jupyter notebook from the repository root — see examples/README.md for dataset sourcing notes and a per-notebook breakdown of what's demonstrated.

Automated EDA

Skyulf Core includes automated exploratory data analysis for Polars frames — data quality, distributions, outliers, correlations/target-association, optional temporal/geospatial analysis, and a PCA-based exploratory clustering pass, all from one .analyze() call.

import polars as pl
from skyulf import EDAAnalyzer, EDAVisualizer

df = pl.read_csv("data.csv")
profile = EDAAnalyzer(df).analyze(
    target_col="target",  # Optional: unlocks target-association analysis
    date_col="timestamp",  # Optional: unlocks temporal analysis
    lat_col="latitude",  # Optional: unlocks geospatial analysis
    lon_col="longitude",  # Optional
)

EDAVisualizer(profile, df).summary()  # Rich terminal dashboard (skyulf-core[viz])
EDAVisualizer(profile, df).plot()  # Matplotlib figures (skyulf-core[viz])

Everything the visualizer renders is also available as plain data on profile, so you can build your own dashboards, logs, or CI gates:

print(f"Rows: {profile.row_count}  Missing cells: {profile.missing_cells_percentage:.2f}%")

# Human-readable, prioritized data-quality findings
for alert in profile.alerts[:5]:
    print("-", alert.message)

# Actionable next-step suggestions (e.g. "consider log-transforming X")
for rec in profile.recommendations[:5]:
    print(f"- [{rec.action}] {rec.column or ''}: {rec.reason} -> {rec.suggestion}")

# Outlier detection (IsolationForest by default) across numeric columns
if profile.outliers is not None:
    print(f"Outliers: {profile.outliers.outlier_percentage:.1f}% ({profile.outliers.method})")

# Feature-vs-target association (ANOVA-style p-value + boxplot data)
income_interaction = next(
    (ti for ti in (profile.target_interactions or []) if ti.feature == "income"), None
)
if income_interaction is not None and income_interaction.p_value is not None:
    print(f"income vs target -> p-value: {income_interaction.p_value:.2e}")

# Rule-tree feature importances (a fast, model-free "what matters" signal)
if profile.rule_tree is not None:
    top = sorted(profile.rule_tree.feature_importances, key=lambda d: -d["importance"])[:5]
    for f in top:
        print(f"  {f['feature']:20s} {f['importance']:.4f}")

See any of examples/00_quickstart.ipynb through 08_online_retail_customer_segmentation.ipynb for the full EDA pass run against real datasets, both with and without a target column.

Features

mindmap
  root((Skyulf Core))
    Preprocessing
      Imputation / Encoding / Scaling
      Outlier detection / Binning
      Date, geospatial, lag/rolling, interaction features
      Text cleaning + Count/TF-IDF/Hashing vectorizers
    Modeling
      Classification / Regression
      Clustering
      Voting + Stacking ensembles
      Naive Bayes (text)
    Tuning
      Grid Search
      Random Search
      Optuna
    Evaluation
      Standardized metrics
      SHAP explainability
    EDA
      Data quality / distributions
      Outliers / temporal / geospatial
      Target correlation + rule-tree importances
    Engine
      Polars-native
      NumPy bridge to sklearn
      Explicit Arrow export

• Unified pipelines: Serializable preprocessing and model artifacts with readable descriptions, fingerprints, model cards, and prediction APIs.
• Leakage-aware execution: Split-first guidance and platform validation guard against fitting learned preprocessing on held-out rows.
• Preprocessing and feature engineering: Cleaning, casting, imputation, encoders, scalers, outliers, selection, binning, dates, geospatial features, time-series lags/rolling windows, and interactions.
• Modeling: sklearn-backed classification, regression, segmentation, and Voting/Stacking ensembles, plus text-specific Naive Bayes models.
• Tuning and evaluation: Grid Search, Random Search, Optuna, standardized classification/regression/clustering metrics, and optional SHAP.
• Automated EDA: Data quality, distributions, outliers, temporal, geospatial, and target analysis for Polars data.
• Polars and Arrow: Native Polars support with direct NumPy bridging for sklearn and an explicit Arrow export path.
• Notebook export: The Skyulf platform can export any pipeline as a runnable Jupyter notebook you own — your exact node config inlined, ready to run in any Jupyter environment with pip install skyulf-core, no Skyulf platform required.

License

skyulf-core — this standalone Python library — is licensed under the Apache License 2.0; see the LICENSE file beside it.

That is the permissive half of a split model: the Skyulf backend and frontend are GNU AGPLv3. You can use skyulf-core in proprietary projects without restriction, while running a modified backend or frontend as a network service carries AGPLv3's copyleft obligations. See COMMERCIAL-LICENSE.md for the full terms.