polars-learn

Prepare Polars data for scikit-learn with reusable feature expressions, column selection, and schema validation. Work with eager DataFrame inputs or keep LazyFrame queries deferred until you need their results.

Requires Python 3.11+, Polars 1.20+, and scikit-learn 1.6+.

Installation

Install the package into your project:

uv add polars-learn
# Or use pip:
pip install polars-learn

For optional pandas input support:

uv add 'polars-learn[pandas]'
# Or use pip:
pip install 'polars-learn[pandas]'

See Development for running this repository from a checkout.

Choose a Class

Class	Use it when	What fitting does
SchemaSpec	You already know the required column names and dtypes.	Checks your explicit contract and records output names.
SchemaValidator	Training data should define the schema for later input.	Learns every training column's name, order, and dtype.
FeatureSpec	You want to describe one feature operation.	This is a description, not a transformer; there is no fit.
FeatureEngineer	You want to execute feature operations in order.	Validates the transformation and records input/output schemas.
ColumnSelector	You want to keep, reorder, or rename selected columns.	Checks selected columns and records their output names.
FeaturePassthrough	You want an explicit unchanged-column pipeline step or branch.	Checks selected columns and records their unchanged names.

The five transformers share fit, transform, fit_transform, get_feature_names_out, get_params, set_params, and set_output. All six classes accept verbosity. See the API reference for every constructor option, fitted attribute, and public method.

Quick Start

import polars as pl

from polars_learn import FeatureEngineer, FeatureSpec, SchemaValidator

X = pl.DataFrame({"price": [10.0, 20.0, 30.0], "quantity": [2, 3, 4]})

validator = SchemaValidator().fit(X)
validated = validator.transform(X)

engineer = FeatureEngineer(
    [
        FeatureSpec(pl.col("price") * pl.col("quantity"), alias="notional"),
        FeatureSpec(
            pl.col("notional").log1p(), alias="log_notional", drop=["quantity"]
        ),
    ]
)
result = engineer.fit_transform(validated)

assert isinstance(result, pl.DataFrame)
assert result.columns == ["price", "notional", "log_notional"]
assert result["notional"].to_list() == [20.0, 60.0, 120.0]
print(engineer.get_feature_names_out().tolist())
# ['price', 'notional', 'log_notional']

fit(X) returns the transformer. transform(X) applies a fitted transformer. fit_transform(X) fits and then transforms the same input. Calling transform or get_feature_names_out before a successful fit raises scikit-learn's NotFittedError. The input frame is never mutated.

Feature Engineering

Create a FeatureSpec with exactly one of expression (a single pl.Expr) or expressions (a nonempty sequence). Use native expression names, an expression's .alias(...), or the specification's alias argument to name its outputs. An explicit alias requires exactly one expression with one output.

import polars as pl

from polars_learn import FeatureEngineer, FeatureSpec

quotes = pl.DataFrame({"ask": [11.0], "bid": [9.0]})
spec = FeatureSpec(
    expressions=[
        (pl.col("ask") - pl.col("bid")).alias("spread"),
        ((pl.col("ask") + pl.col("bid")) / 2).alias("mid"),
    ],
    drop=["ask", "bid"],
    name="quote_features",
)
result = FeatureEngineer(spec).fit_transform(quotes)
assert result.to_dict(as_series=False) == {"spread": [2.0], "mid": [10.0]}

FeatureEngineer accepts a single FeatureSpec directly, as in the example above, a dict of FeatureSpec keyword arguments, or an ordered sequence mixing those forms. Dicts are unpacked with FeatureSpec(**config) and undergo the same validation. For example:

import polars as pl

from polars_learn import FeatureEngineer

X = pl.DataFrame({"price": [10.0, 20.0]})
engineer = FeatureEngineer({"expression": pl.col("price") * 2, "alias": "doubled"})
assert engineer.fit_transform(X)["doubled"].to_list() == [20.0, 40.0]

A dict uses the exact keys expression, expressions, alias, drop, name, and verbosity accepted by FeatureSpec, with exactly one of expression or expressions. Expressions must be actual Polars expression objects; expression strings are not evaluated. Unknown keys and invalid values fail at construction, with the configuration's sequence index included when applicable. An empty dict is invalid; use FeatureEngineer([]) for an identity operation.

Single definitions are treated as one-element sequences internally; constructor parameters, including original dicts, are retained unchanged for scikit-learn cloning and get_params. The same forms are accepted by set_params(features=...); refit after changes.

Expressions within one specification run together. They can reference columns already present, but cannot reference columns newly created by another expression in that same specification. Use separate specifications for dependent features. Specifications run in list order; drops occur after each specification's expressions. Later specifications cannot use dropped columns.

Existing columns can be overwritten: FeatureSpec(pl.col("price") + 1) replaces price. Selectors such as pl.all().name.suffix("_copy") can expand to multiple outputs. Duplicate output names and missing expression or drop inputs fail during schema checking; duplicates detectable when constructing a specification fail there. An empty FeatureEngineer([]) leaves the columns unchanged.

FeatureEngineer.transform requires the fitted input names in exactly the same order, and the resulting output names, order, and dtypes must match the fitted output schema. Input dtypes may differ if the final output schema still matches. Use a SchemaValidator first when input types need a defined conversion policy.

Fitting eager input evaluates the feature expressions. Fitting lazy input checks the plan's schema without evaluating rows. Features do not learn statistics; use deterministic expressions. Currently fit_transform calls both fit and transform, so eager feature expressions are evaluated twice.

Fixed Contracts and Column Selection

SchemaSpec checks an explicit, nonempty schema. Required columns must have exactly the specified dtypes; this class does not cast them. Both its fit and transform validate input, and transform requires a successful fit.

ColumnSelector returns just the columns you choose, in the requested order. Unselected columns are ignored, input order may differ, and dtypes are not validated. Selected and renamed output names must be unique.

import polars as pl

from polars_learn import ColumnSelector, SchemaSpec

X = pl.DataFrame({"price": [10.0, 20.0], "quantity": [2, 3]})
contract = SchemaSpec({"price": pl.Float64, "quantity": pl.Int64})
validated = contract.fit_transform(X)

selector = ColumnSelector(["quantity", "price"], rename={"quantity": "qty"})
selected = selector.fit_transform(validated)
assert selected.columns == ["qty", "price"]

For SchemaSpec, extra columns are rejected by default. If extras are allowed, extra_columns="drop" produces only the required columns; "preserve" puts required columns first and then extras in their input order. Preserved extra column names become part of the fitted output names. Later transforms must produce those same names in the same order; their dtypes are not checked. Use SchemaValidator if inference may introduce new extra columns.

Explicit Feature Passthrough

Use FeaturePassthrough(columns) when a pipeline layer should simply carry named columns forward. It returns just those columns in the specified order, with their Polars values, names, dtypes, nulls, and row order preserved. There is no renaming or casting option. Selected columns must exist; input order and dtypes may differ between calls. It supports the same verbosity, pandas conversion, fitted feature names, and output controls as the other transformers.

import polars as pl
from sklearn.pipeline import Pipeline

from polars_learn import FeaturePassthrough

X = pl.DataFrame({"id": [7, 8], "flag": [True, False], "unused": [1, 2]})
pipeline = Pipeline([("unchanged", FeaturePassthrough(["id", "flag"]))])
result = pipeline.fit_transform(X.lazy())
assert isinstance(result, pl.LazyFrame)
assert result.collect().equals(X.select("id", "flag"))

A sequential Pipeline passes each step's output to the next step, so a passthrough step selecting two columns omits all the others. To keep unchanged columns alongside transformed columns, put the operations in parallel ColumnTransformer branches:

import polars as pl
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler

from polars_learn import FeaturePassthrough

X = pl.DataFrame({"value": [10.0, 20.0], "id": [7, 8], "flag": [True, False]})
preprocessing = ColumnTransformer(
    [
        ("scaled", StandardScaler(), ["value"]),
        ("unchanged", FeaturePassthrough(["id", "flag"]), ["id", "flag"]),
    ],
    verbose_feature_names_out=False,
).set_output(transform="polars")
result = preprocessing.fit_transform(X)
assert result["value"].to_list() == [-1.0, 1.0]
assert result.select("id", "flag").equals(X.select("id", "flag"))
assert result.columns == ["value", "id", "flag"]

The outer column lists tell ColumnTransformer which input to send to each branch; FeaturePassthrough declares its own selected-column contract. Use eager input at this boundary (lazy_X.collect() for a lazy query). Polars output keeps the separate column dtypes. verbose_feature_names_out=False prevents branch prefixes on output names; final names must then be unique. For branch behavior, see the scikit-learn ColumnTransformer reference. Use ColumnSelector when you also need optional renaming.

Learned Schema Validation

SchemaValidator.fit learns all training columns as required columns. Transform checks them against the learned dtypes according to mode:

Mode	Treatment of a different input dtype
"strict" (default)	Reject it; no casting.
"semi-strict"	Allow only the lossless widening policy below.
"permissive"	Attempt a Polars cast with strict=True; some successful casts may lose precision.

Semi-strict permits integer widening when the entire source dtype's range fits inside the fitted integer dtype's range, including compatible unsigned to signed widening. Small observed values alone do not make narrowing acceptable. It also permits Float32 to Float64, and integer to Float64 only when every non-null value round-trips exactly. For example, 2**53 and 2**54 are accepted, but 2**53 + 1 is rejected. String to numeric conversion, narrowing, and Null dtype to another dtype are rejected in this mode.

import polars as pl

from polars_learn import SchemaValidator

train = pl.DataFrame({"x": pl.Series([1], dtype=pl.Int64)})
incoming = pl.DataFrame({"x": pl.Series([2, None], dtype=pl.Int32)})
validator = SchemaValidator(mode="semi-strict").fit(train)
result = validator.transform(incoming)
assert result.schema["x"] == pl.Int64
assert result["x"].to_list() == [2, None]

Required columns must always be present. Extra columns fail unless allow_extra_columns=True; allowing extras does not make fitted columns optional. extra_columns="drop" removes allowed extras, while the default "preserve" keeps them. The drop setting alone does not allow extras.

enforce_column_order=True checks the relative order of required columns, so allowed extras can appear between them. With enforce_column_order=False, output puts fitted columns first, followed by preserved extras in their input order. get_feature_names_out() lists required fitted columns, excluding inference-only extras. Preserving extras can change the output width; drop them before an estimator that expects a fixed number of features.

Null values are preserved when the dtype is compatible. Nullability constraints are not learned. In permissive mode, a column with Null dtype can be cast to the fitted dtype. Failed strict casts raise; they do not silently replace invalid values with nulls. Choose the validation policy through mode: "strict", "semi-strict", or "permissive". The default is "strict".

LazyFrames and Return Types

All five transformers accept a pl.LazyFrame, including a query from pl.scan_csv(...). Default output follows the input to each call, independently of whether fitting used eager or lazy input. Pandas input, when conversion is allowed, is treated as an eager DataFrame.

set_output(transform=...)	DataFrame input	LazyFrame input
"default" (initial behavior)	DataFrame	LazyFrame
"polars"	DataFrame	DataFrame; collects the result
"polars-lazy"	LazyFrame	LazyFrame
None	Keep the existing choice	Keep the existing choice

import polars as pl

from polars_learn import FeatureEngineer, FeatureSpec

X = pl.DataFrame({"price": [10.0, 20.0], "quantity": [2, 3]})
engineer = FeatureEngineer(
    [FeatureSpec(pl.col("price") * pl.col("quantity"), alias="notional")]
)
plan = engineer.fit_transform(X.lazy())  # Build a plan; do not evaluate rows
assert isinstance(plan, pl.LazyFrame)
result = plan.collect()  # Execute when ready
assert result["notional"].to_list() == [20.0, 60.0]

engineer.set_output(transform="polars")
eager_result = engineer.transform(X.lazy())  # Collect inside transform
assert isinstance(eager_result, pl.DataFrame)

engineer.set_output(transform="polars-lazy")
lazy_result = engineer.transform(X)
assert isinstance(lazy_result, pl.LazyFrame)
engineer.set_output(transform="default")

Output settings control transform and fit_transform returns. fit always returns the transformer and does not collect lazy rows just because eager output is configured. Forcing a lazy return from eager input can wrap an already computed result; pass lazy input to keep row evaluation deferred throughout.

Lazy operations check column names, order, dtypes, expression dependencies, and drops using schema metadata. Resolving metadata can access a source's metadata; it does not collect all its rows. Lazy fit validates the plan schema, not every input value. Value-dependent errors surface at collection. A returned LazyFrame is a query plan: collecting it again may execute the query again.

Inspect lazy output with plan.collect_schema() for names and dtypes, and plan.collect() for data. Verbosity reports lazy row counts as unknown rather than evaluating rows to count them.

scikit-learn Pipelines

A pipeline built from these transformers can remain lazy. Set the last native transformer to eager output before a downstream estimator that needs materialized data. "polars-lazy" is a package-specific output option; do not pass it to scikit-learn's standard transformers.

import numpy as np
import polars as pl
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

from polars_learn import FeatureEngineer, FeatureSpec, SchemaValidator

X = pl.DataFrame({"price": [10.0, 20.0, 30.0], "quantity": [2, 3, 4]})
y = np.array([1.0, 2.0, 3.0])
pipeline = Pipeline(
    [
        ("schema", SchemaValidator(mode="semi-strict")),
        (
            "features",
            FeatureEngineer(
                [FeatureSpec(pl.col("price") * pl.col("quantity"), alias="notional")]
            ).set_output(transform="polars"),
        ),
        ("scale", StandardScaler().set_output(transform="polars")),
        ("model", RandomForestRegressor(n_estimators=20, random_state=0)),
    ]
)
pipeline.fit(X.lazy(), y)
predictions = pipeline.predict(X.lazy())
assert predictions.shape == (3,)

Supported input is Polars DataFrames and LazyFrames, plus explicitly enabled pandas DataFrames. NumPy arrays, Series, Python lists, and sparse arrays are not accepted. These transformers bypass scikit-learn's automatic output wrapping and global output-container settings; configure each native transformer with set_output. There is no automatic conversion to NumPy or pandas output.

Cloning keeps constructor parameters and explicit output settings, and discards fitted state. y is accepted for estimator compatibility and ignored. Additional fit_transform keyword parameters, including sample_weight, are unsupported. Refit after changing parameters that affect the fitted contract; set_params does not reset fitted state. Invalid settings are rechecked when processing input.

Verbosity

All six classes accept an integer verbosity:

Level	Output to standard output
0 (default)	No progress diagnostics.
1	Input dimensions and fit/transform progress; specification construction summaries.
2	Level 1 plus schemas and feature operation details.

import polars as pl

from polars_learn import SchemaValidator

X = pl.DataFrame({"x": [1, 2]})
validator = SchemaValidator(verbosity=1)
validated = validator.fit_transform(X)
# [SchemaValidator] Validating input
# [SchemaValidator] Input: 2 rows, 1 columns
# [SchemaValidator] Fit complete; output columns: ['x']
# [SchemaValidator] Validating input
# [SchemaValidator] Input: 2 rows, 1 columns
# [SchemaValidator] Transform complete: 2 rows, 1 columns

FeatureSpec reports its construction separately. An engineer's verbosity does not change its specifications' verbosity. A dict config's verbosity applies when that config is converted into a specification, during construction and each fit/transform call. Frame summaries show metadata rather than printing data rows; feature details can include expression representations. Levels must be 0, 1, or 2; booleans and strings are rejected. Verbosity does not suppress exceptions or pandas conversion warnings.

Errors and When They Occur

Situation	Exception or warning	Timing
Invalid argument type or unsupported input	TypeError	Construction or the next relevant operation
Invalid settings, conflicting policies, or duplicate configured names	ValueError	Construction or revalidation
Transform or feature-name access before fitting	sklearn.exceptions.NotFittedError	Method call
SchemaSpec contract or fitted output-name mismatch	ValueError	Fit/transform schema check
ColumnSelector/FeaturePassthrough missing columns or fitted output-name mismatch	ValueError	Fit/transform schema check
SchemaValidator missing columns, disallowed extras/order/types, or eager cast failure	SchemaValidationError	Transform
Feature dependency, drop, or schema failure; eager feature execution failure	FeatureEngineeringError	Fit/transform
Value-dependent feature failure or permissive cast failure in a returned lazy plan	Native Polars exception	collect() or another execution operation
Lossy semi-strict integer to Float64 cast in a lazy plan	SchemaValidationError	Lazy execution
Enabled pandas conversion with warnings enabled	PandasConversionWarning	Conversion, even at verbosity 0

PolarsLearnError is the base of SchemaValidationError and FeatureEngineeringError. Plain TypeError/ValueError, NotFittedError, and native Polars errors do not inherit from it. Input query and source failures can also propagate as native Polars or source exceptions during schema resolution or collection. Wrapped errors preserve their original exception as __cause__.

To force value evaluation within transform, request "polars" output. FeatureEngineer wraps resulting execution errors in FeatureEngineeringError; SchemaValidator wraps Polars collection errors in SchemaValidationError. SchemaSpec, ColumnSelector, and FeaturePassthrough do not wrap source/collection errors.

Optional Pandas Support

Install the pandas extra shown above, then explicitly allow conversion:

import pandas as pd

from polars_learn import SchemaValidator

X = pd.DataFrame({"price": [10.0, 20.0], "quantity": [2, 3]})
validator = SchemaValidator(convert_pandas=True, warn_on_conversion=False)
result = validator.fit_transform(X)
assert result.columns == ["price", "quantity"]

All five transformers expose convert_pandas and warn_on_conversion. Conversion is disabled by default. When enabled, it emits PandasConversionWarning unless warn_on_conversion=False. Use Polars directly to avoid conversion overhead; fit_transform may convert input more than once. Pandas and PyArrow are imported only when needed. The extra includes PyArrow for nullable, extension, and other dtypes that require it. Conversion follows Polars conventions, including NaN to null conversion and ignoring the pandas index. Unsupported dtypes raise; conversion does not fall back to row operations. An explicit "polars-lazy" output choice returns a lazy wrapper after conversion.

Development

Run from a repository checkout:

uv sync  # Includes pandas and PyArrow for the full test suite
uv run pytest --doctest-modules src tests
uv run ruff check .
uv run ruff format .
uv run ruff format --check .
uv run pyright
uv build

Ruff is the formatter for all Python code and documentation examples. CI checks ruff format --check ., linting, types, tests, and API doctests. Ruff's directory checks cover Python files, not Markdown code fences; format a Markdown example with ruff format --stdin-filename example.py before adding it. The test matrix covers Python 3.11 and 3.12 with all development dependencies, including pandas and PyArrow. Every job runs the full interoperability suite. A subprocess test separately verifies native usage with pandas and PyArrow blocked.

uv.lock pins development dependencies. Pyright includes its Node runtime via the development group. Pandas and PyArrow are required for development; missing pandas causes test collection to fail rather than silently skip coverage. Run uv sync after changing requirements. These remain optional runtime dependencies for users of the package. The package includes py.typed; transformer return annotations cover both DataFrame and LazyFrame because set_output can change the container. Narrow with isinstance(result, pl.LazyFrame) when your type checker needs the specific type. Library dependencies use minimum versions; the lockfile does not constrain downstream applications to those exact versions.

Releases

Run Actions → Create GitHub Release → Run workflow to release the version in pyproject.toml from the selected branch or tag. The workflow validates the package, builds a fresh wheel and source archive, and creates a GitHub release named vX.Y.Z with those files attached. Once that workflow succeeds, Upload Python Package starts automatically and publishes the same built files to PyPI. Existing files in a local dist/ directory are not used.

Before the first release:

1. Push both release and publishing workflows to the repository's default branch. The automatic publisher uses GitHub's workflow_run event, which requires the publishing workflow on that branch.
2. Create the GitHub Actions environment pypi if it does not already exist.
3. Configure the polars-learn PyPI project's trusted publisher with your GitHub repository owner/name, workflow filename python-publish.yml, and environment pypi. See PyPI's setup instructions. Publishing uses OIDC and does not read a PYPI_API_TOKEN secret.

For each new version:

1. Set project.version in pyproject.toml and __version__ in src/polars_learn/__init__.py to the same new X.Y.Z version, then run uv lock.
2. Commit and push the release changes. The runner builds from the selected committed revision, so uncommitted local changes are not included.
3. Run Create GitHub Release from Actions, selecting the release branch or tag.
4. Check both workflow runs. The second run uses the pypi environment; any protection rules configured for that environment still apply.

The release workflow fails on mismatched versions, an existing vX.Y.Z tag, failed tests/checks, or invalid build metadata. The automatic publisher only accepts successful manual release-workflow runs from this repository and downloads artifacts from that exact run. Failed or cancelled release runs do not publish. It works with GITHUB_TOKEN without relying on its release event to start another workflow. See GitHub's workflow_run documentation.

Publishing a GitHub release manually through the Releases page is also supported. Use a vX.Y.Z tag matching the code's package version. That path builds and validates the tagged source before uploading to PyPI.

If PyPI publishing fails after the GitHub release was created, fix the reported setup or upload problem and rerun Upload Python Package, rather than creating the same release tag again. Its automatic path reuses the original release-run artifacts, which must still be available. PyPI versions cannot be overwritten; use a new version for changed distributions.

License

MIT. See LICENSE.