xplainable-preprocessing

Shared preprocessing pipeline library for xplainable. Define, compile, fit, serialise, and apply data transformation pipelines using a declarative JSON spec.

Install

pip install xplainable-preprocessing
pip install "xplainable-preprocessing[excel]"   # io.read_table for .xlsx/.xls

Upgrading from 0.3.x? 0.4.0 has breaking changes; see CHANGELOG.md.

Quick Start

Define a pipeline as JSON

from xplainable_preprocessing.schema import PipelineSpec
from xplainable_preprocessing.compiler import compile_spec

spec = PipelineSpec(
    version="2.0",
    steps=[
        {"id": "drop_ids", "type": "DropColumnsTransformer", "params": {"columns": ["customer_id", "row_id"]}},
        {"id": "fill_numeric", "type": "FillMissingTransformer", "columns": ["tenure", "charges"], "params": {"strategies": {"tenure": "median", "charges": "median"}}},
        {"id": "clip_outliers", "type": "ClipTransformer", "columns": ["charges"], "params": {"min_val": 0, "max_val": 500}},
        {"id": "missing_flags", "type": "MissingFlagTransformer", "columns": ["tenure", "charges"]},
        {"id": "condense_plan", "type": "CategoryCondenseTransformer", "columns": ["plan_type"], "params": {"max_categories": 10}},
        {"id": "extract_dates", "type": "DateTimeExtractTransformer", "columns": ["signup_date"], "params": {"components": ["month", "dayofweek"], "drop_original": true}},
    ]
)

pipeline = compile_spec(spec)

Fit and transform

import pandas as pd

df = pd.read_csv("data.csv")
pipeline.fit(df)
transformed = pipeline.transform(df)

Serialise and load

from xplainable_preprocessing.serialization import save_pipeline, load_pipeline

# Save
binary = save_pipeline(pipeline)

# Load
loaded = load_pipeline(binary)
result = loaded.transform(new_data)

Available Transformers

Data Cleaning

Missing Values

Feature Engineering

Categorical

Scaling (use with caution -- breaks xplainable model explainability)

Note: Scaling transformers are available but should NOT be used with xplainable models. xplainable models are inherently explainable and require raw feature values. Scaling destroys interpretability.

Other sklearn

Pipeline Spec Format

Pipelines are defined as JSON for portability across the xplainable platform, MCP tools, and API:

{
    "version": "2.0",
    "steps": [
        {
            "id": "step_name",
            "type": "TransformerName",
            "columns": ["col1", "col2"],
            "params": {"param1": "value1"},
            "description": "Optional description"
        }
    ]
}

• id: unique step identifier
• type: transformer class name (from the registry)
• columns: optional list of columns to apply to (wraps in DataFrameColumnTransformer)
• params: constructor arguments for the transformer
• description: optional human-readable description

Architecture

PipelineSpec (JSON)
    ↓ compile_spec()
DataFramePipeline (sklearn Pipeline of transformers)
    ↓ fit() / transform()
Transformed DataFrame
    ↓ save_pipeline()
Binary (cloudpickle)
    ↓ load_pipeline()
DataFramePipeline (restored)

Key modules:

• schema.py -- Pydantic models for PipelineSpec and StepSpec
• compiler.py -- compiles JSON spec into sklearn pipeline
• pipeline.py -- DataFramePipeline wrapper around sklearn Pipeline
• registry.py -- maps transformer names to classes, generates LLM-ready catalog
• serialization.py -- cloudpickle save/load; portable across Python versions
• guard.py -- step safety checks and step/spec previews
• preview.py -- before/after preview with schema deltas
• contracts.py -- which engineered columns each step computes (dependency_map, partial_transform)
• detect/ -- detectors that describe a raw frame (scan_all, Issue)
• io.py -- read_table, the shared CSV/Excel reader
• numeric_text.py -- parse numbers stored as text

Development

pip install -e ".[dev]"
pytest

License

MIT