TinyShift

TinyShift is a lightweight, sklearn-compatible Python library designed for data drift detection, outlier identification, and MLOps monitoring in production machine learning systems. The library provides modular, easy-to-use tools for detecting when data distributions or model performance change over time, with comprehensive visualization capabilities.

For enterprise-grade solutions, consider Nannyml.

Features

• Data Drift Detection: Categorical and continuous data drift monitoring with multiple distance metrics
• Outlier Detection: HBOS, PCA-based and SPAD outlier detection algorithms
• Classification Model Evaluation: Calibration curves, confusion matrices, score distributions, and production confidence analysis
• Time Series Analysis: Seasonality decomposition, trend analysis, forecasting diagnostics, and forecast stabilization
• Decomposed Forecasting: DTL-based non-seasonal and DMSTL-based multi-seasonal forecasting for panel and long-horizon series
• Probabilistic Demand Forecasting: Two-stage discrete or continuous forecasts, calibrated distributions, forecast evaluation, and inventory optimization
• Forecast Stability: Metrics and interpolation methods for stable forecasting

Technologies Used

• Python 3.10+
• Scikit-learn 1.3.0+
• Pandas 2.3.0+
• NumPy
• SciPy
• Statsmodels 0.14.5+
• Plotly 5.22.0+ (optional, for plotting)

📦 Installation

Install the core package with pip:

pip install tinyshift

Or with uv:

uv add tinyshift

Optional module extras

TinyShift now separates optional capabilities into extras so you can install only what you need. Some functions also use lazy importing so optional dependencies are loaded only when they are actually used, helping keep the import surface lightweight and avoiding unnecessary dependency overhead.

• series: forecasting and series-specific dependencies

pip install "tinyshift[series]"
# or
uv add "tinyshift[series]"

• plot: interactive plotting and export support

pip install "tinyshift[plot]"
# or
uv add "tinyshift[plot]"

• notebook: notebook support

pip install "tinyshift[notebook]"
# or
uv add "tinyshift[notebook]"

• all: install all optional extras

pip install "tinyshift[all]"
# or
uv add "tinyshift[all]"

Development installation

Clone the repository and install from source:

git clone https://github.com/HeyLucasLeao/tinyshift.git
cd tinyshift
pip install -e ".[dev]"

With uv:

uv sync --extra dev

📖 Quick Start

1. Categorical Data Drift Detection

TinyShift provides sklearn-compatible reference-to-current drift detectors:

from tinyshift.drift import CatDrift

detector = CatDrift(
    n_resamples=999,
    random_state=42,
).fit(reference_values)

result = detector.predict(current_values)
print(result.p_value, result.drift)

CatDrift uses Jensen–Shannon distance with logarithm base 2, producing a distance in [0, 1], and a two-sample permutation test. Use p_value and drift for monitoring decisions.

2. Continuous Data Drift Detection

For numerical features, use the continuous drift detector:

from tinyshift.drift import ConDrift

detector = ConDrift(
    n_resamples=999,
    random_state=42,
).fit(reference_values)

result = detector.predict(current_values)

ConDrift uses normalized Wasserstein distance and the same permutation inference. Monitoring decisions are exposed through p_value and drift. For panel data, CategoricalDriftAnalyzer and ContinuousDriftAnalyzer apply the corresponding detector independently to each ID. See the drift guide for detector and analyzer examples.

3. Outlier Detection

TinyShift includes sklearn-compatible outlier detection algorithms:

from tinyshift.outlier import SPAD, HBOS, PCAReconstructionError

# SPAD (Simple Probabilistic Anomaly Detector)
spad = SPAD(plus=True)
spad.fit(X_train)

outlier_scores = spad.decision_function(X_test)
outlier_labels = spad.predict(X_test)

# HBOS (Histogram-Based Outlier Score)
hbos = HBOS(dynamic_bins=True)
hbos.fit(X_train, nbins="fd")
scores = hbos.decision_function(X_test)
outlier_labels = hbos.predict(X_test)

# PCA-based outlier detection
pca_detector = PCAReconstructionError(n_components=None)
pca_detector.fit(X_train)
pca_scores = pca_detector.decision_function(X_test)
pca_outlier_labels = pca_detector.predict(X_test)

4. Binary Classification Model Evaluation

Evaluate and visualize classification model performance for production deployment:

from tinyshift.plot import (
    reliability_curve,
    score_distribution, 
    confusion_matrix,
    efficiency_curve,
    beta_confidence_analysis
)

# Model calibration assessment
reliability_curve(
    clf=classifier,
    X=X_test,
    y=y_test,
    model_name="RandomForestClassifier",
    n_bins=15
)

# Analyze prediction confidence patterns
score_distribution(clf, X_test, nbins=20)

# Performance evaluation with interactive confusion matrix
confusion_matrix(clf, X_test, y_test, percentage_by_class=True)

# Conformal prediction analysis
efficiency_curve(conformal_classifier, X_test, y_test)

# Production deployment confidence analysis
beta_confidence_analysis(
    alpha=95, 
    beta_param=5, 
    fig_type=None
)

5. Time Series Analysis and Diagnostics

TinyShift provides comprehensive time series analysis capabilities:

from tinyshift.plot import MSTLDiagnostics
from tinyshift.series import (
    IntermittencyAnalyzer,
    RegularityAnalyzer,
    SeasonalityAnalyzer,
    TemporalStabilityAnalyzer,
    TrendAnalyzer,
    trend_significance, 
    foreca, 
    sample_entropy,
    permutation_entropy,
    theoretical_limit,
    variance_ratio,
    VarianceRatioAnalyzer,
    hampel_filter,
    bollinger_bands
)

diagnostics = MSTLDiagnostics(periods=[7, 365]).fit(time_series)
diagnostics.plot(width=1200, height=800)

# Test for significant trends
slope, r_squared, p_value = trend_significance(time_series)

# Assess forecastability
forecastability = foreca(time_series)
print(f"Forecastability (Omega): {forecastability}")

# Measure complexity and regularity
complexity = sample_entropy(time_series, m=2, tolerance=0.2)
print(f"Sample Entropy: {complexity}")

# Measure ordinal complexity
perm_entropy = permutation_entropy(time_series, m=3, delay=1, normalize=True)
print(f"Permutation Entropy: {perm_entropy}")

# Calculate ordinal regularity (the function name is retained for compatibility)
theo_limit = theoretical_limit(time_series, m=3, delay=1)
print(f"Ordinal Regularity Index: {theo_limit}")

# Inspect persistence at one horizon
ratio, z_statistic, p_value = variance_ratio(time_series, horizon=7)

# Compare several horizons across a panel
vr_summary = VarianceRatioAnalyzer().fit(df).summary()

# Diagnose whether persistent changes justify testing recency weighting
stability = TemporalStabilityAnalyzer(
    horizon=14,
    min_reference_windows=4,
    confirmation_windows=2,
).fit(df)
stability_summary = stability.summary()

# Combined diagnostics for a Nixtla-style panel
analyzers = [
    IntermittencyAnalyzer(),
    RegularityAnalyzer(),
    TrendAnalyzer(),
    SeasonalityAnalyzer(top_k=2),
]
summaries = [analyzer.fit(df).summary() for analyzer in analyzers]
summary = summaries[0]
for section in summaries[1:]:
    summary = summary.merge(section, on="unique_id", validate="one_to_one")

# Outlier detection in time series
outliers = hampel_filter(time_series, window_size=5)
outliers = bollinger_bands(time_series, window_size=20)

# Plot lag analysis with PAMI (Permutation Auto-Mutual Information)
from tinyshift.plot import pami
pami(time_series, nlags=20, m=3, delay=1, normalize=False)

6. Forecast Accuracy Metrics

TinyShift also includes forecast evaluation utilities in the forecasting metrics module, implemented in tinyshift/forecasting/metrics.py. This module provides accuracy, bias, stability, economic-loss, and tail-risk measures:

from tinyshift.forecasting import wape, pbias, score, rmae

# Example evaluation dataframe
# df must contain actual values in the 'y' column and model predictions as columns

wape_df = wape(df, models=["model_a", "model_b"], id_col="unique_id", target_col="y")
pbias_df = pbias(df, models=["model_a", "model_b"], id_col="unique_id", target_col="y")
score_df = score(df, models=["model_a", "model_b"], id_col="unique_id", target_col="y")
rmae_df = rmae(df, models=["model_a", "model_b"], baseline_col="naive", id_col="unique_id", target_col="y")

These utilities cover:

• wape: weighted absolute percentage error for overall accuracy
• pbias: percent bias to detect over- or under-forecasting
• score: composite score combining WAPE and absolute bias
• economic_loss: financial loss from understock and overstock costs
• rmae: relative mean absolute error versus a baseline model
• forecast_instability: revision magnitude across consecutive forecasts
• tail_risk: expected cost, dispersion, VaR, CVaR, and worst-case loss

7. Forecast Stabilization

TinyShift includes forecast interpolation methods and a panel instability metric:

from tinyshift.forecasting import (
    forecast_instability,
    hfi,
    hpi,
    vi,
)

# Calculate period-over-period forecast variability (instability)
# `df` should contain `unique_id`, `ds` (ordered dates) and model forecast columns.
# Example: `forecast_instability(df, models=["model_a", "model_b"], ds_col="ds")`
instability_scores = forecast_instability(df, models=["model_a"], ds_col="ds")

# Apply forecast stabilization techniques
# Vertical Interpolation
stable_forecast = vi(y_hat, anchor, w_s=0.3)

# Horizontal Partial Interpolation
smooth_forecast = hpi(y_hat, w_s=0.4)

# Horizontal Full Interpolation
fully_stable_forecast = hfi(y_hat, w_s=0.5)

8. Preprocessing, Features and Forecasting

These responsibilities are exposed through focused packages:

• filter_features_by_vif — remove highly correlated features using VIF filtering
• FeatureResidualizer — residualize correlated predictors while preserving information
• RobustGaussianScaler — robust scaling with winsorization and power transforms
• DTLWrapper — decomposed LOWESS trend plus ML residual forecasting for non-seasonal data
• DMSTLWrapper — decomposed MSTL forecasting wrapper for panel/multi-seasonal data
• TwoStageForecasterWrapper — configurable Negative Binomial or Gamma predictive distributions and inventory optimization on top of MLForecast Use tinyshift.preprocessing for data transforms and tinyshift.forecasting for estimators and predictive distributions.

9. Advanced Modeling Tools

from tinyshift.preprocessing import FeatureResidualizer, filter_features_by_vif
from tinyshift.stats import BootstrapBCA

# Detect multicollinearity
mask = filter_features_by_vif(X_train, threshold=5.0)
selected_columns = X_train.columns[mask]
X_train = X_train.loc[:, selected_columns].astype(float)
X_test = X_test.loc[:, selected_columns].astype(float)

# Residualize correlated features using the training fit
residualizer = FeatureResidualizer(corrcoef=0.70)
X_train.loc[:, :] = residualizer.fit_transform(X_train)

X_test.loc[:, :] = residualizer.transform(X_test)

# Bootstrap confidence intervals
confidence_interval = BootstrapBCA.compute_interval(
    data,
    confidence_level=0.95,
    statistic=np.mean,
    n_resamples=1000,
    random_state=42,
)

10. Decomposed Forecasting with DTL and DMSTL

TinyShift includes decomposed forecasting wrappers for non-seasonal and multi-seasonal panel data. DTLWrapper extracts a robust LOWESS trend and models residuals with MLForecast:

from tinyshift.forecasting import DTLWrapper
from mlforecast import MLForecast
from sklearn.ensemble import RandomForestRegressor

def residual_model_callable(nlags, freq):
    return MLForecast(
        models=[RandomForestRegressor(random_state=42)],
        lags=nlags,
        freq=freq,
    )

model = DTLWrapper(
    residual_model_callable=residual_model_callable,
    freq="D",
    nlags="auto",
    pami_params={"max_tau": 48, "m": 3, "delay": 1},
    trend_frac=0.2,
    robust=True,
)
model.fit(df, id_col="unique_id", time_col="ds", target_col="y")
preds = model.predict(h=14, stabilization_method="hfi", w_s=0.2)

For multiple seasonalities, use DMSTLWrapper:

from tinyshift.forecasting import DMSTLWrapper
from mlforecast import MLForecast
from sklearn.ensemble import RandomForestRegressor

def residual_model_callable(nlags, freq):
    return MLForecast(
        models=[RandomForestRegressor(random_state=42)],
        lags=nlags,
        freq=freq,
    )

model = DMSTLWrapper(
    residual_model_callable=residual_model_callable,
    freq="D",
    season_length="auto",
    seasonal_detection_params={
        "top_k": 2,
        "noise_threshold_factor": 1.5,
        "significance_level": 0.05,
        "fallback": 7,
    },
    nlags="auto",
    pami_params={"max_tau": 48, "m": 3, "delay": 1},
    log_transform=True,
)

model.fit(df, id_col="unique_id", time_col="ds", target_col="y")

preds = model.predict(h=14, stabilization_method="hfi", w_s=0.2)
print(preds.head())

11. Two-Stage Probabilistic Demand Forecasting

TwoStageForecasterWrapper separates the point forecast from uncertainty calibration. An MLForecast model estimates the conditional mean (lambda_t), while temporal cross-validation calibrates global, horizon, series, and series-by-horizon distribution parameters with hierarchical shrinkage. The default Negative Binomial family supports discrete demand and inventory decisions; GammaFamily supports strictly positive continuous targets.

import pandas as pd
from mlforecast import MLForecast
from sklearn.ensemble import RandomForestRegressor
from tinyshift.forecasting import (
    FirstStageForecasterEvaluator,
    NewsvendorOptimizer,
    TwoStageForecasterEvaluator,
    TwoStageForecasterWrapper,
)

fcst = MLForecast(
    models=[RandomForestRegressor(random_state=42)],
    freq="D",
    lags=[1, 7, 14],
)

model = TwoStageForecasterWrapper(fcst)
model.fit(
    df_train,
    id_col="unique_id",
    time_col="ds",
    target_col="y",
    h=14,
    n_windows=5,
)

# Forecast once and derive all probabilistic views from the same object
forecast = model.predict_distribution(h=14)
forecast_df = forecast.to_frame()
quantiles = forecast.ppf([0.50, 0.95])
distribution = forecast.distribution

# Newsvendor-optimal inventory using shortage and holding costs
stock_plan = NewsvendorOptimizer.optimize(
    forecast_df, distribution, underage_cost=10.0, overage_cost=2.0
)

# Exact probabilities from P(Y=0) through P(Y=10)
probabilities = forecast.pmf(range(11))

# Expected value of stocking each additional discrete inventory unit
marginal_value = NewsvendorOptimizer.marginal_benefit(
    forecast_df,
    distribution,
    underage_cost=10.0,
    overage_cost=2.0,
    max_k=10,
)

# Returns P(Y<=5); PPF columns use names such as Q(0.5)
probability_below_five = forecast.cdf(5)
stockout_risk_above_five = forecast.sf(5)  # P(Y>5)
median = forecast.ppf(0.50)

# Cost columns may be supplied without exogenous forecast features.
costs = pd.DataFrame({"cu": [10.0] * len(forecast), "co": [2.0] * len(forecast)})
stock_plan = NewsvendorOptimizer.optimize(
    forecast_df,
    distribution,
    underage_cost="cu",
    overage_cost="co",
    cost_df=costs,
)

# Continuous alternative; cdf/sf/ppf/interval share the same interface.
from tinyshift.forecasting import GammaFamily

continuous_model = TwoStageForecasterWrapper(fcst, distribution=GammaFamily())

Evaluate only held-out or rolling-origin predictions after joining their actual targets. The first-stage evaluator covers conditional-mean diagnostics and its calibration table; the two-stage evaluator evaluates symmetric quantile pairs derived from a panel predictive distribution using MWIS, empirical coverage, and interval width:

mean_metrics = FirstStageForecasterEvaluator.evaluate(backtest_df)
calibration = FirstStageForecasterEvaluator.calibration_table(
    backtest_df, n_bins=10
)
probabilistic_metrics = TwoStageForecasterEvaluator.evaluate_interval(
    backtest_df,
    forecast,
    coverages=(0.8, 0.9, 0.95),
)

Persist the fitted wrapper so its base forecaster, selected family, and calibrated per-series dispersion parameters remain together:

import joblib

joblib.dump(model, "two_stage_forecaster.joblib")
restored_model = joblib.load("two_stage_forecaster.joblib")

Only load joblib files from trusted sources.

Negative Binomial targets must contain non-negative integer counts; Gamma targets must be strictly positive. Install the series extra to use this wrapper: pip install "tinyshift[series]".

📁 Project Structure

The repository is organized by domain. Each package with dedicated documentation links to it below; public objects are exported from the package's __init__.py.

Runnable notebooks and fitted demonstration artifacts are kept in tinyshift/examples, while the test suite lives in tinyshift/tests.

📋 Requirements

• Python: 3.10+
• Core Dependencies:
  • pandas (>2.3.0)
  • scikit-learn (>1.3.0)
  • statsmodels (>=0.14.5)
• Optional Dependencies:
  • plotly (>5.22.0) - for visualization
  • kaleido (<=0.2.1) - for static plot export
  • nbformat (>=5.10.4) - for notebook support

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Inspired by Nannyml