Statistical Context Engineering (SCE)

Cross-fitted, leakage-aware target statistics for tabular regression.

SCE enriches a tabular dataset with group-level statistics of the target variable (means, medians, quantiles, spread, counts) computed over categorical columns — the "context" each row lives in. The hard part of target encoding is doing it without leaking the target: SCE uses out-of-fold cross-fitting on the training set, supports temporal fold layouts for time-ordered data, and ships diagnostics that help you check enrichment behavior on your data (no automated check can prove the absence of leakage).

Installation

pip install stat-context

import sce  # package name on PyPI is stat-context, import name is sce

Optional extras:

pip install stat-context[models]     # LightGBM + CatBoost backends
pip install stat-context[data]       # remote dataset download helpers
pip install stat-context[dev]        # tests, linters
pip install stat-context[equity]     # experimental equity research stack (see below)
pip install stat-context[sentiment]  # experimental sentiment scorers (FinBERT, VADER)

60-second quickstart

Copy-paste runnable — synthetic data, no files needed:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import root_mean_squared_error
from xgboost import XGBRegressor

from sce import StatisticalContextEngine, ContextConfig, AggregationMethod

# --- toy data: price depends on a hidden per-city level the model can't see
rng = np.random.default_rng(42)
n = 8000
cities = rng.choice([f"city_{i}" for i in range(40)], size=n)
city_level = {f"city_{i}": rng.normal(100, 30) for i in range(40)}
df = pd.DataFrame({
    "city": cities,
    "sqft": rng.uniform(20, 120, size=n),
    "rooms": rng.integers(1, 6, size=n),
})
df["price"] = (
    df["city"].map(city_level) * 10 + df["sqft"] * 50 + rng.normal(0, 300, size=n)
)

# --- split FIRST, then enrich (this is the recommended order)
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)

config = ContextConfig(
    target_col="price",
    categorical_cols=["city"],
    aggregations=[AggregationMethod.MEAN, AggregationMethod.STD, AggregationMethod.COUNT],
    use_cross_fitting=True,   # train rows get out-of-fold stats
    n_folds=5,
)
engine = StatisticalContextEngine(config)
train_enriched = engine.fit_transform(train_df)                       # OOF stats
test_enriched = engine.transform(test_df.drop(columns=["price"]))     # full-train stats

# --- single model on the enriched features
features_sce = ["sqft", "rooms"] + [c for c in train_enriched.columns
                                    if c.startswith("city_price_")]
model = XGBRegressor(n_estimators=200, random_state=42)
model.fit(train_enriched[features_sce], train_enriched["price"])
rmse = root_mean_squared_error(
    test_df["price"], model.predict(test_enriched[features_sce])
)
print(f"test RMSE: {rmse:,.0f}")

This example shows the mechanics: split first, fit the engine on train only, transform test without the target. It deliberately does not stage a baseline contest. If you add one, keep it fair: city is visible to any baseline too, so a meaningful comparison must let the baseline encode it (e.g. OneHotEncoder on city, or a target encoder with its own out-of-fold machinery) — a numeric-only baseline is not evidence that enrichment helps.

New features follow the pattern {column}_{target}_{stat}, e.g. city_price_mean, city_price_std, city_price_count, plus fold-variance features (*_fold_std, *_fold_lower, *_fold_upper) that quantify how stable each statistic is across folds. To see what was added, diff the DataFrame columns before and after enrichment.

Time-ordered data

cross_fit_strategy separates two different temporal layouts; both require time_col and build folds in chronological order (rows are sorted by time_col before splitting, and each validation fold follows its training window):

• "time" — expanding-window temporal folds (TimeSeriesSplit): each validation fold is later than its training data, and the training window grows fold by fold.
• "rolling" — same ordering, but the training window is capped (rolling_max_train_size), so old observations drop out of the statistics; an optional rolling_gap can additionally separate train and validation by a buffer.

config = ContextConfig(
    target_col="sales",
    categorical_cols=["store", "dept"],
    use_cross_fitting=True,
    cross_fit_strategy="rolling",   # or "time" for expanding folds
    time_col="date",
    n_folds=5,
)

Caveat: folds are split by row position after sorting, so rows sharing an identical timestamp can land on both sides of a fold boundary. The engine also does not check that labels were observable when each row is scored — ensuring a chronologically sound split and point-in-time label availability is your responsibility.

Current capabilities

• Out-of-fold cross-fitting with four strategies: random (shuffled K-fold), time (expanding temporal folds), rolling (capped sliding temporal window), and off (no cross-fitting).
• Grouped target summaries — per group: mean, median, std, var, CV, quantiles, count.
• Interactions — 2-way categorical crosses (city×property_type) by default (include_interactions=True, max_interaction_depth=2).
• Rare-group backoff — groups smaller than min_group_size (default 5) fall back to global statistics instead of producing noisy estimates.
• Fold-stability features (include_fold_variance=True by default) — *_fold_std/*_fold_cv and *_fold_lower/*_fold_upper (fold mean ± 2× fold std) describe how stable each statistic is across folds. They are stability diagnostics, not calibrated uncertainty or confidence bounds.
• Model factory — sce.build_model("ridge", ...) now returns a RidgeCV that selects its regularization over a built-in alpha grid (a scalar alpha argument is ignored — pass alphas to override). The pure-scikit-learn estimators (ridge, random forest, extra trees, gradient boosting) are wrapped in a median-imputing Pipeline, so the NaN-heavy feature blocks SCE produces for small groups fit cleanly. Details: CHANGELOG.md.
• Optional cleanup — FeatureCleanupPipeline for pruning low-value or unstable context features after enrichment.

Scope warnings:

• Relative features are opt-in and leakage-prone. include_relative_features defaults to False, and when enabled, transform requires the target column at inference time (the engine raises if the target is missing). SCE is therefore not suitable for ordinary predictive deployments with that option on — it is a research/analysis tool in that mode.
• No guaranteed improvements. Enrichment helps when the categorical hierarchy is informative; it can do nothing or hurt otherwise. There is no guarantee in either direction.

Configuration reference

class StatisticalContextEngine(BaseEstimator, TransformerMixin):
    def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None) -> "StatisticalContextEngine": ...
    def transform(self, X: pd.DataFrame) -> pd.DataFrame: ...
    def fit_transform(self, X: pd.DataFrame, y: Optional[pd.Series] = None) -> pd.DataFrame: ...

Scikit-learn compatible; sce.pipeline.create_sce_pipeline(config, model) wraps the engine and an optional model into a Pipeline fit on (X, y). Full API docs: joint-hubs.github.io/sce

How it works

1. Grouping — categorical columns (and optionally their pairwise interactions) define groups: city, city×property_type, …
2. Aggregation — for each group, compute the configured target statistics.
3. Cross-fitting — on the training set, each row receives statistics computed without its own fold (random K-fold by default, or expanding/rolling temporal folds for time-ordered data). Small groups back off to global statistics.
4. Inference — test/production rows receive statistics computed from the full training set; the target is not needed at transform time (unless relative features are explicitly enabled).

The recommended evaluation protocol is split-first: the train/test split happens before any enrichment, encoders and pruning are fit on train only, and the test set is touched once, at scoring time.

Evidence status

The numbers in this repository's results/ directories come from historical, exploratory runs on a small set of datasets. They are illustrative only: they are not a curated benchmark, they predate the current code, and negative or null results were not systematically collected or reported. Do not treat any figure there as an expected gain.

• Exploratory result directories (not checked into git): results/, results_gcp_*/
• Dataset configurations (including ones excluded from that set): configs/, configs/experimental/
• Run metadata (git SHA, config hash, seed, metrics, diagnostics) is written to metadata.json in each results directory, so individual runs remain traceable to an exact commit and config.

Reproducing experiments

Requires a repo clone (datasets are rebuilt locally — Kaggle sources cannot be redistributed):

git clone https://github.com/joint-hubs/sce.git && cd sce
pip install -e .[dev,data]

# rebuild the Kaggle-derived datasets (needs Kaggle API credentials)
python scripts/prepare_new_datasets.py
python scripts/prepare_m5_dataset.py --download

sce datasets list                                  # see what is available
python scripts/run.py --dataset rental_poland_short   # single experiment
python scripts/run.py --all                        # full sweep

Diagnostics

python -m scripts.diagnostics.permuted_target  --dataset rossmann_daily
python -m scripts.diagnostics.shuffled_groups  --dataset rossmann_daily
python -m scripts.diagnostics.crossfit_ab      --dataset rossmann_daily

These checks reduce the risk of silent leakage but do not prove its absence.

Experimental: equity research package

The 0.5.0 release also ships an experimental equity package — point-in-time OHLCV ingestion for equity universes (e.g. S&P 500), SCE feature enrichment, and a walk-forward forecaster with sector-head, residual, and quantile heads. Install its dependencies with the [equity] extra; a companion [sentiment] extra adds FinBERT/VADER sentiment scoring for the same pipeline.

A single-fold smoke run (requires a prices.parquet in the canonical schema documented in equity/data/README.md):

python -m equity.forecaster.run_smoke --prices prices.parquet --sectors sectors.csv

Walk-forward evaluation is available via python -m equity.forecaster.run_walk_forward. This is research tooling for studying SCE on market data: it makes no investment-performance claims and is not trading software.

Research status and roadmap

An earlier version of the accompanying manuscript was submitted to ICML 2026 and was not accepted. There is no publication, acceptance, or under-review status to report.

Planned work:

• Leakage audit, including nested evaluation (selection and reporting on the same fold can itself leak).
• Matched comparisons against hierarchical/mixed-model baselines: target encoding variants, hierarchical GLMMs, and gradient-boosted trees with native categorical support (e.g. CatBoost) under identical splits and budgets.
• Multi-domain regression benchmark with pre-registered inclusion rules so negative results are part of the published set.
• Reproducibility and negative results as first-class outputs.
• Conditional shrinkage and compact context selection — pursued only if supported by the audit and comparison results above.

Project layout

sce/                  # the library (pip install stat-context)
├── engine.py         #   StatisticalContextEngine
├── config.py         #   ContextConfig, AggregationMethod
├── models.py         #   model factory (xgboost core; lightgbm/catboost optional)
├── stats.py          #   aggregation functions, backoff
├── cleanup.py        #   FeatureCleanupPipeline
└── io/               #   dataset registry + loading
configs/              # dataset configs (TOML)
configs/experimental/ # configs excluded from the exploratory set
equity/               # experimental equity research package (see above)
scripts/              # experiment runner, diagnostics, dataset preparation
tests/                # pytest suite (incl. leakage guard tests)

License

This software is released under the Apache License 2.0 — free for academic, research, and commercial use.

The accompanying research manuscript remains under CC BY-NC 4.0.

Citation

If you use SCE in academic work, please cite the software:

@misc{stat_context_2026,
  title        = {Statistical Context Engineering (stat-context)},
  author       = {Stachowicz, Mateusz and Halkiewicz, Stanisław},
  year         = {2026},
  url          = {https://github.com/joint-hubs/sce},
  note         = {Python software}
}

Links

• Source code
• Issue tracker
• Documentation
• Contributing