rgram 0.2.3

A Python library for regressograms and kernel smoothing, designed for data analysis and visualisation.

Rgram

Rgram is a high-performance Python library for nonparametric regression analysis and visualisation. It provides tools for creating regressograms (binned regression estimators) and performing kernel smoothing with the Epanechnikov kernel. Built on top of Polars for rapid data processing, Rgram is designed for exploratory data analysis and statistical visualisation.

Theoretical foundation: Regressograms are discussed in Section 4.4 of García-Portugués, E. (2023). Notes for nonparametric statistics. Carlos III University of Madrid.

Table of Contents

• Features
• When to Use Rgram
• Installation
• Quick Start
• Concepts & Architecture
• API Reference
  • Regressogram
  • KernelSmoother
• Examples
• Guides
  • Choosing a Binning Strategy
  • Custom Aggregation Functions
  • Working with Groups (Hue)
  • Data Input Formats
• Benefits vs Limitations
• Limitations
• Troubleshooting
• Future Improvements
• Contributing
• References
• License

Features

• Regressogram Analysis: Multiple binning strategies (dist, width, unique, int) for flexible bin assignment
• Confidence Intervals: Customisable confidence interval computation via user-defined aggregation functions
• Kernel Smoothing: Epanechnikov kernel smoother with flexible bandwidth selection (silverman, scott, manual)
• Predictions: Apply fitted models to new data points via predict() method
• Grouped Analysis: Support for grouping variables (hue) to analyse multiple subgroups simultaneously
• Polars Backend: High-performance DataFrame operations using lazy evaluation
• Scikit-learn API: Familiar fit(), transform(), and fit_transform() methods
• Array-like or DataFrame Input: Works seamlessly with Polars DataFrames or NumPy/Python arrays
• Composable Design: Clean, focused API allows users to easily compose additional statistical methods

When to Use Rgram

Rgram is ideal for:

1. Exploratory Data Analysis (EDA): Quickly visualise relationships between variables without assuming a specific functional form
2. Non-parametric Regression: When you don't want to assume the underlying relationship is linear or polynomial
3. Binned Estimation: When you need interpretable, step-wise predictions (e.g., age-based analysis, price ranges)
4. Grouped Analysis: When comparing multiple strata or groups simultaneously
5. Robust Estimation: When outliers exist and robust statistics (median, quantiles) are preferred
6. Semi-parametric workflows: As a first step before fitting parametric models or validating assumptions

Rgram is NOT the best choice for:

• High-dimensional feature spaces (use dimensionality reduction + Rgram, or scikit-learn alternatives)
• Time series with temporal dependencies (use specialised time series libraries)
• Classification tasks (Rgram is for regression only)
• When you need real-time predictions on streaming data (requires refitting)
• When extremely fast inference on massive datasets is critical (though Polars is reasonably fast)

Requirements

• Python >= 3.9
• Polars >= 1.28.1
• typing-extensions >= 4.15.0

Optional Dependencies (for development)

• pytest >= 8.4.2 - Testing framework
• matplotlib >= 3.9.4 - Visualisation
• seaborn >= 0.13.2 - Statistical graphics
• scipy >= 1.13.1 - Statistical functions
• ruff >= 0.14.7 - Code linting
• ipykernel >= 6.31.0 - Jupyter support

Installation

Using UV (Recommended)

UV is a fast Python package installer and resolver written in Rust. It's the recommended way to work with this project.

1. Clone the repository:

   git clone https://github.com/JackGreenaway/Rgram.git
   cd Rgram
   

2. Install with UV:

   uv sync
   

3. Install the package in development mode:

   uv pip install -e .
   

4. Verify installation:

   python -c "from rgram import Regressogram, KernelSmoother; print('Installation successful!')"
   

Using pip

If you prefer using standard pip:

git clone https://github.com/JackGreenaway/Rgram.git
cd Rgram
pip install -e .

Quick Start

Basic Regressogram Example

import polars as pl
import numpy as np
from rgram import Regressogram

# Generate sample data
np.random.seed(42)
n = 100
x = np.linspace(0, 10, n)
y = np.sin(x) + np.random.normal(0, 0.5, n)

# Create and fit regressogram
df = pl.DataFrame({"x": x, "y": y})
rgram = Regressogram(binning="dist")
result = rgram.fit_transform(data=df, x="x", y="y").collect()

print(result.head())

Kernel Smoothing Example

from rgram import KernelSmoother

# Apply kernel smoothing to regressogram output
smoother = KernelSmoother(n_eval_samples=100)
smoothed = smoother.fit_transform(data=result, x="x_val", y="y_pred_rgram").collect()

print(smoothed.head())

Concepts & Architecture

Regressogram Overview

A regressogram is a non-parametric regression estimator that:

1. Divides the feature space (x-axis) into bins
2. Aggregates target values (y) within each bin using a function (default: mean)
3. Returns the aggregated value for all points in that bin

Key advantages: Simple, interpretable, and computationally efficient Trade-off: Creates step-wise predictions (discontinuous at bin boundaries)

Binning Strategies

Rgram supports four binning methods:

Strategy	Method	Use Case	Output
"dist" (Default)	Distribution-based with Scott's bandwidth	General purpose, data-driven	Fewer bins in sparse regions
"width"	Fixed bin width from data range	When consistent bin sizes matter	Equal-width bins
"int"	Integer bin assignment	When x values are naturally discrete	Integer-indexed bins
"unique"	Uses x values as unique bins	Per-unique-value statistics	One bin per unique x value

Kernel Smoother Overview

A kernel smoother applies the Epanechnikov kernel to smooth predictions:

1. Defines evaluation points across the x-axis
2. For each point, computes a weighted average of nearby y values
3. Weights decay with distance from the evaluation point

Key advantages: Smooth predictions, continuous derivatives Trade-off: More computationally expensive than regressograms

Bandwidth Selection

The KernelSmoother supports three bandwidth selection methods:

1. Silverman's Rule (default) - Robust and data-adaptive: $$h = 0.9 \min(\sigma, IQR/1.34) \cdot n^{-1/5}$$ Best for most use cases; automatically adapts to data spread

2. Scott's Rule - Simpler and less sensitive to outliers: $$h = 1.06 \cdot \sigma \cdot n^{-1/5}$$ Good for normally distributed data

3. Manual Specification - Full control for expert users Specify exact bandwidth value for fine-tuned smoothness control

Each method balances bias and variance differently. Experiment with bandwidth parameter to find optimal smoothing for your data.

Data Flow Architecture

Input Data (arrays/DataFrame)
    ↓
[_prepare_data] - Normalize to LazyFrame with named columns
    ↓
[fit] - Learn parameters (bins, bandwidth, etc.)
    ↓
[transform] - Apply learned mapping to all rows
    ↓
Output LazyFrame (collect() to materialise results)

Key design patterns:

• Lazy evaluation: Polars LazyFrames enable optimisation and memory efficiency
• Fit-predict separation: Learn on one dataset, apply to another (e.g., train/test split)
• Composable: Chain output of one tool into input of another

Regressogram

The Regressogram class performs binned regression on one or more features and targets with customisable aggregation and optional confidence intervals.

Parameters

Regressogram(
    binning: Literal["dist", "width", "all", "int"] = "dist",
    agg: Callable[[pl.Expr], pl.Expr] = lambda x: x.mean(),
    ci: Optional[tuple[Callable, Callable]] = (lambda x: x.mean() - x.std(), lambda x: x.mean() + x.std()),)

Parameter	Type	Default	Description
binning	str	"dist"	Binning strategy. Options: "dist" (distribution-based), "width" (fixed width), "unique" (unique x values), "int" (integer bins)
agg	callable	lambda x: x.mean()	Aggregation function to apply to y values within each bin. Must accept and return a Polars expression
ci	tuple of callables	(mean-std, mean+std)	Tuple of functions for lower and upper confidence limit calculations. Set to None to disable

Methods

fit(x, y, data=None, hue=None, keys=None) -> Regressogram

Fit the regressogram to data.

• x: Column name(s) if data provided, else array-like
• y: Column name(s) if data provided, else array-like
• data: pl.DataFrame, pl.LazyFrame, or None. If None, x/y treated as arrays
• hue: Optional grouping variable(s) for stratified analysis
• keys: Optional additional grouping columns

Returns: self (fitted estimator)

transform() -> pl.LazyFrame

Returns the fitted regressogram results.

Output columns:

• x_val: Binned x values
• y_pred_rgram: Aggregated y predictions (based on agg function)
• y_pred_rgram_lci, y_pred_rgram_uci: Confidence interval bounds (if ci provided)
• y_val: Original y values

fit_transform(x, y, data=None, hue=None, keys=None) -> pl.LazyFrame

Fit and return results in one step. Recommended for most use cases.

predict(x: Union[Sequence[float], pl.Series]) -> pl.Series

Make predictions on new x values using the fitted binning scheme.

• x: Array-like or pl.Series of new x points to predict

Returns: pl.Series containing predicted values based on learned bins

---

KernelSmoother

The KernelSmoother class performs kernel smoothing using the Epanechnikov kernel with flexible bandwidth selection.

Parameters

KernelSmoother(
    n_eval_samples: int = 100,
    bandwidth: Literal['silverman', 'scott', 'manual'] = 'silverman',
    bandwidth_value: Optional[float] = None,
    hue: Optional[Sequence[str]] = None
)

Parameter	Type	Default	Description
n_eval_samples	int	100	Number of evaluation points where the kernel smoother is evaluated
bandwidth	str	'silverman'	Bandwidth selection method: 'silverman', 'scott', or 'manual'
bandwidth_value	float	None	Manual bandwidth value. Required if bandwidth='manual'
hue	sequence of str	None	Optional grouping variable(s). Can be set here or passed to fit()

Bandwidth Methods:

• 'silverman' (default): 0.9 × min(std, IQR/1.34) × n^(-1/5) - Robust, adapts to data spread
• 'scott': 1.06 × std × n^(-1/5) - Simpler, less sensitive to outliers
• 'manual': User specifies exact bandwidth value for fine-tuned control

Methods

fit(x, y, data=None, hue=None) -> KernelSmoother

Fit the kernel smoother to data using the selected bandwidth method.

• x: Column name if data provided, else array-like (must be univariate)
• y: Column name if data provided, else array-like (must be univariate)
• data: pl.DataFrame, pl.LazyFrame, or None
• hue: Optional grouping variable(s) for stratified smoothing. Overrides hue from __init__ if provided

Returns: self (fitted estimator)

transform() -> pl.LazyFrame

Returns the kernel smoothed results at evaluation points.

Output columns:

• x_eval: Evaluation points
• y_kernel: Kernel-smoothed y values

fit_transform(x, y, data=None, hue=None) -> pl.LazyFrame

Fit and return results in one step.

predict(x_new) -> pl.Series

Apply the fitted smoother to new x values without refitting. Uses the bandwidth value determined during fit().

• x_new: Array-like or pl.Series of new x points

Returns: pl.Series containing smoothed predictions

Note: The bandwidth is determined once during fit() using the selected method and training data. predict() applies this same bandwidth to new points, ensuring consistent smoothing behavior. No refitting is required.

---

Examples

Example 1: Complete Regressogram Workflow with Visualisation

import polars as pl
import numpy as np
import matplotlib.pyplot as plt
from rgram import Regressogram, KernelSmoother

# Generate synthetic data
np.random.seed(42)
n = 150
x = np.linspace(0, 10, n)
y_true = np.sin(x)
y_noisy = y_true + np.random.normal(0, 0.6, n)

# Create DataFrame
df = pl.DataFrame({"x": x, "y_true": y_true, "y_noisy": y_noisy})

# Fit regressogram with different binning strategies
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

for ax, binning in zip(axes.flat, ["dist", "width", "int", "unique"]):
    rgram = Regressogram(
        binning=binning,
        ci=(lambda x: x.mean() - 1.96 * x.std(), lambda x: x.mean() + 1.96 * x.std())
    )
    result = rgram.fit_transform(data=df, x="x", y="y_noisy").collect()

    ax.scatter(x, y_noisy, alpha=0.4, s=20, label="observations")
    ax.plot(x, y_true, "g-", linewidth=2, label="true function")
    ax.step(result["x_val"], result["y_pred_rgram"], "r-", linewidth=2, label="rgram")
    ax.fill_between(
        result["x_val"],
        result["y_pred_rgram_lci"],
        result["y_pred_rgram_uci"],
        alpha=0.2,
        color="red",
        label="95% CI"
    )

    ax.set_title(f"Binning: {binning}")
    ax.set_xlabel("x")
    ax.set_ylabel("y")
    ax.legend(fontsize=8)
    ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Example 2: Grouped Analysis with Hue

import polars as pl
import numpy as np
from rgram import Regressogram
import matplotlib.pyplot as plt

# Generate data with multiple groups
np.random.seed(42)
n = 200
x = np.tile(np.linspace(0, 10, 100), 2)
group = np.repeat(["Group A", "Group B"], 100)
y_a = np.sin(x[:100]) + np.random.normal(0, 0.4, 100)
y_b = np.cos(x[100:]) + np.random.normal(0, 0.4, 100)
y = np.concatenate([y_a, y_b])

df = pl.DataFrame({"x": x, "y": y, "group": group})

# Fit regressogram with grouping
rgram = Regressogram(binning="dist")
result = rgram.fit_transform(data=df, x="x", y="y", hue="group").collect()

# Visualise
fig, ax = plt.subplots(figsize=(10, 6))
for grp in ["Group A", "Group B"]:
    mask = result["group"] == grp
    ax.step(result.filter(mask)["x_val"], result.filter(mask)["y_pred_rgram"],
            label=f"rgram - {grp}", linewidth=2)

ax.set_xlabel("x")
ax.set_ylabel("y")
ax.legend()
ax.grid(True, alpha=0.3)
plt.show()

Example 3: Combining Regressogram with Kernel Smoothing

from rgram import Regressogram, KernelSmoother
import polars as pl
import numpy as np
import matplotlib.pyplot as plt

# Generate data
np.random.seed(42)
x = np.linspace(0, 2 * np.pi, 200)
y = np.sin(x) * np.exp(-x / 5) + np.random.normal(0, 0.3, 200)

df = pl.DataFrame({"x": x, "y": y})

# Step 1: Fit regressogram
rgram = Regressogram(binning="dist", ci=None)  # No CI for clarity
rgram_result = rgram.fit_transform(data=df, x="x", y="y").collect()

# Step 2: Smooth the regressogram predictions
smoother = KernelSmoother(n_eval_samples=200)
smoothed = smoother.fit_transform(
    data=rgram_result, x="x_val", y="y_pred_rgram"
).collect()

# Visualisation
fig, ax = plt.subplots(figsize=(12, 6))
ax.scatter(x, y, alpha=0.3, s=20, label="Raw observations")
ax.step(rgram_result["x_val"], rgram_result["y_pred_rgram"],
        where="post", linewidth=2, label="Regressogram")
ax.plot(smoothed["x_eval"], smoothed["y_kernel"],
        linewidth=2.5, color="green", label="Kernel smoothed")

ax.set_xlabel("x")
ax.set_ylabel("y")
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Example 4: Custom Aggregation Functions

from rgram import Regressogram
import polars as pl
import numpy as np

# Generate data
np.random.seed(42)
x = np.linspace(0, 10, 150)
y = np.sin(x) + np.random.normal(0, 0.5, 150)

df = pl.DataFrame({"x": x, "y": y})

# Use median instead of mean
rgram_median = Regressogram(
    binning="dist",
    agg=lambda x: x.median(),
    ci=(
        lambda x: x.quantile(0.25),
        lambda x: x.quantile(0.75)
    )
)
result = rgram_median.fit_transform(data=df, x="x", y="y").collect()

print("Regressogram with median aggregation:")
print(result.select(["x_val", "y_pred_rgram", "y_pred_rgram_lci", "y_pred_rgram_uci"]).head())

Guides

Choosing a Binning Strategy

Distribution-based binning ("dist") ← Default choice for most cases

• Uses Scott's bandwidth rule to adapt bin width to data density
• Fewer, wider bins where data is sparse; more bins where data is dense
• Handles duplicates: Robust to duplicate x values; uses qcut with allow_duplicates=True internally
• Best for: Normal or near-normal distributions, data exploration, datasets with duplicate values
• Example: Customer age analysis with uneven age distribution

rgram = Regressogram(binning="dist")
# Automatically creates wider bins for underrepresented ages
# Handles duplicate ages gracefully
result = rgram.fit_transform(data=df, x="age", y="purchase_amount").collect()

Fixed-width binning ("width")

• Creates equal-sized bins across the entire range
• Consistent interpretation across bins
• Best for: When bin boundaries have business meaning (e.g., income brackets: $0-50K, $50-100K, etc.)
• Drawback: May have very sparse bins at data extremes

rgram = Regressogram(binning="width")
# Creates consistent income brackets, though some may be nearly empty
result = rgram.fit_transform(data=df, x="annual_income", y="credit_score").collect()

Integer binning ("int")

• Casts x values to integers and groups by integer value
• Best for: Truly discrete integer data (number of items, years of experience)
• Example: Product rating (1-5 stars) vs review count

rgram = Regressogram(binning="int")
result = rgram.fit_transform(data=df, x="product_rating", y="num_reviews").collect()

Unique value binning ("unique")

• Treats each unique x value as its own independent bin
• Best for: Computing statistics at each unique x value without binning across x values
• Use case: When x is already categorical or when you want predictions for each exact x value

rgram = Regressogram(binning="unique", agg=lambda x: x.mean())
result = rgram.fit_transform(data=df, x="x_col", y="target").collect()
# Result has one row per unique x value with its corresponding y statistic
# No binning/grouping across different x values occurs

Custom Aggregation Functions

By default, Regressogram uses the mean, but you can specify any Polars aggregation:

import polars as pl
from rgram import Regressogram

# Median (robust to outliers)
rgram_median = Regressogram(agg=lambda x: x.median())

# Count of observations per bin
rgram_count = Regressogram(
    agg=lambda x: pl.len(),
    ci=None  # Disable confidence intervals for count
)

# Standard deviation
rgram_std = Regressogram(agg=lambda x: x.std())

result = rgram_count.fit_transform(data=df, x="x", y="dummy_col").collect()
# y_pred_rgram now contains bin counts

Working with Groups (Hue)

Use the hue parameter to analyse multiple subgroups simultaneously:

from rgram import Regressogram
import polars as pl

# Sales by region
result = Regressogram(binning="dist").fit_transform(
    data=sales_df,
    x="time_of_week",
    y="revenue",
    hue="region"  # Analyse each region separately
).collect()

# Output includes a "region" column; group results show region-specific trends
for region in result["region"].unique():
    region_data = result.filter(pl.col("region") == region)
    print(f"Region {region} - Mean revenue: {region_data['y_pred_rgram'].mean()}")

Benefits of hue:

• Single fit call vs multiple calls for each group
• Ensures consistent binning across groups
• More efficient than separate analyses
• Output includes group identifier for easy filtering

Data Input Formats

Rgram follows a seaborn-like API where you can use either:

Pattern 1: DataFrame + Column Names (Best for large data and reusable workflows)

import polars as pl
from rgram import Regressogram

df = pl.DataFrame({
    "age": [25, 30, 35, 40, 45],
    "salary": [50000, 55000, 60000, 70000, 80000]
})

# Reference columns by name (like seaborn.kdeplot)
rgram = Regressogram()
result = rgram.fit_transform(data=df, x="age", y="salary").collect()

Pattern 2: Raw Arrays/Series (Best for quick analysis, interactive work)

import numpy as np
from rgram import Regressogram

x = np.array([25, 30, 35, 40, 45])
y = np.array([50000, 55000, 60000, 70000, 80000])

# Pass arrays directly without a DataFrame (like seaborn.kdeplot with just x=)
rgram = Regressogram()
result = rgram.fit_transform(x=x, y=y).collect()

Pattern 3: Mixed with Polars Series

import polars as pl
from rgram import Regressogram

df = pl.DataFrame({
    "age": [25, 30, 35, 40, 45],
    "salary": [50000, 55000, 60000, 70000, 80000]
})

# Use Series directly without wrapping in DataFrame
result = rgram.fit_transform(x=df["age"], y=df["salary"]).collect()

Pattern 4: Multiple features/targets

# Multiple x columns (analysed as separate x-y pair combinations)
result = rgram.fit_transform(
    data=df,
    x=["age", "experience"],
    y="salary"
).collect()
# Creates separate results for each x-y feature pair

When to use which pattern:

• DataFrame + names: Production code, complex pipelines, grouped analysis (hue parameter)
• Raw arrays: Quick exploration, notebooks, when data is already in memory
• Series: Intermediate between the two; good for simple scripts

Benefits vs Limitations

Advantages of Regressogram

Benefit	Description
Interpretability	Step-wise predictions are easy to explain to stakeholders ("if age 25-35, avg salary is X")
Robustness	Can use median or quantiles instead of mean for outlier-resistant estimates
Flexibility	Custom aggregation functions support domain-specific logic (e.g., weighted means)
Multi-group	hue parameter enables simultaneous analysis of multiple subgroups with consistent binning
Speed	Binning is computationally efficient; results scale well with data size
No assumptions	Non-parametric; doesn't assume linearity, polynomials, or other functional forms

Advantages of Kernel Smoother

Benefit	Description
Smoothness	Produces continuous predictions without step discontinuities
Local relationships	Captures local patterns via adaptive weighting
Derivative existence	Smooth function enables gradient-based analysis
Visual appeal	Creates professional-looking curves for plots
Flexible composition	Can be chained after regressogram for two-stage smoothing

Limitations Summary

Limitation	Impact	Workaround
1D only	Cannot handle high dimensions directly	Use dimensionality reduction or analyse features independently
No feature selection	All x variables are used	Pre-select relevant features based on domain knowledge
Binning creates artifacts	Regressogram has artificial step discontinuities	Use KernelSmoother after Regressogram, or use Regressogram only for EDA
Bandwidth sensitivity	Kernel results vary with bandwidth choice	Silverman's rule is automatic; use cross-validation for critical applications
Memory for large data	Lazy evaluation has limits during .collect()	Process data in batches; use Polars partitioning
No missing value handling	NaN values cause errors	Impute or remove missing values before fitting
No real-time predictions	Must refit to add new data	Refitting is fast enough for small->medium datasets
Categorical inputs	X and Y must be numeric	Encode categorical variables (ordinal encoding or one-hot + aggregation)

Limitations

• Univariate Kernel Smoothing: KernelSmoother currently only supports single-variable smoothing. Multivariate kernel smoothing is not yet implemented.

• Bandwidth Selection: Kernel smoothing offers three methods (Silverman, Scott, Manual) but selection remains user-driven; automatic cross-validation is not yet implemented.

• Binning Strategy Selection: The choice of binning strategy can significantly impact results. The library provides multiple strategies but does not automatically select the optimal one. Users should experiment or use cross-validation.

• Memory Efficiency: For very large datasets, even with Polars' optimisations, lazy evaluation may be limited by system memory during collection.

• Multi-dimensional Input: Regressogram and KernelSmoother are designed for 1D→1D mappings. Multi-dimensional feature spaces require feature engineering or multiple univariate analyses.

• Missing Values: The current implementation does not explicitly handle missing values. Pre-processing with appropriate techniques (imputation, removal) is required.

• Categorical Features: Both classes require numerical input. Categorical variables must be encoded numerically before use.

Troubleshooting

Common Issues and Solutions

Q: I'm getting RuntimeError: You must call fit() before transform()

A: Ensure you call .fit() before .transform(), or use .fit_transform() as a shortcut:

# ❌ Wrong
smoother = KernelSmoother()
result = smoother.transform()  # Error!

# ✅ Correct
smoother = KernelSmoother()
smoother.fit(data=df, x="x", y="y")
result = smoother.transform()

# ✅ Alternative (recommended)
smoother = KernelSmoother()
result = smoother.fit_transform(data=df, x="x", y="y")

Q: My results have very few bins or all data in one bin

A: This often happens with highly skewed distributions. Try different binning strategies:

from rgram import Regressogram

# If dist binning creates too few bins, try width or int
rgram_width = Regressogram(binning="width")
result = rgram_width.fit_transform(data=df, x="age", y="income").collect()

# Or check your data distribution first
print(f"X range: {df['age'].min()} to {df['age'].max()}")
print(f"X std: {df['age'].std()}, IQR: {df['age'].quantile(0.75) - df['age'].quantile(0.25)}")

Q: KernelSmoother results look too smooth/too wiggly

A: Adjust the n_eval_samples parameter (more samples = smoother curve with finer detail):

# Too wiggly? Use fewer evaluation points
smoother_smooth = KernelSmoother(n_eval_samples=50)

# Want more detail? Use more evaluation points
smoother_detailed = KernelSmoother(n_eval_samples=500)

Q: Getting ValueError: If data is None, input must be an array-like

A: You mixed column names with array mode. Either pass column names with a DataFrame, or pass arrays without a DataFrame:

import polars as pl
import numpy as np
from rgram import Regressogram

# ❌ Wrong mix
df = pl.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
result = Regressogram().fit_transform(x=df["x"], y="y", data=df)  # Error!

# ✅ Correct: use column names with data
result = Regressogram().fit_transform(data=df, x="x", y="y").collect()

# ✅ Correct: use arrays without data
x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
result = Regressogram().fit_transform(x=x, y=y).collect()

Q: Data collection .collect() is very slow or runs out of memory

A: Process data in batches using Polars filtering:

from rgram import Regressogram

rgram = Regressogram()
rgram.fit(data=large_df, x="x", y="y")

# Process in batches instead of collecting everything at once
for batch_df in large_df.partition_by("date"):  # or any grouping column
    results = rgram.transform().filter(
        pl.col("date") == batch_df["date"][0]
    ).collect()
    print(results.head())

Q: Getting different results between runs with the same data

A: Polars operations are deterministic. Different results usually mean:

1. Random seed not set (if you generated synthetic data)
2. Parallelization order differences (set polars.Config.set_streaming_chunk_size())
3. Data changed between runs (verify your input data)

import polars as pl
import numpy as np

# Set reproducible random seed
np.random.seed(42)
pl.Config.set_random_seed(42)

# Your analysis...

Q: Hue grouping not working as expected

A: Ensure the hue column exists and reference it correctly:

from rgram import Regressogram
import polars as pl

# Verify hue column exists
print(df.columns)

# Make sure data is not None when using column names
result = Regressogram().fit_transform(
    data=df,  # Must provide data when using column names
    x="x_col",
    y="y_col",
    hue="group_col"  # This must be a column in df
).collect()

Future Improvements

Completed

• Bandwidth Selection: Silverman, Scott, and manual bandwidth methods for KernelSmoother (implemented in v0.2+)
• Robust Duplicate Handling: Distribution-based binning now handles duplicate x values via qcut(..., allow_duplicates=True) (implemented in v0.2+)

High Priority

• Multivariate Kernel Smoothing: Extend KernelSmoother to support multi-dimensional input with optimal bandwidth selection for each dimension
• Cross-Validated Bandwidth Selection: Automatic bandwidth tuning via leave-one-out or k-fold cross-validation
• Missing Data Handling: Built-in support for various imputation strategies and missing value indicators
• Auto Binning Strategy Selection: Data-driven method to select optimal binning strategy using cross-validation

Medium Priority

• Adaptive Binning: Implement data-driven bin size selection using information-theoretic criteria
• Confidence Band Methods: Additional methods for computing confidence bands (e.g., bootstrap, Bayesian)
• Plotting Utilities: High-level visualisation functions with matplotlib/plotly backends
• Performance Profiling: Detailed benchmarks and optimisation for large-scale datasets

Lower Priority

• Additional Kernels: Support for Gaussian, Triangular, and other kernel types
• Categorical Grouping Optimisation: Special handling for categorical hue variables
• GPU Acceleration: Polars GPU backend support
• Advanced Statistics: Additional statistical methods (local polynomial regression, etc.)

Contributing

Contributions are welcome! To get started:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Install development dependencies: uv sync
4. Make your changes and add tests
5. Run tests: pytest
6. Lint code: ruff check .
7. Commit: git commit -am 'Add your feature'
8. Push: git push origin feature/your-feature
9. Create a Pull Request

Development Workflow with UV

# Install all dependencies including dev
uv sync

# Run tests
uv run pytest

# Run linter
uv run ruff check .

# Run formatter
uv run ruff format .

# Start interactive shell
uv run ipython

References

• García-Portugués, E. (2023). Notes for nonparametric statistics. Carlos III University of Madrid. Available online
• Polars Documentation: https://pola-rs.github.io/
• Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall/CRC: New York.
• Wand, M. P., & Jones, M. C. (1994). Kernel Smoothing. Chapman and Hall/CRC: New York.

License

This project is licensed under the MIT License. See LICENSE for details.

---

Acknowledgments: This library was inspired by nonparametric regression techniques in statistical computing and is built on the excellent Polars data manipulation library.