quantresearch-thd 0.1.33

Ensemble framework for detecting outliers in grouped time-series data

Quantresearch_thd

quantresearch_thd is a production-grade ensemble framework for detecting outliers in grouped time-series data. It automates the entire workflow from data cleaning and calendar interpolation to running 8 different detection algorithms and generating visual diagnostic reports.

Key Capabilities

• Ensemble Scoring: Combines 8 models (Statistical + ML) to provide a robust Anomaly_Score and a final is_Anomaly consensus.
• Hierarchical Processing: Natively handles grouped data (e.g., detecting anomalies per Region, Product, or Channel).
• Automated Preprocessing: Handles missing dates via linear interpolation and filters out "low-quality" unique_ids automatically.
• Parallel Execution: Leverages joblib for multi-core processing of large datasets.
• Visual Analytics: Generates pie charts, stacked bar plots, and detailed group-level time-series breakdowns.

Included Models

The pipeline utilizes an ensemble of the following methodologies:

• Statistical: Percentile, Standard Deviation (SD), Median Absolute Deviation (MAD), and Interquartile Range (IQR).

• Time-Series Specific: EWMA (Exponentially Weighted Moving Average) and FB Prophet (Walk-forward validation).

• Machine Learning: Isolation Forest (General & Time-series optimized) and DBSCAN.

Detailed Functionality

• Decomposition Engine: Uses STL decomposition (Trend + Seasonality) to isolate residuals before modeling.

• Multiplicative Residual Modeling: Statistical models now operate on residual proportions (relative deviation) rather than absolute units, ensuring scale-invariant detection across high and low-volume series.

• Magnitude-Aware Gating: Integrated thresholding to filter out deviations that meet statistical criteria but lack sufficient business volume.

• Robust Input Validation: Clear error messaging for missing parameters or incorrect data types.

• Quality Control: Automatically generates a Success Report

• Visual Suite: Automated rendering of Pie Charts (Summary), Stacked Bars (Distribution), and Top-5 Anomaly Heatmaps.

🚀 Quick Start

!pip install quantresearch_thd
import pandas as pd
from quantresearch_thd import timeseries_anomaly_detection

 # Load your data
df = pd.read_csv("your_data.csv")

 # Run the pipeline
anomaly_df, success_report = timeseries_anomaly_detection( master_data=df,
                                                                             unique_ids=['category', 'region'],
                                                                             variable='sales',
                                                                             date_column='timestamp',
                                                                             freq='W-MON',
                                                                             eval_period=1  # Evaluate the most recent recor
                                                                             )

📊 Visualizing Results & Deep Dives

Inspecting a Specific Group, if a specific group shows a high anomaly rate, use the evaluation_info tool to render detailed diagnostic plots.

from quantresearch_thd import evaluation_info

# Filter the specific group you want to inspect. Define the group values (must match the order in unique_ids)
group_values = ['appliances', 'TX'] 

# Filter the results for this group
mask = anomaly_df[unique_ids].eq(group_values).all(axis=1)
group_df = anomaly_df[mask]

# Generate detailed diagnostic plots
evaluation_info(group_df,
                unique_ids,
                variable,
                date_column,
                eval_period=1
                )

The Evaluation Dashboard provides:

• Model Breakdown: Individual charts for FB Prophet, EWMA, and Isolation Forest with confidence intervals.

• Ensemble View: A summary highlighting where multiple models overlap.

• Statistical Thresholds: Visual markers for IQR, MAD, percentile and SD limits.

Input_data:

Mandatory

• master_data (pd.DataFrame) : Name of your dataframe containing inputs to be evaluated for anomalies include variables, dates, and unique_ids.
• unique_ids (list[str]) : List of columns used to segment data ['SKU', 'channel', "store_id"].
• variable (str) : The numerical target column name to analyze for presence of anomalies.
• date_column (str) : The datetime column representing the time dimension ["date","week","month"].
• freq (str) : Frequency of the date column. Default: 'W-MON'. Accepts 'D', or 'MS'.

Default

• eval_period (int) : Number of trailing records or periods to evaluate for anomalies. Default: 1.
• max_records (int) : Max history to consider starting from the most recent date. Default: all history
• imputation_method : Technique to fill missing time units. Default: 'linear'. Acceptable values are : 'mean', 'mode', 'zero', 'linear'
• percentile_threshold (int): Percentile parameter, Identifies top 2 and bottom 2 percentile as anomaly. Default: 0.02.
• SD_threshold (int) : SD parameter, controls the number of deviation from mean. Default: 3.
• mad_threshold (int) : MAD parameter, controls Median Absolute Deviation sensitivity. Default: 5.
• mad_scale_factor (int) : MAD parameter, The scaling constant used to normalize the MAD. Default: 0.6745.
• IQR_threshold (int) : IQR parameter, controls IQR multiplicative term. Default: 3.
• magnitude_threshold (int) : statistical model parameter, controls the % deviation from base_signal as a gate for statistical anomaly check. Accepts 0 to 1. Default: 0.
• alpha (float) : EWMA parameter, controls the smoothing factor for EWMA trend. Default: 0.3.
• sigma (float) : EWMA parameter, determines the standard deviation multiplier for upper and lower bounds. Default: 2.
• prophet_CI (float) : Prophet parameter, determines the confidence interval. Range 0 to 1, Default: 0.95.
• contamination (float) : Isolation Forest parameter, expected % of outliers (0 to 0.5). Default: 0.05.
• sensitivity (float) : DBSCAN parameter controls the anomaly detection threshold by inversely adjusting the Epsilon radius. Higher sensitivity results in smaller radius. Default: 0.95.
• random_state (int) : Seed for model reproducibility. Default: 42.
• enable_models (list) : selected model list. Default : ['PERCENTILE', 'SD', 'MAD', 'IQR', 'EWMA', 'FB', 'ISF','DBSCAN']

📤 RETURNS

tuple [pd.DataFrame, pd.DataFrame]:

• final_results : The main output, a dataframe that identifies anomalies with Anomaly_Votes and is_Anomaly.
• evaluation_report : Summary of interpolation %, record counts, and anomaly rates.

---

Output columns of final_results : All the output values are at "unique_ids" level.

is_missing_record True if the original data point was missing and imputed; False otherwise.

---

Percentile_low / Percentile_high Dynamic thresholds used to detect unusually low or high values based on the distribution of the residual proportion. These are calculated in the percentage domain and scaled back to raw units for reporting.

---

Percentile_anomaly Categorical flag indicating the direction of a percentile-based outlier: • Low → value < Percentile_low • High → value > Percentile_high • None → within the range

---

Mean The rolling average of the residual proportion (typically over 52 weeks) used to establish the expected baseline of noise. These are calculated in the percentage domain and scaled back to raw units for reporting.

---

SD (Standard Deviation) The rolling standard deviation of the residual proportion, measuring the volatility of the signal relative to the trend.

---

SD_low / SD_high Volatility-based thresholds (Mean ± SD_threshold × SD). These boundaries expand and contract based on the historical variability of the residual proportion. These are calculated in the percentage domain and scaled back to raw units for reporting.

---

SD_anomaly Categorical flag indicating an outlier detected via standard deviation limits: • Low → value < SD_low • High → value > SD_high • None → within the range

---

Median / MAD (Median Absolute Deviation) Median is the Rolling Median of the residual proportion. Mad is the median of the absolute deviation of residual proprtion at each point from median Deviation. Median is scaled back to raw values for reporting.

---

MAD_low / MAD_high Robust statistical thresholds (Median ± mad threhold × MAD/mad_scale_factor) calculated on the residual proportion and projected back into raw units.

---

MAD_anomaly Categorical flag indicating a robust statistical outlier:: • Low → value < MAD_low • High → value > MAD_high • None → within the range

---

Q1 / Q3 / IQR (Interquartile Range)* • Q1/Q3: 25th and 75th percentiles of the residual proportion. • IQR = The middle 50% range (Q3 - Q1). Used to establish boundaries that ignore the extreme tails of the distribution. Implemented on residual proportion over a rolling window of 52 weeks and scaled back to raw values.

---

IQR_low / IQR_high IQR-based limits: • IQR_low = Q1 − 3 × IQR (floored at 0) • IQR_high = Q3 + 3 × IQR Implemented on residual proportion and scaled back to raw values.

---

IQR_anomaly Categorical flag based on IQR fences. • Low → value < IQR_low • High → value > IQR_high • None → within the range

---

is_Percentile_anomaly / is_SD_anomaly / is_MAD_anomaly / is_IQR_anomaly Boolean indicators (True/False) stating whether the specific statistical method classified the point as an anomaly. Note: These may be suppressed if the Consensus Filter determines volume is below the floor magnitude threshold gates.

---

EWMA_forecast Boolean indicators (True/False) stating whether the specific statistical method classified the point as an anomaly. Note: These may be suppressed if the Consensus Filter determines only statistical models voted and volume is below the floor.

---

EWMA_STD The rolling standard deviation of residuals relative to the EWMA forecast, used to calculate dynamic error bands.

---

EWMA_high / EWMA_low The upper and lower detection bands for the EWMA model (EWMA_Forecast ± Sigma × EWMA_STD).

---

Is_EWMA_anomaly Boolean flag indicating the observed raw value fell outside the EWMA dynamic bands.

---

FB_forecast The expected value generated by the FB Prophet model, incorporating trend, seasonality, and holiday effects.

---

FB_low / FB_high The lower and upper bounds of the Prophet uncertainty interval (Credible Interval)

---

FB_residual The raw difference between the actual observed value and the FB_forecast.

---

FB_anomaly Categorical flag (low/high) based on whether the observation exceeded Prophet’s forecast intervals.

---

Is_FB_anomaly Boolean flag indicating a Prophet-detected anomaly.

---

isolation_forest_score Score from the Isolation Forest model indicating anomaly severity. Typical range: –0.5 to +0.5

• Higher/Positive → More normal.
• Lower/Negative → More anomalous (isolated early in the tree).

---

is_IsoForest_anomaly Boolean flag based on Isolation Forest model output:

• True → model predicts anomaly (prediction = –1)
• False → model predicts normal (prediction = 1)

---

dbscan_score The cluster assignment or density score from the DBSCAN algorithm (-1 indicates the point is "noise" or an outlier).

---

is_DBSCAN_anomaly Boolean flag indicating DBSCAN-detected anomaly.

---

Anomaly_Votes The count of individual models (out of 8) that flagged the point as an anomaly. Ranges from 0 to 8.

---

Vote_Cnt The total number of models enabled and executed for that specific group. Ranges from 0 to 8.

---

is_Anomaly Set to True if the majority of models (typically >= 50%) agree. Automatically flipped to False if a Directional Conflict exists or if the Consensus Filter determines the detection is purely statistical noise at a low volume.

---

Anomaly_Votes_Display A string representation of the ensemble consensus. It displays the number of models that flagged an anomaly against the total number of models enabled (e.g., "3 out of 8").

---

is_conflict A boolean flag indicating a directional disagreement within the ensemble. It is marked True if at least one model flagged a "high" anomaly while another model simultaneously flagged a "low" anomaly for the same data point.

---

consensus_type Indicates the prevailing direction of the anomaly based on the net sum of model votes (High = +1, Low = -1).

• high: More models flagged a positive deviation than a negative one.
• low: More models flagged a negative deviation than a positive one.
• none: No clear directional majority or no anomalies detected.

---

max_threshold The upper boundary used for data cleansing. If a point is a confirmed "high" anomaly, this value represents the most lenient (highest) upper bound among all voting models; otherwise, it defaults to the raw variable value.

---

min_threshold The lower boundary used for data cleansing. If a point is a confirmed "low" anomaly, this value represents the most lenient (lowest) lower bound among all voting models; otherwise, it defaults to the raw variable value.

---

raw_total_views_cleansed The final "corrected" version of the input variable. Anomalous values are clipped to the min_threshold or max_threshold to neutralize the outlier while preserving the underlying trend and seasonality.

---

[Model]_score_scaled (Applies to Percentile, SD, MAD, IQR, EWMA, FB, IsolationForest, DBSCAN) The individual anomaly score from each specific model, normalized to a standard 0–100 scale.

• 0: Perfectly normal according to that specific model.
• 100: Maximum anomalousness according to that specific model.

---

Anomaly_Score The final aggregated ensemble score (0–100). This is typically a weighted average of all enabled [Model]_score_scaled values, representing the overall confidence that a point is anomalous.

---

is_Anomaly The final ensemble decision based on a majority rule.

• True: Flagged as anomalous by 50% or more of the enabled models (and passing the directional conflict check).
• False: Fewer than 50% of models agree, or the models are in directional conflict.

---

anom_score_cat A categorical grouping of the Anomaly_Score used for simplified reporting (e.g., "Normal", "Suspect", "High Anomaly").