Comparative Edge

Discover where outcomes behave reliably. Establish what your model can realistically expect. Compare that expectation with the outside world.

Comparative Edge is a lightweight Python toolkit for discovering validated performance ranges from historical binary outcomes.

It is designed for situations where you have:

• a measurable value or condition,
• a binary outcome represented by 0 or 1,
• historical observations,
• and an external expectation, benchmark, target, or opportunity that you eventually want to compare against your own model.

The central philosophy:

Model first. Comparison second. Decision last.

Comparative Edge does not allow an external expectation to define what your model should believe. It first uses your own historical data to discover and validate behavioral ranges. Only after an expectation has been established should it be compared with an external reference.

---

The Problem

Many decisions involve two sources of information: what your own data says should happen, and what the outside world says should happen.

The outside expectation may be attractive. It may also be wrong for the conditions you're dealing with. The danger is letting that external expectation influence the model itself.

For example, imagine a company running paid advertising. Every day it records:

ad_spend    outcome
───────────────────
50             0
75             1
100            1
125            0
150            1
175            1
200            0

Here 1 means the desired outcome occurred and 0 means it didn't. The business wants to know whether the amount it spends is associated with a reliable outcome.

Comparative Edge can search the historical values and discover that certain spending regions behave differently:

ad spend range      observed outcome
────────────────────────────────────
<= $80                    48%
$80 – $140                61%
$140 – $200               54%
>= $200                   43%

The important discovery isn't simply that one range has the highest percentage. The question is:

Which behavior is sufficiently consistent across historical periods to establish a realistic expectation?

Once that expectation is established, it can be compared with an external one. Suppose an external campaign target assumes:

Expected outcome = 75%

while your validated historical expectation is:

Your model = 61%

Comparative Edge exposes the disagreement:

Your model              External expectation
    61%                         75%
     │                           │
     └───────────┬───────────────┘
                 ↓
             -14% gap

The package doesn't tell the business what to do. It makes the risk/reward difference visible before commitment.

---

Core Philosophy

1. Let your data establish the expectation

The model should establish its own expectation from historical evidence. An external benchmark should not decide what the model ought to produce.

Historical data → Model behavior → Validated expectation

2. Validate the behavior

A range that looks good across the entire dataset may not actually be reliable. Comparative Edge tests candidate ranges across multiple validation folds and eliminates ranges whose behavior falls outside the required performance band.

Candidate range
      ↓
Fold 1 → Pass
      ↓
Fold 2 → Pass
      ↓
Fold 3 → Pass
      ↓
Fold 4 → Pass
      ↓
Fold 5 → Pass
      ↓
Validated range

The objective isn't an impressive historical result. It's repeatable behavior.

3. Compare only after establishing your expectation

Your model → Your expectation → COMPARE ← External expectation

This preserves the independence of the model.

4. Use the difference to understand risk and reward

The comparison doesn't automatically determine whether an opportunity is good or bad. It tells you how far the external expectation is from what your own evidence supports. That difference can be used as a decision boundary.

---

Binary Outcomes

1 = desired outcome occurred
0 = desired outcome did not occur

Applicable to many problems: sale/no sale, conversion/no conversion, success/failure, approved/rejected, retained/churned, delivery/no delivery, event/no event.

The other column represents the value or condition whose relationship with the outcome you want to investigate, e.g. customer_score → conversion, ad_spend → sale, processing_time → success, temperature → failure.

---

What the Algorithm Does

DataFrame
   ↓
Select value + outcome
   ↓
Generate percentile edges
   ↓
Create candidate boundaries
   ↓
Test candidates on validation folds
   ↓
Remove weak or unstable candidates
   ↓
Score surviving candidates
   ↓
Return the winning validated boundary

---

Candidate Directions

Comparative Edge doesn't assume higher values are better. It can investigate three shapes of behavior.

Left

Find behavior below a threshold: value <= edge. Example: ad_spend <= $100.

Right

Find behavior above a threshold: value >= edge. Example: customer_score >= 75.

Range

Find behavior between two boundaries: low <= value <= high. Example: $80 <= ad_spend <= $140.

All

Test all three shapes together and let them compete within the same candidate pool. Useful when you don't want to assume the relationship beforehand; the data determines whether the useful behavior occurs below a threshold, above a threshold, or inside a range.

---

Percentile-Based Candidate Generation

Rather than testing every possible numerical value, Comparative Edge creates candidate boundaries from percentile points, e.g. 0% - 7% - 14% - 21% - ... - 100%. These percentile boundaries are converted into actual dataset values, and candidate ranges are constructed from those values. Duplicate percentile values are removed automatically.

---

Validation

Suppose n_folds = 5. The dataset is divided into five sequential folds. Each candidate must satisfy the configured performance requirements in every fold it's evaluated in; candidates that fail are eliminated.

                    Candidates
                        │
          ┌─────────────┼─────────────┐
          ↓             ↓             ↓
        Fold 1        Fold 1        Fold 1
          │             │             │
         Pass          Fail          Pass
          │             X             │
        Fold 2                       Fold 2
          │                            │
         Pass                         Pass
          │                            │
        Fold 3                       Fold 3
          │                            │
         Pass                         Fail
          │                            X
        Survives

This makes the resulting expectation less dependent on a single portion of the dataset.

---

Auto Fold Count

n_folds=None

is the default. When n_folds is left as None, the fold count is chosen automatically from the number of rows remaining after dropping missing values:

rows	n_folds
< 50	1
50 – 149	2
150 – 349	3
350 – 699	4
≥ 700	5

Pass an explicit int (1 to 9) instead to override auto selection. In display=True mode, the header shows which one happened, e.g. 2-fold (auto: 100 rows) or 3-fold (manual).

---

Target Performance

target_pct=(50, 60)

sets the acceptable performance band directly as a (lo, hi) tuple — here, 50% to 60%. A candidate must fall inside this band during validation to remain a survivor. For a surviving candidate, the reported difference is always measured against the low end of the band (target_pct[0]):

observed = 57%, target = (50, 60)  ->  difference = +7%
observed = 46%, target = (50, 60)  ->  difference = -4%

The sign preserves direction: negative means below the floor, zero means matches the floor, positive means above it.

target_pct is required — there's no default. It must be a two-element tuple/list of numbers, both between 0 and 100, with lo <= hi.

---

Choosing Which Survivor Wins

winner="lowest"   # or "highest"

is also required. Once fold elimination is finished, more than one candidate can still be left standing. winner decides which of those survivors becomes the final result:

• "lowest" — pick the survivor whose overall hit rate sits closest to the low end of the band (target_pct[0]). This favors the more conservative, floor-hugging candidate.
• "highest" — pick the survivor whose overall hit rate sits closest to the high end of the band (target_pct[1]). This favors the strongest-performing candidate that's still validated.

The vs_target value shown in the results table and winner box always measures distance above the floor (target_pct[0]) regardless of which winner mode is used — winner only changes which row is selected, not how that row's difference is displayed.

---

Minimum Sample Protection

min_count=20

A candidate with fewer than the required observations in a validation fold is eliminated, preventing small samples from becoming apparently strong ranges.

---

Installation

pip install comparative-edge

---

Basic Usage

The DataFrame should contain exactly one binary outcome column and one numeric value column. min_count requires that many rows to fall inside a candidate range within every fold, so make sure your dataset is large enough for your chosen n_folds and min_count (roughly n_folds * min_count rows at minimum):

import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 500
ad_spend = rng.uniform(50, 300, n).round(2)
# outcome is more likely when ad_spend falls between 100 and 200
prob = np.where((ad_spend > 100) & (ad_spend < 200), 0.58, 0.30)
outcome = (rng.random(n) < prob).astype(int)

df = pd.DataFrame({"ad_spend": ad_spend, "outcome": outcome})

from comparative_edge import search

result = search(
    df,
    outcome_col="outcome",
    target_pct=(50, 60),
    winner="lowest",
    range_bins=15,
    n_folds=5,
    min_count=20,
)
# result -> (126.63, 231.62)

---

Direction

Default is direction="all". You can restrict the search:

search(df, outcome_col="outcome", target_pct=(50, 60), winner="lowest", direction="left")   # below a threshold
search(df, outcome_col="outcome", target_pct=(50, 60), winner="lowest", direction="right")  # above a threshold
search(df, outcome_col="outcome", target_pct=(50, 60), winner="lowest", direction="range")  # bounded ranges only
search(df, outcome_col="outcome", target_pct=(50, 60), winner="lowest", direction="all")    # everything

---

Comparing Against a Mark

mark=8

is an optional value to compare against the winning edge once it's found. The comparison direction follows the shape of the winning edge:

• left edge (value <= edge) → pass if mark <= edge
• right edge (value >= edge) → pass if mark >= edge
• range edge (lo <= value <= hi) → pass if lo <= mark <= hi

When mark is supplied, the return shape changes from a plain (lo, hi) tuple to a dict:

result = search(
    df,
    outcome_col="outcome",
    target_pct=(50, 60),
    winner="lowest",
    direction="all",
    mark=150,
)
# result -> {"status": True, "edge": (126.63, 231.62)}

Leave mark as None (the default) to keep the original (lo, hi) tuple return. In display=True mode, the mark check prints its own line at the end of the output, with 🟢 for a pass and 🔴 for a fail.

---

Parameters

Parameter	Default	Description
df	n/a	Input DataFrame: exactly one value column plus outcome_col
outcome_col	n/a	Binary 0/1 outcome column
target_pct	n/a	Required. (lo, hi) tuple defining the acceptable performance band, both values 0-100, lo <= hi
winner	n/a	Required. "lowest" or "highest" — which surviving candidate to pick (see Choosing Which Survivor Wins)
range_bins	15	Number of percentile boundaries used to build candidates (>= 2)
n_folds	None	Number of validation folds, 1 to 9. None auto-picks 1-5 from the row count (see Auto Fold Count)
min_count	20	Minimum observations required per candidate per fold (>= 1)
direction	"all"	"left", "right", "range", or "all"
mark	None	Optional value compared against the winning edge (see Comparing Against a Mark)
display	True	Print the analysis output
verbose	False	Print detailed fold-level candidate keep/drop information

Invalid values for any parameter raise ValueError or TypeError up front, before any search runs. This includes a missing target_pct or winner, a target_pct tuple with lo > hi or values outside 0-100, and a winner value other than "lowest"/"highest".

---

Display Mode

display=True

Example output:

Comparative Edge
ad_spend vs outcome

  target       50-60%
  winner       lowest
  direction    all
  range_bins   15
  n_folds      5-fold (manual)
  min_count    20

Results   (total: 10)
  Range                General_pct   Vs_target Coverage_pct  Coverage_n
  <=80.000                   51.20        1.20        20.00           2
  (80.000, 140.000)          58.43        8.43        40.00           4
  (140.000, 200.000)         54.91        4.91        30.00           3
  >=200.000                  47.10       -2.90        10.00           1

Winner   ad_spend (80.000, 140.000)
hit_pct = 58.43%  ·  target 50-60%  ·  +8.43 over floor  ·  coverage 40.0% (4 / 10)

Coverage_pct and Coverage_n show what share (and raw count) of the full dataset each surviving range covers, out of the total printed above the table. The winner line always shows a signed vs_target value measured from the floor of the band: +8.43 when the range beats the target floor, -2.90 if it ends up below it — regardless of whether winner="lowest" or winner="highest" was used to select that row.

When n_folds is left as None, the same line instead reads something like n_folds 2-fold (auto: 100 rows), showing how the fold count was chosen.

When mark is supplied, a Mark check line prints after the winner box, e.g. Mark check mark=150 vs edge (126.630, 231.620) -> 🟢 PASS.

verbose=True

also shows how candidates were kept or eliminated in each fold.

---

Understanding the Result

If search returns (80.0, 140.0), the winning validated region is approximately 80 <= value <= 140:

Range:       80 - 140
Hit rate:    58.43%
Target:      50-60%
Difference:  +8.43% (vs floor)
Coverage:    40.0% of all observations (4 / 10)

This region demonstrated an observed outcome rate inside the target band while surviving the configured validation process, along with how much of your data that region represents.

If no candidate survives validation, search returns None. Unbounded sides come back as None rather than -inf/inf: a left-direction winner looks like (None, 45.2), a right-direction winner like (12.0, None).

---

Comparative Edge Is Not a Prediction Engine

It is a validation and comparison layer, not a replacement for your predictive model. Its role is to help answer:

Where does my model demonstrate reliable behavior, and how does that expectation compare with an external reference?

It does not determine whether an external benchmark is correct, whether an opportunity should be accepted, whether a model is causally correct, or whether historical behavior will continue indefinitely. It provides evidence for those decisions.

---

Model First, External Expectation Second

Consider a model that expects 60% performance, and an external source claiming 75%: a 15-point disagreement. Comparative Edge doesn't respond by changing the model to 75%. Instead, the difference becomes information. Maybe the external expectation is optimistic. Maybe the model is missing a variable. Maybe the historical sample doesn't represent current conditions. Maybe the opportunity genuinely reflects a change in conditions. The package doesn't assume which explanation is correct; it makes the disagreement visible so it can be investigated.

---

Risk / Reward

The comparison becomes:

What my evidence supports   vs   What the opportunity requires

A large disagreement can indicate a poor fit between the opportunity and the model's demonstrated behavior. A close agreement can indicate the opportunity is more consistent with the model's expectation. This doesn't automatically make an opportunity good or bad; it provides a risk/reward reference point for the decision-maker.

---

Applications

Comparative Edge is industry-neutral. The same framework applies anywhere an outcome can be 0 or 1:

• Advertising: ad_spend → conversion / no conversion - does spend correspond with expected outcome?
• Sales: lead_value → sale / no sale - which value ranges consistently produce a sale?
• Operations: processing_time → success / failure - which operating ranges see more consistent success?
• Customer Analytics: customer_score → retention / churn - which score ranges show reliable retention?
• Quality Control: measurement → pass / fail - which measurement ranges correspond with acceptable outcomes?
• Forecasting: forecast_value → event / no event - where does the forecast show consistent historical behavior?

The domain changes; the underlying question stays the same:

What does my own evidence realistically support, and how does that compare with what is expected externally?

---

Limitations

The current design expects exactly one numeric value column besides the outcome column, a binary 0/1 outcome, sufficient observations for validation, and numerical values that can be ordered into meaningful boundaries.

The percentile edges used to build candidates are computed from the full dataset before folding begins, and the winner's reported hit rate, vs_target, and coverage are also scored against the full dataset. Fold elimination genuinely filters out unstable candidates, but the final displayed numbers for the winner are in-sample, not held-out: treat them as a validated screen, not an out-of-sample performance guarantee.

Validation folds are contiguous, sequential slices of the DataFrame in its existing row order, not shuffled or randomly sampled. If your data is ordered by date or by the value column itself, folds may not be independent; shuffle beforehand if that matters for your use case.

It does not attempt to solve causal inference, multivariate feature interactions, time-series forecasting, probability calibration, model training, concept drift detection, or statistical significance testing. Those problems can complement Comparative Edge, but they're outside its core purpose.

---

The Philosophy Behind the Package

Comparative Edge was created around a practical problem: how do you know whether an opportunity's expected reward is consistent with what your own model says is realistically achievable?

The answer shouldn't begin with the opportunity. It should begin with your evidence. First establish what your data supports. Then validate whether that behavior is repeatable. Then compare it with the outside expectation. Only then decide whether the difference represents an opportunity, a warning, or a reason to investigate further.

        YOUR DATA
            ↓
       YOUR MODEL
            ↓
   VALIDATED BEHAVIOR
            ↓
     YOUR EXPECTATION
            │
        COMPARISON
            │
            ↑
 EXTERNAL EXPECTATION
            ↓
    RISK / REWARD VIEW
            ↓
         DECISION

Comparative Edge isn't about making the model agree with the world. It's about finding out where the model agrees with the world, where it disagrees, and how large that difference is.

Model first. Opportunity second. Comparison before commitment.