commcp 1.0.1

Conformal Model Moderation & Human-in-the-Loop Routing Python Library

commCP 🛡️

commCP (Conformal Model Moderation & Human-in-the-Loop Routing) is a post-training wrapper for binary classification estimators. It combines Conformal Prediction (to enforce statistical reliability guarantees) and LLM Refereeing to decide when a prediction can be auto-accepted vs. when it should be escalated for human review.

Inspired by stats-centric tools like MAPIE, commCP bridges the gap between statistical guarantees and LLM verification for the AI era.

---

Features

• Statistical Coverage Guarantees: Enforces target error rates ($1 - \alpha$) via conformal calibration.
• Selective Prediction / HITL: Automatically routes predictions into auto_decided or escalated queues.
• LLM-as-a-Referee: Mediates ensemble disagreements and conformal "gray-zone" uncertainties dynamically.
• Cost-Optimized: Bypasses the LLM completely for obvious acceptances or low-confidence/high-risk rejections, keeping API costs to a minimum.
• Seamless sklearn Compatibility: Works with any estimator exposing a predict_proba method (e.g., LogisticRegression, RandomForest, XGBoost).

---

Installation

# Install from source (or PyPI once published)
pip install .

---

How to Prepare Your Data

Conformal prediction requires a held-out calibration set that the model was not exposed to during training. Before wrapping your model, split your dataset into three distinct partitions:

1. Training Set (e.g., 60% of data): Used to train your base classifier (e.g. RandomForest).
2. Calibration Set (e.g., 20% of data): Used by commCP to calculate safety thresholds. Crucial: Calibrating on the training set violates statistical guarantees.
3. Test Set (e.g., 20% of data): Used for incoming predictions and routing.

---

Quick Start Guide

1. Train Your Classifier

from sklearn.ensemble import RandomForestClassifier
from commcp import CommCP

# Train a standard sklearn classifier
model = RandomForestClassifier()
model.fit(X_train, y_train)

2. Wrap and Calibrate commCP

Initialize CommCP with your trained estimator, along with a task description and class labels (which are required to build high-accuracy prompts for the LLM Referee). Pass a held-out calibration set to establish the conformal threshold.

# Initialize commcp wrapper (configured for significance level alpha=0.05 -> 95% coverage)
ccp = CommCP(
    estimator=model,
    task_description="Predict whether a patient has heart disease based on clinical features",
    class_labels={0: "Healthy", 1: "Heart Disease Present"},
    alpha=0.05,
    llm_provider="groq", # Supports "groq" or "openai"
    verify_margin=0.15   # Trigger LLM verification on predictions within 15% of the threshold
)

# Calibrate
ccp.calibrate(X_calib, y_calib)

3. Predict & Moderate

Predict outcomes for test data. CommCP will execute conformal gating, query the LLM referee on borderline cases, and partition predictions.

# Run predictions
results = ccp.predict(
    X_test, 
    text_dossiers=text_descriptions # Optional natural language dossiers for LLM inspection
)

# Get automation and routing results
print(f"Automation rate: {results.automation_rate:.2%}")

# Access lists of auto-decided and escalated records
auto_cases = results.auto_decided  # list of dicts
human_queue = results.escalated    # list of dicts

Understanding the Results Structure

Each record inside results.auto_decided and results.escalated is a dictionary with the following schema:

• sample_index (int): The index of the sample in the test dataset.
• model_prediction (int): The raw output prediction of your base classifier.
• confidence (float): The probability score assigned to the predicted class by the model.
• route (str): The final decision route. It will be:
  • "ACCEPT": Auto-accepted directly by Conformal Prediction (high confidence).
  • "LLM_VERIFIED": Evaluated by the LLM Referee (due to borderline confidence or ensemble disagreement) and the LLM agreed with the classifier.
  • "HUMAN_REVIEW": Escalated to a human reviewer (due to low confidence or LLM disagreement).
• route_reason (str): A detailed description explaining why this route was selected.
• llm_prediction (int or None): The decision made by the LLM Referee (0 or 1 if called, otherwise None).
• llm_reasoning (str or None): The short reasoning sentence written by the LLM Referee (if called).
• final_prediction (int or None): The final automated prediction value if automated. If escalated to a human, this is None (requiring human resolution).

How Statistics are Calculated

When you call results.stats(y_true=y_test), the library calculates the following metrics under the hood:

1. Automation Rate:

   Automation Rate = (ACCEPT Cases + LLM_VERIFIED Cases) / Total Samples
   

2. Empirical Conformal Coverage: The accuracy of the automated decisions against the true labels. Under conformal prediction, this is mathematically guaranteed to be >= 1 - alpha.
3. Human-in-the-Loop (HITL) System Accuracy:

   HITL System Accuracy = (Correct Automated Predictions + Total Human Escalated Cases) / Total Samples
   

   (Note: This calculation assumes the human reviewer acts as a ground-truth oracle and corrects any escalated case to the right label).

---

4. Evaluate Guarantees

Verify if your target mathematical coverage guarantee was met:

empirical_coverage = results.coverage(y_test)
print(f"Empirical Coverage: {empirical_coverage:.2%}") # Should be >= 95%

Examine system performance details:

print(results.stats(y_test))

---

Customizing Gating Logic

CommCP dynamically adjusts its gating based on your model architecture:

• Single Models: Uses Gray-Zone Gating. Calls the LLM referee only when confidence is close but below the conformal cutoff.
• Ensembles: Uses Disagreement Gating. Automatically inspects ensemble consensus and calls the LLM to referee conflicting model predictions.

---

License

Licensed under the MIT License.