idkmetrics 0.1.0

Plain-English explanations for ML metrics.

idkmetrics

For when the model works... but you still want a human sentence.

idkmetrics is a tiny Python library that translates ML metrics into plain English for people who do not want to decode a wall of acronyms before coffee.

It is made for:

• new ML folks
• PMs and founders trying to understand model results
• engineers who know the metric name but want the "so what?"
• anyone who has ever stared at RMSE = 12.7 and thought "cool, I guess?"

What it does

You pass in one metric or a bundle of metrics.

It gives you:

• what the metric means
• whether higher or lower is better
• a plain-English read on the number
• warnings when the metric is easy to misuse
• optional next-step suggestions

Install

pip install -e .

Quick start

from idkmetrics import explain

print(
    explain(
        {"mae": 4.3, "rmse": 6.1, "r2": 0.81},
        task="regression",
        include_next_steps=True,
    )
)

Example output:

Metric vibes, one by one:
- Quick read: MAE = 4.3. This is in plain English; This one is mostly useful for relative comparison.
  MAE tells you average absolute error between predictions and the real value. For this run, a lower value of 4.3 looks context-dependent. Its meaning depends a lot on the scale of your target, so compare it with typical real-world error.
  Watch out: MAE uses the same units as the target.
  Next: Compare against a baseline like predicting the mean or last known value.
  Next: Plot prediction error so you can see whether the misses are concentrated on certain ranges.
...

Main API

The main wrapper is explain(...).

explain(
    metrics=None,
    *,
    task=None,
    metric=None,
    value=None,
    audience="beginner",
    tone="casual",
    detail="standard",
    include_scale_tips=True,
    include_warnings=True,
    include_next_steps=False,
    thresholds=None,
    aliases=None,
    strict=False,
    sort_metrics=True,
    return_format="text",
)

Useful parameters

• task: hint the model family, like "regression", "classification", "forecasting", "clustering", "ranking", or "generative"
• metric + value: convenient single-metric mode
• audience: tune the wording for "beginner", "product", "data_science", or "exec"
• tone: choose "casual", "neutral", "hype", or "blunt"
• detail: choose "short", "standard", or "detailed"
• include_scale_tips: adds reminders when the score depends heavily on target scale
• include_warnings: includes caveats like class imbalance traps and baseline gotchas
• include_next_steps: adds practical suggestions for what to inspect next
• thresholds: override built-in interpretation bands for your use case
• aliases: teach the wrapper extra metric spellings
• strict=True: raise an error instead of quietly handling unknown metrics
• return_format: "text", "dict", or "list"

Supported model types and metrics

This library is intentionally broad enough for the common "what does this even mean?" situations.

Regression

Good for:

• house price prediction
• demand estimation
• churn value prediction
• any "predict a number" task

Built-in metrics:

• mae
• mse
• rmse
• r2
• medae

Example:

from idkmetrics import explain

print(explain({"mae": 18.2, "rmse": 31.7, "r2": 0.72}, task="regression"))

Classification

Good for:

• spam detection
• fraud detection
• disease prediction
• sentiment classification

Built-in metrics:

• accuracy
• balanced_accuracy
• precision
• recall
• specificity
• f1
• roc_auc
• pr_auc
• log_loss

Example:

print(
    explain(
        {
            "accuracy": 0.94,
            "precision": 0.71,
            "recall": 0.43,
            "f1": 0.54,
            "roc_auc": 0.91,
        },
        task="classification",
        include_warnings=True,
    )
)

That kind of output is useful when a model looks "accurate" overall but still misses a lot of the positive cases.

Forecasting / Time Series

Good for:

• sales forecasting
• traffic forecasting
• inventory planning
• energy usage prediction

Built-in metrics:

• mape
• smape
• wape
• mase

Example:

print(explain({"mape": 0.18, "smape": 0.14, "mase": 0.82}, task="forecasting"))

Clustering

Good for:

• customer segmentation
• behavior grouping
• topic clustering
• anomaly exploration

Built-in metrics:

• silhouette
• davies_bouldin
• calinski_harabasz
• inertia

Example:

print(explain({"silhouette": 0.48, "davies_bouldin": 0.91}, task="clustering"))

Ranking / Retrieval

Good for:

• search ranking
• recommendation systems
• retrieval pipelines
• "what should show up first?" systems

Built-in metrics:

• ndcg
• map
• mrr

Example:

print(explain({"ndcg": 0.79, "mrr": 0.68}, task="ranking"))

Generative / NLP-ish overlap metrics

Good for:

• summarization evaluation
• translation experiments
• text generation comparisons
• rough reference-based NLP scoring

Built-in metrics:

• bleu
• rouge
• rouge_l
• bertscore
• perplexity

Example:

print(explain({"rouge_l": 0.44, "bertscore": 0.92}, task="generative"))

Single metric mode

print(explain(metric="rmse", value=12.6, task="regression"))
print(explain(metric="f1", value=0.87, task="classification", tone="hype"))
print(explain(metric="silhouette", value=0.21, task="clustering", tone="blunt"))

Return structured data instead of text

result = explain(
    {"accuracy": 0.91, "f1": 0.77},
    task="classification",
    return_format="dict",
)

print(result["items"][0]["summary"])

This is handy if you want to:

• display explanations inside a web app
• build your own UI on top
• send metric summaries to Slack, Discord, or logs

Add custom thresholds

Different industries have different definitions of "good".

print(
    explain(
        {"recall": 0.84},
        task="classification",
        thresholds={
            "recall": [
                (0.70, "too many positives are still slipping through"),
                (0.85, "pretty solid catch rate"),
                (0.95, "excellent catch rate"),
            ]
        },
    )
)

Aliases and robustness

The wrapper already normalizes a lot of common naming variants:

• r_squared
• r2_score
• mean_absolute_error
• roc auc
• average_precision
• rouge-l
• and more

You can add your own too:

print(
    explain(
        {"my_team_auc_name": 0.88},
        aliases={"my_team_auc_name": "roc_auc"},
    )
)

If you want it to fail loudly on unknown metrics:

explain({"mystery_score": 42}, strict=True)

Why this exists

A lot of metric libraries are technically correct and emotionally unhelpful.

idkmetrics is for the moment after training, when someone asks:

• "is 0.71 good?"
• "why is accuracy high but recall bad?"
• "does RMSE being 20 mean the model is broken?"
• "what does silhouette score even look like in human words?"

Tiny roadmap ideas

• confidence interval and variance-aware explanations
• confusion-matrix-aware narrative summaries
• baseline comparison helpers
• prettier formatting for notebooks and dashboards

License

MIT