dot-metrics 0.4.0

A lightweight metrics and constraint evaluation framework

dot-metrics

dot-metrics is a lightweight metrics and constraint evaluation framework. Define metrics and constraints, run them against your data, and get structured results with debug info. Visualize results with terminal bar charts or an interactive browser-based explorer with scatter, bar, and heatmap charts.

Install

pip install dot-metrics

Concept

A MetricSet holds metric and constraint definitions. Call compute(data) to evaluate them.

MetricSet
 ├── metrics:     {"coverage": MetricDefinition}
 └── constraints: {"errors":   ConstraintDefinition}
          │
          ▼
     set.compute(data)
          │
          ▼
      EvalResult
       ├── metrics:     {"coverage": Metric}
       └── constraints: {"errors":   Constraint}

Metrics are continuous measurements (e.g. coverage rate, score). Constraints are pass/fail checks against a threshold (e.g. error count ≤ 0).

Quick start

from dot_metrics import MetricSet

metric_set = MetricSet()

@metric_set.metric("coverage")
def coverage(data):
    return data["covered"] / data["total"]

@metric_set.constraint("errors", threshold=0)
def errors(data):
    return data["error_count"]

result = metric_set.compute({"covered": 90, "total": 100, "error_count": 0})

result.score("coverage")     # 0.9
result.constraints_ok        # True

Defining metrics and constraints

Decorator style

metric_set = MetricSet()

@metric_set.metric("latency_ms", unit="ms", higher_is_better=False)
def latency(data):
    return data["total_ms"] / data["requests"]

@metric_set.constraint("error_rate", threshold=0.01, unit="%")
def error_rate(data):
    return data["errors"] / data["requests"]

Imperative style

metric_set = MetricSet()
metric_set.add("coverage", lambda data: data["covered"] / data["total"])
metric_set.add_constraint("errors", lambda data: data["error_count"], threshold=0)

Both styles accept the same keyword arguments.

Parameters

Metrics — metric_set.add(key, fn, *, ...) / @metric_set.metric(key, *, ...)

Parameter	Default	Description
name	""	Human-readable display label
unit	""	Unit of measurement
description	""	Free-text description
higher_is_better	True	Affects terminal chart rendering
normalize	None	Callable[[float], float] — maps raw value to [0, 1] for chart display
metadata	{}	Arbitrary data, passed through to results

Constraints — metric_set.add_constraint(key, fn, *, ...) / @metric_set.constraint(key, *, ...)

Parameter	Default	Description
threshold	required	Pass/fail boundary
name	""	Human-readable display label
unit	""	Unit of measurement
description	""	Free-text description
higher_is_better	False	False: passes when value <= threshold. True: passes when value >= threshold
metadata	{}	Arbitrary data, passed through to results

Computing results

result = metric_set.compute(data)

Every metric and constraint function must accept exactly one argument — the data object. data can be anything: a dict, dataclass, Pydantic model, etc.

Accessing results

result.score("coverage")                  # float — metric value
result.score("errors")                    # also works for constraints
result.metrics["coverage"].value          # same as score("coverage")
result.metrics["coverage"].name           # "" by default
result.metrics["coverage"].unit           # ""
result.metrics["coverage"].debug          # {} by default

result.constraints["errors"].passed       # True/False
result.constraints["errors"].threshold    # 0

result.constraints_ok                     # True if all constraints passed
result.violations                         # list of failed Constraint objects
result.assert_constraints()               # raises ValueError if any failed

# Iteration
len(result)                               # total number of metrics + constraints
result["coverage"]                        # returns Metric or Constraint by key
for key in result:                        # iterates over all keys (metrics first)
    print(key, result[key].value)

Attaching debug info

Return a ComputedValue instead of a plain float to attach structured debug data:

from dot_metrics import MetricSet, ComputedValue

metric_set = MetricSet()

@metric_set.metric("coverage")
def coverage(data):
    missed = [x for x in data if not x["covered"]]
    return ComputedValue(value=1 - len(missed) / len(data), debug={"missed": missed})

result = metric_set.compute(data)
result.metrics["coverage"].debug    # {"missed": [...]}

ComputedValue works the same way for constraints.

Batch evaluation

Evaluate a set of inputs in one call:

# dict of inputs
batch = metric_set.compute_batch({"run_1": data1, "run_2": data2})
batch["run_1"].score("coverage")            # 0.9
batch.scores("coverage")                   # {"run_1": 0.9, "run_2": 0.85}

# list of inputs
batch = metric_set.compute_batch([data1, data2, data3])
batch[0].score("coverage")                 # indexed by position

BatchResult supports iteration, len(), and .items().

Metric documentation

Add a Google-style docstring to auto-populate a help dict on both the definition and the result:

@metric_set.metric("coverage", unit="%")
def coverage(data):
    """Percentage of code paths covered by tests.

    Range: 0-100
    Interpretation:
        - 90-100: Excellent
        - 70-90:  Good
        - <70:    Needs improvement
    Notes:
        - Returns 0 for empty input.
    """
    return sum(1 for x in data if x["covered"]) / len(data)

result = metric_set.compute(data)
result.metrics["coverage"].help
# {"summary": "Percentage of code paths covered by tests.",
#  "range": "0-100",
#  "interpretation": "- 90-100: Excellent\n- 70-90:  Good\n...",
#  "notes": "- Returns 0 for empty input."}

Supported sections: Range:, Interpretation:, Notes:. No docstring produces help = {}.

Typing

MetricSet is generic over the input type T and an optional metadata type M:

from dataclasses import dataclass
from typing import TypedDict
from dot_metrics import MetricSet

@dataclass
class SchedulingData:
    appointments: list
    solution: list

# Type-checked input
metric_set: MetricSet[SchedulingData] = MetricSet()
metric_set.add("rate", lambda d: len(d.solution) / len(d.appointments))

# Type-checked metadata
class MyMeta(TypedDict):
    category: str
    priority: int

ms: MetricSet[dict, MyMeta] = MetricSet()
ms.add("score", lambda d: 1.0, metadata=MyMeta(category="perf", priority=1))

Both type parameters are optional — omitting them is fine and everything still works at runtime.

Terminal chart

from dot_metrics import draw_terminal_chart

print(draw_terminal_chart(result))
# coverage  ████████████████████  90%

draw_terminal_chart(result, width=40, char="█") renders a Unicode bar chart from an EvalResult.

By default, metric values are assumed to be in [0, 1]. For metrics outside this range, provide a normalize function when registering the metric to map the raw value into [0, 1] for display:

metric_set.add("wait_days", compute_wait, normalize=lambda v: 1 / (v + 1))

Interactive explorer

Explore batch results interactively in the browser:

from dot_metrics import serve

batch = metric_set.compute_batch({
    ("gpt4", "en"): data_en,
    ("gpt4", "fr"): data_fr,
    ("llama", "en"): data_en,
    ("llama", "fr"): data_fr,
}, key_names=["model", "language"])

serve(batch)  # opens localhost:8050

The app provides:

• Chart — scatter, bar, or heatmap with configurable X, Y, color, and size axes
• Aggregation panel — group by any categorical column, compute mean/median/min/max
• Data table — sortable, with debug cell inspection and CSV export

key_names labels tuple key components (defaults to key[0], key[1], …). You can also pass a single EvalResult.

serve(data, *, host="127.0.0.1", port=8050, debug=False)

Full example

from dot_metrics import MetricSet, ComputedValue

appointments = [
    {"id": "A1", "patient": "Alice",   "duration": 30},
    {"id": "A2", "patient": "Bob",     "duration": 60},
    {"id": "A3", "patient": "Charlie", "duration": 30},
]

solution = [
    {"appointment_id": "A1", "practitioner": "Dr. Martin", "slot": "09:00", "scheduled": True},
    {"appointment_id": "A2", "practitioner": "Dr. Martin", "slot": "09:00", "scheduled": True},  # conflict!
    {"appointment_id": "A3", "practitioner": "Dr. Martin", "slot": "10:00", "scheduled": True},
]

metric_set = MetricSet()

@metric_set.metric("scheduling_rate")
def scheduling_rate(data):
    scheduled = [e for e in data["solution"] if e["scheduled"]]
    unscheduled = [e["appointment_id"] for e in data["solution"] if not e["scheduled"]]
    return ComputedValue(value=len(scheduled) / len(data["appointments"]), debug={"unscheduled": unscheduled})

@metric_set.constraint("conflicts", threshold=0)
def count_conflicts(data):
    seen = {}
    conflicts = []
    for entry in data["solution"]:
        key = (entry["practitioner"], entry["slot"])
        if key in seen:
            conflicts.append((seen[key], entry["appointment_id"]))
        seen[key] = entry["appointment_id"]
    return ComputedValue(value=len(conflicts), debug={"conflicts": conflicts})

result = metric_set.compute({"appointments": appointments, "solution": solution})

result.score("scheduling_rate")                     # 1.0
result.constraints_ok                               # False
result.constraints["conflicts"].debug               # {"conflicts": [("A1", "A2")]}

Reference

Import	Description
MetricSet	Main class — holds definitions, runs computation
EvalResult	Output of compute() — holds Metric and Constraint dicts
BatchResult	Output of compute_batch() — maps keys to EvalResult
ComputedValue	Wraps a float return value with optional debug data
Metric	Computed metric result
Constraint	Computed constraint result with passed flag
MetricDefinition	Stored metric definition (in metric_set.metrics)
ConstraintDefinition	Stored constraint definition (in metric_set.constraints)
draw_terminal_chart	Renders a Unicode bar chart from an EvalResult
serve	Launches an interactive Dash explorer

Contributing & Development

See docs/CONTRIBUTING.md and docs/DEVELOPMENT.md.

License

See LICENSE for details.

Contact

deepika Team — [e-mail] Project: gitlab.com/deepika6190303/deepika-open-toolbox/dot-metrics