DataEval Plots

Multi-backend plotting utilities for DataEval outputs.

Installation

# Minimal - no plotting backend included
pip install dataeval-plots

# With matplotlib plotting (recommended)
pip install dataeval-plots[matplotlib]

# With multiple backends
pip install dataeval-plots[matplotlib,plotly]

# Everything
pip install dataeval-plots[all]

For development:

pip install -e dataeval-plots[all]

Available Backends

Usage

Option 1: Import from dataeval-plots directly

from dataeval_plots import plot
from dataeval.metrics.bias import coverage

result = coverage(embeddings)
fig = plot(result, images=dataset, top_k=6)
fig.savefig("coverage.png")

Option 2: Import from dataeval core (convenience)

from dataeval import plotting
from dataeval.metrics.bias import coverage

result = coverage(embeddings)
fig = plotting.plot(result, images=dataset)

Option 3: Set default backend

from dataeval_plots import plot, set_default_backend

# Set seaborn as default
set_default_backend("seaborn")
fig = plot(result, images=dataset)  # Uses seaborn

# Override for a specific plot
fig = plot(result, backend="matplotlib", images=dataset)

Features

• Multi-backend architecture: Support for matplotlib (default), seaborn, plotly, and altair
• Optional dependencies: Install only the backends you need
• Clean separation: Core dataeval has zero plotting dependencies
• Protocol-based design: Loose coupling via structural typing (Plottable protocol)
• Extensible: Easy to add new backends via BasePlottingBackend or custom outputs via Plottable
• Lazy loading: Backends are only imported when first used
• Type safe: Static type checking with mypy/pyright via @runtime_checkable protocols
• DRY architecture: Centralized routing logic in BasePlottingBackend

Architecture

The package uses a protocol-based architecture for loose coupling between dataeval and dataeval-plots:

dataeval/                           # Core package
    outputs/
        _bias.py                    # CoverageOutput, BalanceOutput, DiversityOutput
        _stats.py                   # BaseStatsOutput
        _workflows.py               # SufficiencyOutput
        _drift.py                   # DriftMVDCOutput
    plotting.py                     # Convenience hook to dataeval-plots

dataeval-plots/                     # Separate plotting package
    src/dataeval_plots/
        __init__.py                 # Main plot() function
        _registry.py                # Backend registry with lazy loading
        protocols.py                # Protocol definitions (Plottable hierarchy)
        backends/
            _base.py                # BasePlottingBackend (abstract routing)
            _matplotlib.py          # MatplotlibBackend (default)
            _seaborn.py             # SeabornBackend
            _plotly.py              # PlotlyBackend
            _altair.py              # AltairBackend

Protocol-Based Design

All DataEval output classes implement the Plottable protocol, which requires:

• plot_type(): Returns a string identifying the plot type (e.g., "balance")
• meta(): Returns execution metadata

This enables:

• Loose coupling: dataeval-plots doesn't import concrete classes from dataeval
• Type safety: Static and runtime type checking via @runtime_checkable protocols
• Extensibility: Anyone can create custom outputs implementing Plottable
• Zero dependencies: Core dataeval has no plotting dependencies

Supported Output Types

Each output type implements the Plottable protocol and can be plotted using any registered backend.

Reading the Feasibility Plot

plot() draws the result of DataEval's experimental Feasibility().evaluate(...). It answers one question per quantity you asked about: for each accuracy or AP requirement, does the evidence demonstrate it, rule it out, or leave it unknown?

from dataeval.performance import Feasibility
from dataeval_plots import plot

result = Feasibility().evaluate(dataset, target={"accuracy": 0.95}, embeddings=embeddings)

fig = plot(result)  # one band per quantity
fig = plot(result, plot_evidence=True)  # one row per evidence source, in place of the bands

What the plot shows

Each quantity gets its own band, with a "Required ..." axis (for example "Required accuracy" or "Required AP") and a subtitle that says which models the band judges:

The axis is split into three zones. Each zone classifies the requirements that fall inside it:

• Demonstrated runs from the axis minimum up to the lower confidence bound of the best tested model, which the label under the band names, for example "95.7% lower confidence bound". The zone reads "demonstrated".
• Unknown lies between the other two zones. The evidence neither demonstrates nor rules out these requirements.
• Ruled out runs from the upper confidence bound of the lowest applicable ceiling to the top of the axis. The zone reads "ruled out*" wherever the label fits.

The * marks a footnote: "ruled out" holds only under the recorded assumptions. Wherever "ruled out*" appears, the caption under the same band ends with that footnote, "* Ruled out under the recorded assumptions, at the stated confidence."

Each of your requirements is marked on the axis, and the marker states the determination in words:

The caption under the band reports the observed score next to the lower confidence bound it supports, for example "Your model scored 91.0% on 2,000 evaluation images; given sampling uncertainty, its lower confidence bound is 88.5%." The sentence about the observed score appears only when an observed score exists. The caption then states that the bounds apply to new images from the same population, and that confidence holds jointly across the whole evaluation (for example "Bounds apply to new images from the same population, at 95% joint confidence.").

Numbers read as DataEval prints them. Bounds are given to 0.1%, a lower bound rounded down and an upper bound rounded up, so rounding never overstates what the evidence shows. Requirements and confidence keep their own decimals, such as "95.5%" or "at 97.5% joint confidence".

When a ceiling bounds the band, the caption ends with the footnote "* Ruled out under the recorded assumptions, at the stated confidence." A group-based ceiling adds "(group-based ceiling; validity pending lead-scientist sign-off)".

A ceiling that applies only to narrower models than the band judges never draws a zone. It appears as a note instead, for example "Group-based ceiling (72.1%) applies only to models that give one answer per group."

Other outcomes

• Ruled out. A requirement above the lowest applicable ceiling's upper confidence bound is marked "ruled out* ✗". The ruled-out zone covers everything above that bound, which the label under the band names, for example "93.0% upper confidence bound". The caption ends with the footnote "* Ruled out under the recorded assumptions, at the stated confidence."
• Evidence conflict. When bounds from different sources contradict each other, the band draws no zones. A dashed warning outline reads "Evidence conflicts: check assumptions", both clashing bounds are marked, and the caption reads "Bounds disagree: check the assumptions, such as evaluation images entering training. Even when they hold, the chance that any bound fails is at most 5%." The 5% is the stated alpha, at 95% confidence.
• Bounds at 0% or 100%. A lower confidence bound of 0% demonstrates nothing and an upper confidence bound of 100% rules nothing out, so neither draws a zone.

Showing the evidence

plot_evidence=True replaces each band with one row per evidence source. Each row shows the observed score as a dot and the confidence bound as a tick, with the legend "● observed · ┃ lower confidence bound" (and "· ┃ upper confidence bound" when a ceiling has one). The caption says what both mean: "Observed on the evaluation images; bounds apply to new images from the same population, at 95% joint confidence." In a conflict, the conflict caption follows it. Evidence that does not apply to the judged models is drawn faint, and its label names the models it applies to, for example "Group-based ceiling (one answer per group)", so this view has no narrower-scope notes. A source that produced no bound for the quantity is listed faint as "no bound available", and when no source has a bound the view says "No bound is available yet." instead of showing empty axes.

Extending the Package

Creating Custom Outputs

You can create custom output classes that work with the plotting system by implementing the Plottable protocol:

from dataclasses import dataclass
from numpy.typing import NDArray
from dataeval_plots.protocols import Plottable, ExecutionMetadata

@dataclass
class MyCustomOutput:
    """Custom output that reuses existing plot type."""
    uncovered_indices: NDArray

    def plot_type(self) -> str:
        return "balance"  # Reuse existing balance plotting

    def meta(self) -> ExecutionMetadata:
        return ExecutionMetadata.empty()

# Works seamlessly with existing backends
result = MyCustomOutput(uncovered_indices=my_data)
fig = plot(result, images=my_images)

Creating Custom Backends

Extend BasePlottingBackend to create a new plotting backend:

from dataeval_plots.backends._base import BasePlottingBackend
from dataeval_plots.protocols import PlottableBalance
from dataeval_plots import register_backend

class CustomBackend(BasePlottingBackend):
    """Custom plotting backend using your preferred library."""

    def _plot_balance(self, output: PlottableBalance, **kwargs):
        # Implement balance plotting
        return my_figure

    # Implement other _plot_* methods...

# Register and use
register_backend("custom", CustomBackend())
fig = plot(result, backend="custom")

The BasePlottingBackend class handles all routing logic automatically - you just implement the plot-type-specific methods (_plot_coverage, _plot_balance, etc.).