telemeval

Scoring for anomaly detectors on spacecraft telemetry — the grading side, not the detecting side. You bring the labelled anomalies and your model's output; telemeval returns precision, recall and F-scores that count whole events rather than individual timestamps, and refuses to return a number at all when the inputs would produce a misleading one.

For anyone evaluating a time-series anomaly detector who wants the score to be defensible. Anomaly-detection scoring is unusually easy to get wrong — the widely used point-adjusted F1 can rank random guessing above a real detector — and the metrics the literature recommends instead have not had a maintained, permissively licensed, pip-installable home. This is that home.

Status: v0.x, early releases. The API may still change before v1. Maintained and CI-tested, but this is an evaluation library, not a certified tool: nothing here is airworthiness evidence or input to a flight-safety decision, and no metric is claimed to be novel. The value is packaging, correctness against reference fixtures, and maintenance. Released under the Apache 2.0 Licence.

Install

pip install telemeval

Requires Python 3.11+. Core dependencies are numpy and pandas, and nothing else — a CI job installs the wheel into an empty environment and fails if anything more appears.

Use it

import pandas as pd
from telemeval import evaluate

labels = pd.DataFrame(
    {
        "ID": ["anomaly_1"],
        "Channel": ["channel_41"],
        "StartTime": ["2024-01-01T00:02:00"],
        "EndTime": ["2024-01-01T00:03:00"],
    }
)
timestamps = pd.date_range("2024-01-01", periods=6, freq="1min")
predictions = {
    "channel_41": pd.DataFrame({"Timestamp": timestamps, "Score": [0, 0, 1, 1, 0, 0]})
}

result = evaluate(labels, predictions, dataset="my-mission")
print(result.metrics["event_wise"]["event_wise_fbeta"])     # 1.0
print(result.metrics["affiliation"]["affiliation_fbeta"])   # 1.0
result.save(json_path="report.json", markdown_path="report.md")

Both printed values are asserted by the test suite, so this block cannot drift from what the library does.

What it does

• Event-wise precision / recall / F-beta — every labelled anomaly weighs the same regardless of how long it lasted, with documented overlap semantics.
• Affiliation-based precision / recall (Huet et al., KDD 2022) — distance-based scoring that rewards being close to an event. The canonical MIT reference implementation is vendored, wrapped and kept green against its own test suite.
• ADTQC detection-timing quality — scores when each event was first caught, not just whether. Previously available only inside ESA-ADB's research fork.
• Channel- and subsystem-aware F-beta — did the detector point at the right source? Checked against the reference suite's exact expected values.
• A validated ingestion contract — malformed or misaligned inputs raise a typed, actionable error instead of quietly producing a number.
• A train/test-window leakage guard, on by default. See below.
• Deterministic JSON and Markdown reports, stamped with which metrics ran and what they do not cover.
• Format readers and sklearn-style wrappers — ESA-ADB and TimeEval layouts in, score(y_true, y_pred) functions for dropping the metrics into existing code.

Why trust this

This library exists because an evaluation was found to be wrong, not because a metric was missing.

A spacecraft anomaly detector was reporting a recall that looked like a weak model. It was not the model. The scorer was counting anomalies from the training period — events the detector had never been asked to predict — as missed detections. Restricting scoring to the window the model was actually given, which is what the benchmark's own protocol requires, raised the figure substantially. The correction made a published result look better, which is the direction nobody audits.

That audit is what produced the leakage guard here. evaluate() compares the label window against the prediction window and raises WindowLeakageError rather than scoring across the boundary; clipping is possible but has to be asked for explicitly with clip_to_window=True, and the choice is recorded in the report. It is a runtime check with tests behind it, not a claim — but it guards that one leakage class, not every way an evaluation can be wrong, and it does not apply to the thin sklearn-style wrappers, which take raw arrays with no window to check.

The full audit and its measured figures are in the reference pipeline, aerospace-prognostics, which consumes this library as a dependency.

Documentation

• Usage — the leakage guard, continuous scores and thresholds, ESA-ADB and TimeEval loaders, parquet input, sklearn wrappers, the metric registry.
• Metric definitions — exact semantics, including where event-wise recall matches ESA-ADB's definition and where precision deliberately diverges from it.
• Related work — an honest map of prior art and when to use something else.
• Roadmap · Changelog

Scope

telemeval scores detectors; it does not build them, benchmark them, or serve them. For detectors see PyOD, aeon or darts; for benchmark harnesses and dataset collections see TimeEval or TSB-AD. The closest neighbours on metrics are TSADmetrics (GPL-3.0, generic time-series metrics), aeon (range and VUS metrics) and Merlion (point-adjusted F1). What is different here is the permissive licence, the telemetry-domain ingestion contract, and maintained affiliation and ESA-ADB metrics; where metrics overlap, the aim is to reproduce prior-art numbers rather than invent new ones.

Not affiliated with or endorsed by ESA. The ESA Anomaly Dataset is not redistributed here.

Licence and citation

Apache-2.0. The vendored affiliation reference implementation stays MIT, with attribution retained in NOTICE. Dataset licences are separate from this code and nothing is bundled. To cite a specific release, see CITATION.cff or the DOI badge above. Contributions: CONTRIBUTING.md.