Skip to main content

EvalSuite

CI PyPI Python License: MIT

Unified, reproducible evaluation for machine learning, LLM and research.

EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection evaluation into one consistent, validated, documented framework.

Status: stable (0.5.3). Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE. LLM systems arrive in 0.5.0: safety, robustness, calibration and uncertainty, agents and tool use, multilingual, code generation, long context and summarization, and inference efficiency and cost.

Installation

pip install evalsuite-python

The package is installed as evalsuite-python and imported as evalsuite:

import evalsuite as es

Why EvalSuite

  • One consistent API. Every metric returns a result object that behaves like a number and exports to JSON, pandas, Markdown and LaTeX.
  • Explicit conventions. Averaging, label order, the positive class and zero-division behaviour are stated and recorded in every result, never silently assumed.
  • Validated. Each metric is tested against scikit-learn where definitions coincide, plus property-based tests and edge cases.
  • Documented. Every metric carries its definition, formula, range, input requirements and references, available programmatically through metric_info().
  • Efficient. evaluate() validates inputs once and computes the confusion matrix once for all metrics.
  • Lightweight. Requires only NumPy, SciPy and pandas.

Quick start

import evalsuite as es

y_true = [0, 1, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 1]
y_prob = [0.1, 0.9, 0.4, 0.2, 0.8, 0.6]

result = es.evaluate(y_true, y_pred, y_prob=y_prob)
print(result.summary())

result["f1"]  # MetricResult(f1=0.666667)
f"{result['mcc']:.3f}"  # '0.333'
result.to_latex(caption="Test-set performance")
result.to_dataframe()

es.f1(y_true, y_pred)  # individual metrics
es.roc_auc(y_true, y_prob)
es.metric_info("classification.mcc").formula  # documentation
es.list_metrics("regression")

Comparing models

result = es.compare(
    y_true,
    {"logistic": pred_lr, "forest": pred_rf, "boosting": pred_gb},
    probabilities={"logistic": prob_lr, "forest": prob_rf, "boosting": prob_gb},
    random_state=0,
)
print(result.summary())  # estimates with 95% CIs, paired tests, Holm-adjusted p-values
result.to_latex(label="tab:models")

es.bootstrap_ci("f1", y_true, y_pred, average="macro", random_state=0)  # BCa interval for any metric
es.accuracy_ci(y_true, y_pred)  # Wilson interval
es.delong_test(y_true, prob_a, prob_b)  # two correlated AUCs
es.mcnemar_test(y_true, pred_a, pred_b)

Every model is evaluated on the same bootstrap resamples, so differences are paired. Accuracy is compared with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test; p-values are adjusted for multiple comparisons (Holm by default).

Clinical evaluation

report = es.diagnostic_report(y_true, y_pred)  # binary test vs reference standard
print(report)
# Sensitivity, specificity, PPV, NPV (Wilson CIs), LR+ and LR− (log CIs, Simel 1991),
# diagnostic odds ratio (Woolf), Youden's J, accuracy and prevalence

es.lr_positive(y_true, y_pred)
es.youden_j(y_true, y_pred)

dca = es.decision_curve(y_true, {"model": y_prob})  # net benefit vs treat all / treat none
dca.useful_range()  # thresholds where the model beats both
es.plot.decision_curve(y_true, {"model": y_prob})

Ratios that divide by zero are inf or NaN with a warning, never 0.

Calibration

es.calibration_report(y_true, y_prob)  # Brier, ECE, MCE, intercept, slope, Hosmer–Lemeshow
es.calibration_slope(y_true, y_prob)  # ideal 1; < 1 means predictions are too extreme
es.calibration_intercept(y_true, y_prob)  # ideal 0 (calibration-in-the-large)
es.hosmer_lemeshow(y_true, y_prob, n_groups=10)

Statistical tests

es.t_test(scores_a, scores_b)  # Welch by default; mean difference with CI and Cohen's d
es.paired_t_test(fold_scores_a, fold_scores_b)
es.wilcoxon_test(fold_scores_a, fold_scores_b)  # with matched-pairs rank-biserial r
es.mann_whitney_test(a, b)  # with rank-biserial r
es.friedman_test(scores_a, scores_b, scores_c)  # with Kendall's W
es.kruskal_wallis_test(g1, g2, g3)
es.shapiro_wilk_test(residuals)
es.chi_square_test(table)  # with Cramér's V
es.fisher_exact_test([[8, 2], [1, 5]])
es.adjust_pvalues(p_values, method="hochberg")  # also holm, bonferroni, bh, by

Every test returns a TestResult with the statistic, p-value and an effect size, computed with SciPy and checked against SciPy and statsmodels in the test suite.

Segmentation

# label masks: one 2-D image, or images on the first axis (N, H, W) / (N, D, H, W), or a list of masks
es.dice(y_true, y_pred)  # macro over classes, pixel counts summed over the dataset
es.iou(y_true, y_pred, average=None)  # per class; classes absent from both masks are NaN, not 0
es.miou(y_true, y_pred, ignore_index=255)
es.dice(y_true, y_pred, aggregate="image")  # mean of per-image scores (medical imaging convention)
es.boundary_iou(y_true, y_pred)  # Cheng et al. 2021
es.hausdorff_distance(y_true, y_pred, percentile=95, spacing=(0.8, 0.8))  # HD95 in mm

report = es.segmentation_report(y_true, y_pred, class_names={0: "background", 1: "liver"})
print(report)  # mIoU, Dice, pixel accuracy, Boundary IoU, HD95, ASSD + per-class table
es.plot.segmentation(image, y_true[0], y_pred[0])  # prediction fill, truth outline
es.plot.per_class(report, metric="iou")  # per-class bars (also takes a detection report)

Object detection

y_true = [{"boxes": [[x1, y1, x2, y2], ...], "labels": [3, ...]}, ...]  # one dict per image
y_pred = [{"boxes": [...], "labels": [...], "scores": [...]}, ...]

report = es.detection_report(y_true, y_pred)  # the 12 COCO numbers + AP per class
report["map"], report["map_50"], report["mar_100"]
es.mean_average_precision(y_true, y_pred, iou_threshold=0.5)  # mAP@.50
es.average_precision_detection(y_true, y_pred, interpolation="voc")  # per class, VOC-style
es.box_iou(boxes_a, boxes_b, box_format="xywh")
y_true, y_pred = es.from_coco("instances_val.json", "detections.json")
es.plot.detection_pr(y_true, y_pred)

The COCO protocol (crowd regions, area ranges, max detections, 101-point interpolation) matches pycocotools to the last digit in the test suite.

Models can be compared over the same images with intervals and paired tests, as for every other task: es.compare(y_true, {"unet": masks_a, "deeplab": masks_b}) resamples images; for detection it compares mAP.

LLM evaluation

es.bleu(references, predictions)  # sacreBLEU-identical; also chrf, ter, rouge_l, meteor, cider
es.text_report(references, predictions)  # BLEU, chrF(++), TER, ROUGE, METEOR, EM, token F1 at once
es.bertscore(ref_token_embs, pred_token_embs)  # from your encoder's embeddings; also mauve, moverscore
es.model_score(refs, preds, scorer=comet_fn)  # any learned metric or judge, with intervals and compare
es.faithfulness(claim_verdicts)  # also hallucination_rate, citation_recall, answer_correctness
es.bradley_terry(comparisons, scale="elo")  # also win_rate, elo_ratings, krippendorff_alpha
es.ndcg_at_k(relevant, retrieved, k=10)  # also mrr, context_precision, context_recall
es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_compliance_rate
es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
es.spice(reference_tuples, candidate_tuples)  # scene-graph F-score; parser= for captions, synonyms="wordnet"

LLM systems

Safety, robustness, uncertainty, agents, multilingual, code, long context and serving cost. The verdicts these take (harmful or not, test passed or not, confidence) come from your own judge, classifier or sandbox; EvalSuite turns them into rates with Wilson intervals and breakdowns. PII detection, CodeBLEU, cyclomatic complexity and the maintainability index (identical to radon) are computed from the text itself, and EvalSuite never executes generated code.

import evalsuite as es

es.harmful_response_rate([True, False, False, True], harmful_prompt=[True, True, False, True])
es.over_refusal_rate(refused=[True, False, True], should_refuse=[True, False, False])
es.pii_leakage_rate(["mail me at a@b.io", "nothing here"])  # e-mail, phone, Luhn-checked cards, IPs
es.paraphrase_consistency([["Paris", "paris"], ["4", "5", "4"]])
es.aurc(correct=[True, True, False, True], confidence=[0.9, 0.8, 0.7, 0.4])  # also selective_risk
es.plan_adherence(plans=[["search", "read", "answer"]], executed=[["search", "answer"]])
es.language_parity({"en": [1, 1, 0, 1], "sw": [1, 0, 0, 1]})
es.codebleu(["def add(a, b):\n    return a + b"], ["def add(x, y):\n    return x + y"])
es.code_complexity(["def f(x):\n    return 1 if x else 0"])  # cyclomatic complexity + maintainability
es.needle_in_haystack(correct=[1, 0, 1, 1], context_lengths=[4000, 4000, 8000, 8000], depths=[0, 0.5, 0, 0.5])
es.time_to_first_token(request_times=[0.0, 1.0], first_token_times=[0.21, 1.35])
es.inference_cost([1200, 800], [300, 150], input_price=3.0, output_price=15.0)  # prices per 1M tokens

Classification report

report = es.classification_report(y_true, y_pred)
print(report)  # per-class precision, recall, F1, specificity, support + averages
report.save("report.html")  # also .csv .md .tex .json .txt

Plots

pip install "evalsuite-python[plot]"   # adds matplotlib; importing evalsuite never loads it
es.plot.roc(y_true, {"logistic": prob_lr, "forest": prob_rf})  # AUC in the legend
es.plot.pr(y_true, prob)  # AP and the prevalence line
es.plot.calibration(y_true, prob)  # reliability diagram, ECE, Brier
es.plot.confusion_matrix(y_true, y_pred, normalize="true")
es.plot.residuals(y_reg, pred_reg)  # or kind="predicted"
es.plot.comparison(es.compare(...))  # forest plot with CIs

Each function returns a matplotlib Axes (pass ax= to draw into your own figure). The numbers shown are computed with EvalSuite's metrics, so plots and tables always agree. Several models get distinct colours and line styles, so figures stay readable in greyscale print.

Exports

Every result (evaluate, classification_report, compare, single metrics) exports to summary(), to_json(), to_csv(), to_dataframe(), to_markdown(), to_latex() and to_html(), and save(path) picks the format from the extension. HTML pages are standalone (inline CSS, no scripts) and escape all text.

Command line

evalsuite evaluate predictions.csv --y-true label --y-pred pred --y-prob prob
evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
    --prob lr=p_lr --prob rf=p_rf --plot comparison.png
evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
evalsuite diagnostic predictions.csv --y-true label --y-pred pred   # sensitivity, LR+, DOR... with CIs
evalsuite calibration predictions.csv --y-true label --y-prob prob  # slope, intercept, ECE, HL
evalsuite plot decision predictions.csv --y-true label --y-prob prob -o dca.png
evalsuite segmentation true_masks.npy pred_masks.npy --ignore-index 255 --plot per_class.png
evalsuite detection instances_val.json detections.json --plot pr_curves.png
evalsuite metrics --category clinical
evalsuite info classification.mcc
evalsuite benchmark --quick

Input files can be CSV, TSV, Parquet or JSON. Output format follows --format or the -o extension (text, json, csv, markdown, latex, html). Errors are reported in one line with exit code 2.

Performance

Benchmarked against reference implementations on the same data (fastest of 5 runs; EvalSuite 0.5.3, Python 3.13, NumPy 2.5, Linux x86_64). Overall: 81 of 81 rows agree with the reference (largest difference 2.5e-14), 58 are faster, and the geometric-mean speed-up across all 27 cases and three sizes is 2.06×. At 1,000,000 samples (pixels for segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in alphabetical order:

Case Reference EvalSuite (ms) Reference (ms) Speed-up
10 classes: macro F1 scikit-learn 35.9 122.1 3.4×
agreement: Krippendorff's alpha, interval (4 raters × 10000 items) krippendorff 4.4 3.4 0.77×
binary: 8 label metrics via evaluate() scikit-learn 23.3 866.5 37.2×
binary: ROC AUC scikit-learn 187.9 305.7 1.6×
calibration: slope and intercept statsmodels 236.6 1085.8 4.6×
clinical: diagnostic report (7 CIs) statsmodels 26.2 5.7 0.22×
clinical: sensitivity, specificity, LR+, LR− scikit-learn 122.9 353.9 2.9×
decision curve: 99 thresholds NumPy loop 167.3 238.1 1.4×
detection: COCO evaluation (1000 images) pycocotools 684.7 660.8 0.97×
multiple testing: Hochberg (n p-values) statsmodels 71.3 92.4 1.3×
regression: MAE, MSE, RMSE, R² via evaluate() scikit-learn 27.3 18.8 0.69×
retrieval: MRR, MAP@20, NDCG@10 (10000 queries) ranx 129.5 614.7 4.7×
segmentation: Dice and IoU per class (n = pixels) scikit-learn 35.7 235.1 6.6×
segmentation: Hausdorff distance (50 images) SciPy 18.8 13.6 0.72×
statistics: Cramér's V (5×5 table) SciPy 0.2 0.2 0.90×
statistics: Mann–Whitney U SciPy 342.3 325.8 0.95×
statistics: Welch t-test SciPy 8.2 11.9 1.4×
structured: JSON Schema compliance (10000 documents) jsonschema 93.9 223.5 2.4×
systems: CodeBLEU n-gram terms (1000 programs) codebleu 413.8 507.8 1.2×
systems: confidence AUROC (n answers) scikit-learn 154.1 251.8 1.6×
systems: distribution-shift drop, Welch CI (n scores) SciPy 10.9 12.9 1.2×
systems: invalid tool calls vs JSON Schema (10000 tasks) jsonschema 78.4 213.2 2.7×
systems: latency p50 / p95 / p99 (n requests) NumPy 18.0 15.9 0.88×
systems: maintainability index (1000 programs) radon 89.3 96.0 1.1×
text: corpus BLEU and chrF (10000 sentences) sacreBLEU 1728.8 1993.1 1.2×
text: METEOR, exact and stem matches (10000 sentences) NLTK 319.1 358.6 1.1×
text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) rouge-score 716.0 612.5 0.86×

Every metric is benchmarked too (evalsuite benchmark --suite metrics): all 225 registered metrics and statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; all 204 comparisons agree, and against the libraries the geometric-mean speed-up is 2.80× (138 of 172 measurements faster). Per-metric table in BENCHMARKS.md.

evaluate() validates inputs once and builds the confusion matrix once for all metrics, which is where most of the speed-up comes from. The t-tests compute their statistics in one pass and beat SciPy; other hypothesis tests use SciPy underneath. Rows below 1× pay for input validation and the extra intervals and effect sizes EvalSuite reports. Reproduce on your machine with evalsuite benchmark; full table (1k, 100k and 1M samples, peak memory) and notes in BENCHMARKS.md.

Metrics in this release

Classification (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging; sample weights): accuracy, balanced accuracy, precision, recall, specificity, NPV, F1, F-beta, Jaccard, MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matrix, ROC AUC (binary, one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy, calibration curve and expected calibration error.

Clinical (binary; pos_label; sample weights): sensitivity, specificity, PPV, NPV, positive and negative likelihood ratios, diagnostic odds ratio, Youden's J, net benefit and decision curves, and a diagnostic report with confidence intervals for all of them.

Calibration: calibration curve, Brier score, expected and maximum calibration error, calibration slope and intercept, Hosmer–Lemeshow test.

Statistics: confidence intervals (bootstrap percentile/basic/BCa, Wilson, Clopper–Pearson, DeLong), paired tests (McNemar, DeLong, paired bootstrap), t-tests (Welch, Student, paired), Mann–Whitney, Wilcoxon, Kruskal–Wallis, Friedman, Shapiro–Wilk, χ², Fisher's exact; effect sizes (Cohen's d, Hedges' g, Cliff's delta, Cramér's V); multiple-testing corrections (Bonferroni, Holm, Hochberg, Benjamini–Hochberg, Benjamini–Yekutieli).

Segmentation (2-D and 3-D label masks; ignore_index; dataset or per-image aggregation): Dice, IoU, mIoU, pixel accuracy, mean pixel accuracy, Boundary IoU, Hausdorff distance and HD95, average symmetric surface distance (with pixel spacing), confusion matrix and a full report.

Object detection: box IoU (xyxy, xywh, cxcywh), COCO mAP@[.50:.95], mAP@.50, mAP@.75, mAP and mAR by object size, AP per class with COCO or VOC interpolation, precision-recall curves, COCO file import.

Regression (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE, MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball) loss, Huber loss, relative absolute error, relative squared error.

LLM evaluation (0.4.0): BLEU, sentence BLEU, chrF/chrF++, TER, ROUGE-1/2/L/Lsum, METEOR, CIDEr-D, SPICE, perplexity, cross-entropy, Distinct-n, Self-BLEU, MAUVE, BERTScore, embedding similarity, MoverScore, learned metrics via model_score; faithfulness, hallucination rate, groundedness, citation precision/recall, claim verification, knowledge consistency, answer correctness/relevance, abstention, exact match, token F1; win rate, Bradley–Terry, Elo, Krippendorff's alpha, Fleiss' kappa, judge agreement and biases, rubric scores; pass@k, majority vote, benchmark accuracy; Precision/Recall/Hit rate@k, MRR, MAP, NDCG, context precision, recall and relevance; JSON / JSON Schema / XML validity, required fields, tool calls, instruction following.

LLM systems (0.5.0): harmful-response, refusal, over-refusal, attack-success and red-team success rates, expected maximum toxicity, CrowS-Pairs stereotype preference, WEAT, PII leakage, memorization exposure, policy violations; adversarial, typo-noise, OOD and code-switching robustness, distribution-shift drop, paraphrase / counterfactual / cross-lingual consistency, response stability, contradiction rate, failure and recovery rates, prompt sensitivity, truncation sensitivity; adaptive calibration error, risk-coverage curve, AURC and E-AURC, selective risk, risk at coverage, coverage at risk, confidence AUROC; task completion, invalid tool calls, tool-use efficiency, steps per task, plan adherence, state tracking, tool-failure recovery, loop rate, human intervention, cost per task; language ID, bitext mining, language parity, direct assessment, cultural appropriateness, language consistency; unit-test pass rate, syntax validity, static analysis and security findings, execution success, test coverage, patch acceptance, SWE-bench resolved rate, CodeBLEU, cyclomatic complexity and maintainability index, runtime efficiency; long-context accuracy by length, needle-in-a-haystack grid, position accuracy, lost-in-the-middle gap, context utilization, summary coverage, compression ratio, citation coverage, cross-document consistency; TTFT, TPOT, latency percentiles, throughput, token usage, cost, resource utilization, energy per request, requests per second, availability.

Conventions

  • average="auto" resolves to "binary" for binary targets and "macro" otherwise; the resolved value is stored in result.params["average"].
  • Labels are sorted unless you pass labels=[...]; that order defines per-class outputs and the columns of 2-D y_prob.
  • Undefined ratios (zero denominators) return 0 with an UndefinedMetricWarning; pass zero_division=np.nan to propagate NaN, or 0/1 to choose silently.
  • Domain violations raise clear errors instead of being patched over (for example MAPE with zero targets).

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest --cov=evalsuite
ruff check . && ruff format --check . && mypy

Credits

Authors and maintainers: Manoj Kumar C S and Nikhil D Bharadwaj.

License

MIT. See LICENSE.

Metadata

Release files for evalsuite-python 0.5.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalsuite-python 0.5.3
File Size Uploaded Provenance Actions
evalsuite_python-0.5.3.tar.gz 306.4 kB With provenance Details for evalsuite_python-0.5.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalsuite-python 0.5.3
File Interpreter ABI Platform Actions
evalsuite_python-0.5.3-py3-none-any.whl Python 3 none any Details for evalsuite_python-0.5.3-py3-none-any.whl

Total release size: 580.4 kB

Release files / evalsuite_python-0.5.3.tar.gz

Download URL evalsuite_python-0.5.3.tar.gz
Size 306.4 kB
Tags Source
SHA-256 checksum
How to use checksums
d69bafc847900827941b2e8b8acd7ff0deb96df24943294a8506e71f6edd9240
BLAKE2b-256 checksum
How to use checksums
9467afca24ed251637f0f5429627509b4a55794e3a2cd3227a3c06d2693bf87a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Release files / evalsuite_python-0.5.3-py3-none-any.whl

Download URL evalsuite_python-0.5.3-py3-none-any.whl
Size 274.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9ab8a5be7626e8ba6356cf6aff3840dde29274159218e9c0067678806dd776eb
BLAKE2b-256 checksum
How to use checksums
5fc3cb3d7c8cf7f1414ecbe1e9c552ac0d09bb06cbbbc1bccaf1650c6941fbce
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Security for evalsuite-python 0.5.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for evalsuite_python-0.5.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page