Skip to main content

📊 TestGraph

Unified Python Testing Analytics & Regression Diagnostics

Local-first test reporting, failure clustering, flakiness heuristics, and AI/ML model evaluation in a single interactive workflow.


🌟 Why TestGraph?

Modern software engineering teams frequently build hybrid architectures: backend APIs, distributed pipelines, and AI/ML inference models. When automated runs fail or degrade, developers are forced to navigate fragmented tools—traditional unit test runners on one hand, and specialized ML monitoring platforms on the other.

TestGraph unifies software test outcomes and ML evaluation metrics under a single, versioned data model:

  • 🔒 Local-First & Safe: Zero daemons, zero external accounts, zero network calls. Uses defusedxml to neutralize XXE attacks and auto-escapes all untrusted stack traces.
  • 📉 Regression Diagnostics: Automatically catches newly failing tests, execution slowdowns, and ML metric drops against baseline runs.
  • 🔍 Failure Clustering: Normalizes volatile tokens (hex addresses, UUIDs, temporary paths) into deterministic signatures to group related errors.
  • 🎲 Historical Flakiness: Tracks outcome transitions across historical runs, flagging flaky tests and warning when sample sizes are insufficient ($N < 5$).
  • 🤖 AI/ML Evaluation: Built-in calculation of classification metrics (Accuracy, Macro/Weighted F1, Confusion Matrix) and latency benchmarks.
  • 📊 Standalone Interactive HTML: Generates self-contained HTML reports with embedded Plotly charts that work completely offline.

🚀 Quick Start

Installation

pip install testgraph-analytics

For development and additional data science tooling:

pip install "testgraph-analytics[dev]"

💻 Command-Line Interface (CLI)

TestGraph comes with a fast, versatile CLI:

1. Inspect Run Summary

testgraph summary reports/junit.xml

2. Compare Runs & Gate CI on Regressions

# Returns exit code 1 if regressions are detected (ideal for CI gates)
testgraph compare reports/baseline_junit.xml reports/current_junit.xml

3. Generate Standalone Interactive HTML Report

testgraph report reports/junit.xml -o reports/testgraph-report.html --baseline reports/baseline_junit.xml

4. Persist and Query Run History (SQLite)

# Save run to SQLite
testgraph record reports/junit.xml --db testgraph.db

# Inspect recent runs
testgraph history --db testgraph.db

# Detect flaky tests across historical executions
testgraph history --db testgraph.db --flaky

# View history of a specific test case
testgraph history --db testgraph.db --case "tests.test_api::test_health#4a1b8c2d"

🐍 Programmatic Python API

Ingesting JUnit XML & Analyzing Failures

from testgraph.ingestion import JUnitIngester
from testgraph.analytics import compute_run_summary, cluster_failures

# Safe parsing with defusedxml
ingester = JUnitIngester()
run = ingester.parse_file("reports/junit.xml")

# Outcome summary
summary = compute_run_summary(run)
print(
    f"Pass Rate: {summary.effective_pass_rate * 100:.1f}% ({summary.passed}/{summary.total_tests})"
)

# Deterministic failure clustering
clusters = cluster_failures(run)
for c in clusters:
    print(f"[{c.signature_id}] {c.count}x {c.error_type}: {c.normalized_message}")

Unifying ML Evaluations & Software Tests

from testgraph.models import TestRun
from testgraph.ml import create_evaluation
from testgraph.report import ReportRenderer

# Compute ML classification metrics & latency
ml_eval = create_evaluation(
    model_name="fraud_classifier_v2",
    y_true=["legit", "fraud", "legit", "fraud"],
    y_pred=["legit", "legit", "legit", "fraud"],
    latencies_ms=[14.2, 18.5, 12.1, 15.0],
)

# Attach to TestRun
run = TestRun(source="ml_pipeline")
run.ml_evaluations.append(ml_eval)

# Generate interactive standalone HTML report
renderer = ReportRenderer()
renderer.render_to_file(run, "reports/ml_and_tests.html")

🛠️ GitHub Actions CI Integration

Add TestGraph to your GitHub Actions workflow for automated regression gating and HTML report artifacts:

name: Test Suite & Analytics

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install Dependencies
        run: |
          pip install -e .
          pip install pytest

      - name: Run Pytest
        run: |
          pytest -v --junitxml=reports/current.xml

      - name: Gate on Regressions
        if: always()
        run: |
          # Compare with baseline if present
          if [ -f "reports/baseline.xml" ]; then
            testgraph compare reports/baseline.xml reports/current.xml
          fi

      - name: Generate HTML Report
        if: always()
        run: |
          testgraph report reports/current.xml -o reports/testgraph-report.html

      - name: Upload Report Artifact
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: testgraph-html-report
          path: reports/testgraph-report.html

📚 Documentation


📄 License

TestGraph is released under the Apache License 2.0.

Metadata

Release files for krixs-test-for-complet-complaition 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for krixs-test-for-complet-complaition 0.1.0
File Size Uploaded Provenance Actions
krixs_test_for_complet_complaition-0.1.0.tar.gz 47.0 kB Without provenance Details for krixs_test_for_complet_complaition-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for krixs-test-for-complet-complaition 0.1.0
File Interpreter ABI Platform Actions
krixs_test_for_complet_complaition-0.1.0-py3-none-any.whl Python 3 none any Details for krixs_test_for_complet_complaition-0.1.0-py3-none-any.whl

Total release size: 96.0 kB

Release files / krixs_test_for_complet_complaition-0.1.0.tar.gz

Download URL krixs_test_for_complet_complaition-0.1.0.tar.gz
Size 47.0 kB
Tags Source
SHA-256 checksum
How to use checksums
9f45377eca29a6e242b6dfd7bec23898bec5e6c98c41864eef69b5ad2ba32938
BLAKE2b-256 checksum
How to use checksums
94bc93b6021fb563d0ab0a7c074c294e2f939097edcea3b3b3801eb422da9244
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.0

Release files / krixs_test_for_complet_complaition-0.1.0-py3-none-any.whl

Download URL krixs_test_for_complet_complaition-0.1.0-py3-none-any.whl
Size 49.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8d84083941b03e133a0bdc0d2df21446a8d60174222f751739343acbdb070216
BLAKE2b-256 checksum
How to use checksums
39a2980a76623d05a3d55990ee8b2338327a7267a82bc2aa4278f5f09621dc70
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.0

Security for krixs-test-for-complet-complaition 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page