TestPulse

Time your tests, optimize your outcomes.

Turn your current git diff into a single, evidence-based pytest command by learning from locally recorded test timings, failures, and which tests historically correlate with your changes; quantify flakiness and produce explicit risk guidance for skipping tests.

TestPulse is a Python library and CLI (testtime). It stores history in a local SQLite database (stdlib only—no runtime dependencies). There is no web application and no HTTP API server; you use pytest, the terminal, and optional library imports.

Features

Local test recording (pytest plugin)

What it does: Records each test’s nodeid, duration, pass/fail/skip/xfail outcome, and hashes of captured stdout/stderr into SQLite. Associates the run with the current git diff paths and updates a change↔test co-occurrence graph for later recommendations.

How to use:

export PYPI_PACKAGE_NAME=testpulse
pytest --testpulse-record
pytest --testpulse-record --testpulse-ci          # quiet CI mode
pytest --testpulse-record --testpulse-store /path/to/store.sqlite3

Default store: .testpulse/store.sqlite3 (override with TESTPULSE_STORE).

---

Recommended pytest command (testtime run)

What it does: Reads your git diff (or a commit range), maps changed files to pytest targets via static heuristics and learned co-occurrence, weights candidates by expected reliability from flakiness history, and prints one exact pytest ... line.

How to use:

testtime run
testtime run --commit-range main..HEAD
testtime run --min-reliability 0.8 --store .testpulse/store.sqlite3

Copy the printed command and run it in your shell.

---

Flakiness estimates (Bayesian smoothing)

What it does: Per-test flaky probability (inconsistent pass/fail history), confidence band (low / medium / high by sample size), and expected reliability for ranking and skip-risk logic.

How to use (library):

from testpulse import Store, flakiness_estimates, flakiness_for_nodeid
from testpulse.paths import default_store_path

store = Store.open(default_store_path())
try:
    for est in flakiness_estimates(store):
        print(est.nodeid, est.flaky_probability, est.expected_reliability)
finally:
    store.close()

Also surfaced in testtime report and in recommendation / risk scoring.

---

Skip and regret risk

What it does: For tests not included in the recommendation, computes a regret risk score (likelihood of missing a real failure) and an aggregate risk level: low, medium, high, or critical.

How to use (library):

from testpulse import compute_skip_risk_report

report = compute_skip_risk_report(["src/mod.py"], store)
print(report.aggregate_level, report.max_regret_risk)
for item in report.excluded:
    print(item.target, item.regret_risk)

Shown in testtime report (menu 6) and --no-interactive skip-risk section.

---

Regret hits

What it does: Across consecutive recorded runs, finds tests that passed in an earlier run but failed in a later run when diffs link through shared changed paths or learned co-occurrence—tests that “should have caught” a later breakage.

How to use (library):

from testpulse import regret_hits

for hit in regret_hits(store):
    print(hit.nodeid, hit.prior_run_id, hit.later_run_id, hit.linking_path)

Listed in testtime report (menu 5).

---

Interactive report (testtime report)

What it does: Terminal menu to explore:

1. Slowest tests (average duration)
2. Most flaky tests
3. Most change-relevant tests (for current diff)
4. Top drivers of today’s recommendation
5. Regret hits
6. Skip / regret risk summary

Filter: press f to filter by pytest marker or class name (parsed from stored markers and nodeids).

How to use:

testtime report
testtime report --no-interactive    # drivers + skip risk only, then exit
testtime report --commit-range HEAD~5..HEAD

---

Explanations with local citations (testtime explain)

What it does: For each included/excluded candidate, explains why and which changed files relate, with citations pointing at rows in your SQLite history (heuristic maps, co-occurrence links, outcome counts)—no external datasets.

Optional LLM: With --llm and your own API key, adds a short narrative constrained to those citations only.

How to use:

testtime explain
testtime explain --llm

Exit code 3 if --llm is requested but no API key is configured.

---

CI stats export (de-identified)

What it does: Exports aggregated timing and outcome stats (run counts, outcome totals, duration percentiles, flaky buckets). No test logs, no raw nodeids in the JSON (only counts and hashed identifiers internally). Optional HTTPS upload when URL and token env vars are set.

How to use:

testtime ci-export --output testpulse-ci-stats.json
testtime ci-export --upload

GitHub Actions workflow .github/workflows/testpulse-stats.yml records (optional), exports JSON, and uploads an artifact.

---

Brand assets

Static logo and favicon for documentation and packaging: frontend/public/logo.svg, frontend/public/favicon.svg (primary #003366, accent #FFCC00). Not a runnable UI.

Pages

TestPulse does not ship a browser frontend or routes. User-facing surfaces are:

Surface	What you see
Terminal — testtime report	Menu of report sections; optional marker/class filter prompts
Terminal — other commands	Single-shot stdout (pytest command, explanations, CI JSON path)
Pytest session	Optional line testpulse: recorded N test(s) when recording (hidden in --testpulse-ci)

There are no /login, settings, or dashboard URLs—only the CLI and library APIs below.

API

There is no REST backend. The stable Python public API is exported from testpulse (defined in testpulse.api). Open a Store unless noted.

Package metadata

Symbol	Purpose
VERSION, tagline	Version string and product tagline
distribution_name()	Read PYPI_PACKAGE_NAME
import_package_name()	Distribution name with - → _ for imports

Recording and storage

Symbol	Purpose
Store.open(path) / Store.connect_memory()	SQLite store (testpulse.store)
Pytest plugin	Entry point testpulse — flags --testpulse-record, --testpulse-ci, --testpulse-store

Recommendations

Symbol	Purpose
recommend_pytest_command(changed_paths, store, min_reliability=0.0)	One pytest ... shell line

Flakiness

Symbol	Purpose
flakiness_estimates(store)	All tests with FlakinessEstimate
flakiness_for_nodeid(store, nodeid)	Single test or None
flaky_probability(passes, fails)	Smoothed flaky score
beta_smoothed_pass_rate(passes, fails)	Posterior mean pass rate
confidence_band(passes, fails)	low / medium / high
estimate_from_counts(OutcomeCounts)	Build FlakinessEstimate

Skip / regret risk

Symbol	Purpose
compute_skip_risk_report(changed_paths, store, …)	SkipRiskReport with excluded tests
regret_risk(relevance, fail_probability, max_relevance=…)	Single regret score
aggregate_risk_level(excluded)	Overall RiskLevel

Regret hits

Symbol	Purpose
regret_hits(store)	List of RegretHit across consecutive runs

Reports

Symbol	Purpose
build_report_snapshot(changed_paths, store, …)	ReportSnapshot for all sections
run_interactive_report(changed_paths, store, …)	Interactive menu
render_report_view(snapshot, view, filter, …)	Print one section (slowest, flaky, relevant, drivers, regret, skip_risk)
ReportFilter, matches_filter, nodeid_class	Marker/class filtering helpers

Explanations and citations

Symbol	Purpose
explain_recommendation(changed_paths, store, …)	Tuple of TestDecisionExplanation
explain_with_optional_llm(…, client)	Explanations plus optional LLM text
build_test_explanation(store, paths, target, decision)	One explanation
gather_citations(store, paths, target)	HistoryCitation rows from SQLite
format_explanations_text(explanations)	Plain-text rendering
enhance_explanations_with_llm(…, client)	LLM narrative from citations only

LLM (optional, caller credentials)

Symbol	Purpose
llm_config_from_env()	LLMConfig or None
client_from_env()	OpenAICompatibleClient or None
OpenAICompatibleClient, LLMConfig	OpenAI-compatible HTTP client (stdlib)
LLMCompletionError	Request / response errors

CI stats

Symbol	Purpose
build_aggregated_ci_stats(store)	AggregatedCiStats
export_ci_stats_json(store)	JSON string
write_ci_stats_file(path, store)	Write JSON file
upload_ci_stats(store, endpoint_url=…, token=…)	HTTPS POST
upload_ci_stats_from_env(store)	Upload if env configured; else False
hash_nodeid(nodeid)	De-identified hash (not for display of raw ids in export)
CiUploadError	Upload failures

CLI entry point

Command	Purpose
testtime run	Print recommended pytest command
testtime report	Interactive or --no-interactive report
testtime explain	Citations; optional --llm
testtime ci-export	De-identified JSON; optional --upload
testtime health	Deployment health JSON (status, version, distribution)

Exit codes: 0 success, 1 failure, 2 usage, 3 missing optional config.

Configuration

Set variables in the environment (names only in .env.example; do not commit values).

Variable	Required	Purpose
PYPI_PACKAGE_NAME	No (recommended in CI)	Overrides distribution name; defaults to installed package metadata
TESTPULSE_STORE	No	SQLite path (default .testpulse/store.sqlite3)

Third-party keys (optional; you provide your own)

Variable	Feature	Where to get it
TESTPULSE_LLM_API_KEY or OPENAI_API_KEY	LLM explanations (testtime explain --llm)	Your OpenAI or compatible provider account
TESTPULSE_LLM_BASE_URL	LLM API base URL	Provider docs (default OpenAI v1 URL)
TESTPULSE_LLM_MODEL	Chat model id	Provider model list (e.g. gpt-4o-mini)
TESTPULSE_STATS_UPLOAD_URL	CI stats HTTPS upload	Your webhook or ingest endpoint
TESTPULSE_STATS_UPLOAD_TOKEN	Bearer token for upload	Same service that issued the URL
TESTPULSE_STATS_SALT	Salt for hashed ids in CI export	Any secret string you choose (default internal salt if unset)

GitHub Actions can store upload URL/token as repository secrets for .github/workflows/testpulse-stats.yml.

Deployment contract

After build, run bash scripts/bootstrap_deployment_env.sh (or use the manifest install command) so PYPI_PACKAGE_NAME is set when infrastructure did not inject it. Smoke checks: python scripts/verify_deployment_environment.py and testtime health.

TestPulse ships as a PyPI package with a CLI-first workflow and an optional stdlib HTTP API for deployment checks:

• testtime health — CLI health JSON
• testtime serve — GET /api/health, POST /api/auth/sign-up, POST /api/auth/sign-in, protected GET /api/me (set JWT_SECRET)
• python scripts/deployment_environment_verification.py — Factory env verification (loads committed deployment.env)

Contract item	TestPulse behavior
Install / build	uv sync, uv build; CI runs the same checks in .github/workflows/ci.yml
Runtime config	PYPI_PACKAGE_NAME optional at runtime; falls back to installed wheel / pyproject.toml name
Health / smoke	testtime health (JSON), testtime --help, and uv run pytest in CI
Optional upload	De-identified CI stats only when you set upload URL + token

Run it locally

Requires Python 3.11+, git, and uv.

export PYPI_PACKAGE_NAME=testpulse
python3 scripts/sync_project_name.py
uv sync

Record history and get a recommendation:

uv run pytest --testpulse-record
uv run testtime run
uv run testtime report --no-interactive

Development checks:

uv run ruff check .
uv run mypy src
uv run pytest
uv build

Install the console script: uv sync exposes testtime in the project virtualenv.

Architecture

Layer	Technology
Frontend	None (static SVG brand assets only)
Backend / HTTP API	None
Database	Local SQLite via stdlib (testpulse.store)
Vector database	None
Package manager	uv
CLI	testtime → testpulse.cli
Pytest integration	Entry point testpulse → testpulse.recording

Repository layout

src/testpulse/
  api.py              # Public re-exports
  cli.py              # testtime subcommands
  store.py            # SQLite schema and queries
  recording.py        # Pytest hooks
  git_diff.py         # git diff → changed paths
  heuristics.py       # Static path → test mapping
  recommend.py        # Merge heuristics + co-occurrence → pytest command
  flakiness.py        # Bayesian flaky scores
  risk.py             # Skip / regret risk
  regret_hits.py      # Temporal pass-then-fail analysis
  report.py           # Report snapshot and TUI
  citations.py        # SQLite citation records
  explanations.py     # Template + optional LLM explanations
  llm_client.py       # Optional OpenAI-compatible client
  ci_stats.py         # De-identified CI export/upload
  paths.py, config.py # Store path and PYPI_PACKAGE_NAME
tests/                # Unit and CLI tests
scripts/              # sync_project_name.py for PYPI_PACKAGE_NAME → pyproject name
.github/workflows/    # CI and optional TestPulse stats artifact
frontend/public/      # logo.svg, favicon.svg

How components connect

  pytest --testpulse-record
           │
           ▼
  recording.py ──► Store (SQLite) ◄── git_diff.py (changed paths per run)
           │              │
           │              ├── learn change_test_links (co-occurrence)
           │              └── test_results (timing, outcome, marker names)
           │
  testtime run / report / explain / ci-export
           │
           ├── recommend.py ← heuristics + co-occurrence + flakiness
           ├── risk.py, regret_hits.py, report.py, explanations.py
           └── ci_stats.py → JSON artifact (optional HTTPS upload)

All learning and recommendations use your machine’s SQLite file only; nothing is sent unless you opt into LLM or CI upload with your credentials.

License

MIT