tapbench

tapbench is a small fixed-rate benchmark client for IVOA TAP asynchronous jobs. It schedules one ADQL query at an offered rate, follows each admitted UWS job to a terminal phase, optionally streams its primary result, and reports latency and throughput. It is intentionally not a general-purpose load-testing framework.

Python 3.12 or newer is required.

Install

Install the command from PyPI with either uv or pipx:

uv tool install tapbench
# or
pipx install tapbench

From a checkout, uv can create the development environment and run the command:

uv sync
uv run tapbench --help

The service credentials must already be present in the process environment as TAPBENCH_USER and TAPBENCH_PASSWORD. Both must be nonempty. The command does not read dotenv files and does not accept credentials as command-line options.

Run a benchmark

uv run tapbench run \
  --url https://example.org/tap \
  --query-file examples/fast_lookup.sql \
  --rate 20 \
  --duration 60 \
  --max-inflight 200 \
  --poll-interval 0.5 \
  --no-download-results

--url is the TAP base URL; tapbench appends /async. --query-file must contain exactly the ADQL query offered at every opportunity. The query itself is never printed or stored in JSON; the report identifies it by file name/path and SHA-256 digest.

The run options are:

Run uv run tapbench run --help for the installed command's defaults.

Fixed-rate behavior

This is an open-loop offered-rate benchmark. Opportunity k has an absolute deadline of k / rate seconds from the monotonic scheduling epoch, beginning with opportunity zero at t=0. All deadlines strictly before duration are offered. Thus --rate 10 --duration 10 creates exactly 100 opportunities even when earlier queries are slow. Polls and result requests do not count toward the offered rate, and a late event-loop wake-up does not shift later deadlines.

An opportunity is admitted only if an in-flight slot is immediately available. Otherwise it is recorded as local_scheduling_drop; it is never queued to run later. A --max-inflight slot is held from submission start through job creation, phase handling, optional result download, and attempted cleanup. Consequently, slow downloads or cleanup can legitimately cause later drops. This bounds local tasks and network work as well as remote nonterminal jobs.

After the scheduling interval, tapbench drains the bounded set of admitted operations. The job timeout covers waiting for a terminal phase, beginning after successful job creation. Result download and cleanup occur outside that job deadline and are instead bounded by per-operation request timeouts.

Job creation, job start, phase polling, and remote cleanup treat an explicit HTTP 429 response as transient server backpressure through one shared retry policy. tapbench honors a valid Retry-After delay (seconds or HTTP date), adds small jitter to avoid synchronized retries, and otherwise uses bounded jittered exponential backoff. Creation retries remain within the request timeout, start and polling retries remain within the original job deadline, and DELETE retries remain within a separate cleanup deadline. Ambiguous POST timeouts and transport failures are not retried. Each record and the terminal summary report stage-specific rate-limit counts and cumulative wait times.

After observing a terminal phase, tapbench retrieves the standard UWS job document before cleanup and reads creationTime, startTime, and endTime. It reports query queue duration as startTime - creationTime and query execution duration as endTime - startTime. Timing retrieval is best effort: missing, malformed, or unavailable fields remain null and do not change the job outcome. A service's timestamp precision limits these measurements; a whole-second service may report zero for a subsecond query. The queue duration also includes any time the job remained PENDING before the client sent RUN, so it is not necessarily pure database scheduler wait.

Results and success

By default, a COMPLETED job is successful only after {job}/results/result has been completely consumed and no HTTP, TAP, or VOTable result error was detected. Response-header, first-byte, complete-body, and submission-to-download timings are recorded along with the byte count. The body is streamed rather than retained in memory.

With --no-download-results, a job is successful as soon as creation succeeds and COMPLETED is observed. No result-list, result-resource, redirect-target, or size-discovery request is made, and all result timing and byte fields are null. Cleanup failure is reported separately and does not change an otherwise successful operation.

Result inspection is deliberately bounded and does not deserialize a complete VOTable. tapbench retains only a small prefix while streaming and detects a QUERY_STATUS=ERROR marker when it appears there. An error marker appearing only beyond that inspection bound cannot be detected.

Latency summaries ignore missing observations and use nearest-rank p50, p90, p95, and p99 values: sort the samples and select observation ceil(percentile * count). An unavailable metric is shown as unavailable and serialized as JSON null.

Throughput labels have distinct denominators:

• achieved submission rate is submission starts divided by the configured scheduling duration, or by elapsed scheduler-active time after interruption;
• job-creation throughput is created jobs divided by the time from the scheduling epoch through the last successful creation response;
• server execution throughput is completed jobs with valid UWS timing divided by the server work window from the earliest startTime to the latest endTime; it is independent of client polling and unavailable when the server timestamps produce a zero-width window;
• result-completion throughput, when downloading, is successful downloads divided by the download window from the first download start through the last completion;
• download throughput is successfully downloaded bytes divided by the span of that same download window. Download rates are unavailable when there are no successful downloads or the span is zero.

No throughput denominator includes phase-poll sleeps or post-completion cleanup. The report shows how many completed jobs supplied valid UWS timing because server execution throughput may use fewer samples than the completed count.

Progress and terminal output

Progress is operational telemetry; final metrics are recomputed from all records. It is written to stderr so the final human summary remains on stdout. On a terminal, one line refreshes about once per second:

[schedule  12.0/60.0s] offered=241 started=238 dropped=3 active=87 created=235 completed=141 error=2 aborted=0 success=139 failed=5 polls=812 submission_rate_limited=4 start_rate_limited=2 rate_limited=11 cleanup_rate_limited=3 downloaded=812 MiB

Once scheduling ends, the label changes rather than implying submissions are continuing:

[drain  72.4s] offered=1200 started=1184 dropped=16 active=87 created=1179 completed=1068 error=9 aborted=1 success=1059 failed=26 polls=6312 submission_rate_limited=18 start_rate_limited=9 rate_limited=47 cleanup_rate_limited=12 downloaded=3.1 GiB

The downloaded field is omitted in no-download mode, and stage-specific rate limit fields are omitted until nonzero. When stderr is not a TTY, searchable tapbench progress: snapshots are emitted at most every five seconds, plus one at the schedule-to-drain transition. --no-progress suppresses all snapshots but not the final summary.

The final stdout report includes opportunity, submission, terminal, success, failure, HTTP, poll, and byte counts; peak in-flight work; the separate throughput values; client-observed and UWS server-side latency percentiles; and a primary-failure breakdown. Result rows are omitted in no-download mode.

JSON output and exit status

--json-output records sanitized configuration, UTC correlation timestamps, monotonic durations, query identity, aggregates, and one structured record for every opportunity, including local drops. Its schema id is tapbench.run/v1. Enums are strings and unavailable observations are null. Output is flushed and atomically replaces the destination, so a failed write does not leave a partially encoded report. Per-job records retain the three raw UWS timestamp strings alongside the derived queue and execution durations.

Exit codes are:

On the first SIGINT or SIGTERM, scheduling stops immediately, active operations are cancelled, and already-created remote jobs receive bounded best-effort cleanup unless --keep-jobs was selected. Accumulated records are retained, the partial summary and requested JSON are produced, and the command exits 3. A second interruption may bypass cleanup and output.

Security

Basic authentication is applied explicitly to requests on the configured TAP origin. Redirects may cross origins so services can return results from object storage, but cross-origin requests never receive the TAP Authorization header. URLs containing userinfo or using unsupported schemes are rejected. Displayed and serialized URLs omit userinfo, query strings, and fragments, and job identifiers are redacted. Errors are bounded and scrubbed of credential values. Neither credentials, authorization headers, query text, nor complete job URLs belong in terminal output or JSON.

Never commit local credential files. In particular, .env.mppdb and backup variants are local secrets and are ignored by this repository.

APDB example queries

The queries in examples/ use table and column names from the published Rubin APDB schema:

• fast_lookup.sql returns at most ten current object summaries;
• selective_join.sql returns at most 25 reliable r-band detections joined to their current object positions;
• expensive_small_result.sql scans and aggregates source fluxes but returns only one row per band.

The SQL files are included in both source and wheel distributions. In an installed wheel they reside under the tapbench/examples/ package directory; the repository copies remain the most convenient paths for command-line use.

They intentionally use unqualified table names, as shown by the published APDB schema. If a deployment exposes APDB tables only under a TAP schema qualifier, make a local copy and qualify both table names for that deployment. tapbench does not discover or rewrite schemas at runtime.

For a safe live check, first run each example once at a low rate with a single in-flight slot and normal cleanup. Then perform a short no-download benchmark:

uv run tapbench run \
  --url https://mppdb.juriclab.org \
  --query-file examples/fast_lookup.sql \
  --rate 1 \
  --duration 3 \
  --max-inflight 1 \
  --no-download-results

Do not use --keep-jobs for routine validation. Confirm the report contains three opportunities and no sensitive URL or credential material.

Development

The test suite uses a deterministic local fake TAP service; live service behavior is not a replacement for those tests. Run all checks with:

uv run ruff format --check .
uv run ruff check .
uv run mypy src
uv run pytest

Releasing

Package versions are derived from Git tags. Releases use clean tags of the form vX.Y.Z; for example, tag v0.1.0 produces package version 0.1.0. Untagged commits produce development versions and are never published.

Pushing a release tag builds and validates the wheel and source distribution, prints their complete file manifests, and saves them as workflow artifacts. It does not publish anything. Review the artifact names, hashes, and manifests, then create a draft GitHub Release for that tag.

Publishing the GitHub Release is the explicit production approval. It rebuilds and validates the tagged source, publishes the distributions to PyPI through Trusted Publishing, and verifies a fresh installation from PyPI. The publishing workflow stores no PyPI token and is restricted to the pypi environment.

Before the first release, create a pending Trusted Publisher on PyPI with project tapbench, owner mjuric, repository tapbench, workflow publish.yml, and environment pypi. Configure that GitHub environment to permit only tags matching v*.