top-arena

top-arena is the typed Python SDK for Top Arena, an open benchmark for guitar-amplifier and neural-audio models. Your model runs on your own computer. The package downloads the public dry inputs, calls your Python function for every benchmark case, uploads the rendered audio, and returns the scores calculated by the public service.

The SDK does not upload your model, weights, training data, or source code. It only uploads the audio files returned by your callback and the model metadata you provide.

Installation

top-arena supports Python 3.13 and 3.14.

uv add top-arena

or:

python -m pip install top-arena

The PyPI distribution is named top-arena; the Python import uses an underscore:

from top_arena import benchmark

Quick start

Create a run, define a function that renders one dry file, and select the amplifier to benchmark. The callback may be synchronous or asynchronous and may return a Path or string pointing to any audio format supported by SoundFile, including WAV and FLAC.

from pathlib import Path

from top_arena import PipelineOptions, PositionMatrix, benchmark

from my_model import render_audio

run = benchmark.create(
    name="super-model-v1",
    creator="your-name",
    unique_positions_used=1,
    audio_duration_sum=4_000.0,
    turns=1,
    training_time=5_000.0,
    description="A short, useful explanation of the model and training setup.",
    parameter_count=40_000,
    options=PipelineOptions(
        download_concurrency=4,
        run_concurrency=1,
        upload_concurrency=4,
    ),
)


async def model(dry_audio: Path, positions: PositionMatrix) -> Path:
    return await render_audio(dry_audio, positions)


result = run.run("D3D21964-8E80-11EE-B9D1-0242AC120002", model)
print(result.run_id, result.status, result.metrics)

Use await run.run_async(amp_id, model) when your application already has an async event loop, such as a notebook, FastAPI application, or async test. run.run(...) deliberately raises an error when called from an active event loop so it cannot nest event-loop ownership accidentally.

The amplifier IDs currently available from the service can be read from GET /api/v1/amps. A complete runnable identity-model example is available in examples/passthrough_benchmark.py.

What the callback receives

The callback is invoked as callback(dry_audio, positions):

• dry_audio is a cached local path to the dry benchmark input.
• positions is an immutable matrix of normalized control values for that case.
• the return value is a path to the model's wet output for the same input.

Stereo output is folded to mono by the scoring service, and output at a different sample rate is resampled to the 48 kHz reference rate. Returning audio with the same duration and alignment as the dry input produces the most meaningful comparison. The SDK converts the returned file to lossless PCM-24 FLAC before upload.

Run metadata

The fields passed to benchmark.create(...) make leaderboard comparisons auditable:

These values are reported by the submitter; benchmark audio scores are calculated by the server.

How the pipeline works

The SDK uses three bounded stages:

1. Download benchmark inputs and verify/cache them by content hash.
2. Invoke the model callback and measure its wall-clock render speed.
3. Convert the output to PCM-24 FLAC and upload it for scoring.

The stages overlap, while each queue remains bounded so large benchmark runs do not grow memory use without limit. run_concurrency defaults to 1 because many GPU models and plugin hosts are not safe to invoke concurrently. Increase it only when your runtime supports parallel inference. Download and upload concurrency default to 4.

Every stage transition is appended to the run's server-side event log. Dry inputs are cached in the platform-appropriate user cache directory, and completed upload staging files are removed automatically. Set cache_dir= on benchmark.create(...) to choose a different cache location.

Configuration

The public service at https://top-arena.labqoat.com is used by default. To run against a local or private deployment, either pass server_url= or set:

export TOP_ARENA_SERVER_URL=http://127.0.0.1:8000

Explicit server_url= values take precedence over the environment variable.

PipelineOptions controls stage concurrency, queue capacity, score polling, and the overall completion timeout:

from top_arena import PipelineOptions

options = PipelineOptions(
    download_concurrency=8,
    run_concurrency=1,
    upload_concurrency=8,
    queue_capacity=16,
    poll_interval_seconds=1.0,
    completion_timeout_seconds=1_800.0,
)

Scores and results

run(...) returns a BenchmarkResult after server-side scoring completes. Its metrics mapping contains mean, P90, best, and worst summaries for the versioned metric contract. The primary metrics are ESR, human-weighted ESR, and MRSTFT; lower is better. Correlation and render speed are also reported; higher is better.

The run appears on the public leaderboard while it progresses. If the callback, download, conversion, upload, or server-side scoring fails, the SDK raises the underlying error and records a failure event when a run ID has already been created.

Development

The SDK lives in the packages/top-arena workspace package of qforge-dev/top-bench. From the repository root:

uv sync --locked --all-packages --all-groups
uv run pytest packages/top-arena/tests
uv run ruff check packages/top-arena
uv run mypy
uv build --package top-arena

The project is licensed under the MIT License.