Tuning Client Python SDK and demo

Tuning Client is the Python SDK and demo for the hyperopt.dev AutoML service.

Use the Python SDK to request tuning suggestions while keeping model training, datasets, and model weights on your computer. The optional demo trains small PyTorch models and compares tuning strategies on classification, regression, and teaching datasets.

Code provenance: The SDK and demo code were generated with ChatGPT and reviewed. Review does not guarantee correctness; evaluate the code and benchmark results for your use case. The demos are examples, not production ML benchmarks.

Installation

Requires Python 3.11 or newer. Install the SDK alone or include the demo:

python -m pip install tuning-client
python -m pip install 'tuning-client[demo]'

Run the demo

Create an account using the service's signup page and verify your email. Replace https://YOUR_API below with the service URL provided to you:

python -m tuning_demo --api-url https://YOUR_API --email you@example.com --dataset digits
# The installed command is equivalent:
tuning-client-demo --api-url https://YOUR_API --email you@example.com --dataset iris
python -m tuning_demo --list-datasets

The demo prompts for your password. You can instead set COMPUTE_ACCESS_TOKEN, or COMPUTE_EMAIL and COMPUTE_PASSWORD. Set COMPUTE_API_URL to avoid passing --api-url each time. Connections use HTTPS and verified TLS.

Service effort defaults to medium. Use --effort easy for lower effort; the observation budget stays the same. --interactive offers another dataset after each run. Sessions created by the demo are released automatically on completion or failure. --session-id ID uses an existing unused session that you remain responsible for releasing; --keep-session retains a new session.

Results are saved in ./results/. Downloaded datasets are cached in ~/.cache/tuning-client/. Override these with --output-dir and --data-dir. Optional usage events include a run ID, session ID, workload category, and progress event; use --no-telemetry to disable them. Training data, file paths, and model weights are not included in those events.

SDK usage

import os
from tuning_client import TuningClient

with TuningClient(os.environ['COMPUTE_API_URL'],
                   access_token=os.environ['COMPUTE_ACCESS_TOKEN']) as api:
    session_id = api.create_session()
    try:
        stream = api.tuning_session(session_id)
        try:
            configurations = [{'learning_rate': rate} for rate in
                              [0.001, 0.002, 0.003, 0.004, 0.005,
                               0.006, 0.007, 0.008, 0.009, 0.010]]
            stream.initialize(configurations, max_steps=10, seed=42)
            suggestion = stream.suggest()
            # Train the suggested configuration locally, then report validation loss:
            # stream.report(suggestion['idx'], validation_loss)
            # Continue with suggest/report or report_and_suggest as needed.
        finally:
            stream.close()
    finally:
        api.terminate_session(session_id)

initialize and reset default to effort="medium". A report records one observation; max_steps limits observations per model, not optimizer updates. Use status() to inspect progress and rankings. reset(...) starts a fresh run after the previous run completes. api.list_sessions() lists your sessions; api.wait_for_termination(session_id) waits for a release to complete.

Datasets and training budgets

Each demo starts with ten sampled model configurations and allows up to 50 validation observations across them, with at most ten observations per model. After tuning, all models finish to the same update cap for comparison. Classification ranks use validation cross-entropy; regression ranks use MSE in original target units. Lower is better. Test data is used only for final reporting.

Glass and vehicle download on first use. Ordinary datasets use a 70/15/15 train/validation/test split; wine uses 55/30/15. Identical language contexts are grouped into one split. Preprocessing is fitted on training data only.

--block-steps N sets updates per observation and a maximum of 10*N updates. The defaults are bounded workload profiles, not guarantees of convergence. Version 2 uses loss-based classification ranks; these are not comparable with version 1 accuracy-error ranks. --seed sets all seeds; --config-seed, --split-seed, and --training-seed allow independent overrides.

Repeated benchmarks

python -m tuning_demo.stats --api-url https://YOUR_API --email you@example.com \
  --datasets iris wine --runs 2

Without dataset/run options, this runs all eleven datasets ten times, starting at seed 42. Each trial starts with fresh models and tuning state. Sessions are reused per dataset; --session-scope run reuses one session across datasets. Service benchmarks run sequentially.

Local reference schedulers need no service account:

python -m tuning_demo --scheduler asha --dataset digits
python -m tuning_demo.stats --scheduler flaml --datasets iris wine --runs 2 --workers 2

Available schedulers are service (default), asha, median, pbt, pb2, and flaml. --workers applies only to local reference schedulers; it cannot parallelize service benchmarks. Reference statistics default to available CPU cores, capped by trial count; use --workers 1 to limit resource use. See benchmark details for algorithm settings.

Results include configurations, seeds, observation histories, timings, losses, and rankings at the tuning cutoff and after final training. Statistics report the mean final rank of the model selected at the cutoff and the best final rank among models already capped at that cutoff. Ties receive ranks such as 1, 1, 3; unavailable measurements and failed trials do not count as zero. A failed service trial skips the remaining trials for that dataset. The command exits nonzero on trial or cleanup failure and preserves partial results.

Timing includes client training and service round trips; it is not a pure measurement of service compute. The service ranking uses latest reported losses, not predicted fully trained losses. PBT/PB2 compare evolved checkpoint lineages, not the original fixed configurations.

Custom datasets

python -m pip install 'tuning-client[demo,custom]'
python -m tuning_demo --api-url https://YOUR_API --custom-data ./data.csv \
  --target label --task classification --drop-columns id
python -m tuning_demo --api-url https://YOUR_API --custom-data ./data.parquet \
  --target price --task regression

Use one row per independent example, unique column names, a non-missing scalar target, and numeric or categorical input columns. Drop identifiers and columns that leak the target. Custom datasets default to 100 updates per observation. Files stay local. Use an appropriate split outside this demo for time series, grouped observations, or other data that requires special leakage controls.

License and third-party notices

Original SDK and demo code is licensed under the MIT License.

The benchmark implementations include adaptations of Ray Tune (Apache-2.0) and FLAML (MIT). Their license texts and attribution are included in the package; see third-party notices.