For long runs, pass on_result to write each result to disk as it arrives. A crash or interruption only loses the requests that were in-flight at that moment — everything already completed is safe on disk.

generate_batch, embed_batch, and transcribe_batch all support the same per-item callback contract.

import json
from infermesh import LMClient

with open("results.jsonl", "w") as out, \
     LMClient(model="openai/gpt-4.1-mini", max_parallel_requests=32) as client:

    def save(index: int, result, error) -> None:
        row = {"index": index}
        if error is not None:
            row["error"] = str(error)
        else:
            row["output_text"] = result.output_text
        out.write(json.dumps(row) + "\n")
        out.flush()

    client.generate_batch(prompts, on_result=save)

The CLI automates this with --resume — see the CLI section above and the User Guide for the full checkpoint/resume pattern.

Pass any combination of rpm / tpm / rpd / tpd to activate the built-in rate limiter. The client queues requests automatically and respects all four limits simultaneously.

client = LMClient(
    model="openai/gpt-4.1-mini",
    rpm=500,      # requests per minute
    tpm=100_000,  # tokens per minute
)

Find your tier's limits in the provider dashboard: for OpenAI check Settings → Limits; for Anthropic check Console → Settings → Limits.

Use max_request_burst / max_token_burst to allow short bursts above the steady-state rate (token-bucket algorithm). Use default_output_tokens to pre-reserve output tokens for rate-limit accounting when you don't set max_tokens per request.

Provider rate-limit headers (x-ratelimit-*) are read automatically after each response to keep the client's internal counters in sync with the server's view. Use header_bucket_scope to control whether headers are routed to the per-minute or per-day buckets.

CLI flags: --rpm, --tpm, --rpd, --tpd, --max-request-burst, --max-token-burst.

When you run multiple inference servers for the same model, pass a deployments dict to spread load across them. model is the logical name the router exposes; each DeploymentConfig.model is the backend string sent to that server.

from infermesh import DeploymentConfig, LMClient

client = LMClient(
    model="llama-3-8b",
    deployments={
        "gpu-0": DeploymentConfig(
            model="hosted_vllm/meta-llama/Meta-Llama-3-8B-Instruct",
            api_base="http://host1:8000/v1",
        ),
        "gpu-1": DeploymentConfig(
            model="hosted_vllm/meta-llama/Meta-Llama-3-8B-Instruct",
            api_base="http://host2:8000/v1",
        ),
        "gpu-2": DeploymentConfig(
            model="hosted_vllm/meta-llama/Meta-Llama-3-8B-Instruct",
            api_base="http://host3:8000/v1",
        ),
    },
    routing_strategy="least-busy",   # or "simple-shuffle" (default), "latency-based-routing"
)

result = client.generate("Summarise this paper in one paragraph.")
print(result.metrics.deployment)  # e.g. "gpu-1"

DeploymentConfig is a plain dataclass, so it maps naturally to Hydra / OmegaConf structured config. Deployment keys ("gpu-0" etc.) are free-form labels.

CLI — repeated --api-base flags:

infermesh generate \
  --model llama-3-8b \
  --api-base http://host1:8000/v1 \
  --api-base http://host2:8000/v1 \
  --api-base http://host3:8000/v1 \
  --prompt "Hello"

CLI — TOML file for more control:

# deployments.toml
[deployments.gpu-0]
model = "hosted_vllm/meta-llama/Meta-Llama-3-8B-Instruct"
api_base = "http://host1:8000/v1"

[deployments.gpu-1]
model = "hosted_vllm/meta-llama/Meta-Llama-3-8B-Instruct"
api_base = "http://host2:8000/v1"

infermesh generate \
  --model llama-3-8b \
  --deployments-toml deployments.toml \
  --prompt "Hello"

Keep API keys out of TOML files — use environment variables or --env-file instead.

All methods have async counterparts prefixed with a. The sync methods work in notebooks and scripts by running a background event loop thread — you don't need to manage the event loop yourself.

import asyncio
from infermesh import LMClient

async def main():
    async with LMClient(model="openai/gpt-4.1-mini") as client:
        result = await client.agenerate("Hello")
        batch  = await client.agenerate_batch(["prompt A", "prompt B", "prompt C"])
        emb    = await client.aembed("The quick brown fox")
        embs   = await client.aembed_batch(["text a", "text b"])
        txs    = await client.atranscribe_batch(["a.wav", "b.wav"])

asyncio.run(main())

async with calls close() automatically. For sync code, use with or call client.close() when done.

Pass a Pydantic model as response_format and the output is parsed automatically:

from pydantic import BaseModel

class Answer(BaseModel):
    value: int
    confidence: float

result = client.generate(
    "What is 2 + 2? Respond in JSON.",
    response_format=Answer,
)

print(result.output_text)   # raw JSON string
print(result.output_parsed) # Answer(value=4, confidence=0.99)

A plain dict (JSON schema) is also accepted in place of a Pydantic model; the output is returned as a plain Python object. Parse failures are logged as warnings; output_parsed is None if parsing fails.

By default, LMClient retries transient provider errors up to 3 times with exponential backoff. This covers 429 rate-limit spikes, 503 unavailability, 500 server errors, network failures, and timeouts.

client = LMClient(
    model="openai/gpt-4.1-mini",
    max_retries=3,   # default; set to 0 to disable
)

Backoff formula: min(2 ** attempt, 60) seconds plus up to 0.5 s jitter. If the provider returns a Retry-After header its value is used instead (capped at 60 s). Non-transient errors (BadRequestError, AuthenticationError, etc.) are not retried.

result = client.generate("Hello")
print(result.metrics.retries)   # 0 on first-attempt success

CLI flag: --max-retries.

Set a default timeout for every request at construction time:

client = LMClient(
    model="openai/gpt-4.1-mini",
    timeout=30.0,   # seconds
)

Any LiteLLM keyword argument passed to a generate / embed / transcribe call overrides the default for that request:

result = client.generate("Hello", timeout=5.0, max_tokens=64)

Use default_request_kwargs to set persistent overrides for all requests:

client = LMClient(
    model="openai/gpt-4.1-mini",
    default_request_kwargs={"max_tokens": 256, "temperature": 0.7},
)

infermesh bench measures client-side throughput across a concurrency sweep. It is intentionally a client benchmark — it tells you the best max_parallel_requests setting for your workload, not the server's maximum capacity.

infermesh bench generate \
  --model openai/gpt-4.1-mini \
  --api-base https://api.openai.com/v1 \
  --prompt "Write a haiku." \
  --warmup 5 \
  --requests 50 \
  --output-json bench.json

Output:

c=1    rps=3.14  p50=0.401s  p95=0.412s  p99=0.420s  svc_p95=0.410s  q_p95=0.001s  err=0/50  elapsed=15.9s
c=2    rps=5.81  p50=0.470s  p95=0.487s  p99=0.501s  svc_p95=0.480s  q_p95=0.002s  err=0/50  elapsed=8.6s
recommended_max_parallel_requests=8

c is the concurrency level. svc_p95 is the P95 of net provider response time (excluding queue wait). q_p95 is the P95 time a request spent in the client queue. High q_p95 relative to svc_p95 means the client is the bottleneck, not the server.

Use --input-jsonl to benchmark with a real prompt distribution. An embedding benchmark is available as infermesh bench embed.

For server-centric metrics (TTFT, TPOT, ITL, request goodput), use a dedicated server benchmark: vLLM · SGLang · AIPerf