AgniPod Python SDK

AgniPod is a fully managed internal LLM inference platform. You name a model; the platform provisions GPUs, schedules the work, routes it, retries on failure, records metrics and reclaims idle capacity. There is no "pick a provider", no "start an instance", no infrastructure surface.

pip install agnipod

import agnipod

client = agnipod.AgniPod()          # reads AGNIPOD_API_KEY

r = client.generate.create(
    model="qwen3:4b",
    prompt="Name three primary colours.",
    max_tokens=512,
    enable_thinking=False,          # fast, direct answer
)
print(r.content)                    # the answer
print(r.reasoning)                  # thinking (empty when disabled)
print(r.finish_reason, r.truncated) # "stop" / "length", bool
print(r.metrics["tokens_per_second"])

Credentials

Two, and they are not interchangeable.

There is no username and password. The token is issued out of band; agnipod login verifies it against the platform and stores it at ~/.agnipod/credentials.json with mode 0600, so it never has to live in shell history or a dotfile:

$ agnipod login                 # prompts (no echo), verifies, stores
$ agnipod login --admin         # the console JWT, for client.admin
$ agnipod status                # what is configured, and where it came from
$ agnipod logout                # remove it again

Resolution order, for both credentials:

AgniPod(api_key=…, admin_token=…)      explicit, always wins
AGNIPOD_API_KEY / AGNIPOD_ADMIN_TOKEN  environment
~/.agnipod/credentials.json            agnipod login

The environment deliberately outranks the stored file — otherwise an agnipod login a developer ran months ago would silently override what CI exports, which is exactly the "works here, 401 there" failure the ordering exists to prevent. agnipod status names the winning source for that reason.

client = agnipod.AgniPod(api_key="…", admin_token="…")

# Rotating an admin token mid-process, without rebuilding the client:
client.admin.use_token("<new jwt>")

The separation is the safety property: an API-token holder can run inference and submit batches, and only an admin token can terminate instances, delete models or disable a provider. AGNIPOD_BASE_URL overrides the endpoint.

Inference

# Single prompt
r = client.generate.create(model="qwen3:4b", prompt="…", max_tokens=1024)

# Multi-turn. History is managed for you: if the transcript would exceed the
# instance's context window, the oldest user/assistant pairs are dropped
# (system messages and the latest turn are always kept) so the request fits.
c = client.chat.create(model="qwen3:4b", messages=[
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hi"},
])

Sampling parameters (temperature, top_p, top_k, min_p, repeat_penalty, presence_penalty, frequency_penalty, seed, stop) are ordinary keyword arguments.

enable_thinking — reasoning models (Qwen3, DeepSeek-R1) think before answering. False gives a fast, direct answer; omit it for the model's trained default. On a short max_tokens budget the monologue can consume the whole response, so it is worth setting deliberately.

Recoverable conditions arrive as HTTP 200 with an error object, so a caller can retry rather than treat a transient state as a failure:

r = client.generate.create(model="qwen3:4b", prompt="…")
if r.error:
    print(r.error["type"])   # service_unavailable | context_expanding | …

Streaming is not offered, because the platform does not implement it. The parameter used to exist and was silently ignored, which is worse than its absence.

Batches

batch = client.batches.create(
    items=[{"custom_id": "1", "model": "qwen3:4b", "prompt": "…"}],
    batch_name="nightly-scoring",
    priority=5,                      # 1 urgent … 9 whenever
    deadline="2026-08-01 02:00:00",  # escalates priority as it nears
    not_before="2026-07-31 22:00:00" # park for an off-peak window
)

client.batches.wait(batch.batch_id, on_progress=lambda p: print(p.status))

raw = client.batches.results(batch.batch_id)          # bytes
client.batches.results(batch.batch_id, save_to="out.jsonl")   # or stream to disk

Also accepts file_path= (read locally) or url= (fetched server-side). Max 10,000 items inline.

Durability. A batch's input file is its work queue: as each result comes back the item is removed from the input and appended to the output. If the GPU serving it dies, the platform re-provisions and continues from the remaining items — completed work is never re-run, and the delivered file has exactly one result per custom_id.

Administration

client.admin covers every endpoint the operations console uses — the same routes, so the two can never drift.

admin = client.admin

# Is anything wrong right now? One database round trip; no provider API is
# called, so this stays fast when a provider is not.
o = admin.overview()
for a in o.critical:
    print(a["message"])

# Can the platform actually serve a request? Distinct from a liveness check:
# a revoked provider credential leaves /health green while provisioning is
# impossible.
for check in admin.diagnostics().failures:
    print(check["name"], check["message"])

A few behaviours worth knowing:

# Serving, billed, and excluded from routing — the state no status column
# reveals and the one most worth acting on.
for i in admin.instances.list():
    if i.stranded:
        admin.instances.destroy(i.id)

# Enabled is not the same as usable: a row with no driver, an open circuit
# breaker or an exhausted balance is enabled and still never rented from.
for p in admin.providers.list():
    if not p.usable:
        print(p.name, p.health)

# Batch traffic is sampled, so a row count is not a request count. Both are
# reported rather than one being passed off as the other.
page = admin.requests.list(success=False, window_hours=24)
print(page.total, "rows /", page.executions, "executions")

# Only what you pass is written — an absent key means "leave alone", so this
# cannot clobber a concurrent edit.
admin.models.update(7, context_length=16384, has_template=False)

Runnable scripts are in examples/: inference, batches, operations, and a model lifecycle walkthrough.

CLI

agnipod status         # which credentials are configured
agnipod models         # what can be named as `model`
agnipod health         # fleet, spend and anomalies      (admin token)
agnipod diagnostics    # every dependency, with remedies (admin token)

Errors

Everything derives from AgniPodError, so one except catches the lot.

Transient statuses (429, 5xx) are retried automatically with exponential back-off; max_retries controls how many times.

What the platform does for you

• Provisioning & selection. GPUs are chosen by measured throughput-per-dollar (memory bandwidth is the decode bottleneck), preferring cheap community-cloud capacity, tuned by each GPU's historical efficiency and startup reliability — so scheduling improves over time.
• Context sizing. Instances are provisioned at the model's context and each request uses only what it needs; the instance grows its window on demand.
• Concurrency. Each instance runs exactly its GPU's decode slots; interactive requests always keep a reserved slot so a large batch never blocks them.
• Failure handling. Stuck or dead instances are detected and replaced; in-flight batches resume; nothing is left billing.
• Cost control. Idle instances are reclaimed lowest-performer-first; no persistent storage is rented; a per-hour price ceiling is enforced.

See ARCHITECTURE.md in the platform repository for the internals.