olive-compute 0.1.2

Python client for the Olive distributed AI compute platform

Olive Python SDK

Distributed AI compute — embeddings and inference — with one import.

pip install olive-compute

Quickstart

from olive import OliveClient

client = OliveClient(api_key="olv_...")

# Embed text — uses the default embeddings model
vectors = client.embeddings(["hello world", "olive compute"])
print(vectors[0][:4])  # [0.0521, -0.1234, ...]

# Run inference — uses the default chat model
reply = client.inference("What is a neural network?", max_tokens=128)
print(reply)

Choosing a model

Olive supports a catalog of curated open-source models. Browse them at olivecompute.com/models or programmatically:

# List all available chat models
for m in client.list_models(modality="chat"):
    print(m["id"], "—", m["pricing"]["input_per_1m_tokens_usd"], "/1M tokens")

# Get one model's full record
m = client.get_model("meta/llama-3.2-3b-instruct")
print(m["description"])

Pass model= to any inference call to pin a specific model:

reply = client.inference(
    "Write a Python function to reverse a list.",
    model="meta/llama-3.2-3b-instruct",
)

vectors = client.embeddings(
    ["semantic search query"],
    model="baai/bge-small-en-v1.5",
)

If model= is omitted, Olive picks the default (featured) model for the workload.

Authentication

Get an API key from provider.olivecompute.com → Settings → API Keys.

# API key (recommended)
client = OliveClient(api_key="olv_...")

# Email + password (issues a short-lived token automatically)
client = OliveClient(email="[e-mail]", password="...")

Compute tiers

Tier	CPU	RAM	Use case
"light"	1 core	2 GB	Embeddings, small inputs
"medium"	2 cores	4 GB	Standard inference (default)
"heavy"	4 cores	8 GB	Long context, large batches

Async jobs

For long-running workloads, submit and poll separately:

import json

job = client.submit_job(
    workload_type="inference",
    input_data=json.dumps({"prompt": "Write a haiku", "max_tokens": 64}),
    model="meta/llama-3.2-3b-instruct",   # optional — default chat model otherwise
    compute="medium",
)
print(job.id)       # e3b2a1c0-...
print(job.status)   # "running"

# No explicit timeout: waits long enough to cover one automatic retry
# if the assigned device doesn't respond in time.
result = job.wait()
print(json.loads(result["output_data"])["text"])

Error handling

from olive import OliveClient, AuthError, JobError

try:
    client = OliveClient(api_key="olv_bad_key")
    vectors = client.embeddings(["test"])
except AuthError:
    print("Check your API key")
except JobError as e:
    print(f"Job failed: {e}")

Context manager

with OliveClient(api_key="olv_...") as client:
    vectors = client.embeddings(["hello"])