sthai-py

A Python client for the SiteHost AI Platform: inference, embeddings and reranking with typed requests and responses built on msgspec. Check out our PHP client port as well!

The SiteHost AI Platform

The SiteHost AI Platform serves capable open-weight models at their full context windows - not heavily quantised cut-downs - with full control over system prompts and outputs. Everything runs on SiteHost's own hardware in their own New Zealand data centres, so your data never leaves the country, and request bodies are never stored: only usage metrics are kept for billing and performance monitoring.

Models are stable targets, too: each served model has a minimum one-year retention window and at least three months' deprecation notice, with guidance on any adjustments needed.

The platform currently serves three models, one per capability (see the models page for the source of truth):

Qwen 3.6 27B (Qwen/Qwen3.6-27B) is deprecated, and Qwen 3.8 27B replaces it as the default inference model. The platform still accepts Qwen/Qwen3.6-27B as an alias, but models() no longer lists it. InferenceModel.QWEN_3_6_27B stays available until the alias is retired - see the models page for the timeline.

Installation

Requires Python 3.10+. Install from PyPI:

pip install sthai
# or, in a uv project
uv add sthai

Getting started

Create an API key in the SiteHost Control Panel (see API keys). The client reads it from the STHAI_KEY environment variable, or you can pass api_key= explicitly:

from sthai import Client

client = Client()  # or Client(api_key="...")
response = client.chat("What's the tallest mountain in New Zealand?")
print(response.output().text)

Usage

Chat and history

chat() keeps a conversation going: each successful call appends the user and assistant turns to the client's history, and later calls send it back. The system prompt is applied per call rather than stored.

client.chat("I'm planning a tramping trip to Fiordland.")
client.chat("What should I pack?")  # the model sees the earlier turn

client.chat("Standalone question.", use_history=False)  # neither sends nor records
client.clear_history()

client.chat("Be brief: why is the sky blue?", system_prompt="You are terse.")

The stored conversation is accessible: history() returns the turns as role/content dicts, and set_history() replaces them, for restoring a persisted conversation. set_write_history() toggles recording after construction; turning it off keeps the stored history and still sends it, only recording stops. history_usage() sums token usage across the calls that built the stored history. Each call resends the conversation so far, so input tokens count what the server processed (as billed), not unique tokens.

saved = client.history()          # role/content dicts, oldest first
client.set_history(saved)         # restore a persisted conversation
client.set_write_history(False)   # stop recording; stored turns still sent

print(client.history_usage().input_tokens)  # summed across the conversation

Thinking models can reason before answering; the reasoning rides along on the response:

response = client.chat("What is 17 * 23?", use_thinking=True, max_tokens=2000)
print(response.output().reasoning)  # or client.last_reasoning()
print(response.output().text)

Images

Chat and embedding inputs can include images, given as URLs or as local files/bytes (PNG, JPEG, GIF or WEBP - files are inlined as data URIs):

from pathlib import Path

client.chat("What's in this image?", image_files=[Path("photo.png")])
client.chat("Compare these.", image_urls=["https://example.com/a.jpg", "https://example.com/b.jpg"])

One-off and structured responses

response() mirrors chat() without the back-and-forth: the stored history is neither sent nor updated. Pass response_type= to get structured output - a msgspec Struct becomes a JSON schema the server enforces during generation, and the decoded, validated instance is returned:

from msgspec import Struct

class CityInfo(Struct):
    name: str
    country: str
    population: int

city = client.response(
    "Give me basic facts about Wellington.",
    response_type=CityInfo,
)
print(city.population)

data = client.response("List three NZ birds as JSON.", response_type=dict)

If the output is cut off by the token limit, or (rarely, with thinking enabled) the server skips the schema, parsing raises a sthai.ResponseParseError naming the cause.

Embeddings

embed() turns one input - text, images, or both - into a single vector, the sole entry in the response's output(). The embedding model is instruction-trained: document embedding is the default, and query=True switches to the query instruction for search-style lookups. dimensions= truncates the vector server-side (Matryoshka - powers of two work best):

vector = client.embed("The Beehive is New Zealand's parliament building.").output()[0]
query_vector = client.embed("Where does NZ parliament sit?", query=True).output()[0]
small = client.embed("Compact vector, please.", dimensions=512).output()[0]

batch_embed() embeds many texts in one request; the response's output() is one vector per text in order:

vectors = client.batch_embed([
    "Wellington is the capital of New Zealand.",
    "Auckland is the largest city in New Zealand.",
]).output()

Reranking

rerank() scores each document against a query; output() on the response is the results sorted by relevance, with each result's index mapping back to your input list:

response = client.rerank(
    "What is the capital of New Zealand?",
    [
        "The capital of New Zealand is Wellington.",
        "Auckland has the largest population in New Zealand.",
        "The All Blacks are New Zealand's national rugby team.",
    ],
    top_n=2,
)
for result in response.output():
    print(result.relevance_score, result.document.text)

Pass instruction= to steer relevance for a specific task; the model applies a sensible default otherwise.

Response helpers

Every inference, embedding and rerank call returns the full response struct, and every response struct has usage() (input/output/cached token counts) and output() (the useful payload):

response = client.chat("Hello!")
print(response.usage().input_tokens, response.usage().output_tokens)

embedded = client.embed("Hello!")
print(embedded.usage().input_tokens, len(embedded.output()[0]))

The one exception is response(response_type=...), which returns the parsed value directly; the full struct from the most recent inference call remains available via last_response():

city = client.response("Describe Wellington.", response_type=CityInfo)
print(client.last_response().usage().input_tokens)

Sessions

Pinning requests to a server session keeps routing consistent and helps caching. Pass session_pin= with your own identifier, or let the client generate one:

client = Client(auto_session=True)
print(client.session_pin())  # the pin in use; None when unpinned

client.set_session_pin("my-own-pin")  # pin subsequent requests; None unpins
pin = client.new_session()  # switch to a freshly generated pin

Models and health

client.healthy()  # True if the server is up
for card in client.models():
    print(card.id)

Async

AsyncClient mirrors Client method-for-method, with every network method awaitable. Like Client it is session-less: construct it and go - there is no async with or close() to manage.

import asyncio
from sthai import AsyncClient

async def main() -> None:
    client = AsyncClient()
    response = await client.chat("What's the tallest mountain in New Zealand?")
    print(response.output().text)

asyncio.run(main())

Async is most useful for concurrent fan-out, such as asyncio.gather over many embed() or rerank() calls.

Error handling

Every error the client raises subclasses sthai.SthaiError, so catching it is enough to handle anything sthai can raise - you never need to import niquests or msgspec:

from sthai import APIError, Client, ClientError, InputError, ResponseError, TransportError

client = Client()
try:
    response = client.chat("hello", model="no-such-model")
except ClientError as exc:
    print(exc.status_code, exc.error_type, exc)  # 404 invalid_request_error 404: model not found (invalid_request_error)
except APIError:
    ...  # 5xx: the server had a problem
except TransportError:
    ...  # the request never completed: connection failure, timeout, TLS error
except InputError:
    ...  # bad arguments to a client method
except ResponseError:
    ...  # the server returned something the client couldn't use

ClientError (4xx) and APIError (5xx) both subclass APIStatusError, which carries status_code, response, server_message and error_type (parsed from the server's error body when present). ResponseParseError, raised by response()'s structured-output parsing, subclasses ResponseError.

Development

Development uses uv. The test suite runs entirely offline against fixtures captured from the live API:

git clone https://github.com/sitehostnz/sthai-py.git
cd sthai-py
uv sync
uv run pytest
uv run ruff check sthai/ tests/
uv run ruff format --check sthai/ tests/
uv run ty check sthai/ tests/

To refresh the fixtures against the live API (costs tokens, needs a real key), see tests/capture_fixtures.py.

Licence

MIT