aitta-client 0.3.3

Python client for Aitta HPC ML inference platform

Python client for Aitta HPC ML inference platform

A Python client library for the Aitta ML inference platform for HPC systems.

IMPORTANT: Note that both the API and the client library are still under heavy development and while we try to keep changes mostly backwards-compatible, breaking changes may happen. Access to Aitta is currently restricted to selected beta users.

Main client API classes

AccessTokenSource

Used by the client to get (and eventually refresh) access tokens

Client

Implements all requests to the Aitta API servers on a low level and is used by all other classes.

Useful members:

• get_model_list(): Lists all models served by the API that can be accessed with the configured access credentials.

Model

Represents a model and provides methods to perform inference.

Useful members:

• load(model_id, client): Creates a Model instance given the models id, loading the relevant data from the Aitta API server.

Example usage

The below shows an example for usage of the Aitta API using the Python client library.

For accessing the Aitta API the client will need a way to obtain access tokens, which is implemented in the form of an AccessTokenSource. You can obtain an access token by authenticating with your CSC user account at the Generate token page (also accessible from the main website by following the "Generate token" link in the navigation bar after logging in).

With the token thus obtained, then have to create an instance of StaticAccessTokenSource for use with the client library.

Chat completion with the LumiOpen/Llama-Poro-2-70B-Instruct model

This example shows how to start a conversation with the model using OpenAI’s chat completion feature.

NOTE: It is recommended to specify the max_tokens value to a high enough number in order to avoid responses being cut out.

from aitta_client import Model, Client, StaticAccessTokenSource
import openai

# configure Client instance with API URL and access token
access_token = "token_for_given_model"
token_source = StaticAccessTokenSource(access_token)
aitta_client = Client("https://api-aitta.csc.fi", token_source)

# load the LumiOpen/Llama-Poro-2-70B-Instruct model
model = Model.load("LumiOpen/Llama-Poro-2-70B-Instruct", aitta_client)
print(model.description)

# configure OpenAI client to use the Aitta OpenAI compatibility endpoints
client = openai.OpenAI(api_key=token_source.get_access_token(), base_url=model.openai_api_url)
# perform chat completion with the OpenAI client
chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Write a novel about bear talking about how to combine HPC with QC."
        }
    ],
    model=model.id,
    max_tokens=8000,
    stream=False  # response streaming is currently not supported by Aitta
)

print(chat_completion.choices[0].message.content)