Glueco LLM Plugin

Typed LLM chat completions for the Glueco Gateway SDK.

Installation

pip install glueco-plugin-llm

Quick Start

from glueco_sdk import GatewayClient, FileKeyStorage, FileConfigStorage
from glueco_plugin_llm import llm_client

# Setup gateway client
client = GatewayClient(
    key_storage=FileKeyStorage(".gateway/keys.json"),
    config_storage=FileConfigStorage(".gateway/config.json"),
)

# Get transport and create LLM client
transport = client.get_transport()
llm = llm_client(transport)

# Make chat completion
response = llm.chat_completions(
    provider="groq",
    model="llama-3.1-8b-instant",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
    ],
    temperature=0.7,
)

print(response.content)

Streaming

for chunk in llm.chat_completions_stream(
    provider="groq",
    model="llama-3.1-8b-instant",
    messages=[{"role": "user", "content": "Tell me a story"}],
):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Supported Providers

Types

from glueco_plugin_llm import (
    ChatMessage,
    ChatCompletion,
    ChatCompletionChoice,
    ChatCompletionUsage,
)

# Create typed messages
messages = [
    ChatMessage(role="system", content="You are helpful."),
    ChatMessage(role="user", content="Hello!"),
]

# Response is fully typed
response: ChatCompletion = llm.chat_completions(...)
print(response.id)
print(response.model)
print(response.choices[0].message.content)
print(response.usage.total_tokens)

API Reference

llm_client(transport)

Factory function to create an LLM client.

LLMClient

llm.chat_completions(
    provider: str,          # "openai", "groq", "gemini"
    model: str,             # Model name
    messages: list,         # Conversation messages
    temperature: float,     # Optional (0.0-2.0)
    max_tokens: int,        # Optional
    response_format: dict,  # Optional
    **kwargs,               # Provider-specific params
) -> ChatCompletion

llm.chat_completions_stream(...) -> Iterator[ChatCompletionChunk]

License

MIT