Foundry Local Python SDK (v2)

The Foundry Local Python SDK is a native Python binding for the Foundry Local C++ SDK. It lets you discover, download, load, and run inference against local AI models — chat completions (streaming and non-streaming), tool calling, embeddings, and audio transcription — directly in-process via a cffi binding to the Foundry Local native library. No separate service, no HTTP hop.

Features

• Model Catalog and BYOM – browse the public catalog or register existing local model assets without copying them
• Model Management – download, cache, load, and unload models
• Chat Completions – streaming and non-streaming, with first-class tool calling
• Embeddings – text embeddings via a typed tensor API
• Audio Transcription – speech-to-text (offline and live streaming)
• Explicit EP Management – discover, download, and register execution providers on demand
• Built-in Web Service – optional HTTP endpoint for multi-process scenarios
• Native Performance – cffi (API mode) binding to the Foundry Local native library

Installation

pip install foundry-local-sdk

The wheel includes the Foundry Local native library and depends on onnxruntime and onnxruntime-genai-core on every platform. Windows wheels also include the reg-free WinML 2.x runtime.

Building from source

The Python SDK is a cffi binding compiled against sdk_v2/cpp/include/foundry_local/foundry_local_c.h. The compiled extension calls into foundry_local.{dll,so,dylib} at runtime via dlopen — the native library must already exist before the wheel is usable. Build it first:

# Windows
sdk_v2\cpp\build.bat --skip_tests

# Linux / macOS
sdk_v2/cpp/build.sh --skip_tests

The output lands where lib_loader.py will discover it automatically:

• Windows (multi-config MSBuild): sdk_v2/cpp/build/Windows/<Config>/bin/<Config>/foundry_local.dll
• Linux: sdk_v2/cpp/build/Linux/<Config>/bin/libfoundry_local.so
• macOS: sdk_v2/cpp/build/macOS/<Config>/bin/libfoundry_local.dylib

To override the lookup, set FOUNDRY_LOCAL_LIB_DIR to a directory that contains foundry_local.{dll,so,dylib}.

Then build the wheel:

cd sdk_v2/python
python -m build --wheel

For editable installs during development:

pip install -e .

Installing native runtime dependencies for development / CI

foundry-local-install installs the published wheel and verifies that onnxruntime and onnxruntime_genai_core are available.

foundry-local-install

# Add --verbose to print resolved binary paths after installation.

Do not run this against a source-build / editable install. It will overwrite your pip install -e . (or any locally-built wheel install) with the published PyPI version. The source-build flow above (pip install -e . or pip install <local.whl>) already pulls the matching ORT and GenAI runtime packages via pyproject dependencies — no extra step needed.

Requirements

• Python 3.11 or newer (single cp311-abi3 wheel works on every CPython ≥ 3.11)
• Windows (x64), Linux (x64), or macOS (arm64)

Quick start

from foundry_local_sdk import (
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    TextItem,
)

# 1. Initialize
config = Configuration(app_name="MyApp")
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

# 2. Pick and load a model
model = manager.catalog.get_model("qwen2.5-0.5b")
model.download(lambda pct: print(f"\rDownloading: {pct:.1f}%", end="", flush=True))
print()
model.load()

# 3. Run a chat request through a typed session
with ChatSession(model) as session:
    session.set_options(RequestOptions(search=SearchOptions(temperature=0.0, max_output_tokens=128)))

    with Request().add_item(MessageItem.user("Why is the sky blue?")) as req:
        with session.process_request(req) as response:
            for item in response:
                if isinstance(item, TextItem):
                    print(item.text)

# 4. Cleanup
model.unload()

Pass a threading.Event as cancel_event to cancel an active download at the next native progress checkpoint:

from threading import Event

cancel_event = Event()

def on_progress(percent: float) -> None:
    print(f"\rDownloading: {percent:.1f}%", end="", flush=True)
    cancel_event.set()

model.download(progress_callback=on_progress, cancel_event=cancel_event)

Runnable end-to-end examples live under samples/python/.

Usage

Initialization

Create a Configuration and initialize the singleton FoundryLocalManager.

from foundry_local_sdk import Configuration, FoundryLocalManager, LogLevel

config = Configuration(
    app_name="MyApp",
    model_cache_dir="/path/to/cache",                # optional
    log_level=LogLevel.INFORMATION,                   # optional (default: Warning)
)
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

Discovering models

catalog = manager.catalog

# List all models in the catalog
models = catalog.list_models()

# Get a specific model by alias
model = catalog.get_model("qwen2.5-0.5b")

# Get a specific variant by ID
variant = catalog.get_model_variant("qwen2.5-0.5b-instruct-generic-cpu:4")

# Locally cached / currently loaded
cached = catalog.get_cached_models()
loaded = catalog.get_loaded_models()

Bring your own model (BYOM)

Use the local catalog to register an existing ONNX Runtime GenAI model directory. Registration does not copy, download, or take ownership of the model assets. The directory must contain genai_config.json, and the model ID must use the canonical <name>:<version> format. The task metadata property is required and must be one of chat-completion, vision-language-chat, embeddings, or automatic-speech-recognition.

from foundry_local_sdk import CatalogType, ModelInfoBuilder

local_catalog = manager.get_catalog(CatalogType.LOCAL)

with ModelInfoBuilder() as metadata:
    metadata.set_string_property("task", "chat-completion")
    metadata.set_string_property("display_name", "My Local Model")
    metadata.set_int_property("context_length", 4096)
    model = local_catalog.register_model(
        model_path="/path/to/model-directory",
        model_id="my-model-generic-cpu:1",
        metadata=metadata,
    )

print(model.id)
print(model.info.get_string_property("display_name"))

# Existing model handles remain safe for metadata and cleanup queries after
# unregistration, while future catalog lookups no longer return the model.
local_catalog.unregister_model(model.id)

manager.catalog and manager.get_catalog() both return the public catalog for backward compatibility. ModelInfoBuilder() owns a mutable native metadata handle; call close() or use a with block. In contrast, model.info is a frozen point-in-time ModelInfo value that supports normal dataclass operations such as dataclasses.asdict(). Its declared fields remain safe after the native model or manager is released. Arbitrary forward-compatible keys use guarded native lookup and therefore require the owning manager to remain open.

Inspecting model metadata

IModel exposes metadata properties from the catalog:

model = catalog.get_model("phi-3.5-mini")

# Identity
print(model.id)             # e.g. "phi-3.5-mini-instruct-generic-gpu:3"
print(model.alias)          # e.g. "phi-3.5-mini"

# Context and token limits
print(model.context_length)  # e.g. 131072 (tokens), or None if unknown

# Modalities and capabilities
print(model.input_modalities)        # e.g. "text" or "text,image"
print(model.output_modalities)       # e.g. "text"
print(model.capabilities)            # e.g. "chat,completion"
print(model.supports_tool_calling)   # True, False, or None

# Cache / load state
print(model.is_cached)
print(model.is_loaded)

Explicit EP management

# Discover available EPs and registration status
eps = manager.discover_eps()
for ep in eps:
    print(f"{ep.name} - registered: {ep.is_registered}")

# Download and register all available EPs
result = manager.download_and_register_eps()
print(f"Success: {result.success}, Status: {result.status}")

# Download only specific EPs
result2 = manager.download_and_register_eps([eps[0].name])

Per-EP download progress

Pass a progress_callback to receive (ep_name, percent) updates as each EP downloads (percent is 0–100):

current_ep = ""

def on_progress(ep_name: str, percent: float) -> None:
    global current_ep
    if ep_name != current_ep:
        if current_ep:
            print()
        current_ep = ep_name
    print(f"\r  {ep_name}  {percent:5.1f}%", end="", flush=True)

manager.download_and_register_eps(progress_callback=on_progress)
print()

Catalog access does not block on EP downloads. Call download_and_register_eps() when you need hardware-accelerated execution providers.

Chat completions with ChatSession

from foundry_local_sdk import (
    ChatSession, MessageItem, Request, RequestOptions, SearchOptions, TextItem,
)

model = manager.catalog.get_model("qwen2.5-0.5b")
model.load()

with ChatSession(model) as session:
    session.set_options(RequestOptions(search=SearchOptions(temperature=0.0, max_output_tokens=256)))

    # Non-streaming
    with Request().add_item(MessageItem.user("What is 7 multiplied by 6?")) as req:
        with session.process_request(req) as response:
            for item in response:
                if isinstance(item, TextItem):
                    print(item.text)

    # Streaming — yields Item instances as the model produces them
    session.set_streaming(True)
    with Request().add_item(MessageItem.user("Tell me a joke")) as req:
        for item in session.process_streaming_request(req):
            if isinstance(item, TextItem):
                print(item.text, end="", flush=True)
    print()

model.unload()

ChatSession is stateful across turns. session.turn_count reports how many requests have been processed; session.undo_turns(n) rewinds history.

Before processing a request, use preflight_request() to synchronously ask the native runtime for the request's token budget:

model = manager.catalog.get_model("qwen2.5-0.5b")
model.load()
with ChatSession(model) as session:
    with Request().add_item(MessageItem.user("Summarize this conversation.")) as req:
        budget = session.preflight_request(req)
        print(f"Required tokens: {budget.required_tokens}; fits: {budget.fits}")
model.unload()

The SDK reports the native result without estimating tokens, changing options, truncating context, or rewriting the request. context_limit_tokens is the Engine's structural request capacity or the Generator's model context. fits does not reserve cache or guarantee immediate admission while other requests are active.

Multi-turn conversations

Each call to process_request extends the session's turn history. Build a new Request per turn:

with ChatSession(model) as session:
    for prompt in ["Hi!", "What's your favorite color?", "Why?"]:
        with Request().add_item(MessageItem.user(prompt)) as req:
            with session.process_request(req) as resp:
                for item in resp:
                    if isinstance(item, TextItem):
                        print(f"> {prompt}\n{item.text}\n")

Tool calling

Register tool definitions on the session, then watch for ToolCallItem in the response and reply with ToolResultItem:

import json
from foundry_local_sdk import ToolCallItem, ToolResultItem

with ChatSession(model) as session:
    session.add_tool_definition(
        name="get_weather",
        description="Get the current weather for a city.",
        json_schema=json.dumps({
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        }),
    )

    with Request().add_item(MessageItem.user("What's the weather in Seattle?")) as req:
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, ToolCallItem):
                    args = json.loads(item.arguments)
                    result = {"temperature_c": 12, "conditions": "rain"}
                    # Send the result back on the next turn
                    with Request().add_item(
                        ToolResultItem(call_id=item.call_id, content=json.dumps(result))
                    ) as follow_up:
                        with session.process_request(follow_up) as final:
                            for it in final:
                                if isinstance(it, TextItem):
                                    print(it.text)

For a custom tool, register only its name and description. Its ToolCallItem.arguments value is the raw text generated by the model, not JSON:

with ChatSession(model) as session:
    session.add_custom_tool_definition(
        name="apply_patch",
        description="Apply a patch supplied as raw unified-diff text.",
    )

    with Request().add_item(MessageItem.user("Change the greeting to hello.")) as req:
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, ToolCallItem) and item.name == "apply_patch":
                    patch_text = item.arguments
                    print(patch_text)

Tool names, descriptions, and function-tool JSON schemas must be valid UTF-8 strings without an embedded NUL ("\x00"), because the native C ABI represents them as NUL-terminated strings. Removal names have the same restriction. Passing an embedded NUL raises ValueError before native code is called.

Embeddings with EmbeddingsSession

EmbeddingsSession accepts TextItem inputs and returns one TensorItem per input containing the embedding vector. Sessions are stateless — reuse one session for many requests.

from foundry_local_sdk import EmbeddingsSession, Request, TensorItem, TextItem

model = manager.catalog.get_model("qwen3-embedding-0.6b")
model.load()

with EmbeddingsSession(model) as session:
    # Single input
    with Request().add_item(TextItem("The quick brown fox")) as req:
        with session.process_request(req) as resp:
            tensor = next(it for it in resp if isinstance(it, TensorItem))
            print("Dimensions:", tensor.dimensions)
            print("First 5:", tensor.data[:5])

    # Batch input — one TextItem per string, one TensorItem out per input
    with Request() as req:
        for text in ["Machine learning", "Capital of France", "Rust language"]:
            req.add_item(TextItem(text))
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, TensorItem):
                    print(f"  dims={item.dimensions}")

model.unload()

Audio transcription

AudioSession accepts AudioItem input (PCM bytes + sample rate / channels) and produces TextItem output. See samples/python/audio-transcription/ and live-audio-transcription/ for runnable end-to-end examples covering offline files and live PCM streaming through an ItemQueue.

Web service (optional)

Start a built-in HTTP server for multi-process access:

manager.start_web_service()
print(f"Listening on: {manager.urls}")

# ... use the service ...

manager.stop_web_service()

API Reference

Manager and configuration

Sessions

All sessions wrap a native flSession* and are context managers. Closing a session releases the native handle and aborts any in-flight streaming request.

Common session methods:

• process_request(request) -> Response — run synchronously, return the full response.
• process_streaming_request(request) -> Iterator[Item] — yield items as the model produces them. Requires set_streaming(True) first. Abandoning the iterator (break, exception, gen.close()) automatically cancels the request and joins the worker thread.
• set_options(RequestOptions) — apply session-level inference parameters (typed SearchOptions for sampling, optional tool_choice, and additional_options for passthrough).
• set_streaming(enabled) — install or remove the native streaming callback.

Requests and responses

Items

Item is the wire-format type for everything that flows in or out of a session — message turns, individual modality parts, tool calls, embeddings.

Enums: ItemType, TextItemType, MessageRole, TensorDataType.

CLI entry point

Running tests

pip install -r requirements-dev.txt
python -m pytest test/ -v

See test/README.md for detailed test setup and structure.

License

MIT — see LICENSE.txt.

Links

• Foundry Local on GitHub
• Issues