Foundry Local Python SDK (v2)

The Foundry Local Python SDK is a native Python binding for the Foundry Local C++ SDK. It lets you discover, download, load, and run inference against local AI models — chat completions (streaming and non-streaming), tool calling, embeddings, and audio transcription — directly in-process via a cffi binding to the Foundry Local native library. No separate service, no HTTP hop.

Features

• Model Catalog – browse and search the Foundry Local model catalog
• Model Management – download, cache, load, and unload models
• Chat Completions – streaming and non-streaming, with first-class tool calling
• Embeddings – text embeddings via a typed tensor API
• Audio Transcription – speech-to-text (offline and live streaming)
• Explicit EP Management – discover, download, and register execution providers on demand
• Built-in Web Service – optional HTTP endpoint for multi-process scenarios
• Native Performance – cffi (API mode) binding to the Foundry Local native library

Installation

pip install foundry-local-sdk

The wheel includes the Foundry Local native library and depends on onnxruntime and onnxruntime-genai-core on every platform. Windows wheels also include the reg-free WinML 2.x runtime.

Building from source

The Python SDK is a cffi binding compiled against sdk_v2/cpp/include/foundry_local/foundry_local_c.h. The compiled extension calls into foundry_local.{dll,so,dylib} at runtime via dlopen — the native library must already exist before the wheel is usable. Build it first:

# Windows
sdk_v2\cpp\build.bat --skip_tests

# Linux / macOS
sdk_v2/cpp/build.sh --skip_tests

The output lands where lib_loader.py will discover it automatically:

• Windows (multi-config MSBuild): sdk_v2/cpp/build/Windows/<Config>/bin/<Config>/foundry_local.dll
• Linux: sdk_v2/cpp/build/Linux/<Config>/bin/libfoundry_local.so
• macOS: sdk_v2/cpp/build/macOS/<Config>/bin/libfoundry_local.dylib

To override the lookup, set FOUNDRY_LOCAL_LIB_DIR to a directory that contains foundry_local.{dll,so,dylib}.

Then build the wheel:

cd sdk_v2/python
python -m build --wheel

For editable installs during development:

pip install -e .

Installing native runtime dependencies for development / CI

foundry-local-install installs the published wheel and verifies that onnxruntime and onnxruntime_genai_core are available.

foundry-local-install

# Add --verbose to print resolved binary paths after installation.

Do not run this against a source-build / editable install. It will overwrite your pip install -e . (or any locally-built wheel install) with the published PyPI version. The source-build flow above (pip install -e . or pip install <local.whl>) already pulls the matching ORT and GenAI runtime packages via pyproject dependencies — no extra step needed.

Requirements

• Python 3.11 or newer (single cp311-abi3 wheel works on every CPython ≥ 3.11)
• Windows (x64), Linux (x64), or macOS (arm64)

Quick start

from foundry_local_sdk import (
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    TextItem,
)

# 1. Initialize
config = Configuration(app_name="MyApp")
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

# 2. Pick and load a model
model = manager.catalog.get_model("qwen2.5-0.5b")
model.download(lambda pct: print(f"\rDownloading: {pct:.1f}%", end="", flush=True))
print()
model.load()

# 3. Run a chat request through a typed session
with ChatSession(model) as session:
    session.set_options(RequestOptions(search=SearchOptions(temperature=0.0, max_output_tokens=128)))

    with Request().add_item(MessageItem.user("Why is the sky blue?")) as req:
        with session.process_request(req) as response:
            for item in response:
                if isinstance(item, TextItem):
                    print(item.text)

# 4. Cleanup
model.unload()

Runnable end-to-end examples live under samples/python/.

Usage

Initialization

Create a Configuration and initialize the singleton FoundryLocalManager.

from foundry_local_sdk import Configuration, FoundryLocalManager, LogLevel

config = Configuration(
    app_name="MyApp",
    model_cache_dir="/path/to/cache",                # optional
    log_level=LogLevel.INFORMATION,                   # optional (default: Warning)
)
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance

Discovering models

catalog = manager.catalog

# List all models in the catalog
models = catalog.list_models()

# Get a specific model by alias
model = catalog.get_model("qwen2.5-0.5b")

# Get a specific variant by ID
variant = catalog.get_model_variant("qwen2.5-0.5b-instruct-generic-cpu:4")

# Locally cached / currently loaded
cached = catalog.get_cached_models()
loaded = catalog.get_loaded_models()

Inspecting model metadata

IModel exposes metadata properties from the catalog:

model = catalog.get_model("phi-3.5-mini")

# Identity
print(model.id)             # e.g. "phi-3.5-mini-instruct-generic-gpu:3"
print(model.alias)          # e.g. "phi-3.5-mini"

# Context and token limits
print(model.context_length)  # e.g. 131072 (tokens), or None if unknown

# Modalities and capabilities
print(model.input_modalities)        # e.g. "text" or "text,image"
print(model.output_modalities)       # e.g. "text"
print(model.capabilities)            # e.g. "chat,completion"
print(model.supports_tool_calling)   # True, False, or None

# Cache / load state
print(model.is_cached)
print(model.is_loaded)

Explicit EP management

# Discover available EPs and registration status
eps = manager.discover_eps()
for ep in eps:
    print(f"{ep.name} - registered: {ep.is_registered}")

# Download and register all available EPs
result = manager.download_and_register_eps()
print(f"Success: {result.success}, Status: {result.status}")

# Download only specific EPs
result2 = manager.download_and_register_eps([eps[0].name])

Per-EP download progress

Pass a progress_callback to receive (ep_name, percent) updates as each EP downloads (percent is 0–100):

current_ep = ""

def on_progress(ep_name: str, percent: float) -> None:
    global current_ep
    if ep_name != current_ep:
        if current_ep:
            print()
        current_ep = ep_name
    print(f"\r  {ep_name}  {percent:5.1f}%", end="", flush=True)

manager.download_and_register_eps(progress_callback=on_progress)
print()

Catalog access does not block on EP downloads. Call download_and_register_eps() when you need hardware-accelerated execution providers.

Chat completions with ChatSession

from foundry_local_sdk import (
    ChatSession, MessageItem, Request, RequestOptions, SearchOptions, TextItem,
)

model = manager.catalog.get_model("qwen2.5-0.5b")
model.load()

with ChatSession(model) as session:
    session.set_options(RequestOptions(search=SearchOptions(temperature=0.0, max_output_tokens=256)))

    # Non-streaming
    with Request().add_item(MessageItem.user("What is 7 multiplied by 6?")) as req:
        with session.process_request(req) as response:
            for item in response:
                if isinstance(item, TextItem):
                    print(item.text)

    # Streaming — yields Item instances as the model produces them
    session.set_streaming(True)
    with Request().add_item(MessageItem.user("Tell me a joke")) as req:
        for item in session.process_streaming_request(req):
            if isinstance(item, TextItem):
                print(item.text, end="", flush=True)
    print()

model.unload()

ChatSession is stateful across turns. session.turn_count reports how many requests have been processed; session.undo_turns(n) rewinds history.

Multi-turn conversations

Each call to process_request extends the session's turn history. Build a new Request per turn:

with ChatSession(model) as session:
    for prompt in ["Hi!", "What's your favorite color?", "Why?"]:
        with Request().add_item(MessageItem.user(prompt)) as req:
            with session.process_request(req) as resp:
                for item in resp:
                    if isinstance(item, TextItem):
                        print(f"> {prompt}\n{item.text}\n")

Tool calling

Register tool definitions on the session, then watch for ToolCallItem in the response and reply with ToolResultItem:

import json
from foundry_local_sdk import ToolCallItem, ToolResultItem

with ChatSession(model) as session:
    session.add_tool_definition(
        name="get_weather",
        description="Get the current weather for a city.",
        json_schema=json.dumps({
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        }),
    )

    with Request().add_item(MessageItem.user("What's the weather in Seattle?")) as req:
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, ToolCallItem):
                    args = json.loads(item.arguments)
                    result = {"temperature_c": 12, "conditions": "rain"}
                    # Send the result back on the next turn
                    with Request().add_item(
                        ToolResultItem(call_id=item.call_id, content=json.dumps(result))
                    ) as follow_up:
                        with session.process_request(follow_up) as final:
                            for it in final:
                                if isinstance(it, TextItem):
                                    print(it.text)

Embeddings with EmbeddingsSession

EmbeddingsSession accepts TextItem inputs and returns one TensorItem per input containing the embedding vector. Sessions are stateless — reuse one session for many requests.

from foundry_local_sdk import EmbeddingsSession, Request, TensorItem, TextItem

model = manager.catalog.get_model("qwen3-embedding-0.6b")
model.load()

with EmbeddingsSession(model) as session:
    # Single input
    with Request().add_item(TextItem("The quick brown fox")) as req:
        with session.process_request(req) as resp:
            tensor = next(it for it in resp if isinstance(it, TensorItem))
            print("Dimensions:", tensor.dimensions)
            print("First 5:", tensor.data[:5])

    # Batch input — one TextItem per string, one TensorItem out per input
    with Request() as req:
        for text in ["Machine learning", "Capital of France", "Rust language"]:
            req.add_item(TextItem(text))
        with session.process_request(req) as resp:
            for item in resp:
                if isinstance(item, TensorItem):
                    print(f"  dims={item.dimensions}")

model.unload()

Audio transcription

AudioSession accepts AudioItem input (PCM bytes + sample rate / channels) and produces TextItem output. See samples/python/audio-transcription/ and live-audio-transcription/ for runnable end-to-end examples covering offline files and live PCM streaming through an ItemQueue.

Web service (optional)

Start a built-in HTTP server for multi-process access:

manager.start_web_service()
print(f"Listening on: {manager.urls}")

# ... use the service ...

manager.stop_web_service()

API Reference

Manager and configuration

Sessions

All sessions wrap a native flSession* and are context managers. Closing a session releases the native handle and aborts any in-flight streaming request.

Common session methods:

• process_request(request) -> Response — run synchronously, return the full response.
• process_streaming_request(request) -> Iterator[Item] — yield items as the model produces them. Requires set_streaming(True) first. Abandoning the iterator (break, exception, gen.close()) automatically cancels the request and joins the worker thread.
• set_options(RequestOptions) — apply session-level inference parameters (typed SearchOptions for sampling, optional tool_choice, and additional_options for passthrough).
• set_streaming(enabled) — install or remove the native streaming callback.

Requests and responses

Items

Item is the wire-format type for everything that flows in or out of a session — message turns, individual modality parts, tool calls, embeddings.

Enums: ItemType, TextItemType, MessageRole, TensorDataType.

CLI entry point

Running tests

pip install -r requirements-dev.txt
python -m pytest test/ -v

See test/README.md for detailed test setup and structure.

License

MIT — see LICENSE.txt.

Links

• Foundry Local on GitHub
• Issues