OVAT: OpenVINO Agentic Toolkit

Build a tool-calling AI agent on an Intel AI PC from one YAML file and one command.

pip install ovat
ovat setup                                    # install the model server, once
ovat init workflow.yml
ovat serve workflow.yml                       # start it
ovat run workflow.yml -i "what do my notes say about Q3?"

Everything runs locally on your own hardware, through OpenVINO Model Server. No API keys, no cloud, nothing leaving the machine.

GSoC 2026 · OpenVINO Project #18. Full agent on Windows and Linux with an Intel CPU/GPU/NPU; macOS is supported for development. See Platform support.

Contents

• Why OVAT
• Prerequisites · Install · Get a model
• Quickstart
• Examples: RAG, ReAct, audio + vision
• The workflow file
• Four engines, one config
• Tools · Telemetry
• Platform support · Troubleshooting
• Documentation

Why OVAT

A tool-calling agent against OVMS is ~50 lines of boilerplate every project rewrites: build the client, hand-write each tool's JSON schema, call, check finish_reason, dispatch the tool, append the result, loop, guard the iteration count, manage history.

OVAT makes that a config file:

model:
  name: Qwen3.5-4B-int4-ov
  device: GPU
  tool_parser: qwen3coder
tools:
  - name: search_docs
    type: builtin
agent:
  type: native
  max_iterations: 10

The loop, schemas, history and error handling are the toolkit's job now. Moving from a 16 GB GPU box to an 8 GB CPU laptop is a three-line edit: compare examples/workflow.yml with examples/minimal.yml.

Prerequisites

Linux: install Python and OVMS's system libraries first

A minimal Linux image (a container, a fresh VM, a server install) ships with no Python at all, and OVMS needs libxml2. Verified in a clean ubuntu:24.04 container, where python3, pip and venv are all absent:

# Ubuntu 22.04 / 24.04  (the releases Intel builds OVMS for)
sudo apt update && sudo apt install -y python3 python3-venv python3-pip libxml2 curl

# RHEL 9 / Rocky / Alma
sudo dnf install -y python3 python3-pip libxml2 curl

Ubuntu 26.04 and newer are not supported yet. There is no OVMS build for them, the package is named libxml2-16 rather than libxml2, and the ubuntu24 archive cannot start there because it needs that release's system libraries. ovat setup warns before downloading. Use 22.04 or 24.04.

On Linux the interpreter is python3, not python, and python3-venv is a separate package from python3 on Debian and Ubuntu. Then:

python3 -m venv .venv && source .venv/bin/activate

Everything after this point is the same on every platform.

Using GPU or NPU? Update the driver first, an old driver usually shows up as the device simply being absent from ovat doctor, not as an error.

On Windows also install the Visual C++ Redistributable, which OVMS needs in order to start at all.

NPU cannot do tool calling. Use device: CPU or GPU for agents. NPU is fine for embeddings and plain chat.

Install

pip install ovat

Extras, so you install only what you need:

pip install "ovat[langchain,llamaindex,openai-agents,tui]"

OpenVINO Model Server (Windows / Linux)

ovat serve, and every tool-calling run, needs OVMS. One command installs it:

ovat setup

That picks the right archive for your OS (and, on Linux, your distro), checks its SHA-256, and unpacks it into ~/.ovat/ovms - a folder OVAT already searches. Nothing is added to PATH and no environment variable is needed. Run it once per machine; it is safe to run again.

On macOS it prints why there is nothing to install and what to use instead — Intel ships Windows and Linux x86-64 builds only.

If you skip this step, ovat serve notices OVMS is missing and offers to do it for you. It never downloads unattended: with no terminal attached (CI, a pipe) it stops and tells you to run ovat setup.

curl -L https://github.com/openvinotoolkit/model_server/releases/download/v2026.2.1/ovms_windows_2026.2.1_python_on.zip -o ovms.zip
tar -xf ovms.zip

wget https://github.com/openvinotoolkit/model_server/releases/download/v2026.2.1/ovms_ubuntu24_2026.2.1_python_on.tar.gz
tar -xzvf ovms_ubuntu24_2026.2.1_python_on.tar.gz
sudo apt update && sudo apt install -y libxml2 curl

export LD_LIBRARY_PATH=${PWD}/ovms/lib
export PYTHONPATH=${PWD}/ovms/lib/python

export OVAT_OVMS=/path/to/ovms        # or set model.ovms_binary in the YAML

macOS has no OVMS build. See Platform support.

Get a model

ovat serve downloads one for you on the first run. To fetch it yourself (the only option on macOS):

hf download OpenVINO/Qwen3.5-4B-int4-ov --local-dir models/OpenVINO/Qwen3.5-4B-int4-ov

These are pre-converted OpenVINO IR, no conversion needed.

The 4B figures are measured on an Intel AI PC. The peak sits 2.2 GB above steady state, and the load spike is what decides whether a model fits, so 8 GB machines should prefer the 0.8B tier.

Qwen3.5 is a unified model: text, images and tool calling in one export, so all three examples share a single download.

RAG also needs an embedder, the one thing that has to be converted:

pip install "ovat[convert]"
optimum-cli export openvino --model BAAI/bge-small-en-v1.5 \
    --task feature-extraction models/bge-small-en-v1.5

Quickstart

ovat setup                                    # 1. install OVMS (once per machine)
ovat init workflow.yml                        # 2. write a starter config
ovat doctor workflow.yml                      # 3. check machine + config
ovat run workflow.yml -i "hi" --dry-run       # 4. no server, no model needed
ovat serve workflow.yml                       # 5. start OVMS
ovat run workflow.yml -i "summarise my notes" # 6. ask something
ovat serve workflow.yml --stop                # 7. shut it down

Step 5 takes a while the first time, it downloads the model. That is expected: serve shows elapsed time and only gives up after five minutes of no progress at all, so a slow link is fine.

Skipping step 1 is fine too - ovat serve will offer to install OVMS when it finds none.

On macOS there is no OVMS. Use the local path instead. ovat chat answers from an index, so it needs a config with a rag: section, which the example below provides:

git clone https://github.com/Lagmator22/ovat.git && cd ovat   # for the examples
hf download OpenVINO/Qwen3.5-0.8B-int4-ov --local-dir models/Qwen3.5-0.8B-int4-ov

pip install "ovat[convert]"                                   # for the embedder
optimum-cli export openvino --model BAAI/bge-small-en-v1.5 \
    --task feature-extraction models/bge-small-en-v1.5

ovat index ./examples/rag/docs examples/rag/workflow.yml
ovat chat examples/rag/workflow.yml -i "What is OVAT's memory budget?"

Examples

Each folder has its own README with the exact commands.

The examples are not included in the pip package, since they are project files rather than library code. Clone the repo to run them: git clone https://github.com/Lagmator22/ovat.git && cd ovat. Everything else in this README works from pip install ovat alone.

The workflow file

rag is optional. Without it search_docs answers in a documented stub mode, so every quickstart command works on a fresh install with nothing downloaded.

Four engines, one config

agent.type picks the loop; nothing else in your config changes.

• native. OVAT's own loop. Zero extra dependencies, and the only engine that records per-turn token counts.
• react. LangChain (create_agent + ChatOpenAI pointed at OVMS)
• llamaindex. LlamaIndex FunctionAgent + OpenAILike
• openai-agents. OpenAI Agents SDK compatibility model

Override for a single run without editing the file:

ovat run workflow.yml -i "..." --llamaindex
ovat run workflow.yml -i "..." --react              # or --langchain
ovat run workflow.yml -i "..." --openai-sdk         # or --openai-agents
ovat bench workflow.yml -i "..." --out report.json  # all four, side by side

bench prints build time, answer time, peak memory and token counts. A failing engine is a row, not a crash, and an engine that answered nothing scores not ok rather than a misleading green.

Tools

• search_docs, semantic search over your documents, returning source paths
• transcribe, speech-to-text via OpenVINO Whisper; set OVAT_WHISPER_MODEL
• describe_image, caption or answer questions about an image; set OVAT_VLM_MODEL

All three are also standalone MCP servers, so any MCP-aware agent can call them. And OVAT is an MCP client, point it at any server:

tools:
  - name: search_docs
    type: mcp_stdio
    command: ["python", "-m", "ovat.tools.search_docs", "--config", "workflow.yml"]
  - name: anything_else
    type: mcp_stdio
    command: ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/docs"]

Pass --config when serving search_docs over MCP. That server runs in its own process and builds its own retriever; without a config it stays in stub mode.

Telemetry

ovat run workflow.yml -i "..." --trace trace.json      # one run's trace
ovat run workflow.yml -i "..." --telemetry live.jsonl  # sampled alongside
ovat telemetry                                         # live table, no run

Two rules the numbers follow:

• Unknown stays unknown. If the server reports no token counts the field is null, not 0, a zero would read as "used no tokens".
• An unavailable source says why, instead of reporting zeros.

--trace peak RSS measures the OVAT process. With OVMS serving, the model lives in ovms.exe, so measure that instead (Task Manager, or Get-Process ovms). --trace is the right number for ovat chat, where the model runs in-process.

Platform support

OVMS is x86-64 with no macOS build. For day-to-day macOS work, ovat chat runs a local openvino_genai model and needs no server at all.

Troubleshooting

ovat doctor <config> is the fastest diagnosis for anything else.

Documentation

Development

git clone https://github.com/Lagmator22/ovat.git && cd ovat
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate.bat
pip install -e ".[dev]"
pytest -q                    # ~550 tests, no server needed
pytest -m live               # against a running OVMS (AI PC only)

live needs OVMS; rag needs bge-small on disk. Both auto-skip, so a fresh clone runs green.

Licensed under Apache 2.0.