launch ccai with the --expose flag to accept inbound traffic:

ccai --expose

also, configure your operating system's firewall to allow inbound traffic on your chosen port (default is 8000)

tool definition format varies by model; a default create_tool_definition is provided in chat.py for gpt-oss

here is an example chat.ipynb using dynamic tool registration and execution:

async def create_chat_payload(self, nodes: list[dict]):
    # ...

    # add @tool definitions
    for attr_name in dir(self):
        attr = getattr(self, attr_name)
        if callable(attr) and hasattr(attr, '_metadata'):
            data['tools'].append(self.create_tool_definition(attr))

    return data

async def handle_tool_call(self, tool_call: dict):
    return await self.execute_tool(tool_call)

@dataclass
class WeatherToolParams:
    location: str = field(metadata={'description': 'a location'})

@tool({'description': 'gets the current weather in a given location'})
async def weather_tool(self, args: WeatherToolParams):
    return {
        'location': args.location,
        'temperature': f'77 degrees Fahrenheit (25 degrees Celsius)',
    }

llama.cpp is recommended; you can find their precompiled releases here: llama.cpp/releases

for example, a Windows user with an nvidia gpu would download and extract llama-b7000-bin-win-cuda-12.4-x64.zip

(and: if you don't already have CUDA installed, you should download and extract the .dll files from cudart-llama-bin-win-cuda-12.4-x64.zip to be adjacent to your llama-server.exe)

then run llama-server.exe with a configuration appropriate for your hardware and model, for example:

llama-server.exe --model C:\Users\Sky\Downloads\gpt-oss-20b-F16.gguf --host localhost --port 8080 --threads 6 --ctx-size 32768 --flash-attn on --jinja --n-gpu-layers 24 --n-cpu-moe 24 --ubatch-size 2048 --batch-size 2048 --temp 1.0 --min-p 0.0 --top-p 1.0 --top-k 0.0 --no-mmap --mlock --no-webui

The official /v1/chat/completions spec does not support reasoning models, so providers like llama.cpp and Ollama have deviated from the official spec in order to force support, and these deviations are provider-specific. At the time of this writing, Ollama's implementation conflicts with llama.cpp's implementation, so Ollama will not work correctly out-of-the-box. Other providers may have similar incompatibilities. But, it is fairly easy to modify chat.py and chat.ipynb to support any provider/spec if you are so inclined.