needle-openai

An OpenAI-compatible HTTP API in front of Needle 2 — Cactus Compute's 45M-parameter model for tool calling, device use and structured extraction. Point any OpenAI client at it and get function calls back from a 14MB model that runs on CPU in ~100MB of RAM.

📖 Full documentation

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="needle-2",
    messages=[{"role": "user", "content": "what's it like in Lagos right now?"}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather for a city.",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
)
print(response.choices[0].message.tool_calls[0].function)
# Function(arguments='{"city": "Lagos"}', name='get_weather')

[!IMPORTANT] Needle 2 is not a chat model. It emits tool calls or nothing at all — it has no free-form generation head. Read the fidelity notes before building on this; they list every place the OpenAI mapping is lossy and the two failure modes worth guarding against.

Quickstart

docker run -d -p 8000:8000 -v needle-cache:/cache ghcr.io/sirmmo/needle-openai:latest
curl -s localhost:8000/health

Or from a clone:

docker compose up -d

Or as a Python package (needs Python ≥3.10):

pip install needle-openai
pip install --no-deps cactus-needle==2.0.3   # see "Why --no-deps" in the docs
needle-openai --port 8000

First start downloads the native engine and weights from HuggingFace into /cache — a few tens of MB, ~6s on a warm connection, cached across restarts.

Runnable examples: examples/openai_client.py and examples/curl.sh.

What works

Endpoints

Full details in the API reference.

Key things to know

A short version of the fidelity notes:

• No free-form text. When no tool is called, the model's reasoning trace is returned as content. Asked "hello, who are you?" it replies "No tool available to query location or contact info."
• Deterministic. temperature, top_p, seed and n are accepted, ignored, and reported in x_needle.warnings.
• One request at a time per container. The native engine is a single global session that segfaults under concurrent use, so everything serializes through one thread. 24 concurrent requests complete correctly in ~4.2s; scale by running more containers.
• Long inputs truncate silently and return a confident-looking wrong answer with confidence: 0.0. Gate on confidence.
• response_format: {"type": "json_object"} returns 400 — Needle needs an explicit schema.

Development

pip install -r requirements.txt pytest httpx ruff
pytest                    # 57 tests, ~1s, no model download (fake engine)

docker compose up -d      # live suite against the real engine
NEEDLE_TEST_BASE_URL=http://127.0.0.1:8000 pytest -m live

See Development and CONTRIBUTING.md.

License

MIT — see LICENSE. Needle 2 and its weights are licensed by Cactus Compute.