sotto-llm 0.1.0

Structured thinking as a pluggable layer for any OpenAI-compatible API

sotto voce — the passage spoken under the breath

| Getting Started | How It Works | Library | Proxy | Skills | Contributing |

Sotto gives any model a reasoning model's mechanism — a private thinking phase, effort control, interleaved thinking across tool calls, and skills — without changing your model, your SDK, or your code. The core is one stdlib-only file you can drop into any project. An optional proxy adds the same layer to any client just by repointing base_url.

Works with OpenAI · Azure OpenAI · vLLM · Ollama · DeepSeek · Groq · OpenRouter · LM Studio — anything that speaks /v1/chat/completions.

Latest Updates 🔥

• [2026/07] Initial public release of Sotto: single-file thinking engine, OpenAI-compatible proxy, and drop-in SKILL.md support.

About

Reasoning models (o-series, DeepSeek R1, gpt-5) share a set of mechanisms: they think privately before answering, scale that thinking by an effort setting, keep reasoning between tool calls, and hide the thinking from the final output. Sotto reproduces those mechanisms as a thin, portable layer so every model gets them — including the ones that don't reason natively.

Design goals

• Zero dependencies in the core. thinking.py is stdlib only. Drop it into any project, no install.
• No lock-in. Any OpenAI-compatible client works. Repoint base_url and you're done.
• Progressive disclosure. Skills are injected only when a request matches, keeping context small.
• Graceful degradation. Weak models that skip the tags still produce a usable answer.

Getting Started

Install

pip install sotto-llm            # core: zero dependencies, stdlib only
pip install "sotto-llm[proxy]"   # + the proxy server (fastapi, uvicorn, openai, python-dotenv)

The core is still one self-contained stdlib-only file, so vendoring stays an option:

curl -O https://raw.githubusercontent.com/Madhav-Gohel/sotto/main/src/sotto_llm/thinking.py

Try it offline (no API key)

git clone https://github.com/Madhav-Gohel/sotto.git && cd sotto
pip install -e .
python example.py   # scripted mock model runs the full think → tool → think → answer loop

How It Works

Sotto auto-selects one of three reasoning strategies based on the model:

1. visible_native — DeepSeek R1, QwQ, vLLM/Ollama reasoning models. The server already returns message.reasoning_content; Sotto just harvests it.
2. hidden_native — OpenAI o-series / gpt-5 family. Sotto passes reasoning_effort; reasoning happens server-side.
3. prompted — everything else (gpt-4o-mini, llama, mistral…). Sotto injects a thinking protocol, the model reasons inside <thinking> tags, and Sotto parses them out. Unclosed tags (model cut off mid-thought) are handled.

Force a strategy with mode="prompted" | "native" | "off" instead of the default "auto".

Use as a Library

from openai import OpenAI
from sotto_llm import ThinkingEngine, ThinkingConfig

client = OpenAI(base_url="http://localhost:8000/v1")   # vLLM, Ollama, Azure, anything

engine = ThinkingEngine(
    client, "gpt-4o-mini",
    config=ThinkingConfig(effort="high"),
    tools=my_openai_tools,
    tool_handlers={"get_weather": get_weather},   # python callables, run locally
    skills_dir="skills",
)

result = engine.run([{"role": "user", "content": "weather in ahmedabad?"}])

result.content              # final answer, thinking stripped
result.thinking             # list of every thinking block, in order
result.steps                # full interleaved trace: thinking / tool_call / tool_result / answer
result.as_openai_message()  # {"role": "assistant", "content": ..., "reasoning_content": ...}

The engine runs a full think → act → think agentic loop, executing your tool handlers locally and feeding results back until the model produces a final answer (capped by max_tool_iterations).

Use as a Proxy

Any existing system — bots, agents, SDKs — gains thinking without code changes. Just repoint base_url.

pip install "sotto-llm[proxy]"
cp .env.example .env          # set UPSTREAM_BASE_URL / UPSTREAM_API_KEY
uvicorn sotto_llm.proxy:app --port 8088

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8088/v1", api_key="x")
resp = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "..."}],
    extra_body={"thinking": {"effort": "high"}},   # optional; defaults apply without it
)

resp.choices[0].message.content            # clean answer
resp.choices[0].message.reasoning_content  # the thinking (DeepSeek/vLLM convention)

Reasoning comes back in message.reasoning_content — the same field DeepSeek and vLLM use — so existing clients keep working untouched. If the request includes tools, the proxy returns tool_calls to the caller like a normal OpenAI server (thinking still captured); the full agentic loop with local execution lives in library mode, since tools run on your side.

Skills

Each skill is a folder with a SKILL.md in the standard skill format: name and description in the frontmatter, instructions in the body. Existing skills in that format drop in unchanged.

---
name: code-review
description: Structured checklist for reviewing code changes... Use this skill
  whenever the user asks you to review code, look at a PR or diff, or asks
  whether some code "looks right" — even if they don't say "code review."
---

# Code Review
Review code in this order. The order matters: a fast, elegant function that
returns wrong answers is worse than a slow correct one...

The description does the triggering. Terms are extracted from the clause after "use this skill" — including quoted phrases and short domain tokens (pr, ci, sql) — then matched on word boundaries against the user's message. Only matching skills are injected, keeping context small (progressive disclosure).

keywords: a, b, c is supported as an optional extension. When present, it overrides description matching and a single hit triggers — useful when a skill needs to fire on jargon its description doesn't contain.

On matching quality: this is lexical matching, not semantic. It handles clear cases well but misses paraphrases that share no vocabulary with the description. If you have an embedding endpoint, swapping match_skills() for cosine similarity over descriptions is a strict upgrade — the rest of the engine is unaffected.

Configuration Reference

Proxy environment variables (via .env):

Limitations

• Prompted-mode thinking counts as normal output tokens (no separate billing lane like native reasoning).
• Tags are a convention, not enforcement; weak models occasionally skip them (the parser degrades gracefully — everything becomes the answer).
• The proxy is non-streaming; add SSE if you need it.
• The engine does not enforce a hard token cutoff on thinking; effort is a hint, matching how adaptive thinking treats it.

Contributing

Contributions are welcome — issues, feature requests, and PRs all help.

1. Fork the repo and create a branch from main.
2. Keep the core (src/sotto_llm/thinking.py) dependency-free — stdlib only. New dependencies belong in the proxy extra or the examples.
3. If you touch matching or the reasoning loop, add or update a case in example.py so it runs offline.
4. Open a PR describing the change and the motivation.

Good first contributions: a streaming (SSE) proxy mode, an embedding-based match_skills() variant, and more example skills.

License

MIT — see LICENSE.

Citation

If you use Sotto in your work, please cite it:

@software{sotto,
  title  = {Sotto: Structured thinking as a pluggable layer for OpenAI-compatible APIs},
  author = {Madhav Gohel},
  year   = {2026},
  url    = {https://github.com/Madhav-Gohel/sotto}
}

Built for anyone who wants their model to think first. ⭐ Star the repo if it's useful.