oshepherd

The Oshepherd guiding the Ollama(s) inference orchestration.

A centralized FastAPI service that uses Celery and Redis to orchestrate multiple Ollama servers as workers.

Install

pip install oshepherd

Usage

1. Set up Redis:

   Celery uses Redis as a message broker and backend. You'll need a Redis instance, which you can provision for free at redislabs.com.

2. Set up the FastAPI server:

   # define configuration env file
   # use credentials for redis as broker and backend
   cp .api.env.template .api.env
   
   # start api
   oshepherd start-api --env-file .api.env
   

3. Set up the Celery/Ollama workers:

   # install ollama https://ollama.com/download
   # optionally pull the model
   ollama pull mistral
   
   # define configuration env file
   # use credentials for redis as broker and backend
   cp .worker.env.template .worker.env
   
   # start worker
   oshepherd start-worker --env-file .worker.env
   

Logging

Oshepherd uses Python's standard logging library for both the API server and Celery worker.

Set LOGLEVEL in either .api.env or .worker.env to control verbosity:

LOGLEVEL="info"

Valid values include debug, info, warning, error, and critical. API access logs from Uvicorn are disabled by default; enable them with:

UVICORN_ACCESS_LOG=true

Request and response payload bodies are only logged at debug level because they may contain prompts, model output, or other sensitive data.

4. Now you're ready to execute Ollama completions remotely. Point your Ollama client at your oshepherd API server by setting the host. The server will return the requested completions from one of the workers:

   • ollama-python client:

   import ollama
   
   client = ollama.Client(host="http://127.0.0.1:5001")
   
   # Standard request
   response = client.generate(model="mistral", prompt="Why is the sky blue?")
   
   # Streaming request
   for chunk in client.generate(model="mistral", prompt="Why is the sky blue?", stream=True):
       print(chunk['response'], end='', flush=True)
   

   For a complete Python example with streaming support, see examples/pretty_streaming.py.

   • ollama-js client:

   import { Ollama } from "ollama/browser";
   
   const ollama = new Ollama({ host: "http://127.0.0.1:5001" });
   
   // Standard request
   const response = await ollama.generate({
       model: "mistral",
       prompt: "Why is the sky blue?",
   });
   
   // Streaming request
   const streamResponse = await ollama.generate({
       model: "mistral",
       prompt: "Why is the sky blue?",
       stream: true
   });
   
   for await (const chunk of streamResponse) {
       process.stdout.write(chunk.response);
   }
   

   For a complete TypeScript/JavaScript example with streaming support, see examples/ts-scripts/README.md.

   • Raw HTTP request:

   curl -X POST -H "Content-Type: application/json" -L http://127.0.0.1:5001/api/generate/ \
   -d '{"model":"mistral","prompt":"Why is the sky blue?","stream":true}' \
   --no-buffer
   

Decision models (System One)

POST /v1/systemone queues decision requests to workers and returns a single JSON response. Workers serving decision models must run Ollama 0.35 or later and have the requested model pulled. For example, pull a model with ollama pull tev1:4b or ollama pull nimble:latest.

Use the official Ollama Python client, which supports client.systemone() starting with version 0.6.3:

uv pip install --upgrade 'ollama>=0.6.3'  # or: pip install --upgrade 'ollama>=0.6.3'

from ollama import Client

state = {"ticket": "I was charged twice. Please refund the extra payment."}
questions = {
    "team": {
        "type": "choice",
        "instructions": "Which team should handle this ticket?",
        "criteria": {
            "billing": "Payments and refunds",
            "technical": "Bugs and integrations",
            "other": "None of the above",
        },
    },
    "refund": {
        "type": "noul",
        "instructions": "Does the customer explicitly ask for a refund?",
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this ticket?",
        "criteria": ["Routine", "Soon", "Urgent"],
    },
}

with Client(host="http://127.0.0.1:5001", timeout=180) as client:
    result = client.systemone(
        model="tev1:4b",
        state=state,
        questions=questions,
    )

print(result.answers["team"].choice)
print(result.answers["refund"].noul)
print(result.answers["urgency"].score)

state can be text, a JSON object, or an array. Responses contain named answers and token usage. System One returns a single response and does not support streaming.

The optional TypeSafe Python SDK also works. Install and configure it:

uv pip install typesafe-sdk  # or: pip install typesafe-sdk
export TYPESAFE_BASE_URL=http://127.0.0.1:5001
export TYPESAFE_API_KEY=ollama
export TYPESAFE_DEFAULT_MODEL=tev1:4b

Using the same state and questions from above:

from typesafe_sdk import TypeSafeClient

with TypeSafeClient(timeout=180) as client:
    result = client.system_one(state=state, questions=questions)

print(result.choices["team"].choice)

TypeSafe's system_one() call uses only /v1/systemone. Its separate client.models.list() call to /v1/models is not supported; use /api/tags to list the workers' models. The SDK requires an API key, but oshepherd does not authenticate it.

Example: PyCon Austria 2025

For a practical example of how oshepherd can be used to orchestrate on-premise open-source LLMs, see the companion repository from the PyCon Austria 2025 talk "Beyond the Cloud: On-Premise Orchestration for Open-Source LLMs":

• Repository: https://github.com/mnemonica-ai/on-premise-orchestration-os-llms
• Deck: https://speakerdeck.com/p1nox/beyond-the-cloud-on-premise-orchestration-for-open-source-llms

Disclaimers 🚨

This package is in alpha, and its architecture and API might change in the near future. It is currently being tested by real users in a controlled environment, but it has not been audited or thoroughly tested. Use it at your own risk.

As this is an alpha version, support and responses might be limited. We'll do our best to address questions and issues as quickly as possible.

API server parity

• Generate a completion: POST /api/generate
• Generate a chat completion: POST /api/chat
• Generate Embeddings: POST /api/embeddings
• Decision models: POST /v1/systemone
• List Local Models: GET /api/tags
• Version: GET /api/version
• Show Model Information: POST /api/show
• List Running Models: GET /api/ps

Oshepherd supports the endpoints listed above. Official clients such as ollama-python and ollama-js can call these routes by pointing their host at oshepherd. Support for individual request fields varies by route; this is not full Ollama API parity. For example, the Python client's embed() method calls /api/embed, which is not supported. See the Ollama API documentation for the upstream API.

Contribution guidelines

We welcome contributions! If you find a bug or have suggestions for improvements, please open an issue or submit a pull request targeting the development branch. Before creating a new issue or pull request, take a moment to search the existing issues and pull requests to avoid duplicates.

Conda Support

To run and build locally, you can use conda:

conda create -n oshepherd python=3.12
conda activate oshepherd
pip install -r requirements.txt

# install oshepherd
pip install -e .

uv Support

Or you can use uv:

uv venv --python 3.12
source .venv/bin/activate
uv pip install -r requirements.txt

# install oshepherd
uv pip install -e .

Tests

The e2e tests require the following models to be available on your local Ollama instance:

ollama pull mistral        # used by generate, chat, and show tests
ollama pull embeddinggemma # used by embeddings tests
ollama pull tev1:4b         # used by System One tests; requires Ollama 0.35+
ollama pull nimble:latest   # used by System One tests; requires Ollama 0.35+

Then follow the usage instructions to start the API server and Celery worker, and run:

pytest -s tests/

The focused System One tests need no running services:

pytest -q tests/test_systemone.py

Author

This is a project developed and maintained by mnemonica.ai.

License

MIT