ollama-profiler 0.3.0

CLI + TUI tool for benchmarking Ollama models side-by-side

ollama-profiler

Profile and compare Ollama model performance side-by-side.

Measure tokens/sec, time-to-first-token, and 7 other metrics across your local models. Interactive TUI or rich CLI. Export to JSON, HTML, or PNG.

Installation

This Go tool can be installed directly from PyPI using pip or uv.

You can run it without installing it first using uvx:

uvx ollama-profiler --help

Or install it, then run ollama-profiler:

uv tool install ollama-profiler
# or
pip install ollama-profiler

Other install methods

You can also install the Go binary directly:

go install github.com/piercecohen1/ollama-profiler@latest

Compiled binaries are available on the releases page.

Quick Start

# First, make sure you have some models to test
ollama pull gemma4:e2b
ollama pull qwen3.5:2b

# Launch the interactive TUI
ollama-profiler

# Or compare directly from the command line
ollama-profiler gemma4:e2b qwen3.5:2b

# Rigorous comparison with balanced scheduling
ollama-profiler gemma4:e2b qwen3.5:2b \
  -n 4 --rounds 4 --balanced --warmup --cooldown 30

Features

• Side-by-side comparison of multiple models on the same prompt
• TUI + CLI — interactive TUI by default, or pass models for non-interactive CLI output
• Fair scheduling — sequential, round-robin, randomized rounds, or Latin-square balanced
• Statistical summary — mean, stddev, min/max, relative %, color-coded winners
• Export — JSON with metadata, self-contained HTML report, retina PNG charts
• Reproducible — deterministic seed, configurable token limits, warmup & cooldown

Requires Ollama running locally, or pass --url for a remote server. Run ollama-profiler --help for full usage.

License

MIT