perferox 0.1.0

Agentic benchmark explorer for SGLang.

perferox

Perferox is a small agentic benchmark explorer for inference engines and ML systems performance. The first target is SGLang.

This is alpha software. It is not ready for common use yet. Expect rough edges, missing docs, changing APIs, and places where you still need to understand the repo before trusting a run.

The shape is intentionally simple: give an agent a bounded objective, let it delegate benchmark workers, run experiments through narrow tools, and save the parts worth keeping. The point is not to make a giant benchmark platform. The point is to make it easy to poke at weird corners of an inference stack without losing track of what already happened.

The first use case is SGLang serving benchmarks. Perferox is meant to try different models, backends, cache settings, request patterns, chips, and setup conditions, then preserve the useful results in SQLite. It records the normal perf signals, like request throughput, token throughput, TTFT, TPOT, cache hit rate, accept length, and error rate. It also has a place to store human-readable anomalies, because a lot of the interesting bugs are not just "number went down" but "this specific setup behaved strangely in a way a person should inspect."

The important constraint is that the host owns state. Agents do not choose run IDs, write SQLite directly, or decide whether a benchmark should bypass caps. Real experiments go through a benchmark tool. Started runs get logged. Failed runs still count once they actually start. The database is the source of truth for runs, experiments, anomalies, trace references, and compact exploration state.

Current pieces:

• a perferox CLI
• a Textual TUI for starting and stopping exploration
• a LangGraph main coordinator
• tmux-wrapped benchmark subagents
• Paramiko remote sessions
• structured SGLang bench_serving command construction and parsing
• SQLite-backed runs, experiments, anomalies, notifications, and ExplorerState
• local SGLang docs search plumbing

This is still early software. The goal is to keep the code small enough that it stays easy to inspect and change. The preferred path is boring Python, explicit state, small tools, and very few abstractions.

uvx perferox --help