Agentic benchmark explorer for SGLang.
Project description
perferox
Perferox is a simple performance fuzzer for AI systems software, to try to spot performance bugs in prod systems. It can be used as a CI, always running agent, or extensive performance testing for package releases.
It is geared towards:
- Kernel libraries
- RL infra projects
- Distributed training libraries
- Model serving engines
What it does
Perferox is an agentic loop that tests the absolute edges of a system, by using it in increasingly niche ways, looking for recent PRs that regressed perf or capabilities, trying different combinations of settings, trying less-used/accessible chipsets, etc. in an effort of finding hidden regressions that many users would not report, or that often go unnoticed.
Not everyone opens a github issue when something goes wrong...
How it works
A main agent spawns subagents which each live in persistent tmux sessions, and spin up independent VMs through RunPod or Lambda, and based off the instructions of the main agent, tries attacking a certain vulnerability the main agent thinks is worthwhile prodding about, and comes back with the results, and updates a local DB with detailed information on experiments ran, and points out when it notices an anomaly.
All this information is easily visible through the TUI, and can also quickly be seen via CLI The goal is to keep this very much hackable, such that you can customize it for your small OSS project, or your niche cloud provider that you have a bunch of credits for, or your mess of local machines.
I am building this in a way such that the abstractions allow easy extensibility beyond basic functionality, and there are comments just for agents to guide them towards implementing idiomatic code for your setup.
RunPod and Lambda are supported today through runpodctl and lambda-labs. I plan on adding support for other major projects (vLLM, DeepSpeed, flashinfer, etc.) and other neoclouds with a CLI or MCP.
Because of this, I cannot imagine this going beyond 10 kLOC, (As of Jul. 9th we are hovering ~2 kLOC)
Extra Info
As of Jul. 9th, this is built for SGLang and supports RunPod and Lambda. In the future, I plan on making it generalized such that it is very easy to point an agent at perferox source code and it can customize it for whatever project you are working on.
It is clearly pre-beta right now
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file perferox-0.1.1.tar.gz.
File metadata
- Download URL: perferox-0.1.1.tar.gz
- Upload date:
- Size: 15.6 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1f9cbbea46e47e8cafe22d84fdeb656ece9c828adbff55e00576d5348ce399f0
|
|
| MD5 |
0a0fdced8439203fd0a1c5e0a7826566
|
|
| BLAKE2b-256 |
3250e0bb4f7044072b1ab1206deef05de4b940199367e7017f5c312dc1ed9139
|
File details
Details for the file perferox-0.1.1-py3-none-any.whl.
File metadata
- Download URL: perferox-0.1.1-py3-none-any.whl
- Upload date:
- Size: 15.7 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9d22083876fcbb655117bdaafb1429515396bb1be2642d1f4f9552c50cec7f64
|
|
| MD5 |
1ef5d7c8fb2f5af7ca3a3ad8f34a1fa6
|
|
| BLAKE2b-256 |
693dca43b9b4a577a8f0059e03a77caa0e21d1b83642bfe9b32007d02760a636
|
