chatlint 0.1.1

Chat templates are untested code. ChatLint tests them.

ChatLint

Chat templates are untested code. ChatLint tests them.

Every model release ships a Jinja file that turns a structured messages array into the exact string the model sees. It decides where turns begin and end, how tool calls are encoded, whether reasoning is kept or dropped. It is the most load-bearing file in the release and the only one with no tests.

When it is wrong there is no error message. The model just gets quietly worse, and nobody can tell whether the problem is the prompt, the sampler, the quantisation, or the weights.

On 2026-07-15 Google shipped a fix to the Gemma 4 chat template. The weights did not change:

fix: chat template — null handling, reasoning preservation, turn-tag balance, input validation

Users had spent months reporting that Gemma 4 was bad at tool calling. One watched a 3B model beat a 26B model on the same task. The regression was introduced by an earlier commit that was itself titled as a fix, and it survived 96 days.

ChatLint finds that class of defect in 10 milliseconds per template on a laptop, with no GPU and no model weights. A chat template is a Jinja string; rendering it costs microseconds.

Use it as a CI gate

pip install chatlint
chatlint check path/to/chat_template.jinja

Exit code 1 on any error. Drop .github/workflows/chatlint.yml into a model repo and it runs on every change to chat_template.jinja.

It would have blocked the Gemma 4 regression

Walking Google's own template history, gating each revision:

date	revision	gate	errors	vendor's commit title
2026-04-02	419b2efe4	BLOCK	2	Preparing for release!
2026-04-10	e51e7dcdb	BLOCK	7	fix: prefix-preserving tool calls + dialog compliance
2026-04-28	145dc2508	BLOCK	7	fix(chat_template): update SI and tool call handling
2026-05-18	fcf230276	BLOCK	7	fix(chat_template): Emit multimodal placeholders …
2026-07-15	68abe4801	PASS	0	fix: … null handling, reasoning preservation, turn-tag balance

The gate flips to PASS at exactly the commit Google shipped as the fix, and not before. It was not tuned to do that, and it still holds after the suite grew from 13 conversation shapes to 49.

The defect it catches on the April 10 commit: after a tool round-trip the template never closed the model's turn.

broken:  ...<tool_response|><|turn>user
fixed:   ...<tool_response|><turn|>
         <|turn>user

The model read its own turn and the user's next turn as one fused span. It only fires when a user message follows a tool result with no assistant text between them, which is why plain chat felt fine and tool calling felt cursed.

llama.cpp still ships e51e7dcd. Its bundled models/templates/google-gemma-4-31B-it.jinja is byte-identical to the revision that introduced the bug, three vendor fixes behind, 108 days later.

What the corpus says

The board below covers 786 distinct chat templates, harvested from the templates llama.cpp bundles and from the Hugging Face Hub — every release by every major publisher, plus the Hub's most downloaded and most recent models. Those 786 files are shipped by 4,387 model repositories — one row is one distinct template, because scoring a dozen Qwen3 sizes that ship one identical file as a dozen separate rows would pad the numbers rather than say anything.

• 10 / 786 are clean, with no errors and no warnings.
• 80 / 786 have no errors.
• 706 / 786 would fail a CI gate today, covering 4,102 of the 4,387 repositories.

The whole sweep is 231,084 checks and takes 22 seconds on a laptop.

Things it found, with the number of repositories each reaches:

• 477 / 786 templates crash outright when arguments is a JSON string (2,397 repos). Tool-call arguments is sent as a JSON object by some clients and a JSON string by others, and nobody ever specified which. This is not a disagreement about formatting, it is an HTTP 500.
• 421 / 786 crash on an assistant turn that is reasoning with no content (2,640 repos), which is what a model emits when it thinks and then calls a tool.
• 456 / 786 crash when content arrives as a list of typed parts (3,003 repos), which is ordinary OpenAI multimodal input.
• 396 / 786 crash when tool calls are in the history but tools was not resent (2,098 repos). Servers drop tools on follow-up requests all the time.
• 648 / 786 break on parallel tool calls (3,149 repos), a stock OpenAI feature.
• 233 / 786 leak a Python repr into the token stream (921 repos). The clearest renders {fuzzy:true,limit:None,safe:false} — correct JSON for booleans and Python repr for null, in the same object.
• Only 145 / 786 survive all four prompt-injection shapes. Message text is attacker-controlled, and a template that pastes it in raw lets a user close the current turn and open a new one with a forged role. 633 / 786 mishandle special tokens arriving from a tool result, which nobody sanitises at all.
• Apertus, Switzerland's national model, silently disables tool calling for every standard client. Separate write-up and fix in apertus/.

gpt-oss does not render OpenAI's own wire format

openai/gpt-oss-120b and gpt-oss-20b score 34%. They ship one identical template, which llama.cpp also bundles and 24 further repos redistribute, and it crashes on every tool-calling shape in the suite. The cause is one line:

{%- if "content" in message %}
    {%- if "<|channel|>analysis<|message|>" in message.content or ... %}

The guard asks whether the message has a content key. The body then runs a substring match against its value. An assistant message that only calls tools has content: null — that is what the OpenAI API returns, and gpt-oss is OpenAI's model in OpenAI's own harmony format, so the key is present, the guard passes, and "..." in None raises TypeError. Omitting the key renders fine. Sending "" renders fine. Only the spec-compliant form fails. Writing the guard as {%- if message.content %} fixes it.

That one line accounts for 24 of the 30 errors, because a template that will not render cannot be checked for anything else. The remaining six are a second defect, on assistant turns that say something before calling a tool:

• The visible text is emitted into the analysis channel, which is where chain of thought goes. A preamble is not reasoning; harmony has commentary for it.
• Analysis is then dropped once a later final message exists, so text the user was shown is in the prompt for one turn and gone from the next. That invalidates 28% of the previously cached prefix at that turn.

Both are locked down in tests/test_regression.py and reproduce against the templates live on the Hub as of this writing. unsloth/gpt-oss-20b ships the same file with that guard block deleted rather than corrected, and scores 67% — the difference one line makes.

Properties

Each is config-free. Nothing is keyed to a model family, so a finding cannot be an artifact of the suite knowing something about one template.

property	invariant
renders	renders at all, given only what an OpenAI-compatible server sends
marker_balance	every opened turn marker is closed exactly once
no_repr_leak	no Python repr reaches the prompt
prefix_stability	prompt N+1 extends prompt N rather than rewriting it
content_order	message text appears in the order it was sent
time_invariance	the same conversation renders identically on two different days

prefix_stability is the subtlest. At step k the model saw P_k and committed to a continuation, so P_k must be a literal prefix of P_k+1. Otherwise cached KV past the divergence is invalid and the model reads a history it never produced. One violation, both bugs.

marker_balance needs no assumption about how a template marks turns: rendered without a generation prompt a transcript is complete, so opens must equal closes. It deliberately does not require that a generation prompt dangle, because Llama 3, DeepSeek-R1 and Mistral end a turn with a single terminator rather than a wrapping pair, and demanding a dangling open flagged all three wrongly.

Findings are attributed to whoever caused them

Every structural check reads the rendered prompt, and the rendered prompt contains the caller's own text. So a conversation that pastes <|im_end|> into a user turn would otherwise be scored against whichever template copied it faithfully, and a user who writes the word None in a sentence would look like a repr leak.

Before counting, anything traceable to the input is withdrawn. A template that turns a JSON null into None still fails; a template that echoed the user's prose does not. Without that, every adversarial shape added to the suite would manufacture false findings, and one bad row discredits the whole board.

Severity, and why the gate is not maximally strict

A gate that fails on everything arguable gets switched off. So findings are tiered.

Errors are things no vendor would defend: the template crashes on a payload the OpenAI schema permits, emits structurally malformed output, leaks a Python repr, or rewrites conversation history the model was already conditioned on.

Warnings are things that might be deliberate, and they come in two kinds. Ecosystem conventions: content-as-list-of-parts is legal OpenAI, but most serving stacks flatten text parts to a string before rendering, precisely because so many templates assume str. And documented model constraints: Mistral requires strict user/assistant alternation, Gemma 2 genuinely has no system role. Reporting those as defects would mean reporting a model for being what it says it is.

Telling a rewritten history from a dropped prefill marker is what decides the tier for prefix_stability, and it is done exactly rather than by a character budget. The same snapshot is rendered with and without a generation prompt; the difference between the two is the generation prompt. If everything the next turn failed to carry forward lies inside it, only a marker was lost, and that is a warning.

--strict promotes warnings to errors.

Auditing the suite against ground truth

A conformance suite that only checks properties its author imagined is unfalsifiable. Template revision history fixes that: every commit that edits a chat template is a labelled pair, where the version before is known-broken and the version after is the vendor's own correction.

chatlint --audit

Harvesting 13 repos yields 35 labelled pairs, of which the suite separates 18 (51%), up from 40% before the shape set was expanded. That is the number to improve, and the 17 misses each name a property or shape still missing.

For the record, prefix_stability alone separates 0 of 49 shapes on the Gemma 4 pair — it scores the broken template and its fix identically. The structural properties exist because of that result, and they carry the separation: marker_balance on 6 shapes, no_repr_leak on 3.

The audit surfaces its own class of finding: 7 pairs where a vendor's fix newly broke a property. Gemma 4's April 10 commit is one. Qwen3-Coder's 2025-08-07 update is another, repaired two weeks later.

Testing ChatLint itself

A suite that reports on everyone else's untested code has no business being untested. pytest --corpus-slow runs 208 tests:

• Every detector is paired with a hand-written defect and a hand-written control, so a regression in either direction — a detector that stops detecting, or one that starts crying wolf — fails the build.
• The README's numbers are locked down as regression tests, including the whole Gemma 4 walk.
• Whole-corpus sweeps assert the invariant that actually matters at this scale: no template, on any shape, can make a property raise instead of returning a result. A property that throws aborts the run and silently stops reporting on every template after it.
• Property-based fuzzing over generated OpenAI payloads checks the same invariant on inputs nobody thought of, plus the attribution guarantees: redaction can never increase a marker count or invent a repr leak.

Caveats

• Render side only. Parsing model output back into tool_calls and reasoning_content is not covered, and a large share of reported bugs live there.
• Jinja2, not minja. This renders through transformers' sandbox, the same _compile_jinja_template HuggingFace ships, which is what makes findings hard to dismiss. llama.cpp uses minja, its own C++ Jinja subset. Divergence between the two is a finding this suite cannot yet see.
• Some gated repos are missing. chatlint fetch uses HF_TOKEN when it is set, but a token only reaches repos whose licence that account has accepted. Meta Llama and Gemma 3 are gated per-account and are present here only via the copies llama.cpp and downstream fine-tunes ship, so their rows are named after a redistributor rather than the publisher.
• Prefix instability is mostly a correctness problem, not a throughput one. Over a 64-round tool loop, cumulative recompute from prefix divergence is 0% for Qwen3.5-4B and GLM-4.6, 3.4% for MiniMax-M2, 15.3% for Apertus. Anyone quoting large speedups from this is quoting something else.
• Some failures are deliberate model constraints, not defects. Those are tiered as warnings where the constraint is documented, but the suite reports behaviour and judging intent still needs a human.
• marker_balance discovers marker pairs heuristically and skips templates where it finds none, so it is a lower bound.

Usage

chatlint check chat_template.jinja      # CI gate, exit 1 on error
chatlint check --strict *.jinja         # warnings fail too
chatlint fetch                          # rebuild the corpus from llama.cpp + the Hub
chatlint scoreboard --out SCOREBOARD.md # regenerate the board
chatlint                                # full report over the bundled corpus
chatlint --audit                        # score the suite against vendor fixes

Layout

• chatlint/check.py — the CI gate and severity tiers
• chatlint/properties.py — renders, prefix stability, time invariance
• chatlint/structure.py — marker balance, repr leakage, content order, attribution
• chatlint/shapes.py — 49 conversation shapes as client-side snapshot sequences
• chatlint/render.py — Jinja environment matching transformers
• chatlint/rendering.py — one render pass per (template, shape), shared by every property
• chatlint/fetch.py — corpus discovery and deduplication
• chatlint/history.py — harvests labelled pairs from template revision history
• chatlint/audit.py — scores the suite against those pairs
• chatlint/scoreboard.py — generates the public board
• tests/ — the suite's own test suite
• apertus/ — the Apertus findings and a corrected template

The scoreboard

Generated by chatlint scoreboard, also available standalone in SCOREBOARD.md.

786 distinct chat templates, shipped across 4387 model repositories, against 49 conversation shapes and 6 properties. 231,084 checks.

• 10 / 786 templates are clean: no errors, no warnings.
• 80 / 786 have no errors.
• 706 / 786 would fail a CI gate today, covering 4102 of the 4387 repositories.

One row is one distinct template. Identical templates are not scored twice; models is how many repositories ship that exact file. A cell shows the number of findings for that property, and blank means clean.

template	models	score	renders	markers	literals	prefix	order	time
ibm-granite-granite-4.0	14	clean
ibm-granite-granite-4.1	14	clean
ibm-granite-granite-switch-4.1-3b-preview	3	clean
JetBrains-Mellum2-12B-A2.5B-Instruct	3	clean
DarkArtsForge-Zepar-24B-v1	2	clean
kerzgrr-Monostich-2-base	2	clean
utter-project-EuroVLM-9B-Preview	2	clean
FrontiersMind-Lumma-0.6B-Tool	1	clean
ibm-granite-granite-speech-4.1-2b-plus	1	clean
moonshotai-Kimi-K2	1	clean
ai21labs-Jamba-tiny-dev	2	100%			1
naver-hyperclovax-HyperCLOVAX-SEED-Think-14B	1	100%				1
EssentialAI-rnj-1	2	99%					2
groxaxo-MagiSeek-V1.1	1	99%					2
Reka-Edge	1	99%					2
RekaAI-reka-edge-2603	1	99%					2
Pageshift-Entertainment-pagestorm-research-preview-14b-full-book	1	99%			2
LGAI-EXAONE-EXAONE-4.0-32B	3	99%			2	1
tiiuae-Falcon3-10B-Base-1.58bit-prequantized	2	99%			1		2
idealab-cs2-zagreus-0.4B-italic-softkd	1	99%			1		2
Neura-Tech-AI-Neuron-V2-4B-Instruct	1	99%			3
Ayodele01-gemma-4-E4B-Gemini-3.1-Pro-Reasoning-Distill	4	99%			3
JetBrains-Mellum2-12B-A2.5B-Thinking	3	99%				3
LGAI-EXAONE-K-EXAONE-236B-A23B	2	99%			1	2
LGAI-EXAONE-EXAONE-4.5-33B	1	99%			1	2
arcee-ai-Trinity-Large-TrueBase	3	98%			4
nvidia-NVIDIA-Nemotron-Parse-v1.1	2	98%			4
Nanbeige-Nanbeige4.2-3B-Base	1	98%			4
Salesforce-Llama-xLAM-2-8b-fc-r	3	98%			3		2
Salesforce-xLAM-2-32b-fc-r	3	98%			3		2
LiquidAI-LFM2-ColBERT-350M	23	98%	1
LFM2.5-Instruct	15	98%	1
LiquidAI-LFM2-1.2B	5	98%	1
inclusionAI-LLaDA2.2-flash	2	98%	1
LFM2-8B-A1B	1	98%	1
LiquidAI-LFM2-350M-PII-Extract-JP	1	98%	1
wang-yang-Ornith-1.0-35B-MTPLX	1	98%	1
Qwen-Qwen3-4B-Instruct-2507	41	98%	1				1
mistralai-Ministral-3-3B-Reasoning-2512	5	98%	1			1
allenai-tmax-sft-8b	3	98%	1				1
arcee-ai-Trinity-Nano-Preview	3	98%	1				1
meituan-longcat-LongCat-2.0	3	98%	1			1
mistralai-Mistral-Large-3-675B-Instruct-2512	3	98%	1			1
mistralai-Mistral-Small-4-119B-2603	3	98%	1			1
mistralai-Leanstral-1.5-119B-A6B	2	98%	1			1
arcee-ai-Trinity-Large-Preview	1	98%	1				1
C10X-Qwen2.5-0.5B-Instruct	1	98%	1			1
mistralai-Mistral-Medium-3.5-128B	1	98%	1			1
OpenLLM-France-Luciole-23B-Instruct-1.1	1	98%	1				1
OpenPipe-Qwen3-14B-Instruct	1	98%	1				1
rdtand-Mistral-Medium-3.5-128B-PrismaQuant-4.75-vllm	1	98%	1			1
STEVENZHANG904-Qwen3-4B-Instruct-2507-Meta-SecAlign	1	98%	1				1
MiniMaxAI-MiniMax-M3-MXFP8	6	97%	1				2
tencent-Hy3-preview	1	97%	1			1	1
distil-labs-distil-home-assistant-functiongemma	4	97%	1		2
unsloth-functiongemma-270m-it	1	97%	1		2
HuggingFaceTB-SmolLM3-3B	1	97%	1				3
gururaser-qwen3-4b-bilimkurgu	4	97%	1			2	1
kakaocorp-kanana-2-30b-a3b-instruct-2601	2	97%	1			2	1
Qwen-Qwen3-0.6B@a55ee1	240	97%	1			3	1
inclusionAI-GroveMoE-Inst	10	97%	1			3	1
Snowflake-Arctic-AWM-4B	5	97%	1			3	1
amazon-GKA-primed-HQwen3-32B-Reasoner	2	97%	1			3	1
Nanbeige-Nanbeige4.1-3B	1	97%	1			3	1
trl-internal-testing-tiny-Qwen3ForCausalLM	1	97%	1			3	1
tencent-Hy3@7fc351	3	96%	1			4	1
tencent-Hy3	1	96%	1			4	1
SenseLLM-ReflectionCoder-DS-33B	2	96%	2
inclusionAI-Ring-1T	1	96%	2
meetkai-functionary-medium-v3.1	1	96%	2
NousResearch-Kimi-K2-Thinking-Alternate-Tokenizer	1	96%	2
google-gemma-4-E4B-it	18	96%	1			4	2
nabin2004-AOS-gemma4-manim-sft	1	96%	1			4	2
poolside-Laguna-S-2.1@444819	19	96%	2				1
NousResearch-Hermes-4.3-36B	6	96%	2			1
meituan-longcat-LongCat-Flash-Lite	1	96%	2		1
Skywork-MindLink-32B-0801	1	96%	1			6	1
VincentZhu007-minimind-3	1	96%	1			6	1
mistralai-Devstral-Small-2-24B-Instruct-2512	1	96%	2			1
mistralai-Ministral-3-14B-Instruct-2512	1	96%	2			1
mistralai-Ministral-3-3B-Instruct-2512	1	96%	2			1
mistralai-Ministral-3-8B-Instruct-2512	1	96%	2			1
farbodtavakkoli-OTel-LLM-E4B-IT	25	96%		6	2	3	2
NousResearch-Hermes-4-14B	1	95%	2	1		1
inclusionAI-LLaDA2.0-mini	18	95%	2			2
inclusionAI-Ling-2.6-flash	5	95%	2			2
inclusionAI-Ming-flash-omni-Preview	1	95%	2			2
inclusionAI-Ring-2.5-1T	1	95%	2			2
unsloth-Ornith-1.0-9B	1	95%	2		2
OBLITERATUS-gemma-4-E4B-it-OBLITERATED	2	95%		6	3	3	2
vinoku89-svg-code-generator	14	95%	2			3
inclusionAI-Ring-flash-linear-2.0-128k	1	95%	2			3
OsaurusAI-Osaurus-AppleScript-8B-JANG_6M	1	95%	2		3
openbmb-MiniCPM-MoE-8x2B	9	95%	2		3
praxisresearch-hf_seed_36b_sgtr_benign_0	18	94%	2		4
nvidia-Nemotron-Labs-Diffusion-8B-Base	7	94%	2		3	2
deepreinforce-ai-Ornith-1.0-35B	5	94%	3
unsloth-Ornith-1.0-35B	1	94%	3
Qwen-Qwen2.5-7B-Instruct	408	94%	3
Qwen-Qwen3-Embedding-8B	104	94%	3
Lautloserspieler-Nova-Qwen2.5-14B-Secure-Coder	17	94%	3
Qwen-Qwen2.5-Math-1.5B-Instruct	13	94%	3
internlm-OREAL-7B	6	94%	3
arcee-ai-Virtuoso-Small-v2	2	94%	3
NiffyHunt90-codeguard-security-7b	2	94%	3
arcee-ai-SuperNova-Medius	1	94%	3
arcee-ai-Virtuoso-Large	1	94%	3
arcee-ai-Virtuoso-Medium-v2	1	94%	3
dphn-Dolphin3.0-R1-Mistral-24B	1	94%	3
hexera-org-GmshNet-8B-v0.1	1	94%	3
Lumina-Ai-Official-Nexa-AI-4x4B-Instruct	1	94%	3
MiMo-VL	1	94%	3
netease-youdao-Confucius-o1-14B	1	94%	3
Neura-Tech-AI-Neuron-V1-14B-Instruct	1	94%	3
Qwen-QwQ-32B-Preview	1	94%	3
RefalMachine-ruadapt_qwen2.5_7B_ext_u48_instruct	1	94%	3
SakanaAI-TinySwallow-1.5B-Instruct	1	94%	3
Skywork-MindLink-72B-0801	1	94%	3
WeiboAI-VibeThinker-1.5B	1	94%	3
XCurOS-XCurOS0.1-8B-Instruct	1	94%	3
mistralai-Devstral-2-123B-Instruct-2512	1	94%	3			1
deepseek-ai-DeepSeek-V3.1	1	93%	3			1	1
deepreinforce-ai-Ornith-1.0-9B	5	93%	3		2
inclusionAI-Ring-2.6-1T	1	93%	3			2
XiaomiMiMo-MiMo-V2-Flash	2	93%	3		2	2
upstage-Solar-Open2-250B	3	92%	3			3	1
ByteDance-Seed-OSS	4	92%	3		4
Qwen-Qwen3-Coder-Next	4	92%	4
tencent-Hunyuan-0.5B-Pretrain	9	92%	3	3		2	1
tencent-Hunyuan-7B-Pretrain	1	92%	3	3		2	1
nvidia-audio-flamingo-next-hf	3	91%				25
nvidia-audio-flamingo-3-hf	2	91%				25
nvidia-music-flamingo-2601-hf	2	91%				25
naver-hyperclovax-HyperCLOVAX-SEED-Vision-Instruct-3B	1	91%				25
nvidia-nemotron-labs-audio-visual-flamingo-hf	1	91%				25
Qwen-Qwen3-Coder-30B-A3B-Instruct	3	91%	4		2
Kwaipilot-KAT-Dev	1	91%	4		2
PrimeIntellect-Qwen3-0.6B	1	91%	4		2
Qwen3-Coder	1	91%	4		2
NousResearch-Meta-Llama-3-8B-Instruct	86	91%			27
NousResearch-Meta-Llama-3.1-8B-Instruct	34	91%			27
NousResearch-DeepHermes-3-Llama-3-8B-Preview	20	91%			27
Kiznaiver02-llama3-indonesia-finetuned-dicoding	17	91%			27
zai-org-cogvlm2-llama3-chat-19B	17	91%			27
utter-project-EuroLLM-9B-Instruct-2512	6	91%			27
shenzhi-wang-Llama3-8B-Chinese-Chat	3	91%			27
tasmin-jahan-bangla-gsg	3	91%			27
veyra-ai-Veyra-30M-Base-2.5B-Tokens	3	91%			27
Vikhrmodels-Vikhr-Nemo-12B-Instruct-R-21-09-24	2	91%			27
athena129-CyberSecQwen-4B	1	91%			27
bosonai-Higgs-Llama-3-70B	1	91%			27
Goklas17-llama-3.2-1b-alpaca-indonesian-pgabl	1	91%			27
latam-gpt-Llama-3.1-70B-LatamGPT-SFT-1.0	1	91%			27
marin-community-marin-8b-instruct	1	91%			27
SakanaAI-Llama-3-Karamaru-v1	1	91%			27
shanchen-llama3-8B-slerp-biomed-chat-chinese	1	91%			27
tokyotech-llm-Llama-3.1-Swallow-8B-Instruct-v0.5	1	91%			27
tomg-group-umd-huginn-0125	1	91%			27
TrangBui-II-Medical-8B-Finetuned	1	91%			27
typhoon-ai-llama3.1-typhoon2-8b-instruct	1	91%			27
unsloth-Apriel-1.5	1	91%			27
unsloth-GLM-4.5-Air	1	91%			25	2
ServiceNow-AI-Apriel-1.5-15b-Thinker	1	90%			28
meta-llama-Llama-3.1-8B-Instruct	73	90%	4		3		2
nvidia-Llama-3.1-Nemotron-70B-Instruct-HF	5	90%	4		3		2
ApolloRaines-Llama-3.3-8B-Instruct-128K-Jbliterated	2	90%	4		3		2
arcee-ai-Arcee-SuperNova-v1	1	90%	4		3		2
KISTI-KONI-KONI-Llama3.1-8B-Instruct-20241024	1	90%	4		3		2
OpenPipe-Llama-3.1-8B-Instruct	1	90%	4		3		2
Skywork-Skywork-Reward-Llama-3.1-8B	1	90%	4		3		2
utter-project-EuroLLM-1.7B-Instruct	1	90%			27		2
learning-unit-L1-16B-A3B	1	90%	1		24
zai-org-glm-4-9b-chat	5	89%			2	24
zai-org-glm-4-9b-chat-1m	2	89%			2	24
ferscm44-tiny-aya-agent	1	89%	5		2
sandbagging-games-cedar	1	89%	5				2
zai-org-GLM-5.2	13	89%	1		24	2
zai-org-GLM-4.5-Air	11	89%	1		24	2
zai-org-GLM-4.5V	2	89%	1		24	2
GLM-4.6	1	89%	1		24	2
zai-org-GLM-4.6V	1	89%	1		24	2
zai-org-GLM-4.6V-Flash	1	89%	1		24	2
zai-org-GLM-OCR	1	89%	1		24	2
utter-project-TowerVision-2B	6	89%			27
BananaMind-BananaMind-2-Medium-Chat	3	89%			27
ByteDance-Seed-Seed-Coder-8B-Instruct	3	89%			27
microsoft-DialoGPT-medium	3	89%			27
moonshotai-Moonlight-16B-A3B-Instruct	3	89%			27
ytu-ce-cosmos-Turkish-Gemma-9b-v0.1	3	89%			27
zai-org-chatglm3-6b-32k	3	89%			27
zai-org-LongWriter-glm4-9b	3	89%			27
abdelkader-dev-Quark-flash-1	1	89%			27
abhishekai-slm-125m-legal-sft	1	89%			27
ghost-x-ghost-8b-beta-1608	1	89%			27
microsoft-bitnet-b1.58-2B-4T	1	89%			27
mirajbhandari-NER_GEMMA_FINETUNE	1	89%			27
PocketDoc-Dans-PersonalityEngine-V1.3.0-24b	1	89%			27
submarat-gpt2-small-fineweb-edu-10b-chat	1	89%			27
winninghealth-WiNGPT2-Llama-3-8B-Chat	1	89%			27
zai-org-chatglm3-6b	1	89%			27
stepfun-ai-Step-Audio-EditX	1	89%			27	1
google-diffusiongemma-26B-A4B-it	1	88%	1	22		4	2
meituan-longcat-LongCat-Flash-Chat	1	88%	1	27	1
tencent-Hy-MT2-30B-A3B	1	88%	1	26		1	1
KristianS7-Ouro-1.4B	1	87%	2		25
allenai-BAR-2x7B-Math	8	87%	2	1		1	25
llama-cpp-rwkv-world	1	87%	1		1	5	27
ai9stars-G9v3-3B	1	86%	2		27		1
crusoeai-DeepSeek-V4-Flash	1	86%		20	2	20
Nanbeige-Nanbeige4.2-3B	5	86%	3		24
BAAI-Gemma2-9B-IT-Simpo-Infinity-Preference	1	86%	2		25
NightPrince-Muslim-6B-PRO	1	86%	3		24
openbmb-MiniCPM5-1B	10	86%	2		27	2	1
openbmb-MiniCPM-SALA	1	86%	2		27	2	1
openbmb-MiniCPM5-1B-Base	1	86%	2		27	2	1
NousResearch-Hermes-2-Pro-Llama-3-8B-tool_use	1	85%	3			23	2
c4tdr0ut-grok-oss-Revenant-70B	48	85%	7				2
microsoft-FrogBoss-32B-2510	1	84%	2	26	4	3	1
stepfun-ai-RLVR-8B-0926	2	84%				47
tiiuae-Falcon-H1-Tiny-Tool-Calling-90M	8	84%	4			23
stepfun-ai-Step3-VL-10B	1	84%				47
edwardcapriolo-antares-350m-JQ4	2	84%				48
tiiuae-Falcon3-7B-Instruct	8	84%	4			24
arcee-ai-Trinity-Large-Thinking	1	83%			4	45
openbmb-MiniCPM3-4B	3	83%	4		25
Qwen-Qwen2-VL-2B	4	83%					49
ai-sage-GigaChat3.1-10B-A1.8B	1	83%		49
GigaChat3-10B-A1.8B	1	83%		49
GigaChat3.1-10B-A1.8B	1	83%		49
microsoft-VibeVoice-ASR-HF	1	83%					49
HuggingFaceTB-nanowhale-100m-base	3	83%	2			5	26
penfever-grug-67b-a2b-sft-s2-thinking-step630	3	83%			1	45	4
ewald1976-G4-Plainsong-12B	1	83%				48	2
Indexnusrefather-Qwen-3.5-2b-roleplay-tuned-v2-ERP-tolerant	4	83%			2	47	1
lordx64-Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled	5	83%			2	47	1
jica98-qwen3.5-4B-super-coder	3	83%			2	47	1
zai-org-GLM-ASR-Nano-2512	1	83%			25		25
Monibee-Fudgekins-gemma-coder	3	83%			3	48
google-gemma-4-31B-it-interleaved	1	83%			3	48
zai-org-GLM-4.1V-9B-Thinking	4	83%			26		25
zai-org-AutoGLM-Phone-9B-Multilingual	3	83%			26		25
naver-hyperclovax-HyperCLOVAX-SEED-Think-32B	1	82%	1			46
nex-agi-Nex-N2-mini	1	82%	1		2	44
Cohere2MoE	2	82%	1			44	2
zonzonzik-NMC-heretic-2-heretic	2	82%	1			44	2
Kwaipilot-KAT-Coder-V2.5-Dev	2	82%	1			46
favjiw-Llama-3-8B-Instruct-Indonesian-Legal-GRPO	1	82%			27		25
HuggingFaceTB-SmolLM3-3B-GSM8K-SFT	1	82%			27		25
amazon-GDN-primed-HQwen3-8B-Instruct	3	82%	1			46	1
t-tech-T-lite-it-2.1	3	82%	1			46	1
nc-ai-consortium-VAETKI-20B-A2B	1	82%	1			46	1
STEVENZHANG904-Qwen3-8B-Meta-SecAlign	1	82%	1			46	1
Qwen-Qwen3-4B-Thinking-2507	30	82%	1			46	1
FINAL-Bench-Darwin-28B-REASON	7	82%	1			46	1
FINAL-Bench-Darwin-28B-KR-Legal	1	82%	1			46	1
CohereForAI-c4ai-command-r7b-12-2024-tool_use	1	82%			3	48	2
arcee-ai-Trinity-Mini	1	82%	1			47	1
kakaocorp-kanana-2-30b-a3b-thinking-2601	2	82%	1			47	1
CohereLabs-command-a-plus-05-2026-w4a4	1	82%	1			46	2
lancejames221b-HAWQ	1	82%	2			42
StepFun3.5-Flash	2	82%	1		2	46
stepfun-ai-Step-3.5-Flash-Base-Midtrain	1	82%	1		2	46
stepfun-ai-Step-3.7-Flash	1	82%	1		2	46
google-gemma-4-26B-A4B-it	17	81%	1			47	2
MiniMaxAI-MiniMax-M2	1	81%	1			47	2
MiniMaxAI-MiniMax-M2.1	1	81%	1			47	2
Apertus-8B-Instruct	1	81%	5	1		24		1
thunkaboutit-Koleslaw-Nemotron-30B-A3B-PromptEnhance	1	81%	1		3	47
internlm-Intern-S2-Preview-397B	1	81%	1		2	46	2
stepfun-ai-step3	1	81%	1		4	47
Jackrong-Qwopus3.6-35B-A3B-Coder	1	81%	2			45
inclusionAI-Ring-mini-linear-2.0	3	80%	2			46
yuxinlu1-gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2	10	80%		6	2	48	2
Gryphe-Gemma-4-31B-StyleTune	3	80%		6	2	48	2
baseten-Llama-3.2-3B-Instruct-pythonic	1	80%			7		2	49
BAAI-Emu3-Chat-hf	3	80%					49
StanfordAIMI-GREEN-RadLlama2-7b	2	80%					49
google-gemma-4-31B-it	4	80%		6	3	48	2
darthcrawl-Lilith-31B-v1.0	1	80%		6	3	48	2
unsloth-Ornith-1.0-397B	1	80%	2		2	45
empero-ai-Qwythos-9B-Claude-Mythos-5-1M	2	80%	3			42
deepseek-ai-DeepSeek-R1-Distill-Qwen-32B	1	80%	2			46	2
NVIDIA-Nemotron-Nano-v2	1	80%	2	1		45	2
nvidia-Nemotron-Labs-Diffusion-VLM-8B	1	80%	2		2	46
poolside-Laguna-S-2.1	5	80%	2	47			1
NVIDIA-Nemotron-3-Nano-30B-A3B-BF16	5	79%	2		3	46
trl-internal-testing-tiny-NemotronHForCausalLM-super	3	79%	2		3	46
nvidia-Nemotron-Cascade-2-30B-A3B	1	79%	2		3	46
trl-internal-testing-tiny-NemotronHForCausalLM-ultra	1	79%	2		3	46
unsloth-NVIDIA-Nemotron-3-Nano-4B	1	79%	2		3	46
Kwaipilot-SRPO-Qwen-32B	1	79%			2		49
allenai-tmax-9b	11	79%	3		2	42
mkd-ai-Keural-Nova-v1.0	6	79%	3			44
Ddavidich-LOMONOSOV-ZENIT-27B-1M-INDEV	1	79%	3			44
Qwen-Qwen3.6-35B-A3B	44	79%	3			44
Darksp33d-loreweaver-rp-v1	1	79%	3			44
OBLITERATUS-Qwen3.6-27B-OBLITERATED	1	79%	3			44
Qwen-Qwen-AgentWorld-35B-A3B	1	79%	3			44
microsoft-FrogMini-14B-2510	1	79%	6	22		3	1
deepseek-ai-DeepSeek-V4	1	79%	2	47	1	1	1
zai-org-glm-edge-4b-chat	3	79%			27		25
lewtun-talkie-1930-13b-it-hf	1	79%			27		25
rubenroy-NeuraNET-Zero-18B-Preview	1	79%			27		25
Se00n00-TinyLM-1-70M	1	79%			27		25
Skywork-Skywork-R1V3-38B	1	79%	3			45
stepfun-ai-Step-Audio-R1.1	1	79%	3			45
unsloth-DeepSeek-V4-Flash	1	79%	2	47	1	2	1
tencent-Hy-Embodied-0.5-VLA-UMI	1	79%	2	26		24	1
KSP-NMAI-Boris-1.3-75M-Instruct	1	78%			27		26
tencent-HunyuanOCR	1	78%			25		28
Qwen-Qwen3.5-0.8B	21	78%	3		2	44
openbmb-MiniCPM-V-4.6	3	78%	3		2	44
YoozLabs-Yooz-Quality-v3-Qwen3.5-4B	2	78%	3		2	44
MaXoN654-RUSQL-0.8B-Text2SQL	1	78%	3		2	44
Qwen3.5-4B	73	78%	3		2	44
InternScience-Agents-A1-4B	4	78%	3		2	44
Goekdeniz-Guelmez-JOSIE-2-4B-Preview	1	78%	3		2	44
internlm-Intern-S2-Preview	1	78%	3		2	44
openbmb-MiniCPM-V-4.6-Thinking	1	78%	3		2	44
tiiuae-Falcon-H1-Tiny-90M-Instruct	3	78%	2	26		24	2
tencent-Hy-MT2-1.8B	6	78%			27		27
zai-org-GLM-4-9B-0414	2	78%			27	1	26
dots-studio-dots.llm1.base	1	78%			27		27
dots-studio-dots.llm1.inst	1	78%			27		27
sabari2005-cyberslm-33m-instruct	1	78%			27		27
prefeitura-rio-Rio-3.2-Open-0.8B-Instruct	1	78%	3		2	45
nvidia-Llama-3.1-Nemotron-Nano-8B-v1	2	78%	10		3		2
inclusionAI-SingGuard-0.8b	1	78%	3	1	2	44
tencent-Hy-Embodied-RxBrain-1.0	4	78%	2	26		27	1
poolside-Laguna-XS.2	2	78%	2	47		6	1
poolside-Laguna-XS-2.1	1	78%	2	47		6	1
PrimeIntellect-INTELLECT-3	1	77%	4		2	42
PrimeIntellect-INTELLECT-3.1	1	77%	4		2	42
XMB480-Qwen3.6-fixed-tool-calling	1	75%		27	2	44
meta-llama-Llama-3.2-3B-Instruct	55	75%	4		3		2	45
Novaciano-SEX_ROLEPLAY-3.2-1B	2	75%	4		3		2	45
PleIAs-Monad	2	74%			27	48
Motif-Technologies-Motif-2-12.7B-Instruct	1	74%			27	48
Apriel-1.6-15b-Thinker-fixed	1	74%			27	48
EleutherAI-less-replication-7b-warmup	1	74%			27		48
JetBrains-Mellum2-12B-A2.5B-Base-Pretrain	1	74%			27		48
microsoft-UserLM-8b	1	74%			27	48
K-intelligence-Midm-2.0-Mini-Instruct	2	74%			27			49
arcee-ai-AFM-4.5B-Preview	1	74%		49	27
ServiceNow-AI-Apriel-1.6-15b-Thinker	4	74%			28	48
fdtn-ai-Foundation-Sec-8B-Reasoning	1	74%			2	48	27
MiniMaxAI-MiniMax-M2.7	2	73%	1		24	47	2
MiniMax-M2	1	73%	1		24	47	2
MiniMaxAI-MiniMax-M2.5	1	73%	1		24	47	2
ai-sage-GigaChat3.1-Audio-10B-A1.8B	1	73%	6	43
upstage-solar-pro2-tokenizer	1	72%	1	1	1	47	26
allenai-Olmo-3-7B-Think	9	71%	2	1		46	25
allenai-Olmo-3-32B-Think	4	71%	2	1		46	25
allenai-Olmo-3-7B-Think-SFT	4	71%	2	1		46	25
ibm-granite-granite-speech-4.1-2b	2	71%			23		48
ServiceNow-AI-Apriel-Nemotron-15b-Thinker	2	71%	2		28	46
arcee-ai-llama-8b-sft-qlora	1	69%			27	48
upstage-Solar-Open-100B	3	68%	3	26		1	3	46
unsloth-gpt-oss-20b	1	67%	3	26		2	5	46
Motif-Technologies-Motif-3-Beta	1	66%		27	25	47
Qwen-Qwen3-Reranker-0.6B	6	66%		49	1		49
bezzam-VibeVoice-7B-hf	1	66%		49		1	49
Harikrish2727-BetterGPT-150M	1	66%		48	27		25
reaperdoesntknow-Symbiotic-Beta	1	66%		49	27		25
SvalTek-Q3-CharThink-9B-temp	1	65%	1	24	25	47
baidu-ERNIE-4.5-VL-28B-A3B-PT	2	65%		33		45	25
baidu-ERNIE-4.5-VL-424B-A47B-Base-Paddle	1	65%		33		45	25
HuggingFaceTB-SmolLM3-3B@3e0e27	4	64%	1	21	27		3	48
deepseek-ai-DeepSeek-V3.2	1	63%	3	45		45	1
emrevrg-Norovox-Alpha-MoE-13.1B	8	59%			27	48	25
tcclaviger-Laguna-S-2.1-RFA_L	1	59%		27	2	44		49
zai-org-GLM-5.1	2	59%	1	46	24	46
mindlab-research-Macaron-V1-Preview-749B	1	59%	1	46	24	46
GLM-4.7-Flash	11	58%	1	47	24	47
zai-org-GLM-5	1	58%	1	47	24	47
Nekochu-nanochat-d24	1	58%		49	27		48
Motif-Technologies-Motif-2-12.7B-Reasoning	1	57%		49	27	48	1
praxisresearch-hf_olmo_32b_em_insecure_sgtr_1	9	57%	21
ibm-granite-granite-guardian-3.2-8b-factuality-detection	1	57%	1		26	47	47	1
allenai-Olmo-3-7B-Instruct-SFT	5	55%	22
allenai-Olmo-Hybrid-Instruct-SFT-7B	2	55%	22
allenai-olmo-3-tokenizer-instruct-release	1	55%	22
allenai-Olmo-3.1-32B-Instruct	1	55%	22
DeepHat-DeepHat-V1-7B	1	54%	22				2
allenai-Olmo-3-7B-Instruct	4	53%	23
tencent-Hy-Embodied-VLM-1.0	1	51%		49		47	49
kakaocorp-kanana-safeguard-8b	1	50%			49	48	49
kakaocorp-kanana-safeguard-prompt-2.1b	1	50%			49	48	49
kakaocorp-kanana-safeguard-siren-8b	1	50%			49	48	49
skt-A.X-4.0-Light	1	50%	24		3
Qwen-Qwen2.5-VL-7B-Instruct	66	49%	25
internlm-CapRL-InternVL3.5-8B	25	49%	25
Qwen-Qwen2.5-Omni-3B	23	49%	25
OpenGVLab-InternVL3_5-1B	20	49%	25
LFM2.5-8B-A1B	9	49%	25
LiquidAI-LFM2.5-350M	5	49%	25
moonshotai-Kimi-VL-A3B-Instruct	5	49%	25
LiquidAI-LFM2-VL-450M	3	49%	25
LiquidAI-LFM2.5-VL-1.6B	3	49%	25
Salesforce-GTA1-32B	3	49%	25
XiaomiMiMo-MiMo-V2.5	2	49%	25
DireDreadlord-Gallium-350M	1	49%	25
ibm-granite-granite-4.0-3b-vision	1	49%	25
ibm-granite-granite-docling-258M	1	49%	25
ibm-granite-granite-vision-4.1-4b	1	49%	25
moonshotai-Kimi-K2-Base	1	49%	25
nvidia-LocateAnything-3B	1	49%	25
nvidia-NVILA-8B-HD-Video	1	49%	25
Qwen-QVQ-72B-Preview	1	49%	25
tencent-Youtu-Parsing	1	49%	25
tencent-Youtu-VL-4B-Instruct	1	49%	25
trl-internal-testing-tiny-Llama4ForCausalLM	1	49%	25
typhoon-ai-typhoon2.1-gemma3-4b	1	49%	25
unsloth-mistral-Devstral-Small-2507	1	49%	25
XiaomiMiMo-MiMo-V2.5-Pro	1	49%	25
sophosympatheia-StrawberryLemonade-L3-70B-v1.0	1	49%	25		1
Qwen-Qwen3-VL-8B-Instruct	47	48%	25				2
Qwen-Qwen3-VL-Embedding-8B	3	48%	25				2
nvidia-Llama-3_3-Nemotron-Super-49B-v1	1	48%	25		2
Qwen-Qwen3-Omni-30B-A3B-Thinking	2	48%	25			2
Qwen-Qwen3-Omni-30B-A3B-Instruct	1	48%	25			2
kakaocorp-kanana-1.5-v-3b-instruct	1	48%	25				2
BAAI-BGE-VL-MLLM-S1	3	48%	25			1	2
BAAI-BGE-VL-v1.5-mmeb	2	48%	25			1	2
DeSTA-ntu-Llama-3.1-8B-Instruct	1	48%	25		1		2
baidu-Qianfan-OCR	1	48%	25			1	2
ByteDance-Seed-academic-ds-9B	12	48%	25			2	1
yujiepan-deepseek-v3-tiny-random	1	48%	25			2	1
LiquidAI-LFM2.5-VL-450M	3	48%	25					4
LiquidAI-LFM2.5-8B-A1B-Base	1	48%	25					4
stepfun-ai-StepFun-Formalizer-7B	19	47%	25			2	2
deepseek-ai-DeepSeek-V2.5	3	47%	25			2	2
deepseek-ai-DeepSeek-Coder-V2-Instruct-0724	1	47%	25			2	2
OpenGVLab-InternVL3-1B	34	47%	26
Kimi-K2-Instruct	1	47%	26
Kimi-K2-Thinking	1	47%	26
dphn-Dolphin-Mistral-24B-Venice-Edition	2	47%	25		2	3		1
henriqueimoveis-Echoes-1-Instruct-PT-BR	1	47%	25		2		4
allenai-olmo-3.2-tokenizer-think-dev	1	46%	22			27
allenai-olmo-3.2-tokenizer-think-release	1	46%	22			27
nvidia-Llama-3_3-Nemotron-Super-49B-v1_5	1	46%	26		1		2
openchat-openchat-3.6-8b-20240522	1	46%	26		3
korkmazumut-orkhon-tangri	1	46%	26		3
N8Programs-talkie-box	1	46%	26		3
naver-hyperclovax-HyperCLOVAX-SEED-Omni-8B	1	46%	22		3	25
trl-internal-testing-tiny-Cohere2ForCausalLM	10	45%	25		6		3
zai-org-webrl-llama-3.1-8b	2	45%	26		1		4
zai-org-webrl-glm-4-9b	1	45%	26		1		4
internlm-internlm2_5-7b-chat	105	45%	27
openbmb-VoxCPM2	99	45%	27
OpenBuddy-openbuddy-yi1.5-9b-v21.1-32k	60	45%	27
upstage-solar-pro-preview-instruct	52	45%	27
Qwen-Qwen2-1.5B-Instruct	47	45%	27
Adamaja111-legal-qwen2.5-1.5b-ft	36	45%	27
Qwen-Qwen2-0.5B	25	45%	27
tiiuae-falcon-mamba-tiny-dev	22	45%	27
HuggingFaceTB-SmolLM2-135M-Instruct	21	45%	27
inclusionAI-Ling-lite-1.5	17	45%	27
deepseek-ai-deepseek-vl2-small	10	45%	27
openbmb-MiniCPM4.1-8B	10	45%	27
microsoft-Phi-3.5-vision-instruct	9	45%	27
microsoft-Phi-4-multimodal-instruct	9	45%	27
deepseek-ai-deepseek-coder-6.7b-instruct	7	45%	27
NousResearch-Hermes-3-Llama-3.1-8B	7	45%	27
occiglot-occiglot-7b-it-en-instruct	6	45%	27
openchat-openchat-3.5-0106	6	45%	27
SepehrKerachi-olmo3_base	6	45%	27
spitfire4794-Zupra-1.7-50M-Instruct-Ultra-Math-exp	6	45%	27
BAAI-AquilaMoE	5	45%	27
chujiezheng-LLaMA3-iterative-DPO-final-ExPO	5	45%	27
LGAI-EXAONE-EXAONE-3.5-7.8B-Instruct	5	45%	27
microsoft-Phi-3-small-8k-instruct	5	45%	27
mlabonne-NeuralMonarch-7B	5	45%	27
tencent-Hunyuan-A13B-Instruct	5	45%	27
tencent-Hy-MT2-7B	5	45%	27
stabilityai-stable-code-instruct-3b	4	45%	27
admijgjtjtjtjjg-Llama-0.1M-MultiDataset-Instruct	3	45%	27
ai21labs-AI21-Jamba2-3B	3	45%	27
arcee-ai-Gemma-Zephyr-Dolly-Chat-Slerp	3	45%	27
BAAI-bge-reranker-v2-minicpm-layerwise	3	45%	27
inclusionAI-Ling-lite-1.5-2507	3	45%	27
RekaAI-reka-flash-3	3	45%	27
tencent-Hunyuan-7B-Pretrain-0124	3	45%	27
ByteDance-Ouro-1.4B	2	45%	27
ByteDance-Ouro-2.6B-Thinking	2	45%	27
hauser458b-bananamind2-mini-it	2	45%	27
IQuestLab-IQuest-Coder-V1-40B-Instruct	2	45%	27
microsoft-Phi-tiny-MoE-instruct	2	45%	27
Motif-Technologies-Motif-2.6B	2	45%	27
tiiuae-Falcon-E-3B-Instruct	2	45%	27
Abdelkareem-supra-50m-arabic-sft-mix	1	45%	27
Abhayn01-Arka-Specialist-V2-Pretrained	1	45%	27
alpindale-WizardLM-2-8x22B	1	45%	27
amalia-llm-AMALIA-9B-0626-DPO	1	45%	27
AvitoTech-avibe	1	45%	27
BSC-LT-salamandra-2b-instruct	1	45%	27
ByteDance-Seed-M3-Agent-Control	1	45%	27
DireDreadlord-Dragon-1.5-0.5B	1	45%	27
EldritchLabs-KrakenSakura-Maelstrom-12B-v1	1	45%	27
GiorgiDuchidze-kona2-ilia	1	45%	27
inclusionAI-Ling-lite	1	45%	27
inclusionAI-Ring-lite-linear-preview	1	45%	27
IQuestLab-IQuest-Coder-V1-7B-Base	1	45%	27
jinaai-ReaderLM-v2	1	45%	27
ministral-Ministral-3b-instruct	1	45%	27
miromind-ai-MiroThinker-1.7	1	45%	27
Nanbeige-ToolMind-Web-3B	1	45%	27
NLPark-AnFeng_v3_Avocet	1	45%	27
qnguyen3-nanoLLaVA	1	45%	27
ragraph-ai-stable-cypher-instruct-3b	1	45%	27
rombodawg-rombos_Replete-Coder-Llama3-8B	1	45%	27
S4nfs-Neeto-1.0-8b	1	45%	27
SeaLLMs-SeaLLM-7B-v2	1	45%	27
SeaLLMs-SeaLLM-7B-v2.5	1	45%	27
stepfun-ai-Step-Audio-Chat	1	45%	27
trl-internal-testing-dummy-GPT2-correct-vocab	1	45%	27
trl-internal-testing-tiny-DbrxForCausalLM	1	45%	27
vereniaes-phi35-mini-alpaca-id	1	45%	27
moonshotai-Kimi-K2.6	3	45%	27
moonshotai-Kimi-Linear-48B-A3B-Instruct	3	45%	27
moonshotai-Kimi-K2-Thinking	2	45%	27
moonshotai-Kimi-K2-Instruct	1	45%	27
moonshotai-Kimi-K2-Instruct-0905	1	45%	27
moonshotai-Kimi-K2.5	1	45%	27
moonshotai-Kimi-K2.7-Code	1	45%	27
ibm-granite-granite-3.1-8b-instruct	6	45%	27					1
ibm-granite-granite-speech-3.2-8b	5	45%	27					1
ibm-granite-granite-speech-3.3-2b	4	45%	27					1
ibm-granite-granite-3.3-2B-Instruct	1	45%	27					1
ibm-granite-granite-3.3-8b-math-prm-v2	1	45%	27					1
ibm-granite-granite-speech-3.3-8b	1	45%	27					1
IQuestLab-IQuest-Coder-V1-40B-Loop-Thinking	1	45%	27			1
kakaocorp-kanana-1.5-8b-instruct-2505	14	45%	27			1
kakaocorp-kanana-1.5-15.7b-a3b-instruct	1	45%	27			1
meetkai-functionary-medium-v3.2	1	45%	27			1
mistralai-Mistral-Small-3.1-24B-Instruct-2503	4	45%	27					1
ShyliaSafetensors-Ariel-Alloy-V1-24B-Heretic	2	45%	27					1
allenai-Olmo-Hybrid-Think-SFT-7B	1	44%	23			26
nvidia-Nemotron-Mini-4B-Instruct	1	44%	27			1	1
QCRI-Fanar-2-27B-Instruct	1	44%	27				2
Qwen-Qwen3-0.6B	63	44%	27			2
fableforge-ai-ReasonCritic-7B	5	44%	27			2
TeichAI-Qwen3-14B-Claude-4.5-Opus-High-Reasoning-Distill	4	44%	27			2
unsloth-Qwen3-8B	3	44%	27			2
baichuan-inc-Baichuan-M2-32B	1	44%	27			2
EphAsad-Atem-8B	1	44%	27			2
openbmb-MiniCPM-o-4_5	1	44%	27			2
openbmb-MiniCPM-V-4_5	1	44%	27			2
Vikhrmodels-QVikhr-3-1.7B-Instruction-noreasoning	1	44%	27			2
ibm-granite-granite-3.0-8b-instruct	9	44%	27				2
microsoft-phi-4	4	44%	27				2
Delentia-delentia-slm-jitna-v0.4	1	44%	27				2
nvidia-OpenReasoning-Nemotron-32B	1	44%	27				2
spitfire4794-supra-arith2	1	44%	27				2
microsoft-Phi-4-mini-reasoning	4	44%	27			1	1
deepseek-ai-DeepSeek-V2-Lite-Chat	31	44%	27				2
openbmb-OmniLMM-12B	27	44%	27				2
microsoft-Phi-3.5-mini-instruct	24	44%	27				2
microsoft-Phi-3-mini-4k-instruct	18	44%	27				2
ibm-granite-granite-8b-code-instruct-4k	12	44%	27				2
OpenBuddy-openbuddy-deepseek-67b-v18.1-4k	10	44%	27				2
upstage-SOLAR-10.7B-Instruct-v1.0	8	44%	27				2
KSP-NMAI-Boris-125M-Instruct	4	44%	27				2
ibm-granite-granite-7b-instruct	2	44%	27				2
allenai-OLMoE-1B-7B-0924-Instruct	1	44%	27				2
amylynn-Riverfish-Rocinante-12B-SFT	1	44%	27				2
amylynn-Riverfish-Rocinante-12B-SFT-DPO	1	44%	27				2
antoniogr7-zagreus-0.4B-ita-elicit	1	44%	27				2
deepseek-ai-DeepSeek-V2-Chat-0628	1	44%	27				2
eivintobias-heartly-rwkv7-1.5b-v2	1	44%	27				2
exnivo-tinybrain-2b-instruct	1	44%	27				2
FINAL-Bench-Aether-7B-5Attn-it	1	44%	27				2
katuni4ka-tiny-random-phi3	1	44%	27				2
mixtao-MixTAO-7Bx2-MoE-v8.1	1	44%	27				2
MTSAIR-multi_verse_model	1	44%	27				2
sbintuitions-tiny-lm-chat	1	44%	27				2
spitfire4794-supra-arith3	1	44%	27				2
spitfire4794-supra-arith4	1	44%	27				2
sverma8873-slm-125m-legal-sft	1	44%	27				2
tiiuae-falcon-11B	1	44%	27				2
tencent-Youtu-HiChunk	1	44%	27			2	1
tencent-Youtu-LLM-2B	1	44%	27			2	1
arcee-ai-Virtuoso-Lite	4	44%	27			1	2
allenai-Llama-3.1-Tulu-3-8B-SFT	31	44%	27			1	2
allenai-OLMoE-1B-7B-0125-Instruct	18	44%	27			1	2
AmberYifan-capsdnum-marin-8b-base-science_random_b4000_s0	8	44%	27				4
Featherless-Chat-Models-llama2-13b-chat-hf	1	44%	27				4
QCRI-Fanar-1-9B-Instruct	1	44%	27				4
TIGER-Lab-MAmmoTH2-8B-Plus	1	44%	27				4
zai-org-webrl-orm-llama-3.1-8b	1	44%	27				4
Salesforce-xLAM-1b-fc-r	3	43%	27				4
ChenWeiLi-MedPhi-3-mini_v1	1	43%	27				4
Infinirc-Infinirc-Llama3-8B-2G-Release-v1.0	1	43%	27				4
nvidia-Mistral-NeMo-Minitron-8B-Instruct	1	43%	27			1	3
nvidia-Nemotron-H-8B-Base-8K	1	43%	27			1	3
zai-org-androidgen-glm-4-9b	1	43%	27				4
zeng123-PonderLM-2-Pythia-1.4b	1	43%	27				4
jondurbin-bagel-dpo-34b-v0.2	11	43%	27	3			2
augmxnt-shisa-gamma-7b-v1	1	43%	27	3			2
UnicomLLM-Unichat-llama3-Chinese-8B	2	43%	27			5
fdtn-ai-Foundation-Sec-8B-Instruct	1	43%	27			1	4
Bielik-11B-v3.0-Instruct	1	43%	27	2		2	2
tiiuae-Falcon-H1R-7B	1	43%	27	2		3	2
inclusionAI-SingGuard-2b	4	42%	25	1		19
unsloth-r1-1776	1	42%	28			1	1
LukeBailey181Pub-dspv2_guide	1	42%	28			2	1
Qwen-Qwen3-VL-8B-Thinking	12	41%	25			22
internlm-Intern-S1-Pro	1	41%	25			22
apple-sage-ft-mixtral-8x7b	1	41%	28				4
TIGER-Lab-MAmmoTH2-8x7B-Plus	1	41%	28				4
tiiuae-Falcon-H1-0.5B-Instruct	9	41%	27	4	4		2
nvidia-Cosmos3-Edge	2	41%	25			23
baidu-ERNIE-4.5-21B-A3B-Base-PT	5	41%	27	9			2
ArmGPT-ArmenianGPT-1.0-3B	1	41%	29
mistralai-Ministral-3-14B-Reasoning-2512	1	41%	29
internlm-Intern-S1-mini	2	41%	25			22	2
mistralai-Mistral-Large-Instruct-2411	2	41%	29
DarkArtsForge-Paimon-24B-v1	1	41%	29
darkc0de-XORTRON.CriminalComputing.LARGE.2026.3	1	41%	29
darkc0de-XortronCriminalComputingConfig	1	41%	29
ByteDance-Seed-M3-Agent-Memorization	1	41%	25	24
Qwen-Qwen3-ForcedAligner-0.6B-hf	1	41%	25				24
deepseek-ai-DeepSeek-R1-Distill-Qwen-32B@56a144	29	40%	25			23	2
deepseek-ai-DeepSeek-R1-Distill-Llama-8B	1	40%	25			23	2
Skywork-Skywork-R1V-38B	1	40%	25			23	2
BAAI-BGE-VL-Screenshot	1	40%	25			23	2
mistralai-Mistral-Small-24B-Instruct-2501	5	40%	29					1
arcee-ai-Arcee-Blitz	2	40%	29					1
Qwen-Qwen3-ASR-1.7B	4	40%	25			1	24
Qwen-Qwen3-ASR-0.6B-hf	2	40%	25			1	24
atomgptlab-microscopy_gpt_llama3.2_vision_11b	1	40%	25		1		2	24
nvidia-NVIDIA-Nemotron-3-Ultra-550B-A55B-GenRM	1	40%	25		2	23	2
trl-internal-testing-tiny-DeepseekV3ForCausalLM-0528	1	40%	29			2	1
tencent-Penguin-VL-2B	2	39%	26			22
sbintuitions-sarashina2.2-3b-instruct-v0.1	3	39%	29		2	2
zai-org-glm-edge-v-2b	2	39%	25				24
MiniMaxAI-MiniMax-VL-01	1	39%	25				24
nvidia-NVIDIA-Nemotron-Parse-v1.1-TC	1	39%	25				24
Sangsang-R1-7B-thinksafe-r1-7B-ablation-32-pm-3ep-harmful_only	2	39%	25			23	2
Aletheia-Bench-DPO-Think-14B	1	39%	25			23	2
rita-cohere-tya-eng-v1-t06-user	1	39%	25			23	2
swiss-ai-Apertus-v1.1-4B-Instruct	5	38%	30				1
FORMAS-Carapicu-Qwen3-0.6B-CPT-SFT	1	38%	27			20
Huyisbeee-SFT-Guard-Llama-Guard-3-1B	1	38%	27			20
nvidia-Nemotron-Cascade-14B-Thinking	1	38%	27			20
tiny-random-phi-4	1	38%	27	18			2
internlm-internlm2-1_8b-reward	4	38%	27			21
inclusionAI-ASearcher-Web-QwQ	3	38%	27			21
Kwaipilot-HiPO-8B	3	38%	27			21
LGAI-EXAONE-EXAONE-Deep-7.8B	3	38%	27			21
Qwen-WorldPM-72B-UltraFeedback	3	38%	27			21
inclusionAI-Ring-mini-2.0	2	38%	27			21
Qwen-QwQ-32B@16488a	2	38%	27			21
inclusionAI-Ring-lite	1	38%	27			21
inclusionAI-Ring-lite-distill-preview	1	38%	27			21
Qwen-QwQ-32B	1	38%	27			21
ai21labs-AI21-Jamba-Reasoning-3B	1	38%	27			21
Qwen-Qwen2.5-Math-PRM-7B	5	38%	27			21
microsoft-Phi-4-reasoning	3	38%	27				21
microsoft-Phi-4-reasoning-vision-15B	1	38%	27				21
spitfire4794-supra-arith	1	38%	27				21
arcee-ai-AFM-4.5B	1	38%	27	21
nvidia-Qwen3-Nemotron-235B-A22B-GenRM-2603	1	38%	26		1	22	4
HuggingFaceTB-SmolVLM-256M-Instruct	8	38%	26				23
HuggingFaceTB-SmolVLM-256M-Base	6	38%	26				23
HuggingFaceTB-SmolVLM-Instruct-DPO	1	38%	26				23
HuggingFaceTB-SmolVLM-Synthetic	1	38%	26				23
01-ai-Yi-1.5-9B-Chat	10	37%	27	18			4
utter-project-SpireNoBlocks	5	37%	27	22
Salesforce-xgen-small-4B-base-r	3	37%	27	22
01-ai-Yi-Coder-9B-Chat	2	37%	27	18			4
BAAI-AquilaMed-RL	1	37%	27	18			4
Langboat-Mengzi3-8B-Chat	1	37%	27	18			4
qnguyen3-Master-Yi-9B	1	37%	27	18			4
SalmanHabeeb-qwen-llamafiles	1	37%	27	22
Sensente-Swen-28M	1	37%	27	18			4
turboderp-Cat-Llama-3-70B-instruct	1	37%	27	22
nvidia-NVIDIA-Nemotron-Nano-12B-v2	3	37%	27	1		20	2
nvidia-NVIDIA-Nemotron-Nano-9B-v2-Japanese	1	37%	27	1		20	2
nvidia-Nemotron-Content-Safety-Reasoning-4B	39	37%	31
CraneAILabs-ganda-gemma-1b	9	37%	31
c4tdr0ut-grok-oss-Apollyon-24B	3	37%	31
zai-org-GLM-Z1-32B-0414	3	37%	27			21	3
unsloth-gemma-3-270m-it	2	37%	31
debowd-sumeru-rm	1	36%	31		1
CohereForAI-c4ai-command-r-plus-tool_use	1	36%	27		4	21
allenai-Olmo-3-7B-RL-Zero-Math	1	36%	27			21
allenai-Olmo-3.1-7B-RL-Zero-Code	1	36%	27			21
allenai-Olmo-3.1-7B-RL-Zero-Math	1	36%	27			21
OpenBuddy-OpenBuddy-R1-0528-Distill-Qwen3-32B-Preview1-QAT-AWQ4	1	36%	27			21
optimum-intel-internal-testing-tiny-random-Phi3ForCausalLM	12	36%	27				21
llm-jp-llm-jp-3-1.8b-instruct	6	36%	27				21
parasail-ai-GritLM-7B-vllm	5	36%	27				21
microsoft-Phi-3-medium-128k-instruct	4	36%	27				21
arcee-ai-Patent-Instruct-Pro	3	36%	27				21
Azure99-blossom-v5.1-9b	2	36%	27				21
AI-MO-NuminaMath-7B-TIR	1	36%	27				21
bharatgenai-AyurParam	1	36%	27				21
bharatgenai-FinanceParam	1	36%	27				21
bharatgenai-LegalParam	1	36%	27				21
nvidia-Llama3-ChatQA-1.5-8B	1	36%	27				21
OrionStarAI-Orion-14B-Chat	1	36%	27				21
zai-org-GLM-Z1-Rumination-32B-0414	1	36%	27				21
danish-foundation-models-croco-munin-apertus-8b-da-simpo-full-50k	1	36%	31			1		1
AI4Chem-ChemLLM-7B-Chat-1_5-DPO	1	36%	27	22			4
lastmass-Qwen3_Medical_GRPO	1	36%	26		1	22	4
microsoft-MAI-DS-R1	5	36%	28			20	1
OpenGVLab-InternVL3_5-GPT-OSS-20B-A4B-Preview	1	36%	27	3			2	22
Mistral-Small-3.2-24B-Instruct-2506	1	36%	31			2		1
tencent-DeepSeek-V3.1-Terminus-W4AFP8	1	35%	28	21		1	1
YOON1v-Apex-1-DPO	1	35%	27			21	4
codellama-CodeLlama-7b-Instruct-hf	3	34%	32		1
openai-gpt-oss-120b	26	34%	27	3		2	4	22
llm-jp-llm-jp-4-8b-thinking	3	34%	27	3		2	4	22
openai-gpt-oss-safeguard-20b	2	34%	27	3		2	4	22
CrowtherLabs-Atom-Electron-1.0	1	34%	27	3		2	4	22
llm-jp-llm-jp-4-8b-instruct	1	34%	27	3		2	4	22
LLMWildling-gpt-oss-220a20b	1	34%	27	3		2	4	22
microsoft-SchGen	1	34%	27	3		2	4	22
Salesforce-FARE-20B	1	34%	27	3		2	4	22
ibm-granite-granite-guardian-3.3-8b	1	34%	29			19
llama-cpp-deepseek-r1	1	34%	29			19	2
skt-A.X-K1	1	33%	24	25	3	24
microsoft-OptiMind-SFT	8	33%	29	3				20
ibm-granite-granite-guardian-3.0-2b	3	33%	25			23	24
optimum-intel-internal-testing-tiny-random-ArceeForCausalLM	26	33%	33
mistralai-Mistral-7B-Instruct-v0.2	23	33%	33
breitburg-pure-7b-210726	1	33%	33
breitburg-pure-reasoning-7b-230726	1	33%	33
BSC-LT-salamandra-7b-instruct	1	33%	33
hmellor-tiny-random-LlamaForCausalLM	1	33%	33
huggyllama-llama-7b	1	33%	33
humain-ai-ALLaM-7B-Instruct-preview	1	33%	33
Intel-neural-chat-7b-v3-3	1	33%	33
saidutta69-c4ai-command-r7b-12-2024-heretic	1	33%	33
Salesforce-xLAM-8x22b-r	1	33%	33
ServiceNow-AI-Apriel-5B-Instruct	1	33%	33
trl-internal-testing-tiny-CohereForCausalLM	1	33%	33
ibm-granite-granite-guardian-3.1-2b	3	33%	25			23	24	1
tiiuae-falcon-7b-instruct	2	32%	33				1
swiss-ai-Apertus-8B-Instruct-2509	3	32%	33			1		1
RamaBashar22-gemma-ticket-prep	2	31%	31		1	17
codellama-CodeLlama-70b-Instruct-hf	1	30%	30		3	18
nvidia-Riva-Translate-4B-Instruct-v1.1	1	30%	30		3		18
Qwen-Qwen3Guard-Gen-4B	4	29%	27	22		21	4
Huyisbeee-SFT-Guard-Qwen3Guard-Gen-0.6B	2	29%	27	22		21	4
mistralai-Mistral-7B-Instruct-v0.3	16	24%	32		1	15	15
Neelectric-Ministral-8B-Instruct-2410_SFT_mathv00.01	5	23%	33			14	13
mistralai-Mistral-Nemo-Instruct-2407	7	23%	33			14	14
mistralai-Codestral-22B-v0.1	4	23%	33			14	14
0xA50C1A1-Mistral-Nemo-Instruct-2407-Heretic-v2	1	23%	33			14	14
nvidia-Nemotron-3-Content-Safety	1	23%	31			17	17
nvidia-Nemotron-3.5-Content-Safety	1	23%	31			17	17
microsoft-llava-med-v1.5-mistral-7b	53	2%	48
google-gemma-2-2b-it	43	2%	48
openbmb-Eurus-70b-sft	12	2%	48
allenai-MolmoAct-7B-D-Pretrain-0812	9	2%	48
allenai-Molmo-7B-D-0924	5	2%	48
planeB-user_gemma_3_1b_it	3	2%	48
h2oai-h2ovl-mississippi-800m	2	2%	48
abacusai-Smaug-Mixtral-v0.1	1	2%	48
ByteDance-Seed-Seed-Coder-8B-Reasoning	1	2%	48
h2oai-h2ovl-mississippi-2b	1	2%	48
microsoft-wavecoder-ultra-6.7b	1	2%	48
orikibd-mentor-v31-ckpt	1	2%	48
Salesforce-xLAM-7b-r	1	2%	48
Skywork-Skywork-Reward-Gemma-2-27B	1	2%	48
TotallyLegitCo-fighthealthinsurance_model_v0.5	1	2%	48
allenai-Molmo2-8B	28	2%	48			1
MiniMaxAI-MiniMax-M1-80k	5	2%	48				1
ibm-granite-granite-guardian-3.2-5b	2	2%	48			1
alpindale-Llama-Guard-3-1B	1	2%	48				1
llamas-community-LlamaGuard-7b	1	2%	48			1
MiniMaxAI-MiniMax-M1-40k	1	2%	48				1
openguardrails-OpenGuardrails-Text-2510	1	2%	48			1
allenai-MolmoWeb-4B	2	2%	48			1
MiniMaxAI-MiniMax-Text-01-hf	2	2%	48				1
nvidia-llama-nemotron-embed-vl-1b-v2	1	2%	48				1
ServiceNow-AI-AprielGuard	1	1%	48		1	1
IEITYuan-Yuan2-2B-hf	7	0%	49
ibm-granite-granite-vision-3.3-2b	6	0%	49
amd-translategemma-4b-it_rai_1.8.0_npu_4K	1	0%	49
chbae624-vllm-translategemma-12b-it	1	0%	49
fireworks-ai-llama-3-firefunction-v2	1	0%	49
ibm-granite-granite-20b-functioncalling	1	0%	49
NousResearch-Genstruct-7B	1	0%	49

What each column means

column	property	severity
renders	renders at all given only what an OpenAI-compatible server sends	error
markers	every opened turn marker is closed exactly once	error
literals	no Python repr (None, True, dict reprs) reaches the prompt	error
prefix	prompt N+1 extends prompt N rather than rewriting it	error if history is rewritten, else warning
order	message text appears in the order it was sent	error if text is dropped, else warning
time	the same conversation renders identically on two different days	warning