😱 ScreamingFace

Build model fusions, measure them honestly, and reproduce any run from a single line of text.

screamingface is the Python client. You compose a fusion — several models answering the same question, reduced to a single answer — score it against a real benchmark, and get back both the number and the exact expression that produced it.

We built this because the same pattern kept showing up: a fusion beat the best single model inside it, and the result was nearly impossible to reproduce or check. Two datapoints: reproducing DRACO, our best fusion scored 68.6% vs 60.2% for the top single model (+8.4, write-up); and small-model ensembles beat their best member in Beyond Leaderboards (Skurikhin et al., Los Alamos).

It is early, and rough in places. If you find something wrong, good: every result here is meant to be rerun and picked apart.

📚 docs.screamingface.ai · 🏆 leaderboard.screamingface.ai · ⚖️ Apache-2.0

Install

pip install "screamingface[notebook]"   # drop the extra if you are not in Jupyter

Python 3.12 or newer. You also need an engine: the runtime that actually executes a run and holds your provider keys. Run your own, or point at a hosted one — same client code either way, only the URL changes.

Quickstart

import screamingface as sf

sf.configure(engine_url="http://127.0.0.1:9108")   # your own engine, or a hosted one
sf.connect()                                       # bring your keys; the client stores none

gpt = sf.Model("openrouter/openai/gpt-5.5")
opus = sf.Model("openrouter/anthropic/claude-opus-4.8")
fusion = sf.Fusion([gpt, opus], name="frontier-pair", synthesizer=opus)

# score the solo models beside the fusion, on the same cases
report = sf.evaluate([gpt, opus, fusion], benchmark="draco", limit=1)
{c.name: c.score for c in report.candidates}

A fusion always names its synthesizer= — the thing that turns several answers into one. Nothing is guessed on your behalf. draco is the full 100-task benchmark and limit=1 runs one case of it; draco/lite and draco/smoke are cheap development protocols whose scores are deliberately not comparable to the real thing. sf.benchmarks.list() shows what your engine has installed.

What it gives you

• One interface, every provider. Compose across open and closed models, API and local. The client never calls a provider directly; your keys go to the engine, which validates and stores them encrypted.
• Evaluation you can defend. The benchmark is pinned and lives engine-side — the cases, the judge, the rubric, the answer keys. Your candidate only ever sees the prompt. That separation is the whole reason a "verified gain" means anything.
• Start from the frontier. Pull a published run's url4, change one thing, and measure the difference, so the next person starts where the last one finished.
• Real costs. Tokens and USD per candidate and in aggregate, not an estimate.

Right now this is single-turn evaluation. No multi-turn or tool-using agent loops yet.

Compose

Three immutable, network-free values. Each takes one input and returns one answer, which is why they nest.

draft = sf.Model("openrouter/openai/gpt-5.5")
review = sf.Model("openrouter/anthropic/claude-opus-4.8", prompt="Correct the previous answer.")
polish = sf.Model("openrouter/openai/gpt-5.5", prompt="Tighten it. Add nothing new.")

chain = sf.Pipeline([draft, review, polish])
chain = draft.then(review).then(polish)            # identical, just shorthand

candidate = sf.Fusion([chain, draft], synthesizer=polish)

Prompts and sampling parameters belong to the candidate, never to the benchmark:

careful = sf.Model(
    "openrouter/openai/gpt-5.5",
    prompt="Answer from primary evidence and follow every output constraint.",
    params={"reasoning_effort": "high"},
)

Unknown routes, invalid parameters, and shape mismatches all fail before anything is spent.

Read the report

result = report.candidates["frontier-pair"]

result.score
result.usage.cost_usd
result.url4                    # the exact expression that ran
result.operations              # every call underneath it
report.export("runs/draco.json")

Reproduce

Sharing a result is sharing its url4. Replay it, or turn it back into editable Python.

score = sf.leaderboards.get_score(score_id)

score.url4.to_python()      # local and free: hands back the Model/Fusion/Pipeline
sf.evaluate(score.url4)     # replay it as a fresh run

Identical url4 does not guarantee identical model output — but it does guarantee everyone is measuring the same thing.

More

• Docs (quickstart, guides, concepts) → docs.screamingface.ai
• Notebooks → examples/, starting with 00_quickstart and the no-spend 01_client_tour
• Source and issues → github.com/OpenMined/screamingface

Everything synchronous has an asynchronous twin: sf.AsyncClient uses the same names with await.

License

Apache-2.0. Open to whoever shows up to measure the next slice.