screamingface

Evaluate composable Candidate Recipes against URL4-native research Benchmarks.

Development status: immutable Model/Fusion/Pipeline authoring, Engine-backed discovery, the direct evaluation API, and the confirmed url4-cloud lifecycle are implemented. The current MVP Engine publishes canonical draco, canonical ifeval, and the separately named healthbench-worst30 challenge as complete URL4 Benchmark resources. There is no fixture, embedded benchmark runtime, or Client-side execution fallback.

Local notebook runtime

Local Engine result artifacts live in the private <data_dir>/artifacts directory rather than temporary storage. Set URL4_CLOUD_ARTIFACTS_DIR to override it; screamingface status shows the absolute serving directory and its current size. The Engine's retention policy still applies. This configuration is independent of SDK report rendering, export and recovery APIs.

Install the SDK, notebook tools, and local services together:

pip install "screamingface[runtime,notebook]"
screamingface prepare --all  # first run only
screamingface up

draco, the Benchmark the examples below use, is a local inspect Task, and so is ifeval: the plain screamingface up Engine lists neither. From a checkout, just local-stack-notebooks serves an Engine with the inspect extra beside the stack; otherwise point SCREAMINGFACE_ENGINE_URL at such an Engine.

screamingface up starts AI Gateway, Scoreboard, and the Engine in the background. Use screamingface status, screamingface doctor, screamingface logs, screamingface restart, and screamingface down to manage them. Runtime state is stored under ~/.screamingface by default; set SCREAMINGFACE_DATA_DIR or pass --data-dir to override it.

The default service ports are Gateway 9105, Scoreboard 9106, and Engine 9108. Override them with --gateway-port, --scoreboard-port, and --engine-port, or the corresponding SCREAMINGFACE_GATEWAY_PORT, SCREAMINGFACE_SCOREBOARD_PORT, and SCREAMINGFACE_ENGINE_PORT environment variables. screamingface up prints the resolved SDK environment variables; the SDK does not switch away from its hosted defaults automatically.

For scripts and troubleshooting:

screamingface status --json
screamingface doctor
screamingface logs --service engine --tail 100 --no-follow
screamingface prepare --list

Logs are timestamped, tagged by service, rotated at 10 MiB, and retain five backups. The log and its backups are readable only by you. Versions before OME-990 wrote prompts into this log; screamingface logs --purge deletes the backups and empties the live log (it does not securely erase the disk blocks). Three structural prompt carriers are redacted before anything is written: url4 q= query values, litellm's Messages: exception suffix, and litellm's debug curl -d body. The runtime also forces LITELLM_LOG=WARNING. Prompt text that appears without one of those markers is not detected, so review a log before sharing it. Benchmark preparation records a versioned manifest, skips current assets, and supports --force when a fresh download is required.

Target v1 workflow

The approved Python workflow is:

import screamingface as sf

sf.connect()  # notebook panel; connect any providers enabled by this Engine

opus = sf.Model("openrouter/anthropic/claude-opus-4.8")
gpt = sf.Model("openrouter/openai/gpt-5.5")

frontier_pair = sf.Fusion(
    [opus, gpt],
    name="frontier-pair",
    synthesizer="openrouter/anthropic/claude-opus-4.8",
)

report = sf.evaluate(
    [opus, gpt, frontier_pair],
    benchmark="draco",
    limit=1,
)

evaluate(...) requires an explicit Benchmark id, fetches its Candidate-independent URL4 expression once, compiles and structurally links every Candidate, executes those complete URL4s concurrently, and returns one immutable Report in declared order. There is no implicit default Benchmark selection. All no-spend validation finishes before the first paid Run starts. Execution requires a Benchmark Runner image containing the expression's referenced data, grading, and Aggregation routes.

A complete evaluation URL4 is already linked to its Benchmark and can be evaluated directly. The Client validates the embedded Candidate projection, executes that exact expression, and recovers the pinned Benchmark identity from the Engine result:

score = sf.leaderboards.get_score(score_id)
editable_python = score.url4.to_python()
replayed_report = sf.evaluate(score.url4)

Url4.to_python() is local and no-spend: it returns a str of Python source, not live objects — the sf.Model, sf.Fusion, sf.Pipeline, sf.CorrectiveLoop, or sf.SelfCorrective written back out as editable code, plus the recovered Benchmark call. Raw URL4 evaluation does not accept benchmark= or limit= because either would imply recompiling an already-complete expression. Replay starts a new, potentially paid Run; identical URL4 does not guarantee identical model output.

Every Client-compiled Candidate URL4 contains exactly one inert _sf_recipe source with the versioned screamingface.recipe.v1 descriptor. It preserves the public Recipe structure and names for exact replay reporting and to_python() reconstruction. These operations require that descriptor; the Client does not guess authoring structure from an executable call graph.

The installed draco definition always refers to the complete official 100-task Benchmark; limit=1 merely runs one Case. Grading uses five independent Judge passes per criterion. The current executable Judge is openrouter/google/gemini-3.1-pro-preview, Google's official replacement for the paper's retired Gemini-3-Pro Preview. Reports should disclose that Judge version difference when comparing scores with the paper.

Benchmark IDs are flat, complete identities. Bounded development runs use limit without changing the named protocol: benchmark="draco", limit=1 still uses canonical DRACO's full rubric and five Judge passes. The non-canonical HealthBench challenge is therefore named independently as healthbench-worst30, rather than presented as a canonical HealthBench variant.

Candidate policy

Models work without prompt configuration. For a Fusion that produces a final answer, name its synthesizer explicitly; neither the Engine catalogue nor a Benchmark silently chooses one. The SDK supplies general answer and constraint-aware synthesis prompts, then embeds the explicit Model routes and resolved prompt defaults in the final URL4:

plain = sf.Model("openrouter/openai/gpt-5.5")
pair = sf.Fusion(
    [opus, gpt],
    synthesizer="openrouter/anthropic/claude-opus-4.8",
)

Researchers can override only Candidate-owned policy when an experiment needs it:

careful = sf.Model(
    "openrouter/openai/gpt-5.5",
    prompt="Answer from primary evidence and follow every requested output constraint.",
    params={"reasoning_effort": "high"},
)

constraint_aware = sf.Fusion(
    [opus, gpt],
    synthesizer=sf.Model(
        "openrouter/openai/gpt-5.5",
        prompt="Produce one final answer that preserves every constraint in the original request.",
        params={"reasoning_effort": "high"},
    ),
)

These overrides never alter Benchmark-owned Cases, fixed Judge models or prompts, Grading, or Aggregation. Prompt defaults and explicit overrides are embedded in each final URL4. Every Fusion requires an explicit synthesizer= and always produces one answer. Model-call parameters are never invented by the SDK: a parameter-free Model emits no sampling or retrieval parameters and therefore uses the Engine's configured defaults. Benchmarks may still impose explicit execution policy in their own URL4 protocol. Transport, routing, tool, and Benchmark-policy fields remain unavailable through Candidate params.

Fusion quorum and failed members

A Fusion requires every member by default (quorum="all"). Mark individual members with optional=True to tolerate their failures:

resilient = sf.Fusion(
    [
        sf.Model("provider/a", optional=True),
        sf.Model("provider/b", optional=True),
        "provider/c",  # required
    ],
    synthesizer="provider/synth",
    quorum=2,
)

The compiler wraps optional members with URL4's ;optional and gates the member-only expression with ;quorum=2, before running the synthesizer. Required members always have to succeed, even if other successes already meet the quorum. Strings and Recipes without optional=True remain required. Pipeline(..., optional=True) and Fusion(..., optional=True) guard a complete composite when used as a member of another Fusion; their intermediate answers do not count toward the containing Fusion's quorum. Optional Recipes are only valid as Fusion members, not standalone Candidates, pipeline stages, or synthesizers.

The synthesizer receives JSON containing the original input and an outputs string with ordered member_N: answer sections for successful members only. Fusions using the default quorum and only required members retain their existing outputs object.

Quorum is a success floor checked after members finish, not an early response race. quorum="all" still requires every member to succeed, including optional members; quorum=0 permits synthesis with no successful members when every member is optional. None, booleans, negative numbers, and numbers above the member count are rejected. Quorum and per-member optional settings survive Url4.to_python() and replay. A reducer synthesizer (sf.Vote, sf.Reduce, sf.Judge and the other combine blocks) takes quorum and optional members too. It reads only the members that answered, and the evidence lists each failed optional member with the reason missing; a judge scores only the members that answered. sf.Concat joins only the members that answered, so set label=True when the next stage must know which member wrote each part. With quorum=0 and no member answering, the reducer abstains and the Case scores 0.0.

Deployment requirement: before using composed quorum Recipes (such as a Pipeline feeding a quorum Fusion or a quorum Fusion used as a synthesizer), deploy an Engine containing the URL4 nested-expression scope fix shipped with this change. Older runtimes can pass literal binding references instead of resolved upstream answers. Deploy the Engine operation-reporting update as well to retain member outputs and accounting. URL4 already supports quorum and optional sources; the runtime change fixes scope propagation for these compositions.

Independent samples of one model

sf.samples(model, n) builds n Models of one route that differ only in an integer seed, so a vote over them counts n real answers and not one cached answer n times:

candidate = sf.Fusion(
    sf.samples("openrouter/openai/gpt-5.5", 5, temperature=0.7),
    synthesizer=sf.Vote(extract="last_number", normalize="numeric"),
)

The block form is the same Candidate, byte for byte: synthesizer=sf.Reduce("plurality", extract=sf.Extract("last_number", normalize="numeric")).

Seeds default to 0 through n - 1; pass seeds=[...] (unique ints) to choose them. prompts=[...] gives member i its own prompt (for example one per language), temperature= and params= apply to every member, and a sf.Model as the first argument is the base whose prompt and params the samples inherit. A model whose provider does not accept seed fails with sf.PlanningError (code unsupported_model_parameter) before the run starts and before any model is called. The SDK emits sf.DuplicateMembersWarning when a reducer Fusion has two members that send the same request, because the gateway may serve them from one cached answer. sf.DuplicateMembersWarning fires for any reducer synthesizer, a named reducer or an sf.Reduce block.

A call that sets its own seed keeps it, and a run seed (answer_seed=, one "sitting" of an evaluation) only applies to calls without one. So sf.samples() members, and any sf.Model(params={"seed": ...}), send the same requests in every sitting, and a mean and confidence interval across sittings would look tighter than the model really is. When a run seed is set and a Recipe or a replayed URL4 has such calls, the SDK emits sf.FixedSeedSittingWarning before the run starts, naming the calls, and then runs as usual. If you already pass different seeds= to sf.samples() in each sitting, the warning is expected, and you can silence it:

import warnings

warnings.filterwarnings("ignore", category=sf.FixedSeedSittingWarning)

Otherwise drop the explicit seed. The check covers Recipes and raw URL4 replays (client.evaluate("<url4>", answer_seed=...)), with the same text.

Serial and recursive composition

Every complete Recipe accepts one input and returns one final answer. Model is atomic, Fusion runs members in parallel and passes their answers to a synthesizer, and Pipeline passes one answer through ordered serial stages:

draft = sf.Model("openrouter/openai/gpt-5.5")
review = sf.Model(
    "openrouter/anthropic/claude-opus-4.8",
    prompt="Review the previous answer and return a corrected answer.",
)
final = sf.Model(
    "openrouter/openai/gpt-5.5",
    prompt="Polish the previous answer without adding unsupported claims.",
)

review_chain = sf.Pipeline([draft, review, final], name="review-chain")
same_chain = draft.then(review).then(final)

The first Pipeline stage receives the Candidate input; each later stage receives only the previous stage's answer. No original input or accumulated history is injected implicitly. Pipelines and complete Fusions compose recursively, including in the synthesis role:

judge = sf.Model("openrouter/anthropic/claude-opus-4.8")
writer = sf.Model("openrouter/openai/gpt-5.5")

candidate = sf.Fusion(
    [
        review_chain,
        sf.Model("openrouter/google/gemini-3.1-pro-preview"),
    ],
    synthesizer=sf.Pipeline([judge, writer]),
)

Pipeline([...]) is the canonical serial representation; .then(...) is immutable shorthand and never executes work. Every Recipe-valued position also accepts a model-route string as shorthand for sf.Model(...). Shape mismatches, cycles, invalid models, and invalid parameters fail before spend.

Each flat Benchmark resource uses screamingface.benchmark.v1 and carries one canonical url4 plus an opaque immutable revision. The SDK fetches the requested id, compiles a Recipe into an expression accepting $input, and links it through the single universal $candidate binding. A Benchmark invokes that bound expression through /candidate; that route evaluates it inside the same Engine job, not through an additional Client or control-plane request. Old structural member/synthesizer bindings fail with a typed planning error rather than invoking a second Client compilation path.

The Candidate input is normally plain text. Engine-owned Benchmarks that require native chat history wrap structured turns in the versioned Candidate-input envelope; the Runner preserves their roles while the SDK continues to treat $input as opaque. This supports multi-turn and stateful protocols without a Client-interpreted workflow language.

Deterministic synthesizers

A Fusion can combine its members with a rule instead of a model:

routes = [
    "openrouter/openai/gpt-5.5",
    "openrouter/anthropic/claude-opus-4.8",
    "openrouter/google/gemini-3.1-pro-preview",
]

by_vote = sf.Fusion(routes, synthesizer=sf.Vote(extract="last_number", normalize="numeric"))
by_median = sf.Fusion(routes, synthesizer=sf.Median(extract="last_number", select="value"))

The combine step makes no model call. The answer is one member's text byte for byte, or the median number itself for select="value". A tie goes to the first member in author order and sets tie. A member with no usable value does not vote. When no member votes, the Case abstains: its output is "", its score is 0.0 and case.abstained is true. It does not fail. On HealthBench a harmful answer can score below 0, so an abstain can rank above a bad answer; this is intended. Only the final step of a Candidate can mark a Case as abstained: a combine step that feeds another stage (a Rank or a Filter) never does.

To group answers the way the benchmark grades them, let the benchmark choose the extract rule. The SDK reads the rule from the benchmark before any spend:

by_grader = sf.Fusion(routes, synthesizer=sf.Vote(extract="benchmark"))
report = client.evaluate(by_grader, benchmark="inspect-gsm8k")

A benchmark with no answer rule raises ValueError before any spend. For a rule with arguments, use sf.Reduce("plurality", extract=sf.Extract("regex", pattern=...)).

Regex patterns. sf.Extract("regex", pattern=...) and sf.matches(pattern) run on RE2 in the Engine. RE2 has no lookarounds and no backreferences. \d and \w match ASCII only: use \p{L} and \p{N} for Unicode letters and digits. $ is the end of the text, so write \s*$ to allow a trailing newline. The repeat limit is 1000. The Engine refuses a compiled program over 5000 instructions. The SDK checks the pattern before any spend, and it is a little stricter than RE2 for \Q...\E, \C, \x{...} and POSIX [[:class:]]. The Engine compile is authoritative: a pattern that the SDK accepts and RE2 refuses fails the Case with combine_spec_invalid.

The evidence (the Answer row per member, the tally, the result, the agreement and the chosen member) is in the combine field of each case's operations entry (op_synthesis_1 for a single reducer). case.answers() joins each member's text to its Answer. case.trace() and report.trace_dict() show what each block did, in order. report.trace_dict() is screamingface.report-trace.v1, and report.to_dict() and report.export() stay report.v1. This needs an Engine that serves GET /v1/operators.

Give a vote weights to make some members count more. A list gives each member a fixed weight, one positive number per member in author order. A sf.signal weighs each member by a number on its own Answer:

fixed = sf.Fusion(routes, synthesizer=sf.Vote(weights=[2, 1, 1]))

# Each member replies with JSON such as {"answer": "B", "confidence": 0.4}.
by_confidence = sf.Fusion(
    routes,
    synthesizer=sf.Reduce(
        "weighted",
        extract=sf.Extract("field", path="answer", normalize="text"),
        weights=sf.signal("field:confidence"),
    ),
)

# The same rule in the named form, with the cost of each member's own model calls as the weight.
by_cost = sf.Fusion(routes, synthesizer=sf.Vote(weights=sf.signal("cost_usd")))
by_cost_block = sf.Fusion(
    routes,
    synthesizer=sf.Reduce(
        "weighted",
        extract=sf.Extract("last_line", normalize="text"),
        weights=sf.signal("cost_usd"),
    ),
)

Each answer scores the sum of the weights of the members who gave it. The answer is the text of the heaviest member who gave the winning answer (the first in author order on equal weights). A member with no weight does not vote, and when no member is left the Case abstains. The report shows each member's weight and each answer's summed score.

A weight that is zero, negative, not finite or above sys.float_info.max / 256 raises ValueError at sf.Vote(...), and a weight that is not a number raises TypeError. A wrong number of weights raises ValueError when the Fusion is built. All of these raise before any call. Weights apply to sf.Vote only; sf.Median(weights=...) raises TypeError. sf.signal("score:judge") needs a judge on the same vote (see "Judge synthesizer" below).

The accounting keys are cost_usd, latency_ms, tokens_in and tokens_out. For a member they are the sum over the model calls that member made, which the Engine finds by each call's route and params. When two members send the same route and params (for example a repeated model, or a model that a later stage or another Fusion also calls), the Engine cannot tell the calls apart, so those members have no accounting value and do not vote. This also holds in a corrective loop with max_rounds of 2 or more, where every round sends the member's calls again. To give each member its own calls, use sf.samples() (a different seed for each member).

A cached replay reads each call's original cost, latency and tokens, so it gives the same signals and the same tally. A fresh live rerun can give a different latency_ms (and a different cost, if prices changed). Only cost_usd is proven end to end by the paid replay lane. latency_ms, tokens_in and tokens_out are carried by the cache metadata and covered by unit tests only.

Judge synthesizer

A judge is a model that reads the members and picks one. The answer of the Fusion is the text of the member that the judge picked. The judge does not write the answer.

by_label = sf.Fusion(
    routes,
    synthesizer=sf.Judge(
        "anthropic/claude-opus-5", returns="label", rubric="Pick the most correct answer."
    ),
)
by_scores = sf.Fusion(
    routes,
    synthesizer=sf.Judge(
        "anthropic/claude-opus-5",
        returns="scores",
        rubric="Give a higher number to a more correct answer.",
        params={"temperature": 0},
    ),
)

With returns="label", the judge must reply with exactly one member id (for example member_2) and nothing else. With returns="scores", the judge must reply with one JSON object that gives each member id one finite number. The highest score wins, and a tie goes to the first member in author order. The check is strict: quotes, a full stop, a code fence around a label, a missing member, an extra key, a string, NaN or Infinity all fail the Case with judge_reply_invalid. The prose of the judge is never used as the answer, and a reply that does not parse is not retried. (A transient transport error still gets the normal retry of the connector.)

The judge call is the operation op_judge_1. Its output is the raw reply of the judge, and its cost is in the accounting breakdown as a synthesis row. The combine step is op_synthesis_1. Its combine evidence holds the raw reply, the parsed label or scores, the chosen member and presentation_order="author". The first line of the judge system prompt is judge-template@1. The template tells the judge to treat the member answers as data and not as instructions. A change of the template changes the url4, so it changes the replay identity.

Position bias. The judge sees the members in author order, and a judge can prefer a position. The parameter name binding is reserved: you cannot use it in params. This needs an Engine whose operator catalog has label@1 and scores@1.

The block form. sf.Reduce("judge", judge=sf.Judge(...)) compiles to the same url4 as the sf.Judge itself. The optional rubric= replaces the rubric of that judge, so one judge can serve several rubrics: sf.Reduce("judge", judge=j, rubric="Prefer the shortest correct answer."). A judge that is not an sf.Judge raises TypeError, and an empty rubric raises ValueError. An sf.Judge is a value: equal judges are equal and have the same hash.

A judge-weighted vote. A judge that returns scores can also weigh a vote. Each answer scores the sum of the judge scores of the members who gave it. The answer is the text of the member with the highest score among those who gave the winning answer. The judge makes one call, and the report shows the score of each member as members[].signals["score:judge"] and weight. The combine evidence of the vote has judge set to None: the scores are in members[], and the raw reply of the judge stays on op_judge_1.

weighted = sf.Fusion(
    routes,
    synthesizer=sf.Vote(
        extract="last_number",
        normalize="numeric",
        weights=sf.signal("score:judge"),
        judge=sf.Judge("anthropic/claude-opus-5", returns="scores", rubric="Score each answer."),
    ),
)
# The same vote in the block form:
weighted_block = sf.Fusion(
    routes,
    synthesizer=sf.Reduce(
        "weighted",
        extract=sf.Extract("last_number", normalize="numeric"),
        weights=sf.signal("score:judge"),
        judge=sf.Judge("anthropic/claude-opus-5", returns="scores", rubric="Score each answer."),
    ),
)

The weights and the judge must go together. weights=sf.signal("score:judge") without a judge, a judge without that weight, or a judge that returns labels each raise ValueError when the vote is built, before any call is paid (the Engine reads score:judge strictly, so a lone half would fail after the member calls). A judge that is not an sf.Judge raises TypeError. The same strict read applies to the reply: a malformed or too large reply, a NaN or an infinite score fails the Case with judge_reply_invalid. It never counts as "no weight", which would hide the defect behind an abstain.

Pick and merge JSON fields

If the members reply with JSON, a rule can choose or build the answer. No model writes it.

by_confidence = sf.Fusion(routes, synthesizer=sf.Pick("confidence", order="max"))
by_answer = sf.Fusion(routes, synthesizer=sf.Pick("answer", equals="yes", normalize="text"))

# argmax: the member with the lowest value of a named signal (here, the cheapest member)
cheapest = sf.Fusion(routes, synthesizer=sf.Pick(by_signal=sf.signal("cost_usd"), order="min"))
# The same pick in the block form:
cheapest_block = sf.Fusion(
    routes, synthesizer=sf.Reduce("argmax", by_signal=sf.signal("cost_usd"), order="min")
)

merged = sf.Fusion(
    routes,
    synthesizer=sf.Merge(
        fields={
            "answer": sf.FieldRule.majority(normalize="text"),
            "score": sf.FieldRule.max(),
            "source": sf.FieldRule.from_member(2),
            "synonyms": sf.FieldRule.union(normalize="text"),
            "summary": sf.FieldRule.model("anthropic/claude-sonnet-5", prompt="Write one summary."),
        }
    ),
)

sf.Pick(field, order="max"|"min") reads one field from each member. It returns the text of the member with the largest or smallest value, byte for byte. sf.Pick(field, equals=value) returns the first member whose value is equal to value. field is a JSON path such as confidence, a.b or a[0].b. With normalize="text" or normalize="numeric", the value of each member and value itself go through the same normalizer before the compare. A number or a boolean in equals is compared in its JSON spelling (1 is not 1.0), unless you use normalize="numeric". Because json_field@1 returns the contents of a string without its quotes, equals=7 also matches a member whose value is the string "7". An empty equals, or equals=True with normalize="numeric", raises ValueError before any call. A tie goes to the first member in author order and sets tie. A field that is longer than 256 characters raises ValueError before any call.

sf.Pick(by_signal=sf.signal(key), order="max"|"min") (argmax) picks the member whose named signal is the highest ("max", the default) or the lowest ("min"). The signal can be an accounting key (cost_usd, latency_ms, tokens_in, tokens_out) or field:<path>. The result is the text of that member, byte for byte. A member with no value for the signal is skipped. When no member has a value, the Case abstains. A tie goes to the first member in author order. For an accounting key, the SDK adds sources to the combine context (the model calls of each member), as it does for a weighted sf.Vote; a member with a call that appears twice in the Candidate gets no value. score:judge raises ValueError: this reducer has no judge. Use sf.Judge(..., returns="scores"), which picks the member with the highest judge score. A Pick takes a field or by_signal=, not both.

The block forms compile to the same url4 as the named classes, and Url4(...).to_python() writes the named class back: sf.Reduce("argmax", by_signal=..., order=...) is sf.Pick(by_signal=..., order=...), sf.Reduce("pick_field", field=..., order=...) or sf.Reduce("pick_field", field=..., equals=..., normalize=...) is sf.Pick(field, ...), and sf.Reduce("merge", fields={...}) is sf.Merge({...}). A pick_field block keeps equals as text, so equals=7 and equals="7" are one block. The errors of the Pick carry the prefix sf.Reduce('pick_field'):. sf.FieldRule is a hashable value.

sf.Merge(fields={...}) builds one JSON object. The object has exactly the fields that you declare, in sorted key order. Each field has one rule:

• majority(normalize=None): the value that most members give. The first group wins a tie.
• max(): the largest number. A value that is not a number is skipped.
• from_member(n): the value of member n (the first member is 1). It is null when that member is not JSON or has no such field. n that is larger than the member count raises ValueError when the Fusion is built.
• union(normalize=None): the items of all list values. Each item is kept once, in its first spelling, in author order.
• model(route, prompt=..., params=None): a model writes the field (see below).

A member whose reply is not a JSON object is skipped, and the report shows the status not_json. A field with a JSON null value counts as absent. If no member gives a usable value for a majority, max or union field, or no member is usable at all, the combine step abstains with no_votes. A from_member field that is null, and the empty text of a writer, are values and never cause an abstain. The report shows the value of each member for each field and the result. The evidence of a Merge holds the JSON object that each member gave (in raw and value), for the projection step and for the final merge step. A member that replies with large JSON makes a large report.

Model-written field. A FieldRule.model field adds two calls for that field. The first call is a pure projection (synthesis_N): it builds one JSON object of the values of that field, by member. The second call is the writer model (writer_N). The writer sees only that projection. It does not see the question and it does not see the other fields. Put the task in the prompt. The reply of the writer is the value of the field, as text. If the writer call fails, the case fails and no partial object is returned. Each writer call is a synthesis row in the accounting breakdown. The first line of the writer system prompt is writer-template@1 field=<field>. The template tells the writer to treat the member values as data and not as instructions, and your prompt follows it. Url4(...).to_python() shows only your prompt. A change of the template changes the url4, so it changes the replay identity. The name binding is reserved in params. This needs an Engine whose operator catalog has json_field@1, json_object@1, values@1, by_field@1, field_rules@1, project_field@1 and built@1. Argmax needs by_signal@1 and best_holder@1.

Rank and Concat selections

A selection is a hand-off to the next stage. It is not an answer. sf.Rank keeps the top k original member texts. sf.Concat joins all the member texts. Neither one rewrites a text. A Fusion with a selection must be followed by another stage of a sf.Pipeline, and that stage receives the selection and nothing else.

ranked = sf.Pipeline(
    [
        sf.Fusion(
            routes,
            synthesizer=sf.Rank(
                k=2,
                by=sf.Judge(
                    "anthropic/claude-opus-5",
                    returns="scores",
                    rubric="Give a higher number to a more correct answer.",
                ),
            ),
        ),
        sf.Model("anthropic/claude-sonnet-5", prompt="Write the final answer from these drafts."),
    ]
)
by_field = sf.Fusion(routes, synthesizer=sf.Rank(k=2, by_field="score"))  # members reply with JSON
joined = sf.Fusion(routes, synthesizer=sf.Concat(separator="\n\n---\n\n", label=True))

sf.Rank(k, by=sf.Judge(..., returns="scores")) scores the members with a judge, and sf.Rank(k, by_field="score") reads a number from the JSON reply of each member (by_field is a field path like score or result.score[0]). The next stage receives this JSON, best member first: {"selected":[{"id":"member_2","text":"..."},{"id":"member_4","text":"..."}]}. A tie at the cut goes to the earlier member, and the evidence sets tie. k is from 1 to 256 and must not be more than the number of members. If fewer than k members have a usable score, the combine step abstains with cut_not_met, and the next stage gets an empty text. The judge call and the scores of all the members stay in the report (op_judge_1 and the combine evidence), also for the members that do not continue. A judge reply that is not valid fails the Case with judge_reply_invalid, as for sf.Judge.

sf.Concat(separator="\n\n---\n\n", label=True) makes no model call. With label=True, each part starts with [member_1] on its own line. The parts are in author order, and an empty member text keeps its place. The separator is 1 to 1024 characters. If a member text contains the separator, the text is joined unchanged and the evidence sets separator_collision, because then the boundaries are not unique. A joined text of more than 8 MiB fails with combine_input_too_large.

sf.Rank(k, by_signal=sf.signal(key), order="max") ranks by a named number on the Answer of each member: a field signal such as field:score, or an accounting key such as cost_usd. With order="max" (the default) it keeps the members with the highest values. With order="min" it keeps the members with the lowest values, for example the cheapest members. order= also works with by= and by_field=. Give exactly one of by, by_field and by_signal; two, or none, raise ValueError. A member with no value for the signal is not ranked. sf.signal("score:judge") is refused, because a Rank with no judge has no judge score: use by=sf.Judge(..., returns="scores").

cheapest_two = sf.Pipeline(
    [
        sf.Fusion(routes, synthesizer=sf.Rank(k=2, by_signal=sf.signal("cost_usd"), order="min")),
        sf.Model("anthropic/claude-sonnet-5", prompt="Write the final answer from these drafts."),
    ]
)

The block forms are sf.Reduce("top_k", k=2, by_signal=..., order=...) (also by= and by_field=) and sf.Reduce("concat", separator=..., label=...). Each compiles to the same url4 as sf.Rank and sf.Concat, and a url4 is written back as sf.Rank(...) or sf.Concat(...).

The compiler checks the place of a selection before any call. A Fusion with Rank, Concat or Filter cannot be the last step of a candidate, cannot be a member of another Fusion (also not through a sf.Pipeline that ends in one), and cannot be a member of a corrective loop. Each case raises ValueError. The next stage does not receive the original question or the other member texts, so put the task in the prompt of that stage. This needs an Engine whose operator catalog has scores@1 and top_k@1 for Rank by a judge or by a field, by_signal@1 and top_k@1 for Rank by a signal, or join@1 and all@1 for Concat.

Filter selection

sf.Filter keeps the members that pass a test. The next stage gets only those members, as the same JSON that sf.Rank writes, in author order. where= is a regex or a signal comparison:

confident = sf.Pipeline(
    [
        sf.Fusion(
            routes,
            # Each member replies with JSON such as {"answer": "B", "confidence": 0.4}.
            synthesizer=sf.Filter(where=sf.signal("field:confidence") >= 0.6),
        ),
        sf.Model("anthropic/claude-sonnet-5", prompt="Write the final answer from these drafts."),
    ]
)
formatted = sf.Filter(where=sf.matches(r"ANSWER: \d+", ignore_case=True))
checked = sf.Filter(where=sf.matches("yes"), extract=sf.Extract("field", path="verdict"))

sf.signal(key) compares with >, >=, <, <= and ==. The number must be finite. There is no !=, and no and or or: a Filter has one test. sf.signal(key) == sf.signal(key) stays a plain bool, and a comparison is not a bool (if sf.signal("cost_usd") < 1: raises TypeError). A threshold is stored as a float, so 1 and 1.0 are one test. sf.signal("field:x") == True also passes a member whose number is 1.0, because a signal holds a float. A member with no value for the signal, or that failed to answer, is dropped. An accounting key (cost_usd, latency_ms, tokens_in, tokens_out) reads the model calls of each member, and sf.signal("score:judge") is refused. A regex reads the original text of the member, never the extracted answer. extract= reads each answer first: a member whose answer cannot be read is dropped, and the report shows the value it read.

A Filter that keeps no member abstains: its combine evidence has the result abstain with the reason no_votes, and the next stage gets an empty text. (The Case is not marked abstained, because a Filter is never the final step.) The report shows a pass mark in members[].score of the combine evidence: 1.0 when the member is kept, 0.0 when it is dropped, and None when the member failed to answer. The block form is sf.Reduce("filter", where=..., extract=...). The place rules of "Rank and Concat selections" apply to a Filter too. This needs an Engine whose operator catalog has filter@1 and kept@1.

Fusion as a named recipe

sf.Fusion(members, synthesizer=sf.Model(p)) is the named recipe Fan-out, then Reduce.compute(context), then Model. The block form says the same in blocks:

block = sf.Pipeline(
    [
        sf.Fusion(routes, synthesizer=sf.Reduce("context")),
        sf.Model("openrouter/openai/gpt-5.5", prompt="Write the final answer from the drafts."),
    ]
)
same = sf.Fusion(
    routes,
    synthesizer=sf.Model(
        "openrouter/openai/gpt-5.5", prompt="Write the final answer from the drafts."
    ),
)

Before the compile, the SDK rewrites the block form to the Fusion. The two give the same url4, byte for byte, and the same operation ids (op_synthesis_N). The rewrite keeps the stages after the synthesizer (Pipeline([Fusion(m, Reduce("context")), S, *rest]) is Pipeline([Fusion(m, S), *rest])), and a named Pipeline keeps its name. A Fusion that was made with sf.Fusion(...) is not changed by the rewrite. The candidate is named from the rewritten value, so both forms report a+b, and report.candidates[...] uses that name (the block form's own name is a+b->s). Two block forms that get the same name raise ValueError: give each one name=. The context block makes no combine call: it needs no operator catalog (no GET /v1/operators) and it has no member limit.

Use sf.Reduce("context") only as the synthesizer of a Fusion that is followed by the real synthesizer in the same Pipeline. In every other place (the last stage, a Fusion member, a loop member, or before another context Fusion) it raises ValueError before any call. To write a multi-stage synthesizer in the block form, give it a name: sf.Pipeline makes an unnamed nested Pipeline flat, so sf.Pipeline([sf.Fusion(m, sf.Reduce("context")), sf.Pipeline([j, w])]) is read as [Fusion, j, w]. Use sf.Pipeline([j, w], name="judge-write"), or a Fusion, as the synthesizer.

Install

pip install screamingface

Python 3.12 or newer is required. This command installs the hosted client. The local stack (screamingface up) also needs the runtime services and notebook tools:

pip install "screamingface[runtime,notebook]"

Troubleshooting

SCREAMINGFACE_RUNTIME_ERROR No module named 'X' from screamingface up — common on Colab, which preinstalls some of the runtime dependencies: the [runtime] extra is not installed. Run pip install "screamingface[runtime,notebook]" and start again with screamingface up. screamingface doctor names the missing modules.

Client configuration

The module-level interface constructs one process-wide Client lazily against the hosted development Engine and public Scoreboard. Configure either origin explicitly when selecting local or alternate deployments:

import screamingface as sf

sf.configure(
    engine_url="http://127.0.0.1:9108",
    scoreboard_url="http://127.0.0.1:9106",
)
report = sf.evaluate(candidates, benchmark="draco", limit=1)
sf.close()

sf.configure(...) replaces and closes any existing default Client. sf.close() releases the default Client and clears it so the next module-level operation can construct a fresh one. Setting SCREAMINGFACE_ENGINE_URL and SCREAMINGFACE_SCOREBOARD_URL may be set before the first operation for environment-driven configuration. The hosted development Scoreboard defaults to https://leaderboard.dev.screamingface.ai.

The Client hides Benchmark fetching, URL4 compilation, REST/WebSocket transport, Event replay, and Report decoding behind sf.evaluate(...).

Hosted caller authentication

Hosted Engines may be protected by Cloudflare Access. No authentication selector, Cloudflare service token, or provider key is passed to the Client:

client = sf.Client(engine_url="https://fusion.dev.screamingface.ai")
client.login()  # optional: the first protected request also starts login

The Client discovers the Access application audience from the Engine redirect, creates an ephemeral encryption keypair, and opens Cloudflare Access login in the user's browser. It polls Cloudflare's encrypted transfer service and decrypts the returned application token locally. The token is held only in process memory and sent as Cf-Access-Token on REST requests and WebSocket handshakes. client.logout() forgets it and opens the Engine's Cloudflare Access logout endpoint in the browser. Concurrent callers share one browser login, a server-rejected token starts one fresh login even before its local expiry, and an Access-specific WebSocket rejection is retried once after reauthentication.

The login URL is always printed. Desktop Python also attempts to open it automatically; in Jupyter or Colab, click the displayed URL and complete the configured Access login. This flow does not use a localhost callback and does not require dynamic client registration or Allow loopback clients. The user's email or identity must be allowed by the Cloudflare Access policy for the hosted Engine. The Client prints a confirmation after it receives and validates the transferred token; client.authenticated then returns True. Local Engines that do not advertise Access continue to work without authentication.

The public authentication boundary is the URL4 Cloud origin, not AI Gateway. After Cloudflare Access authenticates the caller, the deployment passes the verified identity to URL4 Cloud as X-User-Email. URL4 Cloud forwards that identity—not the Access token—to the internal AI Gateway. Consequently, the Python Client never calls AI Gateway or a model provider directly:

Python Client -- Cf-Access-Token --> Cloudflare Access --> URL4 Cloud
                                                          -- X-User-Email --> AI Gateway
                                                                             -- provider key --> Provider

These credentials have deliberately different lifetimes and owners:

• The Cloudflare Access token exists only in Client memory and authenticates calls to URL4 Cloud.
• X-User-Email is derived from the edge-verified identity and selects the AI Gateway account.
• A provider key entered through sf.connect() travels through URL4 Cloud once for AI Gateway to validate and store; URL4 Cloud does not retain it.
• No shared Cloudflare key, provider key, or administrator key is configured on the Client.

If the hosted application later adopts Cloudflare Managed OAuth, the Client can migrate to OAuth discovery and authorization code + PKCE. That is a possible future protocol, not an additional authentication mode implemented by this package today.

In a notebook, sf.connect() displays the connection panel bound to the lazy default Client. For a remote Engine, the panel first checks noninteractively whether Cloudflare Access is present. An unprotected remote Engine loads its providers normally; a protected Engine shows the Engine login row and loads provider rows only after login succeeds. Login waits in the background, so the notebook remains usable and the row becomes Cancel while the encrypted transfer is pending. Opening sf.connect() again reflects that same in-progress login, and all open panels follow its eventual login/logout state. Cancel stops only the pending transfer without opening another browser page; Log out clears the completed token and opens Cloudflare Access logout. An explicit Access rejection is shown in the panel, while an abandoned browser flow remains cancelable until its timeout. Local Engines omit this row. This one-time transfer polling is separate from the authenticated WebSocket used for model execution. The Engine derives its connection catalogue from AI Gateway's enabled provider plugins. API keys are supported for any provider that advertises api_key, and the panel can start OAuth for any provider that advertises oauth. A key is sent only to the SF Engine, which asks AI Gateway to validate and store it; the Python Client does not persist it and never calls AI Gateway or a provider directly. One OpenRouter key covers every enabled openrouter/... route, but does not authorize direct routes owned by other providers.

sf.connect()
flow = sf.connect("codex", method="oauth")
flow.authorize_url
connection = flow.wait()  # or flow.cancel()
sf.connections.list()
sf.connections.get("openrouter")
sf.disconnect("openrouter")

The same panel retains OAuth, pending, authorization, cancellation, and reauthentication states. The Engine catalogue remains authoritative about which methods each provider supports.

Local and hosted Engines use the same Client contract. Local mode may run the URL4 executor in-process with an in-memory event bus; hosted mode may use the REST/WebSocket control plane with NATS and scheduled workers. Engine execution is selected through engine_url; public Leaderboard discovery is configured independently through scoreboard_url. Generic URL4 execution remains benchmark-agnostic; installed Engine definitions own Benchmark semantics.

Progress and Reports

evaluate consumes the Engine's REST and WebSocket lifecycle internally. An optional callback receives typed CloudEvents views in sequence:

def observe(event: sf.Event) -> None:
    print(event.kind)


report = sf.evaluate(
    candidates,
    benchmark="draco",
    limit=1,
    on_event=observe,
)

If a callback raises, the Client attempts to cancel all active Candidate Runs and re-raises the original exception.

One Report shape covers one or many Candidates:

report.ok
report.benchmark
report.case_count
report.candidates[0]
report.candidates["frontier-pair"]
report.candidates["frontier-pair"].run_id
report.candidates["frontier-pair"].url4
report.candidates["frontier-pair"].operations
report.candidates.only
report.failures
report.usage
report.started_at
report.completed_at
report.to_dict()
report.to_json()
artifact_path = report.export()  # Path("report.json")
report.export("runs/draco.json")

Report.export(...) writes the exact complete JSON document returned by to_json(), creates parent directories, replaces an existing selected file for deterministic reruns, and returns its Path. A Report remains one JSON document even when it contains multiple Candidates; JSONL is reserved for a future collection of independent Reports.

JSON export serializes one Case at a time and replaces the destination only after the complete file is flushed and fsynced. Existing output symlinks and ordinary file permissions are preserved; new files honor the process umask. A failure before replacement keeps the previous file. On POSIX, failure during the final directory fsync means replacement happened but its durability is unconfirmed. to_json() uses the same serializer, but its returned string still occupies memory; use export() for large reports.

CandidateResult.client_version records the caller-reported Client version retained by the Engine for that run. It is also present in each candidate in report.json; missing or ambiguous evidence is null. It is not the version of the Client later exporting the report, nor the version that originally generated cached model answers.

Each entry in CandidateResult.operations is a public immutable sf.OperationInfo value.

Authentication, validation, transport, execution, protocol, and invalid-result failures raise typed exceptions.

Expected SDK failures inherit from ScreamingFaceError and always carry a stable error code, plus an optional HTTP status, structured details, remediation hint, and permanent/retryable classification. The public classes reflect distinct recovery actions:

• EngineUnavailableError: start, reconfigure, or retry the Engine.
• AuthenticationError: authenticate the caller again.
• PlanningError: change the Candidate, Benchmark, Model, or evaluation configuration.
• ExecutionError: inspect or retry a Run that failed after reaching the Engine.
• ProviderConnectionError: change a provider credential or provider connection.

These classes arrive directly when the failure happens before any Candidate runs — for example while the Client loads the Benchmark or the Model catalogue — and when an Evaluation has one Candidate. A multi-Candidate Evaluation is different once its Candidates run: one failed Candidate does not stop the others. They run to their end, the progress output marks the failed row at once, and then the Evaluation raises ExecutionError with code="candidates_failed":

• error.details["failed"] maps each failed Candidate's name to its error code, for example websocket_disconnected, engine_at_capacity, authentication_failed or engine_unreachable (unexpected_error when the failure had no SDK code);
• error.__cause__ is the first failure itself, so an AuthenticationError or EngineUnavailableError from a running Candidate is found there, not raised directly;
• error.partial_report is a Report of the Candidates that succeeded, or None when none did. It holds nothing for a failed Candidate.

To keep the paid results, export the Partial Report; details["failed"] names the Candidates to run again:

try:
    report = sf.evaluate(candidates, benchmark="draco")
except sf.ExecutionError as error:
    if error.code != "candidates_failed":
        raise
    if error.partial_report is not None:
        error.partial_report.export("partial-report.json")
    failed = error.details["failed"]  # {candidate name: code}

Ctrl-C, task cancellation, and an exception raised by your own on_event callback still stop every Run of the Evaluation and re-raise that exception unchanged.

IPython and Jupyter render these failures as a concise message, hint, and code instead of exposing dependency tracebacks. Notebook panels render the same safe text inline. Programmatic callers can catch a specific recovery class or catch ScreamingFaceError for every expected SDK failure; translated low-level failures remain attached through error.__cause__ for debugging. Programmer errors such as invalid Python argument types retain their normal tracebacks.

Every CandidateResult exposes the Engine-owned top-level coverage ratio. A partial score remains available alongside the Cases that could not be graded, and the notebook Report panel labels the result as partial rather than silently presenting it as a complete evaluation.

Browse completed Case results

Install screamingface[notebook] and display a completed report in a live notebook to browse 25 Case results per page. Use Previous/Next, enter an exact Case ID in Go to case number, or filter by Candidate using the dropdown or a Candidate name. The answer/grading and cost views remain available for every result. Numeric and string Case IDs retain their original identity.

Download prepares the complete report JSON on disk, including every Candidate and Case, then provides a Jupyter file link (or the saved path when no local server serves it). In Colab it uses the native browser download API; the button can download the saved file again without re-exporting it. Jupyter prepares exports and pages asynchronously. Colab publishes bounded page changes and streaming exports inside the click callback, as its widget transport requires. Failures leave the current page available and allow retry. report.export('report.json') also writes the complete portable artifact directly.

Colab's custom widget manager is enabled automatically for this browser. The notebook extra accepts ipywidgets 7.7.1 and 8, preserving Colab's compatible preinstalled version. If you previously upgraded Colab to ipywidgets 8 and see a blank report, install ipywidgets==7.7.1 and restart the Colab session before displaying the report. Ordinary Jupyter supports ipywidgets 8. Case clicks use Colab’s native callback API there and ipywidgets DOM events in Jupyter. This compatibility verification covers completed reports; older-widget live-progress accessibility has separate existing limitations.

Static HTML, including environments without the notebook extra, previews the first 25 results and shortens oversized Case panes. Small JSON downloads remain inline; larger ones use explicit export. The complete retained data remains available through the Report and JSON export.

Completed accounting breakdown

Each Candidate exposes derived views of the Engine's retained operation and grading records:

breakdown = report.candidates[0].accounting
breakdown.by_stage  # generation, synthesis, grading
breakdown.by_operation  # (stage, operation/check id)
breakdown.by_member  # direct model-member operation ids
breakdown.by_model  # request model or reliable declared identity; None when unknown
breakdown.by_case  # original Case ids
breakdown.unattributed_cost_usd

Each group has usage, calls, cache, provider_latency_ms, and provider_attempts. These summarize retained observations; they do not replace the authoritative run total. Missing observations remain unknown. A missing record stays in its declared model group when that identity is unambiguous and agrees with retained requests. If any row's model cannot be identified, all named by_model summaries are unknown: the unidentified work could belong to any of them. The None bucket summarizes only its anonymous observations. Calls count consumed responses, not provider retries. Provider time sums attempt latencies and is not wall time. Unknown or unpriced costs prevent an exact remainder; inconsistent records disable the breakdown (consistent=False). Loop internals and composite-member ownership remain unattributed where the retained contract cannot prove their scope. Derived group views are not added to Report JSON.

Each completed Report Case has Answer & grading and Cost & usage views. The latter shows labelled activity blocks; whole-run totals remain above the Cases. Try the offline review notebook without credentials or paid calls. Its synthetic figures are explicitly labelled; actual evaluations use the same UI.

Ownership boundary

Researcher or SF App
        ↓
ScreamingFace Python Client
  Recipe authoring · URL4 compilation · Events · Reports · Leaderboard reads
        ├─ REST + WebSocket → SF Engine → AI Gateway
        └─ public HTTPS GET → Scoreboard

Evaluation, discovery, and provider-connection operations call the configured SF Engine. Leaderboard discovery calls the configured public Scoreboard. The Client never calls AI Gateway, model providers, Tavily, or Benchmark datasets directly. Local and hosted Engines expose the same Client-visible contract; in-memory channels, NATS, workers, and deployment topology are Engine details.

Models, Fusions, and Pipelines are immutable, structurally comparable, Client-independent, and network-free. Models select routes and optional answer policy; Fusions declare parallel topology and an explicit synthesizer; Pipelines declare serial topology. Every placement compiles to a distinct logical invocation, even when two Recipe values are equal or reused. The SDK compiles the complete Recipe into one Candidate expression. Benchmarks are immutable Engine protocols that own Cases, Candidate Invocation order, fixed Judge configuration, Grading, Aggregation, and execution policy. Reports record the exact Engine-pinned Benchmark revision.

Durable reuse across graph positions, Candidates, retries, and resumed Evaluations belongs to the Engine's provenance-aware response cache; Client compilation never merges authored positions.

Discovery

An Engine implementing the provisional catalogue contract exposes typed discovery:

models = sf.models.list()
gpt_details = sf.models.get("openrouter/openai/gpt-5.5")
benchmarks = sf.benchmarks.list()
boards = sf.leaderboards.list()
draco_board = sf.leaderboards.get("draco", top=50)

# After evaluating a Benchmark whose Scoreboard accepts submissions:
submission = sf.leaderboards.submit(
    report.candidates.only,
    authors=["alice@example.com", "bob@example.org"],
)
same_submission = sf.leaderboards.get_score(submission.id)

# Add or fix the paper link and authors of a score you submitted, then read the edit log:
sf.leaderboards.edit(submission.id, paper_url="https://arxiv.org/abs/2610.01234")
edit_log = sf.leaderboards.metadata_events(submission.id)
editable_python = same_submission.url4.to_python()
replayed_report = sf.evaluate(same_submission.url4)

Explicit Clients provide the same interface through client.models.list() and client.models.get(model_id) alongside client.benchmarks.list(); asynchronous Clients use the same names with await. ModelInfo rows are lightweight summaries containing the supported parameter and tool names. ModelDetails is the profile-specific contract for one Model, including typed parameter schemas, gateway policy, provider evidence, tools, transport, and freshness.

sf.leaderboards uses the separate public Scoreboard: list() returns its registered benchmark summaries, while get(benchmark_id, top=...) returns one immutable Leaderboard containing ranked best-per-spec entries and imported single-Model baselines. submit(candidate_result) publishes an already-evaluated result — the Benchmark-native score exactly as the Engine graded it, fractional or negative included — without asking the caller to repeat its Benchmark, URL4, models, or run identity; get_score(id) retrieves the resulting immutable LeaderboardScore. Omit authors to credit the authenticated submitter by default. When supplied, the list is the exact credit line—the submitter is not added automatically—and the public LeaderboardScore.authors and LeaderboardEntry.authors values contain the Scoreboard's privacy-trimmed author identifiers. A score's .url4 property is a string-compatible Url4 value: .to_python() produces an editable fork, while passing the value to sf.evaluate(...) replays it through the configured Engine. Submitting a limited or incompletely graded Candidate surfaces a Partial submission advisory because its score is not directly comparable with a full run. In a notebook the Client displays a branded notice even when the score is assigned to a variable; headless callers receive sf.EvaluationWarning attributed to their submission line. If that warning category is configured as an error, submission stops before the Scoreboard is changed. submit(..., paper_url=...) links a paper you have already published (an http or https link with a host and no user info or whitespace). Later, the submitter can call edit(score_id, authors=..., paper_url=...): an argument you leave out stays unchanged, paper_url=None clears the link, and authors=None is refused (pass [submitter] to go back to the default). edit returns the updated LeaderboardScore, which carries paper_url and metadata_updated_at. metadata_events(score_id) returns the owner-only edit log as ScoreMetadataEvent rows with the old and new values. The SDK does not check that the paper exists or that the authors wrote it. A Scoreboard deployment may keep writes closed, in which case submit() raises a typed LeaderboardError. Explicit Clients expose the same interface at client.leaderboards; asynchronous Clients use await. The Scoreboard is the deployed data system, while a Leaderboard is the ranked domain resource returned to callers.

Provider access and explicit Candidate parameters are preflighted before execution. The SDK fetches one detail document per distinct required Candidate Model, including parameter-free Models, and reuses any document fetched during model admission. Authoritative missing provider access raises ProviderConnectionError before any Candidate dispatch or progress events, with a hint to configure BYOK via sf.connect() or enable the selected hosted provider profile. ModelDetails.execution_access is "configured", "missing", or None when an older Gateway omits the field. Unknown access preserves existing behavior; the early missing-access guarantee requires a Gateway that publishes the field. Configured access is not credential validation. Missing, disabled, wrong-type, or out-of-range parameters also fail before any paid Run begins. Access and capability data come from Engine/AI Gateway discovery, with no provider-specific credential or parameter rules in the SDK. Sync and async Clients use the same checks.

The returned catalogues are immutable ordered sequences: iteration, indexing, slicing, and len() work normally in scripts and sidecars. Evaluating one in Jupyter automatically renders a searchable catalogue when the notebook extra is installed, with escaped static HTML and compact terminal representations as fallbacks. Notebook rendering does not change the underlying values or introduce a separate discovery operation.

Examples

From a checkout, just local-stack-notebooks runs every notebook below: it bakes the frozen benchmark assets, brings the local stack up, serves an Engine carrying the imported inspect_evals boards alongside ours, and opens JupyterLab against it.

• examples/00_quickstart.ipynb: one Candidate through the first canonical draco Case, from discovery through Report evidence.
• examples/01_client_tour.ipynb: a no-spend tour of Client lifecycle, hosted authentication, discovery, connections, authoring, events, errors, Reports, and the asynchronous API.
• examples/06_draco_full_e2e.ipynb: the complete seven-solo, nine-Fusion canonical DRACO experiment and audit workflow, with execution disabled by default.
• examples/07_ifeval_e2e.ipynb: canonical deterministic IFEval across the solo/panel × plain/corrective Recipe grid.
• examples/08_healthbench.ipynb: both HealthBench boards — the worst-30% open-Fusion challenge and the full 525-case exam with the official score — rehearsed cheaply with limit=1 first.
• examples/12_inspect_evals_benchmarks.ipynb: the catalogue as two origin groups — ours and the imported inspect_evals boards — then a Fusion against one of the imported boards.

All notebooks are deterministic outputs of scripts/build_notebooks.py.

Development

uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest --cov=screamingface --cov-fail-under=95 -q
uv run --extra notebook python scripts/check_notebooks.py
uv build
uv run python scripts/check_distribution.py

Reopen completed results

Public Client and AsyncClient instances save completed-run metadata before downloading results. Artifact downloads are streamed and checked against the Engine ticket's size and SHA-256. Cases are validated incrementally into a disk index and loaded on demand, preserving prompts, answers, grading, operations, accounting, and the SDK's run provenance.

import screamingface as sf

entries = sf.reports.list()
report = sf.reports.get(entries[0].id)
# In async code: report = await sf.reports.get_async(entries[0].id)
# Explicit removal: sf.reports.delete(entries[0].id)

The default location is ~/.screamingface/results; SCREAMINGFACE_RESULTS_DIR overrides it, or use SCREAMINGFACE_DATA_DIR to choose the containing data directory. list() reads lightweight metadata and groups Candidates from the same Evaluation. get() and get_async() use local copies when available and obtain fresh authentication when a missing artifact needs downloading. They never start model runs. destination='/another/disk' lets recovery copy retained results to another location. Incomplete recovery raises the existing candidates_failed error with a partial_report and named Candidate failures.

New saved evaluations track whether evaluation, notebook display or JSON export is in progress. When a POSIX notebook process exits unexpectedly, the next persisted Client for the same Engine shows an informational notice with an explicit sf.reports.get(...) command for retained completed results. It does not rerun models or recover automatically. Healthy, active and legacy results stay quiet; headless Clients do not scan results at startup. Owners that cannot be verified dead (including non-POSIX environments) are treated conservatively. Process-ID reuse can also suppress a notice; saved results remain available through sf.reports.list(). On POSIX, overlapping notebooks sharing the same directory preserve one another's unfinished work, including when a saved Candidate is recovered, displayed or exported from another process.

Local copies remain until explicit deletion. Deleting a report also makes existing Reports backed by its files unreadable. Recovery requires a saved completion record and either a valid local raw result or an unexpired Engine artifact. An unavailable artifact reports result_expired with its age. Death before completion is recorded cannot be recovered. Disk failures report result_storage_failed; collection never silently falls back to RAM. Only completion/result metadata is saved, never authentication credentials.

Set Client(save_results=False) or AsyncClient(save_results=False) for the previous in-memory collection path. These options do not change a custom supplied run_transport. Calling to_dict(), to_json(), converting Cases to a list, or requesting a Case slice still materializes the requested data. CandidateResult.accounting also materializes its full accounting breakdown. Notebook cost disclosures instead derive a compact context by streaming Cases and cache that context beside a saved Case index on first use. Warm run-level disclosures reuse the cache without decoding Cases; missing, stale or malformed caches rebuild, and a read-only cache location still returns derived values. Per-Case HTML still materializes its rendered output. Automatic lifecycle discovery remains a separate follow-up.