Skip to main content

FrontierKit

CI PyPI version Python 3.10+ License: MIT

Find out which model, prompt and pipeline answer your own questions best, and how sure you can be.

FrontierKit runs your questions through every setup you want to compare, scores the answers against your references, and tells you which setups are really better and which are too close to call. It then lets you check, weeks later, that the choice still holds. It works with free model endpoints, so trying it costs nothing.

Website and guide: frontierkit.github.io/frontierkit

What you get

Thirty reading-comprehension questions, two free models, with and without reading material:

Setup Quality Likely range Time per answer Against the top row
Gemma 4 31B, three passages 0.95 0.74 to 1.00 362 ms
Gemma 4 31B, one passage 0.90 0.69 to 1.00 285 ms too close to call
Llama 3.1 8B, three passages 0.34 0.12 to 0.48 247 ms worse
Llama 3.1 8B, one passage 0.30 0.11 to 0.50 259 ms worse
Gemma 4 31B, no passages 0.14 0.01 to 0.35 257 ms worse
Llama 3.1 8B, no passages 0.01 0.00 to 0.22 225 ms worse

A leaderboard would name the first row the winner. FrontierKit says the first two cannot be told apart on these questions, so you may as well take the faster one, and that both clearly beat the smaller model and the same model without reading material. Quality is word overlap with the reference answers, from 0 to 1. The benchmark behind the table runs on free models and is in benchmarks/qa_frontier/.

What it does

  • Uses your questions, not a leaderboard. A CSV of questions and reference answers is enough. Scorers cover short answers and extracted triples, or pass your own.
  • Says how sure it is. Every score comes with a range. Two setups are compared question by question, and the answer can be "too close to call". The best of many setups is reported with the luck of the pick taken out.
  • Weighs quality, speed and cost together. You see the setups that are not beaten on all three, and pick the trade-off you want.
  • Never pays for an answer twice. Every model reply is kept in a local file. Run the comparison again and no model is called; add one setup and only its new steps are.
  • Lets you re-check a choice. Save what you chose and why, then run frontierkit verify on a schedule. It reports whether quality dropped, the setup got slower, or the model is no longer there.
  • Names failures instead of hiding them. A rate limit, an outage or a retired model is recorded as what it is, not as a wrong answer.
  • Works with any vendor and your own code. Any service or local server that accepts OpenAI-style chat requests: SAIA, NVIDIA, Groq, OpenRouter, Ollama and more. Each pipeline step can also be a plain Python function: a retriever, a re-ranker, a check on the answer.

Try it in five minutes

pip install "frontierkit[real]"
git clone https://github.com/frontierkit/frontierkit
cd frontierkit

Set a free API key from any of SAIA, NVIDIA, Groq or OpenRouter, for example:

export GROQ_API_KEY=...          # PowerShell: $env:GROQ_API_KEY = "..."

Then run it on the sample questions, or on your own CSV of questions and reference answers:

python examples/pick_models.py --max-questions 4
python examples/pick_models.py --questions my_questions.csv

No key yet? These two run offline:

python examples/scored_pipeline.py     # rank pipelines by measured quality
python examples/verify_decision.py     # save a choice and check it three times

The get-started guide walks through each step: getting keys, writing questions, reading the results, and changing what is compared.

Use it from Python

from frontierkit import (
    EvalSet, Example, enumerate_pipelines, evaluate_pipelines,
    paired_difference, quality_frontier, record_decision, token_f1,
)

eval_set = EvalSet(
    examples=[
        Example("Which organ produces insulin?", expected="pancreas"),
        Example("What is the largest ocean?", expected=["Pacific Ocean", "Pacific"]),
    ],
    scorer=token_f1,
)

pipelines = enumerate_pipelines(registry, task)            # every combination
evaluations = evaluate_pipelines(pipelines, executor, eval_set)

for kept in quality_frontier(evaluations):                 # not beaten on quality, speed and cost
    print(kept.components, kept.quality, kept.latency_ms)

best, runner_up = sorted(evaluations, key=lambda e: e.quality, reverse=True)[:2]
print(paired_difference(best.scores, runner_up.scores).verdict)   # may be "too close to call"

record_decision(evaluations, eval_set).save("decision.json")      # to verify later

registry, task and executor describe what you are comparing; examples/scored_pipeline.py is a complete version that runs as it is. Using FrontierKit from Python covers the rest: running your own functions, calling models from several vendors, measuring cost from token usage, keeping answers, and checking a saved choice.

Why measure

On twelve reading-comprehension questions from SQuAD, the same Gemma model scored 0.95 on two vendors, but answered in 230 ms on one and 14,516 ms on the other. Without reading material it scored 0.10. A smaller model was no faster and scored 0.33. None of that is visible from a model card.

Where it stands

FrontierKit is an alpha-stage research library; its API may change between minor versions.

  • Trying every combination is complete and dependable.
  • A few dozen questions can only separate large differences. The ranges in the table above are wide for that reason, and the output says what it cannot tell apart.
  • For spaces too large to try in full, run_closed_loop chooses what to try from the scores it has seen. On the project's benchmark it currently finds the best setup in about as many calls as random picking.
  • Pipelines are straight chains of steps; branching pipelines are not searched.

Documentation

Coming from ChatCortex

FrontierKit was called ChatCortex up to version 0.12. pip install chatcortex still works: it installs FrontierKit and keeps import chatcortex running, with a reminder to switch.

Citing

If you use FrontierKit in your research, please cite it; see CITATION.cff.

License

MIT. Developed by Siddharth Saraswat.

Metadata

Release files for frontierkit 0.13.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for frontierkit 0.13.1
File Size Uploaded Provenance Actions
frontierkit-0.13.1.tar.gz 121.3 kB With provenance Details for frontierkit-0.13.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for frontierkit 0.13.1
File Interpreter ABI Platform Actions
frontierkit-0.13.1-py3-none-any.whl Python 3 none any Details for frontierkit-0.13.1-py3-none-any.whl

Total release size: 211.1 kB

Release files / frontierkit-0.13.1.tar.gz

Download URL frontierkit-0.13.1.tar.gz
Size 121.3 kB
Tags Source
SHA-256 checksum
How to use checksums
ae48e018ad5ecf48f3e3f455afe047e3fbb01644f95905b2d5575ce1a0af2d63
BLAKE2b-256 checksum
How to use checksums
2c84a1278d99ac69cb50ee09b3a08fb6cc984c9c88444b3c9855d9d5fd2a0b08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / frontierkit-0.13.1-py3-none-any.whl

Download URL frontierkit-0.13.1-py3-none-any.whl
Size 89.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1beee0bc3e1f94adeac4abe7202974719f4dc6a73f069dcc60ee3d751f4f0143
BLAKE2b-256 checksum
How to use checksums
ba58abfde2c25e3368440a5d09627a813fcf85d303f840385a2edfa72d91bbd6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for frontierkit 0.13.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for frontierkit-0.13.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.13.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page