FrontierKit
Find out which model, prompt and pipeline answer your own questions best, and how sure you can be.
FrontierKit runs your questions through every setup you want to compare, scores the answers against your references, and tells you which setups are really better and which are too close to call. It then lets you check, weeks later, that the choice still holds. It works with free model endpoints, so trying it costs nothing.
Website and guide: frontierkit.github.io/frontierkit
What you get
Thirty reading-comprehension questions, two free models, with and without reading material:
| Setup | Quality | Likely range | Time per answer | Against the top row |
|---|---|---|---|---|
| Gemma 4 31B, three passages | 0.95 | 0.74 to 1.00 | 362 ms | |
| Gemma 4 31B, one passage | 0.90 | 0.69 to 1.00 | 285 ms | too close to call |
| Llama 3.1 8B, three passages | 0.34 | 0.12 to 0.48 | 247 ms | worse |
| Llama 3.1 8B, one passage | 0.30 | 0.11 to 0.50 | 259 ms | worse |
| Gemma 4 31B, no passages | 0.14 | 0.01 to 0.35 | 257 ms | worse |
| Llama 3.1 8B, no passages | 0.01 | 0.00 to 0.22 | 225 ms | worse |
A leaderboard would name the first row the winner. FrontierKit says the first two cannot be told apart on these questions, so you may as well take the faster one, and that both clearly beat the smaller model and the same model without reading material. Quality is word overlap with the reference answers, from 0 to 1. The benchmark behind the table runs on free models and is in benchmarks/qa_frontier/.
What it does
- Uses your questions, not a leaderboard. A CSV of questions and reference answers is enough. Scorers cover short answers and extracted triples, or pass your own.
- Says how sure it is. Every score comes with a range. Two setups are compared question by question, and the answer can be "too close to call". The best of many setups is reported with the luck of the pick taken out.
- Weighs quality, speed and cost together. You see the setups that are not beaten on all three, and pick the trade-off you want.
- Never pays for an answer twice. Every model reply is kept in a local file. Run the comparison again and no model is called; add one setup and only its new steps are.
- Lets you re-check a choice. Save what you chose and why, then run
frontierkit verifyon a schedule. It reports whether quality dropped, the setup got slower, or the model is no longer there. - Names failures instead of hiding them. A rate limit, an outage or a retired model is recorded as what it is, not as a wrong answer.
- Works with any vendor and your own code. Any service or local server that accepts OpenAI-style chat requests: SAIA, NVIDIA, Groq, OpenRouter, Ollama and more. Each pipeline step can also be a plain Python function: a retriever, a re-ranker, a check on the answer.
Try it in five minutes
pip install "frontierkit[real]"
git clone https://github.com/frontierkit/frontierkit
cd frontierkit
Set a free API key from any of SAIA, NVIDIA, Groq or OpenRouter, for example:
export GROQ_API_KEY=... # PowerShell: $env:GROQ_API_KEY = "..."
Then run it on the sample questions, or on your own CSV of questions and reference answers:
python examples/pick_models.py --max-questions 4
python examples/pick_models.py --questions my_questions.csv
No key yet? These two run offline:
python examples/scored_pipeline.py # rank pipelines by measured quality
python examples/verify_decision.py # save a choice and check it three times
The get-started guide walks through each step: getting keys, writing questions, reading the results, and changing what is compared.
Use it from Python
from frontierkit import (
EvalSet, Example, enumerate_pipelines, evaluate_pipelines,
paired_difference, quality_frontier, record_decision, token_f1,
)
eval_set = EvalSet(
examples=[
Example("Which organ produces insulin?", expected="pancreas"),
Example("What is the largest ocean?", expected=["Pacific Ocean", "Pacific"]),
],
scorer=token_f1,
)
pipelines = enumerate_pipelines(registry, task) # every combination
evaluations = evaluate_pipelines(pipelines, executor, eval_set)
for kept in quality_frontier(evaluations): # not beaten on quality, speed and cost
print(kept.components, kept.quality, kept.latency_ms)
best, runner_up = sorted(evaluations, key=lambda e: e.quality, reverse=True)[:2]
print(paired_difference(best.scores, runner_up.scores).verdict) # may be "too close to call"
record_decision(evaluations, eval_set).save("decision.json") # to verify later
registry, task and executor describe what you are comparing;
examples/scored_pipeline.py
is a complete version that runs as it is.
Using FrontierKit from Python covers the rest: running your
own functions, calling models from several vendors, measuring cost from token
usage, keeping answers, and checking a saved choice.
Why measure
On twelve reading-comprehension questions from SQuAD, the same Gemma model scored 0.95 on two vendors, but answered in 230 ms on one and 14,516 ms on the other. Without reading material it scored 0.10. A smaller model was no faster and scored 0.33. None of that is visible from a model card.
Where it stands
FrontierKit is an alpha-stage research library; its API may change between minor versions.
- Trying every combination is complete and dependable.
- A few dozen questions can only separate large differences. The ranges in the table above are wide for that reason, and the output says what it cannot tell apart.
- For spaces too large to try in full,
run_closed_loopchooses what to try from the scores it has seen. On the project's benchmark it currently finds the best setup in about as many calls as random picking. - Pipelines are straight chains of steps; branching pipelines are not searched.
Documentation
- Website
- Get started: compare free models and prompts on your own questions
- Using FrontierKit from Python
- How it works
- Benchmarks
- Examples
- Contributing and Changelog
Coming from ChatCortex
FrontierKit was called ChatCortex up to version 0.12. pip install chatcortex
still works: it installs FrontierKit and keeps import chatcortex running, with
a reminder to switch.
Citing
If you use FrontierKit in your research, please cite it; see CITATION.cff.
License
MIT. Developed by Siddharth Saraswat.
Metadata
Release files for frontierkit 0.13.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| frontierkit-0.13.1.tar.gz | 121.3 kB | With provenance | Details for frontierkit-0.13.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| frontierkit-0.13.1-py3-none-any.whl | Python 3 | none | any | Details for frontierkit-0.13.1-py3-none-any.whl |
Total release size: 211.1 kB
Release files / frontierkit-0.13.1.tar.gz
| Download URL | frontierkit-0.13.1.tar.gz |
|---|---|
| Size | 121.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ae48e018ad5ecf48f3e3f455afe047e3fbb01644f95905b2d5575ce1a0af2d63
|
|
BLAKE2b-256 checksum How to use checksums |
2c84a1278d99ac69cb50ee09b3a08fb6cc984c9c88444b3c9855d9d5fd2a0b08
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / frontierkit-0.13.1-py3-none-any.whl
| Download URL | frontierkit-0.13.1-py3-none-any.whl |
|---|---|
| Size | 89.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1beee0bc3e1f94adeac4abe7202974719f4dc6a73f069dcc60ee3d751f4f0143
|
|
BLAKE2b-256 checksum How to use checksums |
ba58abfde2c25e3368440a5d09627a813fcf85d303f840385a2edfa72d91bbd6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for frontierkit 0.13.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for frontierkit-0.13.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.