Skip to main content

FrontierKit

CI PyPI version Python 3.10+ License: MIT

Find out which model and prompt answer your questions best.

FrontierKit runs your questions through every combination of model, prompt and pipeline step you want to compare, scores the answers against your references, and tells you which setup to use. It works with free model endpoints, so trying it costs nothing.

Website and guide: frontierkit.github.io/frontierkit

FrontierKit was called ChatCortex up to version 0.12. pip install chatcortex still works: it installs FrontierKit and keeps import chatcortex running, with a reminder to switch.

What you get

One command, four sample questions, one free model on Groq:

Prompt Context given Quality Time per answer
"shortest possible phrase" yes 0.92 292 ms
"shortest possible phrase" no 0.67 275 ms
plain question yes 0.14 369 ms
plain question no 0.06 508 ms

Same model, same questions: the prompt and the context moved quality from 0.06 to 0.92. Quality is word overlap with the reference answers, from 0 to 1.

Try it in five minutes

pip install "frontierkit[real]"
git clone https://github.com/frontierkit/frontierkit
cd frontierkit

Set a free API key from any of SAIA, NVIDIA, Groq or OpenRouter, for example:

export GROQ_API_KEY=...          # PowerShell: $env:GROQ_API_KEY = "..."

Then run it on the sample questions, or on your own CSV of questions and reference answers:

python examples/pick_models.py --max-questions 4
python examples/pick_models.py --questions my_questions.csv

The get-started guide walks through each step: getting keys, writing questions, reading the results, and changing what is compared.

What you can compare

  • Models and vendors. Any vendor or local server that accepts OpenAI-style chat requests: SAIA, NVIDIA, Groq, OpenRouter, Ollama and more. No extra dependencies.
  • Prompts and context. Prompt templates, with or without reading material.
  • Your own code. Each pipeline step can be a plain Python function: a retriever, a re-ranker, a check on the answer.
  • Quality, speed and cost together. Results show the setups that are not beaten on all three, so you can pick the trade-off you want.

Use it from Python

from frontierkit import (
    EvalSet, Example, enumerate_pipelines, evaluate_pipelines,
    quality_frontier, token_f1,
)

eval_set = EvalSet(
    examples=[
        Example("Which organ produces insulin?", expected="pancreas"),
        Example("What is the largest ocean?", expected=["Pacific Ocean", "Pacific"]),
    ],
    scorer=token_f1,
)

pipelines = enumerate_pipelines(registry, task)            # every combination
evaluations = evaluate_pipelines(pipelines, executor, eval_set)

best = quality_frontier(evaluations)[0]                    # highest quality first
print(best.components, best.quality, best.latency_ms)

Using FrontierKit from Python covers the rest: describing components and tasks, running your own functions, calling models from several vendors, measuring cost from token usage, and letting the library choose what to try when there are too many combinations.

Why measure

On twelve reading-comprehension questions from SQuAD, the same Gemma model scored 0.95 on two vendors, but answered in 230 ms on one and 14,516 ms on the other. Without reading material it scored 0.10. A smaller model was no faster and scored 0.33. None of that is visible from a model card. The benchmark that produced these numbers runs on free models and is in benchmarks/qa_frontier/.

Where it stands

FrontierKit is an alpha-stage research library; its API may change between minor versions.

  • Trying every combination is complete and dependable.
  • For spaces too large to try in full, run_closed_loop chooses what to try from the scores it has seen. On the project's benchmark it currently finds the best setup in about as many calls as random picking.
  • Pipelines are straight chains of steps; branching pipelines are not searched.

Documentation

Citing

If you use FrontierKit in your research, please cite it; see CITATION.cff.

License

MIT. Developed by Siddharth Saraswat.

Metadata

Release files for frontierkit 0.13.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for frontierkit 0.13.0
File Size Uploaded Provenance Actions
frontierkit-0.13.0.tar.gz 119.6 kB With provenance Details for frontierkit-0.13.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for frontierkit 0.13.0
File Interpreter ABI Platform Actions
frontierkit-0.13.0-py3-none-any.whl Python 3 none any Details for frontierkit-0.13.0-py3-none-any.whl

Total release size: 208.4 kB

Release files / frontierkit-0.13.0.tar.gz

Download URL frontierkit-0.13.0.tar.gz
Size 119.6 kB
Tags Source
SHA-256 checksum
How to use checksums
acfd23ecbd6d40df6b903da38996994185d671e20662adfd168993903ff31f22
BLAKE2b-256 checksum
How to use checksums
33308825790489ecd3a714b09f7c2c9dcd17ff028008fb4a283c6ad5b9e7d14b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / frontierkit-0.13.0-py3-none-any.whl

Download URL frontierkit-0.13.0-py3-none-any.whl
Size 88.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bb8b2d0a578ceead3184719278cfbea2aafb6fee43745cab1e0bd0e527404fb7
BLAKE2b-256 checksum
How to use checksums
1e9c4e88af0df4b803b42e011c34c2b0873b042786698deeb965452a4fe9a9ae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for frontierkit 0.13.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for frontierkit-0.13.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.13.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page