ChatCortex
Find out which model and prompt answer your questions best.
ChatCortex runs your questions through every combination of model, prompt and pipeline step you want to compare, scores the answers against your references, and tells you which setup to use. It works with free model endpoints, so trying it costs nothing.
Website and guide: siddharth1012.github.io/chatCortex
What you get
One command, four sample questions, one free model on Groq:
| Prompt | Context given | Quality | Time per answer |
|---|---|---|---|
| "shortest possible phrase" | yes | 0.92 | 292 ms |
| "shortest possible phrase" | no | 0.67 | 275 ms |
| plain question | yes | 0.14 | 369 ms |
| plain question | no | 0.06 | 508 ms |
Same model, same questions: the prompt and the context moved quality from 0.06 to 0.92. Quality is word overlap with the reference answers, from 0 to 1.
Try it in five minutes
pip install "chatcortex[real]"
git clone https://github.com/siddharth1012/chatCortex
cd chatCortex
Set a free API key from any of SAIA, NVIDIA, Groq or OpenRouter, for example:
export GROQ_API_KEY=... # PowerShell: $env:GROQ_API_KEY = "..."
Then run it on the sample questions, or on your own CSV of questions and reference answers:
python examples/pick_models.py --max-questions 4
python examples/pick_models.py --questions my_questions.csv
The get-started guide walks through each step: getting keys, writing questions, reading the results, and changing what is compared.
What you can compare
- Models and vendors. Any vendor or local server that accepts OpenAI-style chat requests: SAIA, NVIDIA, Groq, OpenRouter, Ollama and more. No extra dependencies.
- Prompts and context. Prompt templates, with or without reading material.
- Your own code. Each pipeline step can be a plain Python function: a retriever, a re-ranker, a check on the answer.
- Quality, speed and cost together. Results show the setups that are not beaten on all three, so you can pick the trade-off you want.
Use it from Python
from chatcortex import (
EvalSet, Example, enumerate_pipelines, evaluate_pipelines,
quality_frontier, token_f1,
)
eval_set = EvalSet(
examples=[
Example("Which organ produces insulin?", expected="pancreas"),
Example("What is the largest ocean?", expected=["Pacific Ocean", "Pacific"]),
],
scorer=token_f1,
)
pipelines = enumerate_pipelines(registry, task) # every combination
evaluations = evaluate_pipelines(pipelines, executor, eval_set)
best = quality_frontier(evaluations)[0] # highest quality first
print(best.components, best.quality, best.latency_ms)
Using ChatCortex from Python covers the rest: describing components and tasks, running your own functions, calling models from several vendors, measuring cost from token usage, and letting the library choose what to try when there are too many combinations.
Why measure
On twelve reading-comprehension questions from SQuAD, the same Gemma model scored 0.95 on two vendors, but answered in 230 ms on one and 14,516 ms on the other. Without reading material it scored 0.10. A smaller model was no faster and scored 0.33. None of that is visible from a model card. The benchmark that produced these numbers runs on free models and is in benchmarks/qa_frontier/.
Where it stands
ChatCortex is an alpha-stage research library; its API may change between minor versions.
- Trying every combination is complete and dependable.
- For spaces too large to try in full,
run_closed_loopchooses what to try from the scores it has seen. On the project's benchmark it currently finds the best setup in about as many calls as random picking. - Pipelines are straight chains of steps; branching pipelines are not searched.
Documentation
- Website
- Get started: compare free models and prompts on your own questions
- Using ChatCortex from Python
- How it works
- Benchmarks
- Examples
- Contributing and Changelog
Citing
If you use ChatCortex in your research, please cite it; see CITATION.cff.
License
MIT. Developed by Siddharth Saraswat.
Metadata
Release files for chatcortex 0.12.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| chatcortex-0.12.0.tar.gz | 107.8 kB | With provenance | Details for chatcortex-0.12.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| chatcortex-0.12.0-py3-none-any.whl | Python 3 | none | any | Details for chatcortex-0.12.0-py3-none-any.whl |
Total release size: 187.9 kB
Release files / chatcortex-0.12.0.tar.gz
| Download URL | chatcortex-0.12.0.tar.gz |
|---|---|
| Size | 107.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
911e333383d8759a01f937a701c6c3703fa59960a0a25f7280f7ac9d9e056219
|
|
BLAKE2b-256 checksum How to use checksums |
b8841cb6393f3ff6e6a95900dc044122431a5952cb9e9eba2688435a1e47aae7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / chatcortex-0.12.0-py3-none-any.whl
| Download URL | chatcortex-0.12.0-py3-none-any.whl |
|---|---|
| Size | 80.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f2963e7127b065ecd5f5a5bd0d29f038c6128549321d17b1405a97139e25e08a
|
|
BLAKE2b-256 checksum How to use checksums |
ab12afda2bd602c3d22811c2fd1bd9e1b4bf4b0e0384489d8147127d5fcd9770
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for chatcortex 0.12.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for chatcortex-0.12.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.