Skip to main content

ChatCortex

CI PyPI version Python 3.10+ License: MIT

Find out which model and prompt answer your questions best.

ChatCortex runs your questions through every combination of model, prompt and pipeline step you want to compare, scores the answers against your references, and tells you which setup to use. It works with free model endpoints, so trying it costs nothing.

Website and guide: siddharth1012.github.io/chatCortex

What you get

One command, four sample questions, one free model on Groq:

Prompt Context given Quality Time per answer
"shortest possible phrase" yes 0.92 292 ms
"shortest possible phrase" no 0.67 275 ms
plain question yes 0.14 369 ms
plain question no 0.06 508 ms

Same model, same questions: the prompt and the context moved quality from 0.06 to 0.92. Quality is word overlap with the reference answers, from 0 to 1.

Try it in five minutes

pip install "chatcortex[real]"
git clone https://github.com/siddharth1012/chatCortex
cd chatCortex

Set a free API key from any of SAIA, NVIDIA, Groq or OpenRouter, for example:

export GROQ_API_KEY=...          # PowerShell: $env:GROQ_API_KEY = "..."

Then run it on the sample questions, or on your own CSV of questions and reference answers:

python examples/pick_models.py --max-questions 4
python examples/pick_models.py --questions my_questions.csv

The get-started guide walks through each step: getting keys, writing questions, reading the results, and changing what is compared.

What you can compare

  • Models and vendors. Any vendor or local server that accepts OpenAI-style chat requests: SAIA, NVIDIA, Groq, OpenRouter, Ollama and more. No extra dependencies.
  • Prompts and context. Prompt templates, with or without reading material.
  • Your own code. Each pipeline step can be a plain Python function: a retriever, a re-ranker, a check on the answer.
  • Quality, speed and cost together. Results show the setups that are not beaten on all three, so you can pick the trade-off you want.

Use it from Python

from chatcortex import (
    EvalSet, Example, enumerate_pipelines, evaluate_pipelines,
    quality_frontier, token_f1,
)

eval_set = EvalSet(
    examples=[
        Example("Which organ produces insulin?", expected="pancreas"),
        Example("What is the largest ocean?", expected=["Pacific Ocean", "Pacific"]),
    ],
    scorer=token_f1,
)

pipelines = enumerate_pipelines(registry, task)            # every combination
evaluations = evaluate_pipelines(pipelines, executor, eval_set)

best = quality_frontier(evaluations)[0]                    # highest quality first
print(best.components, best.quality, best.latency_ms)

Using ChatCortex from Python covers the rest: describing components and tasks, running your own functions, calling models from several vendors, measuring cost from token usage, and letting the library choose what to try when there are too many combinations.

Why measure

On twelve reading-comprehension questions from SQuAD, the same Gemma model scored 0.95 on two vendors, but answered in 230 ms on one and 14,516 ms on the other. Without reading material it scored 0.10. A smaller model was no faster and scored 0.33. None of that is visible from a model card. The benchmark that produced these numbers runs on free models and is in benchmarks/qa_frontier/.

Where it stands

ChatCortex is an alpha-stage research library; its API may change between minor versions.

  • Trying every combination is complete and dependable.
  • For spaces too large to try in full, run_closed_loop chooses what to try from the scores it has seen. On the project's benchmark it currently finds the best setup in about as many calls as random picking.
  • Pipelines are straight chains of steps; branching pipelines are not searched.

Documentation

Citing

If you use ChatCortex in your research, please cite it; see CITATION.cff.

License

MIT. Developed by Siddharth Saraswat.

Metadata

Release files for chatcortex 0.11.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chatcortex 0.11.0
File Size Uploaded Provenance Actions
chatcortex-0.11.0.tar.gz 88.2 kB With provenance Details for chatcortex-0.11.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for chatcortex 0.11.0
File Interpreter ABI Platform Actions
chatcortex-0.11.0-py3-none-any.whl Python 3 none any Details for chatcortex-0.11.0-py3-none-any.whl

Total release size: 154.4 kB

Release files / chatcortex-0.11.0.tar.gz

Download URL chatcortex-0.11.0.tar.gz
Size 88.2 kB
Tags Source
SHA-256 checksum
How to use checksums
797068cfa9a8472e8853ecbf83713ce45ba5dd057da4e21ea36da167a8ef5c74
BLAKE2b-256 checksum
How to use checksums
2b94e6ba2eb46245a5cce4bdbb9d9214b43492118f4e4e67a4df61cb99d34b76
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / chatcortex-0.11.0-py3-none-any.whl

Download URL chatcortex-0.11.0-py3-none-any.whl
Size 66.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dafaf81d5c23e36eb6924fbe9127e874a32c420389f2fede7d3f5b1dc931ac5a
BLAKE2b-256 checksum
How to use checksums
ae4c95230a0ab5627ea825e716bbd03e9cc54fd555e21a4e9d1b72739256fe03
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for chatcortex 0.11.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for chatcortex-0.11.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page