Skip to main content

touchstone

Small, reproducible tests for LLMs in Hindi and Hinglish.
One command. A clear report. Every prompt and reply saved.

License: Apache-2.0 Python 3.10+ PyPI Status: alpha Data: CC-BY-4.0


What is this, in one picture?

A touchstone is the dark stone jewellers use to test gold. This project does the same for AI models: it gives them a small exam in Hindi and Hinglish and checks their answers.

In an exam In touchstone
Question paper Our own written questions (the dataset)
Student The AI model you choose (Gemini, Llama, Qwen, ...)
Teacher checking answers Fixed scoring rules, no AI judge
Report card A score table, an HTML report, and a JSON file with everything

How one test run works

flowchart TD
    A([You start a run<br/>command or web UI]) --> B[1. Load questions<br/>only ones reviewed by 2 people]
    B --> C[2. Build the prompt<br/>same format for every model]
    C --> D[3. Ask the AI model<br/>Gemini · Groq · OpenRouter · Ollama]
    D --> E[4. Check the answer<br/>fixed rules, no AI judge]
    E --> F([5. Save the report<br/>table · HTML · JSON])
    D -. reply saved at once .-> G[(Local cache)]
    G -. rerun resumes here .-> D
What happens in each step? (click to open)
  1. Load questions. Only questions approved by two reviewers (other than the author) are used. Drafts are never scored.
  2. Build the prompt. Every model gets the exact same text and the same settings: temperature 0, seed 42, top_p 1.
  3. Ask the model. Free APIs allow only a few requests a minute, so touchstone waits between calls and retries politely. Each reply is saved straight away, so if your internet or quota runs out, the same command continues where it stopped.
  4. Check the answer. Multiple choice understands B, (b), Answer: B, उत्तर: B and similar. If the model names two options or none, the answer is marked unparsed and counted wrong. Translation uses chrF. Instructions use rule checks.
  5. Save the report. You get a table in the terminal and three files in runs/.

Quickstart (2 steps)

You need Python 3.10 or newer. Nothing else. Open a terminal (on Windows: PowerShell) and run:

python -m pip install "touchstone-llm[ui]"
python -m touchstone_llm ui

Keep that terminal window open. Your browser opens at http://127.0.0.1:7860 (if that port is busy, the terminal shows the address it used instead). Then:

  1. Open New test and pick a provider. If it says "needs a key", click the link to get a free key, paste it into the box and press Save key.
  2. Pick a model and tick some topics, then press Start test.
  3. Watch the progress bar. The result opens by itself.

No key at all? Choose Practice (mock), a pretend model that needs no key.

Something went wrong? (click to open)
What you see What to do
python is not recognized (Windows) Use py instead of python in both commands, or install Python from python.org and tick "Add python.exe to PATH".
No matching distribution found for touchstone-llm Your Python is older than 3.10. Check with python --version and install a newer one.
Browser says "refused to connect" The server is not running. Run python -m touchstone_llm ui again and keep the window open.
touchstone is not recognized Normal on many Windows setups. Use python -m touchstone_llm ui (the short touchstone command only works if pip's Scripts folder is on PATH).
macOS/Linux says pip or python not found Use python3 -m pip install "touchstone-llm[ui]" and python3 -m touchstone_llm ui.

To update later: python -m pip install -U "touchstone-llm[ui]".

Prefer the terminal? (click to open)
python -m touchstone_llm list-models groq                  # model ids change often, pick from the live list
python -m touchstone_llm run --model groq/<model-id> --tasks all --drafts --limit 2
python -m touchstone_llm run --model mock/fixed --tasks all --drafts   # practice, no key needed
python -m touchstone_llm reproduce runs/<result>.json      # rerun the exact same test
python -m touchstone_llm --help                            # every command

If the touchstone command works on your machine, you can use it as a shortcut, e.g. touchstone run ....

Keys for the terminal can be saved once from the web UI, or put in a .env file (see .env.example), or set as environment variables.

Where do I get a free API key?
Provider Get a key Environment variable
Gemini aistudio.google.com/apikey GEMINI_API_KEY
Groq console.groq.com/keys GROQ_API_KEY
OpenRouter openrouter.ai/keys OPENROUTER_API_KEY
Ollama (local) no key needed runs at http://localhost:11434

Keys you save in the web UI are stored only on your computer, in ~/.touchstone/.env. Free-tier limits and terms change, so check the provider's page. Never paste keys into issues or chats.

For contributors: run from the source code

You also need Node.js 20+ to build the web UI once.

git clone https://github.com/harshita2012sharma-max/Touchstone.git
cd Touchstone
python -m venv .venv
.venv/Scripts/activate            # macOS/Linux: source .venv/bin/activate
cd ui && npm ci && npm run build && cd ..
pip install -e ".[dev,ui]"
pytest

See CONTRIBUTING.md.

What it tests

Six small tasks. Click a task to see a real example question.

hindi_comprehension: read a short Hindi passage, answer a question

Passage: मीना हर रविवार अपनी दादी के साथ बाज़ार जाती है। ... Question: मीना अपनी दादी के साथ बाज़ार कब जाती है? A. हर सोमवार · B. हर रविवार · C. हर शनिवार · D. हर दिन

Scoring: the chosen letter must match exactly.

hinglish_understanding: understand Roman-script chat, including sarcasm

Message: Kabir (bus chhootne ke baad): Wah, kya perfect timing hai meri! Question: Kabir asal mein kya keh raha hai? A. Woh frustrated hai kyunki uski bus chhoot gayi · B. Woh khush hai ki time pe pahuncha · ...

Scoring: the chosen letter must match exactly.

translation_hi_en: translate Hindi into English

Hindi: कृपया दरवाज़ा बंद कर दीजिए। Accepted English: "Please close the door." / "Please shut the door."

Scoring: chrF (character overlap with the best reference), from 0 to 100%.

codemixed_instructions: follow a Hinglish instruction exactly

Instruction: Teen Indian sweets ke naam batao, comma se alag karke. Sirf naam likhna, aur kuch nahi.

Scoring: rule checks (exactly 3 comma-separated items, at most 10 words). All rules must pass. Writing 5 sweets fails.

hindi_reasoning: solve a word problem

Question: एक कमीज़ की कीमत 800 रुपये है। उस पर 25 प्रतिशत की छूट मिल रही है। छूट के बाद कमीज़ कितने रुपये की पड़ेगी? Answer: 600

Scoring: the final number must match. Devanagari digits (६००) and Indian grouping (1,00,000) are understood.

cultural_context: neutral facts about everyday India

Question: इडली बनाने में मुख्य रूप से किन चीज़ों का इस्तेमाल होता है? A. गेहूँ और बेसन · B. मक्का और सूजी · C. जौ और चना · D. चावल और उड़द दाल

Only neutral facts: festivals, food, geography, institutions. No religion comparisons, caste, politics or stereotypes.

How the questions are made

Trust in a benchmark starts with its questions. Every question goes through this path, and the code enforces it:

flowchart LR
    A[Draft question<br/>datasets/pending] --> B[Review sheet<br/>ok · fix · reject]
    B --> C{2 approvals<br/>from people other<br/>than the author?}
    C -- no --> A
    C -- yes --> D[Shipped data<br/>now it can be scored]
The rules every question follows
  • Written for this project. Nothing is copied from exams, books or websites.
  • Licensed CC-BY-4.0, so anyone can reuse it with credit.
  • Approved by at least two reviewers who are not the author. The test suite fails if this is not true.
  • Some first drafts were written with help from an AI assistant. A human author edits and fact-checks every item, and two humans review it. This is stated openly in DATASETS.md.

Want to help review? See the review guide. No coding needed, just a spreadsheet.

How results stay fair

Promise How it is kept
Same test for every model Fixed temperature 0, seed 42, top_p 1 and identical prompts. The CLI cannot change temperature or seed.
Nothing hidden Every result saves the model you asked for, the model id the provider actually used, the date, settings, number of items, and every exact prompt and reply.
Anyone can check touchstone reproduce runs/<file>.json reruns the same config and reports whether the prompts and settings match.
No hype Reports show numbers, sample sizes and limitations. touchstone never names a "best" model, and ties are shown as ties.

Results

No official results yet. They will be published here only after every question has been reviewed, and each result will show the date, the model id, settings, the number of items and the prompts.

Limitations

FAQ

Does it need a GPU or a powerful laptop?

No. It was built on an Intel i3 laptop with 8 GB RAM and no GPU. Models run on the provider's servers, or locally through Ollama if you use a small model.

Is it free?

touchstone itself is free and open source. Gemini, Groq and OpenRouter have free tiers (with limits), and Ollama is free and runs on your machine.

Are my API keys safe?

Keys are read only from your local .env file, which git ignores. touchstone never prints, logs or saves them, sends no telemetry, and the web UI only listens on 127.0.0.1. See SECURITY.md.

Why can the same model get different scores on different days?

Providers can update a model without changing its name, and some ignore the seed. That is why every result records the date and the exact model id the provider reported.

Can I add my own task or model provider?

Yes. Tasks and providers are plugins (touchstone_llm.tasks and touchstone_llm.backends entry points). See CONTRIBUTING.md.

touchstone is a small, laptop-first tool, not a replacement for these larger efforts. It shares no data or code with them.

  • MILU (AI4Bharat and IBM Research India): multiple-choice Indic language understanding across 11 languages. Verma et al., NAACL 2025. arXiv:2411.02538
  • IndicGenBench (Google Research India): generation tasks across 29 Indic languages. Singh et al., ACL 2024. arXiv:2404.16816 · code and data
  • indic_eval (GitHub: adithya-s-k): an Indic LLM evaluation suite built on Hugging Face LightEval. github.com/adithya-s-k/indic_eval. This project was first planned as "indic-eval" and renamed so the two are not confused.
  • HinglishEval: code-generation models on Hinglish prompts, COMPUTE 2024. github.com/mrigankpawagi/HinglishEval

Cite, contribute, license

  • Cite: use CITATION.cff, or GitHub's "Cite this repository" button.
  • Contribute: see CONTRIBUTING.md and the Code of Conduct. Reviews from fluent Hindi and Hinglish speakers are the most valuable help right now.
  • License: code under Apache-2.0, dataset items under CC-BY-4.0.

Metadata

Release files for touchstone-llm 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for touchstone-llm 0.1.2
File Size Uploaded Provenance Actions
touchstone_llm-0.1.2.tar.gz 790.9 kB With provenance Details for touchstone_llm-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for touchstone-llm 0.1.2
File Interpreter ABI Platform Actions
touchstone_llm-0.1.2-py3-none-any.whl Python 3 none any Details for touchstone_llm-0.1.2-py3-none-any.whl

Total release size: 1.5 MB

Release files / touchstone_llm-0.1.2.tar.gz

Download URL touchstone_llm-0.1.2.tar.gz
Size 790.9 kB
Tags Source
SHA-256 checksum
How to use checksums
d7be978c67700afaef5e64140c9ef427a1150d0eb5bc46b32a8f10aafc0d8edc
BLAKE2b-256 checksum
How to use checksums
67a35d6ae5cfd03877cd09531d447c44e91c53fbf13d0c44b1262949dbd6c07d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / touchstone_llm-0.1.2-py3-none-any.whl

Download URL touchstone_llm-0.1.2-py3-none-any.whl
Size 738.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fe15868a8c26f88459831346cf245427dbece57c01e490c70398362639eb1fd0
BLAKE2b-256 checksum
How to use checksums
8850cfccc1d1c4d8fe40205740ae56751b069d2b7759eb354cb14c8c27e3aded
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for touchstone-llm 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for touchstone_llm-0.1.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page