touchstone
Small, reproducible tests for LLMs in Hindi and Hinglish.
One command. A clear report. Every prompt and reply saved.
What is this, in one picture?
A touchstone is the dark stone jewellers use to test gold. This project does the same for AI models: it gives them a small exam in Hindi and Hinglish and checks their answers.
| In an exam | In touchstone |
|---|---|
| Question paper | Our own written questions (the dataset) |
| Student | The AI model you choose (Gemini, Llama, Qwen, ...) |
| Teacher checking answers | Fixed scoring rules, no AI judge |
| Report card | A score table, an HTML report, and a JSON file with everything |
How one test run works
flowchart TD
A([You start a run<br/>command or web UI]) --> B[1. Load questions<br/>only ones reviewed by 2 people]
B --> C[2. Build the prompt<br/>same format for every model]
C --> D[3. Ask the AI model<br/>Gemini · Groq · OpenRouter · Ollama]
D --> E[4. Check the answer<br/>fixed rules, no AI judge]
E --> F([5. Save the report<br/>table · HTML · JSON])
D -. reply saved at once .-> G[(Local cache)]
G -. rerun resumes here .-> D
What happens in each step? (click to open)
- Load questions. Only questions approved by two reviewers (other than the author) are used. Drafts are never scored.
- Build the prompt. Every model gets the exact same text and the same settings: temperature 0, seed 42, top_p 1.
- Ask the model. Free APIs allow only a few requests a minute, so touchstone waits between calls and retries politely. Each reply is saved straight away, so if your internet or quota runs out, the same command continues where it stopped.
- Check the answer. Multiple choice understands
B,(b),Answer: B,उत्तर: Band similar. If the model names two options or none, the answer is marked unparsed and counted wrong. Translation uses chrF. Instructions use rule checks. - Save the report. You get a table in the terminal and three files in
runs/.
Quickstart (2 steps)
You need Python 3.10 or newer. Nothing else.
pip install "touchstone-llm[ui]"
touchstone ui
Your browser opens at http://127.0.0.1:7860. Then:
- Open New test and pick a provider. If it says "needs a key", click the link to get a free key, paste it into the box and press Save key.
- Pick a model and tick some topics, then press Start test.
- Watch the progress bar. The result opens by itself.
No key at all? Choose Practice (mock), a pretend model that needs no key.
Prefer the terminal? (click to open)
touchstone list-models groq # model ids change often, so pick from the live list
touchstone run --model groq/<model-id> --tasks all --drafts --limit 2
touchstone run --model mock/fixed --tasks all --drafts # practice, no key needed
touchstone reproduce runs/<result>.json # rerun the exact same test
touchstone --help # every command
Keys for the terminal can be saved once from the web UI, or put in a .env file (see .env.example), or set as environment variables.
Where do I get a free API key?
| Provider | Get a key | Environment variable |
|---|---|---|
| Gemini | aistudio.google.com/apikey | GEMINI_API_KEY |
| Groq | console.groq.com/keys | GROQ_API_KEY |
| OpenRouter | openrouter.ai/keys | OPENROUTER_API_KEY |
| Ollama (local) | no key needed | runs at http://localhost:11434 |
Keys you save in the web UI are stored only on your computer, in ~/.touchstone/.env. Free-tier limits and terms change, so check the provider's page. Never paste keys into issues or chats.
For contributors: run from the source code
You also need Node.js 20+ to build the web UI once.
git clone https://github.com/harshita2012sharma-max/Touchstone.git
cd Touchstone
python -m venv .venv
.venv/Scripts/activate # macOS/Linux: source .venv/bin/activate
cd ui && npm ci && npm run build && cd ..
pip install -e ".[dev,ui]"
pytest
See CONTRIBUTING.md.
What it tests
Six small tasks. Click a task to see a real example question.
hindi_comprehension: read a short Hindi passage, answer a question
Passage: मीना हर रविवार अपनी दादी के साथ बाज़ार जाती है। ... Question: मीना अपनी दादी के साथ बाज़ार कब जाती है? A. हर सोमवार · B. हर रविवार · C. हर शनिवार · D. हर दिन
Scoring: the chosen letter must match exactly.
hinglish_understanding: understand Roman-script chat, including sarcasm
Message: Kabir (bus chhootne ke baad): Wah, kya perfect timing hai meri! Question: Kabir asal mein kya keh raha hai? A. Woh frustrated hai kyunki uski bus chhoot gayi · B. Woh khush hai ki time pe pahuncha · ...
Scoring: the chosen letter must match exactly.
translation_hi_en: translate Hindi into English
Hindi: कृपया दरवाज़ा बंद कर दीजिए। Accepted English: "Please close the door." / "Please shut the door."
Scoring: chrF (character overlap with the best reference), from 0 to 100%.
codemixed_instructions: follow a Hinglish instruction exactly
Instruction: Teen Indian sweets ke naam batao, comma se alag karke. Sirf naam likhna, aur kuch nahi.
Scoring: rule checks (exactly 3 comma-separated items, at most 10 words). All rules must pass. Writing 5 sweets fails.
hindi_reasoning: solve a word problem
Question: एक कमीज़ की कीमत 800 रुपये है। उस पर 25 प्रतिशत की छूट मिल रही है। छूट के बाद कमीज़ कितने रुपये की पड़ेगी? Answer: 600
Scoring: the final number must match. Devanagari digits (६००) and Indian grouping (1,00,000) are understood.
cultural_context: neutral facts about everyday India
Question: इडली बनाने में मुख्य रूप से किन चीज़ों का इस्तेमाल होता है? A. गेहूँ और बेसन · B. मक्का और सूजी · C. जौ और चना · D. चावल और उड़द दाल
Only neutral facts: festivals, food, geography, institutions. No religion comparisons, caste, politics or stereotypes.
How the questions are made
Trust in a benchmark starts with its questions. Every question goes through this path, and the code enforces it:
flowchart LR
A[Draft question<br/>datasets/pending] --> B[Review sheet<br/>ok · fix · reject]
B --> C{2 approvals<br/>from people other<br/>than the author?}
C -- no --> A
C -- yes --> D[Shipped data<br/>now it can be scored]
The rules every question follows
- Written for this project. Nothing is copied from exams, books or websites.
- Licensed CC-BY-4.0, so anyone can reuse it with credit.
- Approved by at least two reviewers who are not the author. The test suite fails if this is not true.
- Some first drafts were written with help from an AI assistant. A human author edits and fact-checks every item, and two humans review it. This is stated openly in DATASETS.md.
Want to help review? See the review guide. No coding needed, just a spreadsheet.
How results stay fair
| Promise | How it is kept |
|---|---|
| Same test for every model | Fixed temperature 0, seed 42, top_p 1 and identical prompts. The CLI cannot change temperature or seed. |
| Nothing hidden | Every result saves the model you asked for, the model id the provider actually used, the date, settings, number of items, and every exact prompt and reply. |
| Anyone can check | touchstone reproduce runs/<file>.json reruns the same config and reports whether the prompts and settings match. |
| No hype | Reports show numbers, sample sizes and limitations. touchstone never names a "best" model, and ties are shown as ties. |
Results
No official results yet. They will be published here only after every question has been reviewed, and each result will show the date, the model id, settings, the number of items and the prompts.
Limitations
FAQ
Does it need a GPU or a powerful laptop?
No. It was built on an Intel i3 laptop with 8 GB RAM and no GPU. Models run on the provider's servers, or locally through Ollama if you use a small model.
Is it free?
touchstone itself is free and open source. Gemini, Groq and OpenRouter have free tiers (with limits), and Ollama is free and runs on your machine.
Are my API keys safe?
Keys are read only from your local .env file, which git ignores. touchstone never prints, logs or saves them, sends no telemetry, and the web UI only listens on 127.0.0.1. See SECURITY.md.
Why can the same model get different scores on different days?
Providers can update a model without changing its name, and some ignore the seed. That is why every result records the date and the exact model id the provider reported.
Can I add my own task or model provider?
Yes. Tasks and providers are plugins (touchstone_llm.tasks and touchstone_llm.backends entry points). See CONTRIBUTING.md.
Related work
touchstone is a small, laptop-first tool, not a replacement for these larger efforts. It shares no data or code with them.
- MILU (AI4Bharat and IBM Research India): multiple-choice Indic language understanding across 11 languages. Verma et al., NAACL 2025. arXiv:2411.02538
- IndicGenBench (Google Research India): generation tasks across 29 Indic languages. Singh et al., ACL 2024. arXiv:2404.16816 · code and data
- indic_eval (GitHub: adithya-s-k): an Indic LLM evaluation suite built on Hugging Face LightEval. github.com/adithya-s-k/indic_eval. This project was first planned as "indic-eval" and renamed so the two are not confused.
- HinglishEval: code-generation models on Hinglish prompts, COMPUTE 2024. github.com/mrigankpawagi/HinglishEval
Cite, contribute, license
- Cite: use CITATION.cff, or GitHub's "Cite this repository" button.
- Contribute: see CONTRIBUTING.md and the Code of Conduct. Reviews from fluent Hindi and Hinglish speakers are the most valuable help right now.
- License: code under Apache-2.0, dataset items under CC-BY-4.0.
Metadata
Release files for touchstone-llm 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| touchstone_llm-0.1.1.tar.gz | 790.1 kB | With provenance | Details for touchstone_llm-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| touchstone_llm-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for touchstone_llm-0.1.1-py3-none-any.whl |
Total release size: 1.5 MB
Release files / touchstone_llm-0.1.1.tar.gz
| Download URL | touchstone_llm-0.1.1.tar.gz |
|---|---|
| Size | 790.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
54ac8ee452e1844ee32b216caf39beda0de01e0eeb8f5cfda90bc1effbca61f4
|
|
BLAKE2b-256 checksum How to use checksums |
9f13b36671b287a10801398a2306905aeb9bd59b485974a4d4e8fbcb521a99a0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / touchstone_llm-0.1.1-py3-none-any.whl
| Download URL | touchstone_llm-0.1.1-py3-none-any.whl |
|---|---|
| Size | 737.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b93381a6ef24007e4717a7e6d81542cb447a0b1810402d9072890c3b665619f4
|
|
BLAKE2b-256 checksum How to use checksums |
3a91d3515e00c9bb595079ca04047393d5be2c2033480d3d73e0d668ec888961
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for touchstone-llm 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for touchstone_llm-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.