Tiny-Eval
Tiny-Eval is a minimal framework for evaluating language models. It provides a clean, async-first API for interacting with various LLM providers and running evaluation experiments.
Features
-
Multi-Provider Support
- OpenAI API integration
- OpenRouter API integration for access to multiple model providers
- Extensible interface for adding new providers
-
Robust API Handling
- Automatic rate limiting with configurable parameters
- Built-in exponential backoff retry logic
- Async-first design for efficient request handling
-
Evaluation Utilities
- Log probability calculation support
- Async function chaining for complex evaluation pipelines
- Batch processing capabilities
-
Experiment Framework
- Progress tracking for long-running experiments
- Structured data collection and analysis
- Built-in visualization tools using Streamlit
Installation
git clone https://github.com/dtch1997/tiny-eval.git
cd tiny-eval
pip install -e .
Usage
Minimal usage is shown as follows:
import asyncio
from tiny_eval.core.constants import Model
from tiny_eval.model_api import build_model_api
async def main():
model = Model.GPT_4o_mini
api = build_model_api(model)
question = "What is the capital of France?"
response = await api.get_response(question)
print("Question:", question)
print("Response:", response)
if __name__ == "__main__":
asyncio.run(main())
See examples for more examples.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tiny_eval-0.5.0.tar.gz.
File metadata
- Download URL: tiny_eval-0.5.0.tar.gz
- Upload date:
- Size: 21.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ea687a59f45179d13e2bad0df17e93ad0412e82544a43a94316be23ba5480b3f
|
|
| MD5 |
a7d4e6bb6f87f32388dba35752b2b77f
|
|
| BLAKE2b-256 |
950e5109ca5af9f0043569841f6c7355dc134fe82d03a04eda95bac738fa869a
|
Provenance
The following attestation bundles were made for tiny_eval-0.5.0.tar.gz:
Publisher:
ci.yaml on dtch1997/tiny-eval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
tiny_eval-0.5.0.tar.gz -
Subject digest:
ea687a59f45179d13e2bad0df17e93ad0412e82544a43a94316be23ba5480b3f - Sigstore transparency entry: 171215938
- Sigstore integration time:
-
Permalink:
dtch1997/tiny-eval@d15b8db113d09dd4cfa376034d1b6a2549bf2f6b -
Branch / Tag:
refs/heads/main - Owner: https://github.com/dtch1997
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
ci.yaml@d15b8db113d09dd4cfa376034d1b6a2549bf2f6b -
Trigger Event:
push
-
Statement type:
File details
Details for the file tiny_eval-0.5.0-py3-none-any.whl.
File metadata
- Download URL: tiny_eval-0.5.0-py3-none-any.whl
- Upload date:
- Size: 21.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1e8bf61a4aacfab22cb941aa6eb09f7d759b86530d4a537f31bd4cc95c2903c6
|
|
| MD5 |
723a10aab21dd1f077b7c9bce01a6c63
|
|
| BLAKE2b-256 |
1b446a61db37e2d48fa4ed5ebe323f50a14a7bdfcbcba7e4f4bd61532d48fe71
|
Provenance
The following attestation bundles were made for tiny_eval-0.5.0-py3-none-any.whl:
Publisher:
ci.yaml on dtch1997/tiny-eval
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
tiny_eval-0.5.0-py3-none-any.whl -
Subject digest:
1e8bf61a4aacfab22cb941aa6eb09f7d759b86530d4a537f31bd4cc95c2903c6 - Sigstore transparency entry: 171215939
- Sigstore integration time:
-
Permalink:
dtch1997/tiny-eval@d15b8db113d09dd4cfa376034d1b6a2549bf2f6b -
Branch / Tag:
refs/heads/main - Owner: https://github.com/dtch1997
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
ci.yaml@d15b8db113d09dd4cfa376034d1b6a2549bf2f6b -
Trigger Event:
push
-
Statement type:
