isaivam
Supercharge your LLM application evaluations.
isaivam is an evaluation framework for RAG and LLM applications. The name isaivam means music — a nod to its roots in "ragas". It provides objective metrics, test data generation, and data-driven insights so you can move away from slow, subjective assessments toward efficient, repeatable evaluation workflows.
Key Features
- Objective metrics: Evaluate LLM applications using both LLM-based and traditional metrics.
- Test data generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
- Integrations: Works with popular LLM frameworks like LangChain and major observability tools.
- Feedback loops: Leverage production data to continually improve your LLM applications.
Installation
From source (editable):
pip install -e .
Dependencies are tracked in requirements.txt:
pip install -r requirements.txt
Quickstart
isaivam comes with pre-built metrics for common evaluation tasks. For example, DiscreteMetric evaluates any aspect of your output:
import asyncio
from openai import AsyncOpenAI
from isaivam.metrics import DiscreteMetric
from isaivam.llms import llm_factory
# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)
# Create a custom aspect evaluator
metric = DiscreteMetric(
name="summary_accuracy",
allowed_values=["accurate", "inaccurate"],
prompt="""Evaluate if the summary is accurate and captures key information.
Response: {response}
Answer with only 'accurate' or 'inaccurate'."""
)
# Score your application's output
async def main():
score = await metric.ascore(
llm=llm,
response="The summary of the text is..."
)
print(f"Score: {score.value}") # 'accurate' or 'inaccurate'
print(f"Reason: {score.reason}")
if __name__ == "__main__":
asyncio.run(main())
Note: Make sure your
OPENAI_API_KEYenvironment variable is set.
CLI
The library installs an isaivam command for running experiments and evaluations:
isaivam evals <eval_file> --dataset <name> --metrics <fields>
Testing Locally
There is no automated test suite yet, so testing locally means installing the package and exercising the library and CLI.
-
Install in editable mode so code changes take effect immediately:
pip install -e .
-
Verify the import and version:
python -c "import isaivam; print(isaivam.__version__)"
-
Verify the CLI is available:
isaivam --help -
Run the Quickstart example end-to-end. It calls OpenAI, so set your key first:
export OPENAI_API_KEY=your_key_here python your_script.py
-
Run an eval via the CLI (see
examples/for reference eval files):isaivam evals <eval_file> --dataset <name> --metrics <fields>
Analytics
isaivam collects minimal, anonymized usage data to guide development. To opt out, set the ISAIVAM_DO_NOT_TRACK environment variable to true.
Acknowledgements
isaivam is derived from the open-source ragas project (Apache-2.0). See LICENSE for details.
Metadata
Release files for isaivam 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| isaivam-0.1.0.tar.gz | 329.9 kB | Without provenance | Details for isaivam-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| isaivam-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for isaivam-0.1.0-py3-none-any.whl |
Total release size: 786.4 kB
Release files / isaivam-0.1.0.tar.gz
| Download URL | isaivam-0.1.0.tar.gz |
|---|---|
| Size | 329.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c0c3a8ae92c300adfa9a07ca59a60a12fa63e4eaf6725ed592fd286533c08e3b
|
|
BLAKE2b-256 checksum How to use checksums |
474b5557ec9e670b912912919e5c4ee58658b348362d60dc7c0865aab3eddbf0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.12
|
Release files / isaivam-0.1.0-py3-none-any.whl
| Download URL | isaivam-0.1.0-py3-none-any.whl |
|---|---|
| Size | 456.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2a9ccd182110d925a4cc9a2be18b5eb51bbdf02f87f5bae9997821a65ae5ae8e
|
|
BLAKE2b-256 checksum How to use checksums |
79ddc037fed38c23bc2de393a146db3380dabd391502eab9ebd6db3e8743c8f9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.12
|
Security for isaivam 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.