openvals 0.0.2

Open evaluation and benchmarking framework for AI systems

OpenVals

OpenVals is an open evaluation and benchmarking framework for LLMs, SLMs, and AI systems, designed to help organizations measure, compare, and trust AI models before deployment.

Evaluate. Benchmark. Trust. Deploy with Confidence.

---

🚀 Why OpenVals?

Most AI evaluation tools stop at metrics.

OpenVals goes further:

• ✅ Aligns evaluation with business objectives
• ✅ Enables multi-model benchmarking
• ✅ Quantifies trust, risk, and performance
• ✅ Supports deployment decision-making

---

🔷 Core Capabilities

1. Model Evaluation

Evaluate model outputs against structured datasets using:

• Accuracy
• Semantic similarity
• Latency

---

2. Multi-Model Benchmarking

Compare multiple models under the same conditions:

• Side-by-side evaluation
• Normalized scoring
• Model ranking
• Performance insights

---

3. Scoring Engine

Weighted scoring aligned to business priorities:

[ \text{Trust Score} = \sum_{i=1}^{n} w_i \cdot m_i ]

• Customize weights per use case
• Balance accuracy, cost, and latency

---

4. Extensible Architecture

• Plug-and-play model adapters
• Custom metrics support
• Scalable evaluation pipelines

---

📦 Installation

bash pip install -e .

---

⚡ Quick Start

1. Run Evaluation

bash openvals run --dataset examples/sample_eval.json

---

2. Run Multi-Model Benchmark

bash openvals benchmark --dataset examples/sample_eval.json

---

🧪 Example Dataset

json [ { "id": "1", "input": "hello", "expected_output": "olleh" } ]

---

🧠 Python API Usage

python from openvals.core.evaluator import Evaluator from openvals.models.dummy_model import DummyModel from openvals.datasets.loader import load_dataset dataset = load_dataset("examples/sample_eval.json") model = DummyModel() evaluator = Evaluator(model, dataset) result = evaluator.run() print(result)

---

🔷 Multi-Model Benchmarking Example

python from openvals.benchmarking.benchmark import BenchmarkRunner from openvals.benchmarking.normalization import normalize_scores from openvals.benchmarking.ranking import rank_models models = { "model_a": DummyModel(), "model_b": DummyModel() } runner = BenchmarkRunner(models, dataset) results = runner.run() normalized = normalize_scores(results) ranking = rank_models(normalized, { "accuracy": 0.5, "semantic": 0.3, "latency": 0.2 }) print(ranking)

---

🏗️ Project Structure

openvals/ │ ├── core/ # Evaluation engine ├── models/ # Model adapters ├── datasets/ # Dataset loading & schema ├── metrics/ # Evaluation metrics ├── benchmarking/ # Multi-model benchmarking layer ├── scoring/ # Scoring logic ├── safety/ # Risk & safety checks (WIP) ├── reporting/ # Output & reports (WIP) ├── cli.py # Command-line interface

---

🔐 Roadmap

v0.3

• OpenAI / Ollama / HuggingFace adapters
• Config-driven benchmarking (config.yaml)
• Parallel execution

v0.4

• Safety layer (prompt injection, hallucination)
• Cost tracking
• Advanced reporting

v1.0

• Trust Score engine
• Industry-specific benchmarks
• Certification layer

---

🌐 Vision

OpenVals is evolving into:

A Trust Layer for AI Systems

Where organizations can answer:

• Which model should be deployed?
• Is it safe?
• Is it aligned with business goals?
• Can it be trusted in production?

---

🤝 Contributing

Contributions are welcome.

• Fork the repo
• Create a feature branch
• Submit a pull request

---

📄 License

MIT License

---

🧠 Backed by

Developed as part of DrPinnacle’s AI Trust & Validation Initiative, focused on building secure, scalable, and trustworthy AI systems.

---

⚡ Final Thought

AI models are easy to build.

Trusting them is the hard part.

OpenVals exists to solve that.