alquimia-fair-forge 1.0.0

Alquimia Fair Forge library - Performance measurement for AI models and assistants

Fair Forge

Performance-measurement library for evaluating AI models and assistants

Documentation • GitHub

What is Fair Forge?

Fair Forge provides comprehensive metrics for evaluating AI systems:

• Toxicity - Detect toxic language patterns with DIDT fairness framework
• Bias - Analyze biases across protected attributes (gender, race, religion, etc.)
• Context - Assess how well responses align with provided context
• Conversational - Evaluate dialogue quality using Grice's maxims
• Humanity - Measure how natural responses are through emotional analysis
• BestOf - Tournament-style comparison to find the best response

Quick Start

# Install with pip
pip install alquimia-fair-forge

# Or install specific modules
pip install "alquimia-fair-forge[toxicity]"
pip install "alquimia-fair-forge[bias]"
pip install "alquimia-fair-forge[all]"

from fair_forge import Retriever, Dataset
from fair_forge.metrics.toxicity import Toxicity

class MyRetriever(Retriever):
    def load_dataset(self) -> list[Dataset]:
        # Load your dataset here
        pass

metrics = Toxicity.run(MyRetriever, verbose=True)

Documentation

For complete documentation, guides, and API reference visit:

https://fairforge.alquimia.ai

License

MIT License - see LICENSE for details.