This project has been archived by its maintainers, and is no longer receiving any updates.
Performance-measurement library for evaluating AI models and assistants
What is Fair Forge?
Fair Forge provides comprehensive metrics for evaluating AI systems:
- Toxicity - Detect toxic language patterns with DIDT fairness framework
- Bias - Analyze biases across protected attributes (gender, race, religion, etc.)
- Context - Assess how well responses align with provided context
- Conversational - Evaluate dialogue quality using Grice's maxims
- Humanity - Measure how natural responses are through emotional analysis
- BestOf - Tournament-style comparison to find the best response
- Agentic - Evaluate autonomous agent behavior and tool usage
Quick Start
# Install with pip
pip install alquimia-fair-forge
# Or install specific modules
pip install "alquimia-fair-forge[toxicity]"
pip install "alquimia-fair-forge[bias]"
pip install "alquimia-fair-forge[all]"
from fair_forge import Retriever, Dataset
from fair_forge.metrics.toxicity import Toxicity
class MyRetriever(Retriever):
def load_dataset(self) -> list[Dataset]:
# Load your dataset here
pass
metrics = Toxicity.run(MyRetriever, verbose=True)
Documentation
For complete documentation, guides, and API reference visit:
Contributing
Setup
# Clone and install dependencies
git clone https://github.com/Alquimia-ai/fair-forge.git
cd fair-forge
uv sync
# Install pre-commit hooks (required)
uv run pre-commit install --hook-type pre-commit --hook-type commit-msg
Pre-commit Hooks
This project uses pre-commit to enforce code quality and commit conventions. The following hooks run automatically:
| Hook | Stage | Purpose |
|---|---|---|
trailing-whitespace |
pre-commit | Remove trailing whitespace |
end-of-file-fixer |
pre-commit | Ensure files end with a newline |
check-yaml / check-toml |
pre-commit | Validate config files |
check-added-large-files |
pre-commit | Block files >1MB |
detect-private-key |
pre-commit | Prevent accidental secret commits |
ruff |
pre-commit | Lint and auto-fix Python code |
ruff-format |
pre-commit | Format Python code |
commitizen |
commit-msg | Enforce Conventional Commits format |
Commit Convention
All commits must follow the Conventional Commits specification, enforced by commitizen:
type(scope): description
Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
Scopes: metrics, runners, generators, storage, schemas, core, llm, guardians, deps
Examples:
feat(metrics): add agentic evaluation metric
fix(toxicity): handle empty batches
docs: update installation guide
test(runners): add integration tests
Claude Code Skills
This project includes Claude Code skills to automate common workflows. If you use Claude Code as your development tool, the following skills are available:
| Skill | Command | Description |
|---|---|---|
| Commit | /commit |
Analyzes staged changes and generates a properly formatted Conventional Commit message |
| Docs | /docs [type] [name] |
Generates Mintlify MDX documentation for new modules (e.g., /docs metric agentic) |
| Lambda | /fair-forge-lambda |
Scaffolds AWS Lambda deployment boilerplate for Fair-Forge modules |
License
MIT License - see LICENSE for details.
1 maintainer
Metadata
1 maintainer
Release files for alquimia-fair-forge 3.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| alquimia_fair_forge-3.0.0.tar.gz | 796.7 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| alquimia_fair_forge-3.0.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.6 MB
Release files / alquimia_fair_forge-3.0.0.tar.gz
| Download URL | alquimia_fair_forge-3.0.0.tar.gz |
|---|---|
| Size | 796.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c19bdb37ae1510b6d156c9302eefff1f1600cf62a6b38886fa3433ec598e9439
|
|
BLAKE2b-256 checksum How to use checksums |
2d3af4f8e7abecb887199b13aa61bfc09582ba3057ddb8c3147330edead8c923
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / alquimia_fair_forge-3.0.0-py3-none-any.whl
| Download URL | alquimia_fair_forge-3.0.0-py3-none-any.whl |
|---|---|
| Size | 830.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5e8808c1fcf05e3e327f6ce084a931aad12f56ed688b463e5ba3e5718dbd469c
|
|
BLAKE2b-256 checksum How to use checksums |
8e183fba62738541cc02c0f4003668aa3b3882583abf859d09ae286f9f3996f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for alquimia-fair-forge 3.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
