Skip to main content

NiriZan

Continuous Evaluation Infrastructure for Production AI

"Inspection through Measurement" "Engineering Trust Through Continuous Evaluation"


CI Packaging Cross-platform
Security (CIA) Scorecard supply-chain security
Documentation Wikidata License


Why NiriZan?

Modern AI systems are probabilistic rather than deterministic. Traditional software testing alone cannot determine whether a retrieval pipeline, language model, or AI agent is performing correctly. NiriZan exists to provide continuous, reproducible evaluation infrastructure that enables teams to measure quality, detect regressions, compare experiments, and build confidence in production AI systems.

NiriZan is an open-source framework to provide automated judge drift attribution, fixed anchor sets with repeatable, on-demand rescoring, rigorous statistical gating (Mann-Whitney + Holm-Bonferroni), multivariate structure-drift detection, trust-weighted health scoring, and CI/CD-integrated regression gating in a single, architecturally disciplined Python package.


Installation

pip install nirizan

Package: pypi.org/project/nirizan


What NiriZan Does

  1. Automated judge-drift attribution. AttributionEngine uses bootstrap confidence intervals and Mann-Whitney U tests with Holm-Bonferroni correction to produce a five-state verdict: NONE, SYSTEM_DRIFT, JUDGE_DRIFT, JOINT_DRIFT, or INCONCLUSIVE. It distinguishes a quality drop in the system under test from a change in the judge measuring it, and separately flags when both shifted at once or when there wasn't enough data to tell.
  2. Fixed evaluation anchors, rescored on demand. A versioned AnchorSet is never edited in place; updating it means creating a new anchor_set_id, so historical comparisons stay meaningful.
  3. Statistically rigorous regression gating. Mann-Whitney U tests with Holm-Bonferroni correction for multiple comparisons, Cohen's d effect sizes, and bootstrap confidence intervals (5,000 resamples), not a bare threshold on a single score.
  4. Multivariate structure detection. A second regression track catches changes in how metrics vary together (variance-only and correlation-only drift) that per-metric tests cannot see, using permutation-calibrated tests. In its default mode it can raise a warning but never block a deploy on its own, because these tests are undirected; an opt-in strict mode allows blocking.
  5. Trust-weighted health scoring. compute_system_health_score discounts the aggregate score when the attribution verdict signals judge unreliability, not just system degradation.
  6. An 8-layer, unidirectional architecture, instrumentation → orchestrator → metrics → trust → storage → regression → gate → reporting, enforced by import-linter in CI, not just documented as a diagram.

What is NiriZan?

NiriZan is an open-source continuous evaluation infrastructure for production AI systems. It enables engineers and researchers to systematically measure, benchmark, validate, and monitor the quality of:

  • Retrieval-Augmented Generation (RAG) pipelines
  • AI agents
  • Large Language Model (LLM) applications
  • Custom AI workflows

Unlike orchestration frameworks that focus on building AI applications, NiriZan focuses on engineering confidence in AI systems. It provides:

Capability Description
Reproducible evaluation pipelines Consistent, repeatable test runs across environments
Benchmark execution Standardized quality benchmarking for AI systems
Regression detection Automated flagging of quality drops between versions
Experiment tracking Full history of runs, configs, and results
Quality reporting Clear, actionable reports on system performance
Deployment-aware validation Checks tuned to pre-, during-, and post-deployment stages

Vision

The long-term vision of NiriZan is to become the engineering quality layer for production AI, ensuring that every AI application can be continuously measured before, during, and after deployment.


Where the Name Comes From

NiriZan is a fusion of two words from two languages, each contributing a core idea behind the project.

Niri Zan
Origin: নিরীক্ষা (Nirikkha) - Bangla/Bengali Origin: ميزان (Mīzān) - Arabic
Meaning: Inspection · Evaluation · Verification · Audit Meaning: Scale · Balance · Measurement · Criterion

Together, Niri + Zan captures the essence of the project: inspecting AI systems and measuring them against a balanced standard of quality.


Read More

For the complete user guide, see the NiriZan User Manual.

For architecture, contracts, module reference docs, and the evaluation results behind the claims above, see docs/.

See CHANGELOG.md for release history and notable changes between versions.

The Ruler Can Change Too: Navigating Judge Drift in Production AI Evaluation, the first NiriZan engineering post, covering the judge-drift problem and the fixed-anchor, statistical-attribution approach this project takes to it.


License

Copyright 2026 Redwan Rahman

Licensed under the Apache License, Version 2.0. See LICENSE for the full text.

Releases 0.1.0 to 0.3.0 were published under GPL-3.0-or-later and remain available under those terms. Version 0.4.0 and later are licensed under Apache-2.0.


Author

Redwan Rahman github.com/Red1-Rahman

Contributors

NiriZan contributors

Metadata

Release files for nirizan 0.5.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nirizan 0.5.1
File Size Uploaded Provenance Actions
nirizan-0.5.1.tar.gz 2.7 MB With provenance Details for nirizan-0.5.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nirizan 0.5.1
File Interpreter ABI Platform Actions
nirizan-0.5.1-py3-none-any.whl Python 3 none any Details for nirizan-0.5.1-py3-none-any.whl

Total release size: 2.8 MB

Release files / nirizan-0.5.1.tar.gz

Download URL nirizan-0.5.1.tar.gz
Size 2.7 MB
Tags Source
SHA-256 checksum
How to use checksums
c75cae92faf237c5c7a5379d7d1077c24f8e3f5c47af3d1ce4f911621d24e258
BLAKE2b-256 checksum
How to use checksums
6b0d987ce2aedc6574afeb09ee957f9e50f852a7240b1028ae692e9dac83ae92
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / nirizan-0.5.1-py3-none-any.whl

Download URL nirizan-0.5.1-py3-none-any.whl
Size 112.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
01a41649e2bbad8a18cd7d3ddd1ae85144ba0be9d7fcb01601a2c1824c6d30d9
BLAKE2b-256 checksum
How to use checksums
30ffc50f021b6df19642f0189d135dd491f2061c37acf9b5e32a13a35b4e76b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Security for nirizan 0.5.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for nirizan-0.5.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page