nkama-fact-benchmark 0.1.6

Evidence-gated benchmark for testing whether AI assistants can prove what they claim.

Nkama Fact Benchmark

Evidence-gated tools for testing whether an AI assistant can prove what it claims.

The package gives you four public command-line tools:

nkama-fact-benchmark
nkama-prompt-filter
nkama-evidence-layer
nkama-truth-filter

It is designed for use through uvx:

uvx nkama-fact-benchmark --profile public
uvx nkama-fact-benchmark agent
uvx nkama-fact-benchmark agent-run "Create a Cambridge A2/B1 past perfect lesson plan." --provider claude --allow-external-model
uvx nkama-fact-benchmark start
uvx nkama-fact-benchmark prompt "Build a browser game with tests."
uvx nkama-fact-benchmark run "Build a browser game with tests." --output nkama_run_browser_game
uvx --from nkama-fact-benchmark nkama-prompt-filter "Build a browser game with tests." --output prompt_check

Before publishing or sharing a built package, audit the release files:

nkama-fact-benchmark security-audit dist/*.whl dist/*.tar.gz

What It Does

Nkama Fact Benchmark does not promise that an AI is always right. It makes AI work more testable by asking for evidence, running local checks, and marking unavailable proof as blocked instead of pretending it passed.

Typical flow:

raw prompt
  -> evidence-wrapped prompt
  -> AI answer or generated files
  -> evidence manifest / validator
  -> pass, fail, or blocked report

Agent Protocol

Use agent when an AI coding agent has terminal access and should treat Nkama Fact Benchmark as its working protocol:

uvx nkama-fact-benchmark agent

With no task, it prints the protocol an AI agent should follow. With a task, it prepares the evidence workspace and writes AGENT_PROTOCOL.md:

uvx nkama-fact-benchmark agent "Create a Cambridge A2/B1 past perfect lesson plan." --output nkama_agent_lesson

The AI agent should read AGENT_PROTOCOL.md and evidence_prompt.md, build in ai_output/, update ai_output/evidence_manifest.json, run nkama-evidence-layer, and report pass/fail/blocked honestly.

Agent Run

Use agent-run when you want Nkama Fact Benchmark to call an external model through a local provider CLI and capture the answer:

uvx nkama-fact-benchmark agent-run "Create a Cambridge A2/B1 past perfect lesson plan." --provider claude --allow-external-model --output nkama_agent_lesson

The first public provider is claude, using the local Claude CLI. External model calls are blocked unless you pass --allow-external-model. By default Claude receives no tools; this public mode is text-only. It asks the model to produce a final report with Answer, Evidence, Limitations, Files changed or created, and Tests or checks run sections.

The package captures the text answer, writes it to ai_output/ANSWER.md, writes MODEL_RUN_REPORT.json, and verifies ai_output/evidence_manifest.json. If the provider is not logged in, asks for unavailable tools, omits the required report sections, or otherwise fails the contract, the run is marked blocked or fail instead of being treated as verified.

If you do not pass --allow-external-model, the workspace is still created, but the model run is marked blocked instead of pretending it happened.

Start For Normal Users

Use start when you want the tool to ask for your prompt:

uvx nkama-fact-benchmark start

It asks what you want the AI to build, answer, or verify. Then it creates a run folder containing the AI-ready evidence prompt, starter output folder, evidence manifest, and verification instructions.

You can also pass the prompt directly:

uvx nkama-fact-benchmark start "Build a browser game with tests." --output nkama_run_browser_game

Run Folder

Use the run command when you want a complete folder for one AI task:

uvx nkama-fact-benchmark run "Build a browser game with tests." --output nkama_run_browser_game

This writes:

nkama_run_browser_game/
  original_prompt.md
  evidence_prompt.md
  prompt_analysis.json
  run_contract.json
  README.md
  ai_output/
    ANSWER.md
    evidence_manifest.json

Paste evidence_prompt.md into the AI assistant, put the generated files in ai_output/, update ai_output/evidence_manifest.json, then verify:

uvx --from nkama-fact-benchmark nkama-evidence-layer nkama_run_browser_game/ai_output/evidence_manifest.json

Prompt Filter

Use the prompt filter before sending a task to an AI:

uvx --from nkama-fact-benchmark nkama-prompt-filter "Build a browser game with tests." --output prompt_check

This writes:

prompt_check/
  original_prompt.md
  evidence_prompt.md
  prompt_analysis.json
  README.md

Paste evidence_prompt.md into your AI assistant.

Python Library

from nkama_fact_benchmark.prompt_filter import analyze_prompt, wrap_prompt, write_prompt_package

prompt = "Build a browser game with tests."
analysis = analyze_prompt(prompt)
evidence_prompt = wrap_prompt(prompt)
write_prompt_package(prompt=prompt, output_dir="prompt_check")

Evidence Layer

If an AI generates files, ask it to include an evidence_manifest.json, then verify it:

uvx --from nkama-fact-benchmark nkama-evidence-layer path/to/evidence_manifest.json
uvx --from nkama-fact-benchmark nkama-evidence-layer path/to/evidence_manifest.json --allow-commands

Command checks are disabled unless you explicitly pass --allow-commands.

Truth Filter

Use the truth filter to compare multiple AI submissions against the same task:

uvx --from nkama-fact-benchmark nkama-truth-filter init "Browser Game Comparison"
uvx --from nkama-fact-benchmark nkama-truth-filter run browser-game-comparison

Public Safety Defaults

The public profile is designed to be portable:

• no private documents are read by default
• no external model calls are made by default
• no shell commands run unless explicitly allowed
• blocked evidence is not counted as success
• reports are written as JSON and Markdown
• release artifacts can be audited for private paths, internal package names, unexpected commands, and dependencies

Private/local profiles can be used for a specific developer's own machine, but those checks are opt-in.

Status

This package is alpha software. It is useful for evidence-gated AI workflows, prompt testing, and local verification experiments. It is not a guarantee of truth, correctness, safety, legal validity, or production readiness.

License: Apache-2.0.