manifold-guard 0.1.3

ManifoldGuard reference-bounded output regulator utilities

ManifoldGuard

Rebrand note: ManifoldGuard is the public package/product name for the project previously published as mbt-ai-tools. The mbt_ai_tools import path and mbt-check / mbt-eval CLI aliases remain available for compatibility.

ManifoldGuard tests whether AI candidate outputs remain inside a supplied semantic and relational reference manifold.

It runs at inference time:

• no training
• no fine-tuning
• no model-weight inspection
• no hidden classifier

The regulator checks candidate outputs and either emits the safest supported candidate or blocks when every candidate is unsafe.

30-Second Quickstart

Install ManifoldGuard from PyPI:

python -m pip install manifold-guard

Run an offline reference-bounded check:

manifold-check \
  --reference "The capital of France is Paris." \
  --candidate "The capital of France is London." \
  --candidate "The capital of France is Paris." \
  --no-embeddings

Expected result:

EMIT | The capital of France is Paris. | score=0.0000

Python API:

from mbt_ai_tools import regulate_candidates

result = regulate_candidates(
    [
        "The capital of France is London.",
        "The capital of France is Paris.",
    ],
    ["The capital of France is Paris."],
    use_embeddings=False,
)

print(result.action)
print(result.emitted_text)

ManifoldGuard is useful when you already have trusted reference statements and want to reject outputs that drift away from them. For a fuller onboarding path, see docs/getting_started.md. For a starter personal corpus, see examples/personal_corpus_template.jsonl. For extension guidance, see docs/extending.md.

Core Claim

ManifoldGuard treats hallucination as semantic or relational drift from supplied reference structure. It is not a fact oracle and does not claim direct access to external truth.

Universe does facts.
Humans describe the universe.
AI describes human descriptions.
ManifoldGuard regulates AI descriptions against supplied human reference structure.

Supported public claim:

ManifoldGuard v11 blocked hallucinated AI outputs against supplied reference manifolds
and relation constraints. In the frozen EXP20 ledger, it achieved confusion
[[97, 0], [0, 160]] over 257 labelled candidates across 53 cases.

Claims and Scope

See CLAIMS.md for the claim register. The short version: ManifoldGuard regulates outputs against supplied references and the public offline corpus; it is not claimed to be a universal fact checker.

Continuous integration for the offline corpus lives in .github/workflows/tests.yml.

Current Locked Result

Frozen ledger:

MBT5_EXP20_combined_guarded_master_ledger_v11

Candidates: 257
Cases:      53

Candidate-level confusion:
[[TN, FP], [FN, TP]]
[[97, 0], [0, 160]]

Accuracy:   1.0000
Precision:  1.0000
Recall:     1.0000
F1:         1.0000

Case-level:
Correct: 53 / 53
Emitted: 28
Blocked: 25

The current public claim is limited to the supplied test suites and reference manifolds included in the project.

Known Limitations

• ManifoldGuard is not a universal fact checker.
• It only regulates against the references you provide.
• If the references are wrong, incomplete, or ambiguous, outputs are judged against that flawed reference structure.
• Offline mode emphasizes literal, relation, negation, numeric, and unit drift; embedding-backed semantic geometry is optional.
• Public performance claims are limited to the frozen supplied corpus and documented experiment lineage.
• Token-shock diagnostics require embedding dependencies and are not available in strict --no-embeddings mode.

What ManifoldGuard Checks

ManifoldGuard combines:

• semantic geometry
• internal consistency scoring
• token-level shock analysis
• literal drift guards
• entity, number, and unit protection
• overclaim detection
• copular relation checks
• non-copular relation checks
• relation polarity checks
• unsupported negation clamps
• abstention when every candidate is unsafe

Examples of blocked drift:

The capital of France is London.
The Sun is a planet.
Earth is flat.
Water boils at 90 degrees Celsius at sea level.
Gravity is fully solved by modern physics.
Scientific descriptions do not use measurements.
DNA contains the nucleus.
The Sun orbits Earth.

Core Mechanisms

Semantic Shock

Candidate outputs are embedded into semantic space. ManifoldGuard measures distance from the reference manifold:

shock = Gamma * ||candidate_embedding - reference_center||^2

Higher shock means stronger semantic drift.

Literal Drift Guards

Geometry alone can miss small but important substitutions. ManifoldGuard protects numbers, units, named entities, and key content tokens.

Relation Clamps

ManifoldGuard checks relation structure, not just semantic similarity.

Copular examples:

The Sun is a planet.
A dog is a bird.
Rome is the capital city of France.

Non-copular examples:

Earth orbits the Moon.
The Sun orbits Earth.
DNA contains the nucleus.
Heat produces friction.
Photosynthesis converts oxygen into carbon dioxide and water.

Negation Clamp

ManifoldGuard blocks unsupported negations of positive reference support.

Water is not liquid at room temperature.
Sound does not need a material medium to travel.
Scientific descriptions do not use measurements or predictions.
General relativity proves gravity has no connection to mass or energy.

Abstention

When every candidate is unsafe, ManifoldGuard blocks instead of emitting the least-bad candidate.

Installation

Install modes:

Public PyPI install:

• Offline/core mode: python -m pip install manifold-guard
• Optional semantic mode: python -m pip install "manifold-guard[embeddings]"

Source checkout install:

• Offline baseline (default): python -m pip install -e . --no-deps
• Optional semantic mode: python -m pip install -e .[embeddings]

If you need a plain editable install for local experimentation, use pip install -e . or python -m pip install -e .; both remain supported for compatibility.

The optional extra currently installs sentence-transformers>=2.6.0,<3 for model-backed operation.

If sentence-transformers is unavailable, use offline literal/relation-only regulation with --no-embeddings / use_embeddings=False.

from mbt_ai_tools import evaluate_candidate

evaluate_candidate(
    "Paris is the capital city of France.",
    ["The capital of France is Paris."],
    use_embeddings=False,
)

When embedding-backed operation is requested without sentence-transformers, you'll now get a direct error directing to install the dependency or use offline mode.

Python Usage

from mbt_ai_tools import regulate_candidates

references = [
    "The capital of France is Paris.",
    "Paris is the capital city of France.",
]
candidates = [
    "The capital of France is London.",
    "The capital of France is Paris.",
]

result = regulate_candidates(candidates, references, use_embeddings=False)
print(result.action)        # emit
print(result.emitted_text)  # The capital of France is Paris.

CLI Usage

Check the installed CLI version:

manifold-check --version

manifold-check \
  --reference "The capital of France is Paris." \
  --candidate "The capital of France is London." \
  --candidate "The capital of France is Paris." \
  --no-embeddings

Expected output:

EMIT | The capital of France is Paris. | score=0.0000
[0] blocked | ...
[1] safe | ...

JSON report output:

manifold-check \
  --reference "The capital of France is Paris." \
  --candidate "The capital of France is London." \
  --candidate "The capital of France is Paris." \
  --no-embeddings \
  --format json

See examples/cli_json_report.md for a complete offline JSON report demo.

Decision explanations:

manifold-check \
  --reference "The capital of France is Paris." \
  --candidate "The capital of France is London." \
  --candidate "The capital of France is Paris." \
  --no-embeddings \
  --explain

--explain adds per-candidate summaries and guard reasons to text, JSON, and Markdown regulation reports without changing the default report shape. See examples/explain_report.md for a complete offline explanation example.

Optional token-level shock details can be included in regulation reports when embedding dependencies are installed:

manifold-check \
  --reference "The capital of France is Paris." \
  --candidate "The capital of France is London." \
  --format json \
  --token-shock \
  --token-shock-top-k 5

Token-level shock is embedding-backed, so keep --no-embeddings off when using --token-shock.

Batch JSONL evaluation:

manifold-check \
  --input-jsonl examples/batch_input.jsonl \
  --no-embeddings \
  --output batch-report.jsonl

CI guard mode:

manifold-check \
  --input-jsonl examples/batch_input.jsonl \
  --no-embeddings \
  --summary \
  --fail-on-block

--fail-on-block exits with status 2 when a single regulation run blocks or any batch row blocks. --summary appends a final batch summary JSON object.

Markdown audit report:

manifold-check \
  --input-jsonl examples/batch_input.jsonl \
  --no-embeddings \
  --format markdown \
  --output audit.md

See examples/markdown_audit_report.md for a complete Markdown audit demo.

CSV audit export:

manifold-check \
  --input-jsonl examples/batch_input.jsonl \
  --no-embeddings \
  --format csv \
  --output audit.csv

See examples/csv_audit_report.csv for a spreadsheet-friendly batch audit demo.

The JSON/Markdown/CSV report schema is documented in docs/report_schema.md. The release support contract is captured in docs/product_readiness_manifest.json. Release quality expectations are captured in docs/quality_gates.md and docs/release_checklist.md. The maintainer release flow is documented in RELEASE_PROCESS.md.

Local Validation

For schema-driven sanity checks before publishing docs or changing report contracts:

python -m pip install jsonschema
python scripts/validate_reports.py \
  --schema docs/report_schema.json \
  examples/single_report_example.json \
  examples/batch_report_example.jsonl

Run the full local docs-quality check (includes manifest and example consistency):

python scripts/docs_quality.py

Run the full preflight (docs-quality + pytest) before review:

python scripts/preflight.py

Run only docs checks:

python scripts/preflight.py --docs-only

Run only tests:

python scripts/preflight.py --tests-only

Run the frozen offline regression corpus evaluation:

manifold-eval
manifold-eval --output regulator-evaluation.json
manifold-eval --list-families
manifold-eval --family unsupported_negation
manifold-eval --case-id unsupported_negation_water
manifold-eval --failures-only --format json --output failing-cases.json
python scripts/evaluate_regulator.py
python scripts/evaluate_regulator.py --output regulator-evaluation.json
python scripts/build_eval_report.py --input regulator-evaluation.json --output docs/evaluation_report.md

manifold-eval uses the packaged regression corpus by default and accepts --corpus path/to/corpus.jsonl for custom offline checks. Use --family or --case-id to narrow a regression pass, --failures-only to export only failing case details, and --list-families to inspect available taxonomy groups. The generated benchmark report lives at docs/evaluation_report.md. Benchmark/evidence tiers and claim boundaries are documented in docs/benchmark.md.

Compare two saved evaluator JSON reports:

python scripts/compare_eval_reports.py \
  --before baseline-evaluation.json \
  --after candidate-evaluation.json \
  --output evaluation-diff.json

Build a compact replay pack for failed or selected evaluator cases:

python scripts/build_eval_replay_pack.py \
  --evaluation regulator-evaluation.json \
  --corpus examples/regression_corpus.jsonl \
  --output eval-replay.md

Package Build and Publish

Package distribution artifacts are built by .github/workflows/package-publish.yml on version tags and manual runs. The build job also installs the built wheel in a clean virtual environment and runs scripts/install_smoke.py before artifacts are uploaded or published. Publishing is manual-only: run the workflow with publish=true and choose testpypi or pypi after configuring PyPI Trusted Publishing environments named testpypi and pypi.

The exact Trusted Publishing setup values are documented in docs/package_publishing.md. Package-index install commands and smoke checks are documented in docs/package_installation.md. The v0.1.2 post-release verification note is recorded in docs/v0.1.2_release_verification.md.

Recommended release order:

python -m build
python -m twine check dist/*

Use TestPyPI first for release-candidate publishing. Do not publish to PyPI until tag CI, release evidence, and the regulator evaluation report are green.

Run the canonical release check sequence:

python scripts/release_check.py --output release-evidence.json

Release Evidence Snapshot

Use this command block before release candidates or public claims:

python -m pytest -q
python scripts/docs_quality.py
python scripts/evaluate_regulator.py
python scripts/preflight.py
python scripts/preflight.py --docs-only
python scripts/release_check.py --output release-evidence.json
python -m pytest -q tests/test_docs_quality.py

Expected evidence in a healthy release path:

• A full pytest success summary (all tests passed, no failures).
• docs-quality validator exits successfully.
• regulator evaluation reports all frozen corpus cases passing, with taxonomy metrics in JSON output.
• preflight prints Preflight completed successfully. for both full and docs-only modes.
• no schema or URL drift errors.

Regression Corpus

The lightweight public regression corpus lives in examples/regression_corpus.jsonl. It currently contains 229 offline cases covering entity swaps, multi-word capital handling, all-bad abstention, numeric drift, unit drift, role swaps, shared-subject relation repair, unsupported negation, historical-date drift, supported paraphrase, relation-direction boundaries, negation boundaries, and overclaim blocking.

Regenerate the corpus:

uv run python examples/build_regression_corpus.py

An exploratory EXP21 challenge seed lives in examples/challenge_corpus.jsonl. It is separate from the frozen release/regression evidence and is intended for harder development probes, not current public pass-rate claims. The EXP21 workflow is documented in docs/exp21_challenge.md.

The next post-release challenge seed lives in examples/exp22_challenge_corpus.jsonl. EXP22 is broader and intentionally rougher than EXP21, with guidance in docs/exp22_challenge.md.

uv run python examples/build_challenge_corpus.py
manifold-eval --corpus examples/challenge_corpus.jsonl --output challenge-evaluation.json
python scripts/build_eval_replay_pack.py \
  --evaluation challenge-evaluation.json \
  --corpus examples/challenge_corpus.jsonl \
  --output challenge-replay.md

Run the tests:

uv run --with pytest python -m pytest -q

Experiment Lineage

The full EXP01-EXP20 record is in MBT5_EXP01_EXP20_TECHNICAL_LEDGER.md. The expanded CSV experiment exports live in data/csv_exports/.

Key frozen output artifacts:

data/csv_exports/mbt5_exp20_master_candidate_ledger.csv
data/csv_exports/mbt5_exp20_master_case_ledger.csv
data/csv_exports/mbt5_exp20_summary_metrics.csv
data/csv_exports/mbt5_exp20_case_summary.csv
data/csv_exports/mbt5_exp20_clamp_counts.csv
data/csv_exports/mbt5_exp20_failure_table.csv
data/csv_exports/mbt5_exp20_patch_lineage.csv

A docs-quality workflow also validates manifest JSON and referenced docs/examples so support artifacts stay consistent.

Project Layout

mbt_ai_tools/
  mbt/
    embeddings.py      SentenceTransformer loader
    geometry.py        geometric median, shock, distance
    stability.py       self-consistency / entropy scoring
    tokens.py          leave-one-out token shock
    consensus.py       multi-agent / council logic
    regulator.py       v11 candidate regulator
  data/
    regression_corpus.jsonl
  cli.py               manifold-check command
  eval.py              manifold-eval offline frozen corpus evaluator
.github/workflows/
  tests.yml            GitHub Actions offline regression test workflow
  docs-quality.yml     docs and manifest quality workflow
  package-publish.yml  package build and manual publish workflow
CHANGELOG.md          release notes
CLAIMS.md             scoped public claims register
RELEASE_PROCESS.md    maintainer release flow
data/csv_exports/     expanded EXP01-EXP20 CSV exports
scripts/
  docs_quality.py
  build_eval_report.py
  evaluate_regulator.py
  release_evidence.py
  release_readiness.py
  release_check.py
  install_smoke.py
  preflight.py
  validate_reports.py
docs/
  product_readiness_manifest.json
  report_schema.json
  quality_gates.md
  release_checklist.md
  report_schema.md
examples/
  batch_input.jsonl
  build_regression_corpus.py
  cli_json_report.md
  csv_audit_report.csv
  markdown_audit_report.md
  batch_report_example.jsonl
  single_report_example.json
  regression_corpus.jsonl
tests/
  test_regulator.py
  test_regression_corpus.py
REPLICATION.md        local/GitHub/Colab replication instructions
pyproject.toml
README.md
LICENSE

License

See LICENSE.