bh-sentinel-ml

Transformer-based clinical safety signal detection for behavioral health systems.

bh-sentinel-ml is the Layer 2 add-on for bh-sentinel-core. It runs ONNX-Runtime zero-shot NLI inference over clinical sentences to catch signals that deterministic pattern matching misses: implied distress, indirect language, contextual meaning.

The core package always runs without bh-sentinel-ml installed. L2 is opt-in.

Installation

pip install bh-sentinel-ml

This pulls in bh-sentinel-core>=0.1.1, onnxruntime, tokenizers, huggingface-hub, and platformdirs.

Optional extras:

pip install "bh-sentinel-ml[eval]"   # adds numpy + scikit-learn for calibrate/evaluate CLIs

Compatibility

bh-sentinel-ml 0.2.0 depends on the Pipeline(transformer_model_path=..., transformer_auto_download=...) kwargs that were added in bh-sentinel-core 0.1.1. Pairing it with bh-sentinel-core 0.1.0 will break at import/construction time.

Enforcement:

• Install time: pip install bh-sentinel-ml resolves bh-sentinel-core>=0.1.1,<1 from the wheel metadata. This is the primary guard.
• Import time: import bh_sentinel.ml verifies the installed bh-sentinel-core version via importlib.metadata and raises ImportError with an actionable upgrade message if it's too old. This catches the --no-deps, vendored, and editable-monorepo cases that bypass the pip resolver.

Quick Start

from bh_sentinel.core import Pipeline, AnalysisConfig

pipeline = Pipeline(enable_transformer=True)  # auto-downloads the pinned model on first run
result = pipeline.analyze_sync("I just can't see the point anymore.")

for flag in result.flags:
    print(flag.flag_id, flag.severity, flag.confidence, flag.corroborating_layers)

Model distribution

bh-sentinel-ml uses a hybrid distribution strategy. The ~390MB INT8 ONNX model is not bundled in the wheel.

v0.2.2 ships the canonical pinned artifact: a quantized ONNX export of FacebookAI/roberta-large-mnli hosted at bh-healthcare/roberta-large-mnli-int8-onnx on HF Hub. model_revision and model_sha256 in config/ml/ml_config.yaml are real values pinned to the v0.2.2 release. Production auto_download=True works end-to-end — the verify-on-load SHA check passes against the published artifact. See docs/ml-artifact-provenance.md for the full source / license / verification chain (including why the original distilbart and bart-large candidates were rejected), and scripts/export_onnx.py for the re-export workflow.

Upgrade note for v0.2.1 users: the v0.2.1 release shipped a non-functional L2 path — the ONNX had static input axes that crashed inference, and L2 fell back to graceful-degradation FAILED status for every request. v0.2.2 ships a re-exported artifact from a different source (RoBERTa-large-MNLI, an encoder-only architecture that quantizes cleanly under INT8) with verified end-to-end discrimination. v0.2.1 has been yanked from PyPI. pip install --upgrade bh-sentinel-ml lands on v0.2.2.

v0.2.3 adds the score_flags() / FlagScore Layer 2 diagnostic API — per-flag raw entailment + calibrated scores (including sub-threshold flags that classify() drops) for threshold and precision/recall analysis. No change to production emission behavior, and no bh-sentinel-core bump required.

Dev / CI (unrestricted network): pip install bh-sentinel-ml → first analyze() call fetches the pinned HuggingFace revision into a local cache directory. One-time ~30s, zero config.

Production / VPC-isolated / Lambda: Pre-bake the model into your container image at docker build time. Lambda cold starts must never hit HuggingFace Hub.

FROM python:3.12-slim
RUN pip install bh-sentinel-ml
RUN bh-sentinel-ml download-model \
      --revision <PINNED_SHA> \
      --output /opt/bh-sentinel-ml/model \
      --verify-sha256 <PINNED_ONNX_SHA256>
ENV BH_SENTINEL_ML_OFFLINE=1

At runtime the pipeline reads the baked-in model:

from pathlib import Path
from bh_sentinel.core import Pipeline

pipeline = Pipeline(
    enable_transformer=True,
    transformer_model_path=Path("/opt/bh-sentinel-ml/model"),
    transformer_auto_download=False,
)

Production safety rails

• BH_SENTINEL_ML_OFFLINE=1 -- set once in the Dockerfile. When set, auto_download=True is forced to False; huggingface_hub is never even imported. Any accidental future code change that tries to download over the network will fail immediately with a static PHI-safe error.
• Verify-on-load SHA256. TransformerClassifier computes the SHA256 of the ONNX file at pipeline construction and compares it to the pinned digest in ml_config.yaml. Mismatch raises ModelIntegrityError before any InferenceSession is created -- a stale or tampered container bake fails fast, not silently.
• Graceful L2 failure. If the model is missing, the SHA mismatches, or inference throws, the pipeline still returns a 200-shaped response with L1+L3+L4 flags and PipelineStatus.layer_2_transformer == FAILED. No exception ever propagates.

Calibration (Phase A)

Architecture §4.8 prescribes FixedDiscount(0.85) for v0.2 -- raw softmax probabilities multiplied by a conservative factor. This is the default in ml_config.yaml.

TemperatureScaling is fully implemented and wired into the calibrate CLI, but it is not validated against clinical data in v0.2. ECE numbers produced today reflect the fixture data, not clinical reality; treat them as mechanism tests, not calibration claims. Real calibration ships in v0.3 once clinical labels are available per the roadmap.

bh-sentinel-ml calibrate --labels labels.jsonl --out calibration.json

Evaluation

Run the pipeline against a fixture file and get a per-entry report (human-readable, matches the style of core's bh-sentinel test-patterns).

bh-sentinel-ml evaluate --fixtures my_fixtures.yaml
bh-sentinel-ml evaluate --corpus config/eval/real_world_corpus.yaml --enable-transformer

The shared real-world corpus at config/eval/real_world_corpus.yaml (public-domain literature + synthetic clinical vignettes + true negatives) is what the L1 vs L2 diagnostic runs against.

Clinical Use Notice

This is clinical decision support software. It is not a diagnostic tool, not a substitute for clinical judgment, and not FDA-cleared or approved. Organizations deploying this software in clinical settings are responsible for their own clinical validation, regulatory compliance, and patient safety protocols. See CLINICAL_DISCLAIMER.md in the main repository.

Documentation

See docs/architecture.md for the full Layer 2 design, docs/release-process.md for release mechanics, and the main repository for everything else.

License

Apache License 2.0.