xbse 0.1.0

Modular framework for domain-specific BERT Sentence Embeddings (LaBSE/LeBSE/MoBSE): shared encoder + contrastive objective + one validation gate, per-domain PairSource plugins

xbse — modular domain-specific sentence embeddings, validated by one shared gate

Abstract

xbse trains and — more importantly — validates small, specialized text encoders, one per moral dimension (physical harm, fairness, privacy, honesty, autonomy, …). Each encoder turns a sentence into a signed score for its dimension: how much does this text uphold or violate this value? These encoders are the perception layer of a larger AI-ethics engine (DEME / erisml): every number that engine reasons over has to come from somewhere, and xbse is where it comes from.

The distinctive contribution is not the encoders but the discipline around them. One shared, adversarial validation gate every encoder must clear; pass/fail bars derived from the data and pre-registered before results are seen; a hard rule that an unvalidated encoder cannot be used downstream; and validation provenance carried into the ethics engine's cryptographic audit trail. On a held-out cross-dataset test — the honest measure of whether an encoder learned a transferable moral concept instead of one dataset's vocabulary — 8 of 9 dimensions clear their pre-registered bar, beating both an untrained baseline and a bag-of-words control by wide margins (+0.16 to +0.33 AUROC). The 9th (rights_respect) fails, and the framework reports it as failed.

What this is, in plain terms

AI systems that make or assist moral decisions must read a situation and notice things like "this harms someone," "this is unfair," "this exposes private information." A sentence-embedding encoder is a model that turns text into a list of numbers (a vector) capturing its meaning, so that texts meaning similar things get similar vectors. xbse builds one such encoder per moral concept, tuned so the vector captures that concept's valence — is the value upheld or violated? — while ignoring surface details like wording or topic.

The hard part isn't building an encoder; it's knowing whether to trust it. An encoder can look excellent on its training data yet be worthless on anything new, because it memorized that dataset's vocabulary rather than the moral idea. xbse's core contribution is a validation regime built to catch exactly that: it tests every encoder on a second, independent dataset; it compares the encoder against a deliberately "dumb" bag-of-words model to prove it learned meaning, not just words; and it refuses to use any encoder until it clears a pass mark written down before the results were seen. When an encoder fails, the framework says so. Because this is the perception layer of a safety-critical ethics engine, that honesty is the whole point.

For practitioners

*-BSE models (LaBSE, LeBSE, MoBSE, …) are the same architecture trained to be invariant to one axis and sensitive to another. xbse factors out everything they share and leaves each domain as a small plugin — a new *-BSE is a PairSource + a config, not a new repo — and every instance clears the same validation gate, so no encoder can grade itself easier than another.

Each moral *-BSE is the encoder for one DEME MoralVector dimension; the dimension↔feeder map lives in erisml-lib/docs/moralvector_reference.md.

The idea in one table

model	invariance (→ positives)	sensitivity (→ negatives)
LaBSE	language (translation pairs)	meaning
LeBSE	surface / citation form	legal holding
MoralVector feeders	wording / framing / topic / which corpus	one moral dimension's valence

Only the PairSource (what is invariant vs sensitive), an optional domain adversary, and the validation labels differ. The encoder, contrastive objective, and gate are written once.

Architecture

flowchart LR
    PS["PairSource plugin<br/>(invariance → +, sensitivity → −)"] --> ENC["BSEEncoder<br/>BGE-M3 · mean-pool · L2-norm"]
    ENC --> OBJ["InfoNCE<br/>contrastive objective"]
    OBJ --> GATE{"Shared gate<br/>structure-AUROC + fuzz-ratio"}
    GATE -->|pass| OK["validated encoder<br/>earns encode / distance / probe"]
    GATE -->|fail| STOP["HARD STOP<br/>no tools, no claims — iterate"]
    classDef s fill:#e3f2fd,stroke:#1565c0
    classDef o fill:#c8e6c9,stroke:#2e7d32
    classDef x fill:#ffcdd2,stroke:#c62828
    class PS,ENC,OBJ s
    class OK o
    class STOP x

xbse/
  encoder.py        BSEEncoder: base transformer → pooling → projection → L2-norm   (shared)
  pairs.py          PairSource ABC yielding (anchor, positive, negative)            (shared iface)
  objective.py      InfoNCE contrastive loss                                        (shared)
  adversarial.py    gradient-reversal domain head + polar severity readout          (shared)
  validate.py       THE GATE: structure-vs-surface AUROC + fuzz ratio               (shared)
  train.py          single-corpus training loop                                     (shared)
  train_adv.py      joint cross-dataset + domain-adversarial loop                   (shared)
  instances/
    joint.py          JointPairSource — train one dimension on ≥2 corpora at once
    joint_builders.py per-dimension corpus assembly (privacy, care, rights, …)
    mobse.py rights.py physharm.py epistemic.py socenv.py privacybse.py darkpattern.py …
  experiments/
    data_sourcing_plan.md    ≥2 independent corpora per dimension (+ legal-evidence route, APIs)
    rank_test.py             empirical moral-dimension rank test (bifactor result)

Why cross-dataset validation — the correction that reshaped this project

Single-corpus contrastive fine-tuning of BGE-M3 learns a corpus's surface, because that is the easiest way to separate its pairs. A feeder can score a beautiful within-dataset AUROC yet have learned nothing transferable. We proved this the hard way: every moral feeder scored 0.75–0.955 within its own corpus but collapsed to ~0.47–0.55 (random) on a second, independent corpus of the same dimension. The within-dataset numbers were artifacts.

The fix — joint cross-dataset training (JointPairSource): train each dimension on ≥2 corpora at once, drawing every same-sign positive from a different corpus. To pull an A-text next to a same-valence B-text the encoder cannot use A's surface — only the shared moral structure. Held-out AUROC is then cross-dataset by construction. An optional gradient-reversal domain head (adversarial.py) adds explicit corpus-invariance.

flowchart TB
    A["Corpus A<br/>(e.g. privacy RoTs)"] --> J["JointPairSource<br/>same-sign positives ACROSS corpora"]
    B["Corpus B<br/>(e.g. AITA scenarios)"] --> J
    J --> ENC["shared BGE-M3 encoder"]
    ENC --> GRL["gradient-reversal<br/>domain head (optional)"]
    ENC --> XG["cross-dataset held-out gate<br/>= the honest number"]
    classDef s fill:#e3f2fd,stroke:#1565c0
    class A,B,J,ENC,GRL,XG s

Status — honest cross-dataset scorecard

Cross-dataset held-out AUROC (evaluate on a second corpus). BoW is a TF-IDF bag-of-words model on the same held-out pairs — the adversarial lexical control. margin = encoder − BoW:

dimension	corpora (2 independent)	baseline	cross-dataset	BoW (lexical)	margin
privacy_protection	privacy RoTs + AITA scenarios	0.55	0.853	0.54	+0.31
epistemic_quality	Social-Chem honesty + ETHICS	0.48	0.817	0.53	+0.29
societal_environmental	ClimateBERT + dual-judged env-claims	0.43	0.817	0.48	+0.33
virtue_care	Social-Chem care-harm + ETHICS	0.47	0.811	0.53	+0.28
fairness_equity	Social-Chem fairness + ETHICS	0.47	0.789	0.51	+0.28
autonomy_respect	ec-darkpattern + MentalManip	0.52	0.747	0.53	+0.22
legitimacy_trust	Social-Chem authority + ETHICS	0.52	0.708	0.53	+0.18
physical_harm	BeaverTails + ETHICS-harm	0.50	0.622	0.46	+0.16
rights_respect	ECHR + ETHICS-justice	0.52	0.475 ✗	0.49	−0.01

The adversarial control holds: bag-of-words is near-random (0.46–0.54) on every dimension, so the encoder's signal is provably non-lexical — it is not learning valence-correlated vocabulary. This is self-enforcing: JointPairSource pairs an anchor from corpus A with a same-sign positive from corpus B, which share little surface, so BoW is helpless while the encoder must find the shared moral structure. The 8 rehabilitated dimensions beat BoW by +0.16 to +0.33; rights (−0.01, below BoW) is flagged as the honest failure. bow_auroc and lexical_margin are standing gate metrics.

Eight of nine dimensions rehabilitated from artifact to real (0.62–0.85). rights_respect (0.475, below baseline) is the honest exception: a corpus-choice failure, not a method failure — ECHR European case-facts and everyday ETHICS-justice scenarios share no transferable structure. The fix is a better second corpus (US civil-rights via CourtListener), not a different method. Lessons banked: (i) within-dataset AUROC is not evidence of a real dimension — always cross-test; (ii) the load-bearing fix is cross-corpus same-sign positives, not the adversary (which only de-confounds when the two corpora are surface-similar); (iii) physical_harm (0.622) is weakest — the widest genre gap (QA-pairs vs scenarios). Full roadmap: experiments/data_sourcing_plan.md.

Pass/fail is against a per-dimension, pre-registered Bar (xbse.bar), not a universal 0.97. A feeder is VALIDATED iff its cross-dataset AUROC beats BOTH nulls — the untrained-encoder baseline AND the bag-of-words control — by ≥ 0.10 on the same held-out pairs (policy = baseline_relative). This tests the exact claim a cross-dataset feeder makes, and is immune to a subtlety that sank the alternative we tried first (a label-noise ceiling): the ceiling is a within-corpus quantity, but the AUROC is a harder cross-corpus measurement, so requiring "90% of the ceiling" unfairly penalizes cross-corpus difficulty the ceiling never modeled. The bar, its two nulls, its margin, and its date were committed before the re-gate — git history is the pre-registration record; a bar may be tightened, never loosened. Under this policy 8 of 9 dimensions validate (rights_respect fails). The bar provenance (source, derivation, registered) plus the AUROC, BoW margin, and PASS/FAIL travel into every Report and are bound into the ethics engine's hash-chained audit artifact (erisml.ethics.decision_proof.FeederValidationRecord) — a scored dimension can never silently rest on an unvalidated encoder.

The non-negotiable discipline (read before adding code)

Built in falsification order with hard stops (see docs/MOBSE_PLAN.md):

1. Build an instance (a PairSource).
2. Validate on a second, independent dataset against the instance's pre-registered Bar — beat the untrained baseline and the bag-of-words control by the registered margin. HARD STOP: fail the gate → no downstream tools, no claims — iterate.
3. Only a validated embedding earns encode / distance / probe — enforced by require_pass.
4. Any geometric tool on top is a pre-registered, falsifiable hypothesis about the geometry.

Language discipline: operations are named for what they measure, not for retired claims — a harm operation computes a representation-invariant harm ledger, never a "conserved quantity."

Quickstart

pip install -e ".[dev]"          # torch (CPU wheel is fine for tests) + package
pytest -q                        # offline gate/objective/circularity tests
ruff check src tests

from xbse.encoder import BSEEncoder
from xbse.instances.joint_builders import build_privacy_joint
from xbse.train_adv import train_adversarial

src = build_privacy_joint()                 # ≥2 corpora, cross-dataset positives
enc = BSEEncoder(base_model="BAAI/bge-m3", max_len=src.max_len, device="cuda")
report = train_adversarial(enc, src, max_steps=1400)   # trains, then RUNS THE GATE
print(report.metrics["structure_auroc"])    # cross-dataset held-out AUROC — the honest number

Training a *-BSE and reporting its gate number are one operation — a run that doesn't end in gate output is not a finished run.

Testing & CI

pytest runs offline (no model download / network): the gate math, the InfoNCE objective, and the circularity guard (assert_heldout_disjoint fails loud if any held-out eval text leaked into training). CI (.github/workflows/ci.yml) runs ruff + pytest on Python 3.10–3.12.

Kinship (not merged, on purpose)

*-BSE learns invariance — it quotients a nuisance group out. Its equivariant cousins (latte, arc-equivariant-search) covary with a symmetry group. Same principle — a transformation group acts; you either mod it out (here) or transform with it (there) — different codebase.

License

MIT © Andrew H. Bond (SJSU).