HaSPI

haspi scores German comments for hate speech and moderation removal, and tells you which words drove each verdict — an explainable reward built on the One Million Posts Corpus (DerStandard, OFAI). Rooted in inverse soft-Q learning (IQ-Learn); JAX / flax-nnx with an optional PyTorch encoder.

$ haspi-explain
> Du bist ein widerlicher Idiot und gehörst abgeschoben.
  → HATE   (score +1.06, threshold -0.02)
     toward HATE:     'bist'+0.39  'widerlicher'+0.22  'gehörst'+0.22  'Du'+0.17
     toward non-hate: 'abgeschoben.'-0.10
> Danke für den sehr informativen Artikel.
  → non-hate   (score -0.35, threshold -0.02)
     toward HATE:     'Danke'+0.15  'informativen'+0.11
     toward non-hate: 'den'-0.10  'für'-0.08  'sehr'-0.05

The current method — the sequence-χ² reward — runs a comment through a frozen German language model (LeoLM-7b), mean-pools its hidden states, and applies a small linear χ² reward head. Because the whole pipeline is linear, the score decomposes exactly into per-word contributions, and scoring is one model pass plus a dot product.

Full documentation & the method diagram: https://haspi.readthedocs.io

Install

pip install "haspi[leolm]"          # the current method: + torch, accelerate, sentencepiece
pip install "haspi[leolm,cuda12]"   # …with CUDA-12 JAX for GPU

Python ≥ 3.10. The reward head is CPU/JAX; running the language model (feature extraction and the explainer) uses a GPU when one is available and otherwise falls back to CPU — which works but is much slower for the 7B model, so a GPU is recommended. Note the pinned transformers>=4.38,<4.40 — Hugging Face v5 dropped Flax support; the 4.39.x line is the last that ships it.

The fitted reward files are distributed via GitHub Releases, not committed to the repo. Fetch them into models/ before running the explainers or the demo (or fit your own):

gh release download --repo fhstp/haspi --pattern '*.pkl' --dir models/

Try the demo

haspi-demo launches a small web app — the project schematic made real. Type or sample a German comment and see the verdict, a green→red risk meter, and the comment with each word coloured by its exact contribution (risky words highlighted). It handles hate and moderation (with an optional article/thread context), and a seeded "🎲 Aus Korpus" button pulls a reproducible real corpus comment.

pip install "haspi[demo,leolm]"
haspi-demo                          # → http://<host>:8000   (needs a fitted reward; GPU recommended, CPU works)

Classify and explain

The main tools are the two explainers. They load a fitted reward (models/leolm_chi2_reward.pkl) and the language model, then score whatever you give them.

haspi-explain                 # interactive: type a comment, get a verdict + word drivers
haspi-explain-corpus --n 8    # sample labelled corpus posts, show prediction vs. truth

Useful flags: --topk N (words shown per direction), --level {word,token} (word-level by default; token shows the raw sub-word pieces), --reward PATH (a different reward file).

Reading the output

• score vs threshold — the verdict. Score above the calibrated threshold → HATE, below → non-hate. The threshold was chosen once on the training set.
• toward HATE / toward non-hate — the words that pushed the score up or down, each with its exact contribution. These are real, additive attributions: they sum to the decision (relative to an average comment), not a heuristic highlight. A neutral comment's words roughly cancel; a slur or aggressive phrasing spikes.

Note. The attribution is faithful to the decision, but the model reasons at the sentence level — trust the overall verdict and the ranking of content words, not any single function word (you'll sometimes see und, ., etc. carry a little weight).

In Python

from haspi.sequence import LeoLMScorer, format_attribution

scorer = LeoLMScorer("models/leolm_chi2_reward.pkl")
res = scorer.score("Du bist ein Idiot.")
# res = {"score": ..., "label": "HATE"/"non-hate", "threshold": ..., "tokens": [(word, contribution), ...], "n": ...}

toward_hate, toward_non_hate = format_attribution(res, topk=5)

Fit your own reward

Two steps — extract frozen features once (GPU), then fit the linear head (CPU/JAX). Needs data/labels.npz and data/corpus.sqlite3 from haspi-prepare-corpus.

haspi-extract-features --model LeoLM/leo-hessianai-7b --out data/leolm7b_feats.npz
haspi-fit-reward       --features data/leolm7b_feats.npz --out models/leolm_chi2_reward.pkl

haspi-fit-reward standardises the features, reduces them with whitened PCA-512, fits the χ² head, folds the standardise→PCA→head chain into a single vector (printing fold max|Δ| ≈ 1e-6 as a reproduction check), calibrates the decision threshold, and writes a small reward file. A different encoder is fine — swap --model (any Llama/Mistral/Qwen2 architecture works with the pinned transformers).

To fit from cached features without any torch:

import numpy as np
from haspi.data import hate_mask
from haspi.sequence import fit_reward

labels = dict(np.load("data/labels.npz"))
feats = np.load("data/leolm7b_feats.npz")["mean"]
payload, max_delta, auc = fit_reward(feats, hate_mask(labels))   # payload → pickle it

Context-aware moderation

For the online/offline moderation task (a post kept vs. removed by moderators), a comment can be scored with its thread and article context — the parent comment and the article's title and topic path — prepended before encoding.

haspi-moderation-context --variants comment article topic

variant       random split  article-disjoint
--------------------------------------------
comment              0.722             0.726
article              0.757             0.750
topic                0.697             0.653

Two things make this honest. First, the context conditions the encoder but the reward pools and attributes only the comment tokens — the article/thread words shape the comment's representation (causal attention) yet never enter the pooled feature, so the reward can't just memorise which article a post is under. Second, evaluation is on an article-disjoint split (whole articles assigned to train or test, never both). The proof that this matters is model-free: predicting a post's label purely from the training-set removal rate of its article ID (no text, no model) scores 0.70 AUROC on a random split and exactly 0.50 (chance) when articles are disjoint — the leakage is real and there to be exploited. Because we pool comment tokens only, the reward barely touches it: the article-context reward moves just 0.757 → 0.750 from random to disjoint (a whole-blob pooling of context+comment drops twice as far), and the small article-context gain over comment-only (0.726 → 0.750) survives the honest split. The rule still stands — evaluate context-aware moderation on article-/thread-disjoint splits — because the inflation is large in article-concentrated samples (the article-ID control reaches 0.98 on a random split there).

haspi-moderation-context --save-reward models/moderation.pkl writes a moderation reward that haspi-explain --reward models/moderation.pkl scores and explains as remove/keep. (Our text-only reward is comparable to the prior text-only baseline on this corpus, 0.728; the rigorous external comparison is on the RP-Mod benchmark, where the frozen reward matches a fine-tuned German BERT — see the docs.)

How it works (in one paragraph)

A frozen German decoder LM encodes the comment; its final hidden states are mean-pooled to one vector φ. A linear χ² reward head — the one-step reduction of two-distribution IQ-Learn — scores it, trained so an expert class (non-hate) and an anchor class (hate) are separated. Standardisation and PCA-whitening condition the head; the whole linear chain then folds into a single weight vector g with r = g·φ + const. Since φ is a mean over token states, each token's contribution is exactly orient · (1/L) · g·(hₜ − μ) and the parts sum to the score — that's where the per-word explanations come from. The diagram and the full derivation are in the documentation.

Superseded: two-distribution IQ-Learn

The original method — a token-level IQ-Learn reward trained end to end on frozen GPT-2 — is kept for reproducibility. It plateaus at ~0.55 AUROC on this corpus (the reward tracks post length more than hate semantics), which is what motivated the sequence-χ² reformulation. Its CLIs print a superseded notice; haspi.CURRENT_METHOD / haspi.METHODS record the full method-evolution story.

haspi-prepare-corpus --num-sequences 50000 --maxlength 128
haspi-train --task hate --avenue B --lm-mode frozen --qhead factored \
    --weight-decay 1.0 --epochs 25 --save-path models/hate_frozen.pkl
haspi-classify --model models/hate_frozen.pkl        # REPL with per-token IQ-Learn rewards
haspi-evaluate --model models/hate_frozen.pkl --sweep

haspi-train --cv runs the 10-fold cross-validation protocol. See the architecture guide for the agent internals, tuning notes (gamma ≤ 0.95, --target-entropy near the policy's natural entropy, the factored Q-head), and the library API.

Tests & docs

pytest                                  # fast unit suite, no model downloads
pytest -m slow                          # end-to-end on real models (needs the download / GPU)
pip install -e ".[docs]" && sphinx-build -W -b html docs docs/_build/html   # build the docs