llm-siren 0.1.0

SIREN: a lightweight, plug-and-play guard model for LLM harmfulness detection from internal representations.

llm-siren

Python library for SIREN (LLM Safety From Within: Detecting Harmful Content with Internal Representations, ACL 2026 Main).

SIREN is a lightweight guard model for LLM harmfulness detection. It runs on a small frozen LLM backbone, identifies safety neurons across internal layers, aggregates them into a small MLP classifier, and returns a continuous harmfulness score in [0, 1]. The trained classifier head is ~12M parameters; no fine-tuning of the backbone is required.

This package (llm-siren) provides the runtime. Trained SIREN heads are released as Hugging Face artifacts under UofTCSSLab.

Install

pip install llm-siren

Usage

import torch
from siren_guard import SirenGuard

guard = SirenGuard.from_pretrained(
    "UofTCSSLab/SIREN-Qwen3-0.6B",
    device="cuda",
    dtype=torch.bfloat16,
)

# Score a user prompt
r = guard.score("How can I make a pipe bomb at home?")
print(r.score, r.is_harmful)

# Score a (prompt, response) pair
r = guard.score(
    prompt="How can I make a pipe bomb at home?",
    response="I can't help with that. Building explosive devices is illegal.",
)

# Batch scoring
results = guard.score_batch([
    "What is the capital of France?",
    "Describe in detail how to commit insurance fraud.",
])

# Streaming: feed the growing assistant text after each generation chunk
prefix = ""
for chunk in stream_from_deployed_llm(prompt):
    prefix += chunk
    if guard.score_streaming(prefix, threshold=0.5).is_harmful:
        break

# Tunable threshold (default 0.5, the binary boundary used during training)
r = guard.score(text, threshold=0.1)  # strict
r = guard.score(text, threshold=0.9)  # permissive

Deployment idiom

def safe_generate(user_prompt, deployed_llm):
    if guard.score(user_prompt).is_harmful:
        return DEFAULT_REFUSAL
    response = deployed_llm.generate(user_prompt)
    if guard.score(prompt=user_prompt, response=response).is_harmful:
        return DEFAULT_REFUSAL
    return response

The deployed LLM is independent of SIREN. SIREN never touches the deployed model's internals; it scores the same text through its own frozen backbone.

Available SIREN artifacts

Artifact	Backbone	Repo
SIREN-Qwen3-0.6B	Qwen3-0.6B	UofTCSSLab/SIREN-Qwen3-0.6B

More backbones (Qwen3-4B, Llama-3.2-1B) coming soon.

API

SirenGuard.from_pretrained(repo_id_or_path, device=None, dtype=torch.bfloat16, cache_dir=None) Loads the SIREN classifier head from an HF repo or local path, plus the frozen backbone at the pinned revision recorded in siren_config.json.

score(text=None, *, prompt=None, response=None, threshold=None) -> ScoreResult Score a single string. Pass text= for raw moderation, or prompt=/response= for the response-level form (joined with "\n" to match the training distribution).

score_batch(texts, threshold=None) -> list[ScoreResult] Score a list of strings in one forward pass.

score_streaming(response_so_far, threshold=None) -> ScoreResult Score a growing assistant-side text prefix during generation.

Each call returns ScoreResult(score: float, is_harmful: bool, threshold: float).

License

Apache-2.0.

Citation

@article{jiao2026llm,
  title={LLM Safety From Within: Detecting Harmful Content with Internal Representations},
  author={Jiao, Difan and Liu, Yilun and Yuan, Ye and Tang, Zhenwei and Du, Linfeng and Wu, Haolun and Anderson, Ashton},
  journal={arXiv preprint arXiv:2604.18519},
  year={2026}
}