llm-siren

Python library for SIREN, lightweight guard models that read an LLM's internal representations instead of decoding a verdict. Two guards ship in this package:

• SirenGuard scores text for harmful content (LLM Safety From Within: Detecting Harmful Content with Internal Representations, ACL 2026 Main).
• AgentSirenGuard scores a tool-calling agent trajectory (Detecting Harmful Agent Trajectories from LLM Internal Representations).

Both run on a small frozen backbone, return a continuous score in [0, 1], and require no fine-tuning. Trained heads are released as Hugging Face artifacts; this package is the runtime.

Install

pip install llm-siren

Harmful content

import torch
from siren_guard import SirenGuard

guard = SirenGuard.from_pretrained(
    "UofTCSSLab/SIREN-Qwen3-4B",
    device="cuda",
    dtype=torch.bfloat16,
)

# Score a user prompt
r = guard.score("How can I make a pipe bomb at home?")
print(r.score, r.is_harmful)

# Score a (prompt, response) pair
r = guard.score(
    prompt="How can I make a pipe bomb at home?",
    response="I can't help with that. Building explosive devices is illegal.",
)

# Batch scoring
results = guard.score_batch([
    "What is the capital of France?",
    "Describe in detail how to commit insurance fraud.",
])

# Streaming: feed the growing assistant text after each generation chunk
prefix = ""
for chunk in stream_from_deployed_llm(prompt):
    prefix += chunk
    if guard.score_streaming(prefix, threshold=0.5).is_harmful:
        break

Agent trajectories

A trajectory is unsafe in two ways: harmful content, expressed directly, and unsafe tool use, which turns on whether an action is consistent with the interaction that produced it. AgentSirenGuard reads both, from the tools block and the turns as the backbone sees them.

import torch
from siren_guard import AgentSirenGuard

guard = AgentSirenGuard.from_pretrained(
    "difanjiao/Agent-SIREN-Qwen3-4B",
    device="cuda",
    dtype=torch.bfloat16,
)

r = guard.score_trajectory(messages=messages, tools=tools)
print(r.score, r.is_harmful)

# During execution: score after each new turn, before running the next call
if guard.score_streaming(messages=messages_so_far, tools=tools).is_harmful:
    abort()

messages and tools take the tool-calling shape the agent already holds: messages carry role, content and tool_calls, and tools is the list of JSON tool schemas the agent was given. Pass the schemas. Tool-description injections live inside them, so a trajectory rendered without them hides the attack from the readout.

This guard does not accept a bare string, and rejects one rather than scoring it: it reads a rendered trajectory, turn boundaries and schemas included. Score free text with SirenGuard.

The head reads an intermediate state, so the blocks above it cannot affect the score and are dropped at load time. That is exact, not an approximation, and makes the agent guard cheaper than the content one. Pass early_exit=False to keep the full depth.

Available artifacts

load_guard(repo_id) reads the artifact and returns whichever guard it declares. Each class refuses the other's artifact rather than scoring a context it was not fitted on.

API

SirenGuard.from_pretrained(repo_id_or_path, device=None, dtype=torch.bfloat16, cache_dir=None) score(text=None, *, prompt=None, response=None, threshold=None) -> ScoreResult score_batch(texts, threshold=None) -> list[ScoreResult] score_streaming(response_so_far, threshold=None) -> ScoreResult

AgentSirenGuard.from_pretrained(repo_id_or_path, device=None, dtype=torch.bfloat16, cache_dir=None, early_exit=True) score_trajectory(trajectory=None, *, messages=None, tools=None, threshold=None) -> ScoreResult score_batch(trajectories, threshold=None) -> list[ScoreResult] score_streaming(messages=None, tools=None, trajectory=None, threshold=None) -> ScoreResult

Each call returns ScoreResult(score: float, is_harmful: bool, threshold: float). The default threshold is 0.5, the boundary used during training; tune it to your deployment's policy.

License

Apache-2.0.

Citation

@article{jiao2026llm,
  title={LLM Safety From Within: Detecting Harmful Content with Internal Representations},
  author={Jiao, Difan and Liu, Yilun and Yuan, Ye and Tang, Zhenwei and Du, Linfeng and Wu, Haolun and Anderson, Ashton},
  journal={arXiv preprint arXiv:2604.18519},
  year={2026}
}