RednBlue CLI v3.4.0rc1

Adversarial Security Testing for AI Models: local execution, minimal data upload

RednBlue CLI is a command-line tool for testing the adversarial robustness of machine learning models and the security posture of LLM/RAG/Agent systems. Attacks run locally, so model weights and test inputs never leave your machine. Only per-test metrics are uploaded, and only when you submit (see What is uploaded).

███████████               ███████████
▒▒███▒▒▒▒▒███             ▒▒███▒▒▒▒▒███
 ▒███    ▒███  ████████    ▒███    ▒███
 ▒██████████  ▒▒███ ▒▒███  ▒██████████
 ▒███▒▒▒▒▒███  ▒███  ▒███  ▒███▒▒▒▒▒███
 ▒███    ▒███  ▒███  ▒███  ▒███    ▒███
 █████   █████ ████  █████ ███████████
▒▒▒▒▒   ▒▒▒▒▒ ▒▒▒▒  ▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒

Adversarial Security Testing

---

Features

• Local execution, minimal data upload: attacks run on your machine; model weights, images and file paths are never uploaded
• Image Classifiers — Test ResNet, VGG, EfficientNet, and custom architectures
• YOLO Detection — Ultralytics detection models; tested with YOLOv8n and YOLO11n
• LLM / RAG / Agent Red-Team Suite — Jailbreak resistance, system prompt extraction, indirect prompt injection, token-level obfuscation, RAG pipeline security, agentic tool-hijacking/goal-drift/privilege-escalation, backdoor auditing, and model extraction resistance — against hosted providers (OpenAI/Anthropic/Gemini), a local Python target file, or any HTTP endpoint
• Tier-Based Testing — Freelancer (quick scan) and Enterprise (comprehensive)
• Visual Explanations (XAI) — See exactly which pixels confused the model, including the real perturbed image, not a placeholder
• Blue Team Hardening (experimental): rnb defend attaches smoothing/clipping hooks to a model's conv layers for the current process only; defended evaluation is not yet available
• Encrypted Submission: results are encrypted in transit with keys derived from your token. That is transport protection, not a signature. They carry a run attestation (tool and library versions, model and input hashes, attack parameters and a results hash). Certificates are signed by the RednBlue server; see docs/SERVER_SPEC.md
• Compliance Reports: map test results onto EU AI Act articles and NIST AI RMF functions. The report is marked self-assessment, unverified: it is generated locally and is not signed. The grade is recomputed from the raw per-test results, and a verdicts fingerprint (SHA-256 of the pass/fail verdicts, the same as verdicts_sha256 in an upload) identifies the results it was built from

---

Installation

This is a pre-release (release candidate). pip skips pre-releases unless asked:

# Install this release candidate from PyPI
pip install --pre rednblue

# For YOLO testing, install the ultralytics versions the attacks are tested with
pip install --pre "rednblue[yolo]"

# Verify installation
rnb --version

Plain pip install rednblue (without --pre) installs the latest stable release (3.3.1), not this release candidate.

YOLO attacks are tested with ultralytics 8.4.174 to 8.4.176. Other versions are allowed, but the CLI warns, and the upload's attestation records that the run used an untested version.

Requirements

• Python 3.10+ (tested on 3.10, 3.11 and 3.12)
• PyTorch 2.0+ (for classifier/YOLO testing)
• CUDA (optional, for GPU acceleration)
• For LLM testing against a hosted provider: pip install openai / pip install anthropic / pip install google-generativeai as needed — not bundled by default, since most users only test one provider
• For rnb explain --method gradcam: pip install grad-cam (the default method, robust-ig, does not need it)
• For ONNX models: pip install "rednblue[onnx]"

---

Quick Start

1. Choose your interface

Desktop UI (recommended for non-CLI users):

rnb ui

Command line: see below.

2. Set your token (always required for rnb llm; required for submitted runs on rnb preview)

# Linux/Mac
export RNB_TOKEN=RB-XXXXXX-YYYYYY

# Windows
set RNB_TOKEN=RB-XXXXXX-YYYYYY

Get a token at https://dashboard.rednblue.io/dashboard/credits.

3. Run a free preview (Classifier / YOLO)

Image Classifier:

rnb preview --model resnet50.pth --input ./test_images --model-type classifier

YOLO Detection Model:

rnb preview --model yolov10n.pt --input ./test_images --model-type yolo

4. Submit for a RednBlue certificate (Classifier / YOLO)

rnb preview --model yolov10n.pt --input ./images --model-type yolo --submit

5. Run an LLM / RAG / Agent security assessment

Against a hosted provider:

rnb llm --provider openai --model gpt-4o-mini --api-key sk-xxxxx --submit

Against a local Python file exposing a chat function (auto-discovered, or specify --function):

rnb llm --file my_target.py --submit

Against an HTTP endpoint:

rnb llm --endpoint http://localhost:8000/chat --input-field prompt --output-field response --submit

Restrict to specific attack classes (see Assessment Dimensions below for codes):

rnb llm --file my_target.py --attacks JB,SPE,TM --submit

6. View your report

Go to: https://dashboard.rednblue.io/dashboard/tests to download your PDF report and certificate, for both Classifier/YOLO and LLM/RAG/Agent sessions.

---

Commands

Command	Description
rnb	Show welcome banner and quick start
rnb --version	Print the installed version
rnb-welcome	Same as rnb — convenience alias usable outside the main group
rnb ui	Launch the desktop graphical interface
rnb preview --help	Run adversarial attacks (classifier or YOLO)
rnb llm --help	Run the full LLM/RAG/Agent adversarial security assessment
rnb status	Check CLI version and token validity/tier
rnb explain --help	Generate a standalone visual-explanation panel for one image (Robust Integrated Gradients)
rnb defend --help	Experimental: attach hardening hooks to a model in-process (not saved, not scored)
rnb report --help	Generate a compliance report + self-assessment seal from saved classifier/YOLO results

rnb preview options for what is measured

Option	Models	What it does
--normalize auto|imagenet|none|m1,m2,m3/s1,s2,s3	classifier	Input normalisation, applied inside the model so epsilons stay in pixel space. auto (default) recognises 1000-class torchvision/timm models and otherwise stops and asks.
--labels labels.csv	classifier	Labels mode. One filename,class_index row per test image (an optional header row; the file name is the image's base name). The metric becomes robust accuracy: an image the model gets wrong without attack counts as not robust on every test, and the grade is not capped at SILVER (the gradient caps still apply). The labels are your claim and are not verified; the certificate says so. Without labels the metric is prediction stability (does the output change under attack) and the grade is at most SILVER. Every selected image needs a label; fewer than 3 images that are right without attack gives INCOMPLETE, and fewer than 10 labelled images prints a warning that the grade is statistically weak. Uploaded: a correct/incorrect flag per image and a salted hash of the file, not the file (for an image the model gets right, its label equals the uploaded clean prediction).
--gt DIR	YOLO	Labels mode for detectors. One ultralytics-format file per test image, DIR/<image stem>.txt, with a class cx cy w h row per object (normalised to [0, 1]). Only clean detections that match a ground-truth box (same class, IoU ≥ 0.5) are references the attack must keep, so false positives no longer count; an image where no object is detected without attack counts as not robust. A ghost (YNMS) must match neither a clean detection nor a ground-truth box. Every selected image needs a file with at least one object. Same SILVER-cap, INCOMPLETE and weak-sample rules as --labels; uploaded: a correct/incorrect flag and the numbers of ground-truth and missed objects per image, and a salted hash of the files, never the boxes.
--input-size N|native	classifier, YOLO	Side in pixels the images are resized to: classifier default 224, or native to attack each image at its own size (e.g. 32×32 for CIFAR models); YOLO default 640, a multiple of 32, or native for the training size stored in the checkpoint. Patch-attack sizes (UAP: 16/32/48/64 px at 224) scale with the input. Recorded in the results. rnb explain takes the same option for classifiers.

# CIFAR-10 model, 32x32 PNGs, labels
rnb preview --model cifar_net.pt --input ./cifar_png --normalize none --input-size native --labels labels.csv

Other command options

rnb <command> --help lists every option. Options not shown in the examples above:

Command	Option	What it does
rnb preview	--device auto|cpu|cuda|mps	Compute device (default auto: CUDA, then MPS, then CPU). rnb explain and rnb defend take the same option
rnb preview	--tier	Has no effect: the tier always comes from RNB_TOKEN (a different value is ignored with a warning). The same holds for rnb llm --tier
rnb llm	--skip-recon	Skip the 10-probe recon phase
rnb llm	--output PATH	Save the raw results JSON to this path
rnb explain	--image PATH	The image to explain (required)
rnb explain	--attack CODE	Attack to visualise (default GNI; e.g. PGD, FSP for classifiers, YGNI, YPGD, YFGS for YOLO)
rnb explain	--epsilon FLOAT	Attack strength (default 0.03)
rnb explain	--method robust-ig|gradcam	Explanation method (default robust-ig; gradcam needs pip install grad-cam)
rnb explain	--output PATH	Panel image path (default xai_panel.png)
rnb report	--results PATH	Results JSON from rnb preview --save-results (required)
rnb report	--model-name TEXT	Model name in the report (default model)
rnb report	--output-dir DIR	Output directory (default ./compliance_report)
rnb defend	--state PATH	A vulnerability state.json; loaded but not yet used by the hooks

---

Supported Model Formats

Type	Extensions
PyTorch	.pt, .pth, .safetensors
ONNX	.onnx
Ultralytics YOLO	.pt (auto-detected)

Architectures: ResNet, VGG, DenseNet, EfficientNet, MobileNet, Inception, SqueezeNet, ShuffleNet, GoogLeNet, AlexNet, and common timm backbones.

TensorFlow (.h5, .pb) is not yet supported. The ingestion layer (rnb/core/ingestion/wrapper.py) is architected to accept a TensorFlow adapter, but tensorflow is intentionally not bundled as a dependency today — adding real support is tracked as a follow-up, not claimed here.

LLM/RAG/Agent targets: hosted providers (OpenAI, Anthropic, Gemini), a local Python file exposing a chat(user_message: str) -> str-style function, or any HTTP endpoint accepting a JSON prompt field and returning a JSON response field.

---

Assessment Dimensions

Classifier Models

Every listed attack runs against a tiny reference model in the test suite (tests/contracts, finding 3) and must lower its accuracy at its listed epsilons. The noise baselines GNI, SHFP (and YOLO's YGNI) are corruption sanity checks, not adversarial attacks. They are listed on Enterprise only (up to epsilon 0.10), because at Freelancer epsilons of 0.01 or less they do not measurably change any model.

Code	Attack	What it does	Blue team
GNI	Gaussian noise	Random noise with standard deviation epsilon (a corruption baseline, not adversarial)	planned
SHFP	High-frequency pattern	A fixed sinusoidal pattern of amplitude epsilon (corruption baseline)	planned
CCM	Colour channel shift	Shifts each colour channel by up to epsilon (corruption baseline)	planned
UAP	Adversarial patch (per image)	A square patch optimised by gradient steps; patch size 16–64 px (not a universal patch)	planned
FSP	FGSM (white-box)	One signed-gradient step of size epsilon on the model itself; a gradient-cap attack since scoring spec 1.1	planned
PGD	Projected gradient descent	40 signed-gradient steps, projected to the L-inf epsilon ball	planned
CW	C&W margin loss, L-inf	PGD on the Carlini-Wagner margin loss within epsilon (not the original C&W-L2 optimisation)	planned
DEEP	DeepFool, L-inf	Steps to the nearest linearised decision boundary, clipped to epsilon; validated against Foolbox	planned
SQA	Square Attack	Score-based black-box random search on the logit margin, within epsilon	planned
APGD	Auto-PGD (cross-entropy)	Croce & Hein 2020 step-size schedule within epsilon; validated against torchattacks	planned

YOLO Detection Models

Evasion succeeds when a clean detection has no same-class IoU ≥ 0.5 match after the attack, or its match kept less than half its confidence.

Code	Attack	What it does	Blue team
YGNI	Gaussian noise (Enterprise only)	Corruption baseline with standard deviation epsilon	planned
YFGS	FGSM	One signed-gradient step on the detection loss	planned
YPGD	PGD	40 projected signed-gradient steps	planned
YDAG	Dense iterative FGSM	Pushes every anchor's class scores down (inspired by DAG)	planned
YTOG	Class vanishing	Removes the most confident clean class (inspired by TOG)	planned
YNMS	Ghost injection	Adds detections with no match among the clean ones	planned
YNES	NES black-box	Gradient estimated from queries only	planned
YSQA	Square Attack	Score-based black-box random search within epsilon	planned
YGMD	Gradient-masking diagnostic	Compares white-box and black-box removal; not scored	n/a

LLM / RAG / Agent

Each class runs against the target's actual responses — no attack ever counts as a success unless the response shows positive evidence of engaging with the malicious request (see v3.3.0 Changelog entry below); a target that's unreachable or merely non-committal is never scored as compromised.

Code	Class	Sub-attacks	Blue team
JB	Jailbreak Resistance	Template Battery, Many-Shot Jailbreaking, Crescendo Multi-Turn, Multilingual Jailbreak, Persona Transfer, Competing Objectives	planned
SPE	System Prompt Protection	Direct Extraction, Differential Probing, Token Fishing	planned
LPI	Indirect Injection Defense	Document Body Injection	planned
TM	Token Manipulation Robustness	Unicode Homoglyph Substitution, Zero-Width Character Injection, Leetspeak Substitution, Context Dilution, Bidirectional Text Override	planned
RAG	RAG Pipeline Security	Knowledge Boundary Probe, Context Window Stuffing, RAG Prompt Injection (Corpus Poisoning is unlisted until an inject_fn harness exists)	planned
AGT	Agentic Safety	Tool Hijacking via Injected Content, Goal Override / Objective Drift, Privilege Escalation Chain	planned
BD	Backdoor Integrity	STRIP Consistency Test, Trigger Enumeration, Input Sensitivity Probe	planned
EXT	Model Extraction Resistance	Extraction Feasibility Assessment, Response Consistency Test, Model Fingerprinting	planned

What a pass means for injection tests. LPI, RAG-PI, RAG-CS, AGT-GD and AGT-PE plant a per-run canary or secret. Those sub-attacks, and the LPI class, report a pass as "no canary leak observed"; RAG and AGT also contain non-canary sub-attacks, so their class reads "no finding observed (not proof of robustness)". Neither means "robust": it shows only that the target did not follow these specific injections in this run. A target that follows an injection but deliberately leaves out the canary would not be caught, and other phrasings may still succeed.

Blue team. No defense is implemented yet. Every listed attack family carries an explicit waiver ("blue team scheduled", rnb/defenses/registry.py).

Not yet supported

• RAG corpus poisoning (RAG-CP). It needs an inject_fn harness to insert documents into the target's vector store.
• Audio / speech-recognition models. The audio attacks are not listed until they are validated on a real speech-recognition model.

Use --attacks with a comma-separated list of codes to restrict a run (e.g. --attacks JB,SPE,TM). Omit --attacks to run all classes.

---

Tier Comparison

Feature	Freelancer	Enterprise
Classifier Attacks	5 (incl. PGD)	10 (incl. PGD, plus the GNI/SHFP noise baselines)
YOLO Attacks	3 (YFGS, YPGD, YGMD diagnostic)	9 (incl. YFGS, YPGD, plus the YGNI noise baseline)
Epsilon Values	2	4
Total Scenarios	~10-20	~30-70
LLM / RAG / Agent Testing	❌	✅ (all 8 classes, 27 sub-attacks)
Epsilon Optimization	❌	✅
Continuous Integrity Monitoring	❌	🔜 Coming soon

---

Output Example

============================================================
  RednBlue Security Preview — YOLO Detection
============================================================
  Passed  : 19
  Failed  : 1  (attack succeeded)
  Errored : 1  (not counted as passes)
  Skipped : 0
  Prediction stability: 95% of 20 scored tests
  Label mode: none (prediction stability: whether the output changes under attack, not whether it is correct; grade at most SILVER)
  Estimated Grade: SILVER  (prediction stability 95%; label-free run: grade capped at SILVER (run with --labels or --gt for a full grade))

  ⚠️  This is a preview only
  → Visit: https://rednblue.io/checkout
  → Re-run with: rnb preview --model-type yolo --submit

---

Grades

Grade	Score	Meaning
🥇 GOLD	≥90%	Excellent robustness, deployment ready (needs labels: --labels / --gt)
🥈 SILVER	≥75%	Good robustness, minor improvements recommended
🥉 BRONZE	≥60%	Moderate robustness, improvements needed
❌ FAIL	<60%	Not robust
⚠️ INCOMPLETE	more than 5% of tests errored, no test produced a result, too few tests to confirm a gradient break, (classifier/YOLO) no gradient-attack result, or (with labels) fewer than 3 inputs the model gets right without attack	No grade can be given; fix the errors and re-run

How the grade is computed (scoring spec 1.2, one scorer for every model type, rnb/core/scoring.py, full rules in docs/SCORING.md):

• Inputs: every run attacks the first 10 images in the folder by file name (.jpg, .jpeg, .png), on every path (CLI and desktop UI, classifier and YOLO). With fewer than 10 the grade is shown as provisional (CLI, desktop UI and the compliance report) and cannot be submitted.

• Uncertainty: the score is shown with a 95% Wilson interval, for example "Robust accuracy: 46% of 400 scored tests (95% CI 41–51%, test-level)". It counts each test as independent; tests on the same image are correlated, so the real uncertainty is wider. The interval is shown, not used for grading.

• Every test is passed (the attack failed), failed (the attack succeeded), errored or skipped. Errored and skipped tests are never counted as passes.

• Score = passed / (passed + failed). What it measures depends on the label mode:

  • Without labels (default), an attack succeeds when the model's output changes from its own clean output. The score is prediction stability: it cannot tell a wrong answer from a right one, so the grade is capped at SILVER.
  • With labels (--labels for classifiers, --gt for YOLO), an input the model gets wrong without attack fails every test, and the score is robust accuracy, shown with the clean accuracy. Fewer than 3 inputs right without attack gives INCOMPLETE; fewer than 10 labelled inputs prints a warning that the grade is statistically weak.

• Required evidence (classifier and YOLO): a run is graded only if at least one PGD, FGSM, YFGS or YPGD test at epsilon 8/255 or below produced a real result (pass or fail). If every such test errored or was skipped, or none ran, the grade is INCOMPLETE. Audio and LLM runs are exempt.

• Gradient cap: applies to the white-box gradient attacks FGSM, FSP and PGD (classifier) and YFGS/YPGD (YOLO). The score is checked per attack at each epsilon up to 8/255 that has at least 3 tests. For this check only, an errored test counts as broken when the same attack and epsilon also produced at least one real result, so a crash cannot hide a weak result. If every test at that epsilon errored, there is no evidence and the cap does not apply; the error-rate limit still does:

  • At 10% or less the grade is FAIL, whatever the other attacks score and even if too many tests errored for a grade.
  • At 30% or less the grade is at most BRONZE.
  • If a group has only 1 or 2 tests (fewer than 3 images) and its score is 30% or less, the grade is INCOMPLETE: there are too few tests to confirm the break. Use at least 3 images. The exception: when the other completed tests already give FAIL, the grade stays FAIL.

• The YOLO gradient-masking diagnostic (YGMD) is reported but not scored.

• For LLM/RAG/Agent runs, each attack class is one test.

Grades shown by the CLI and desktop UI are estimates. The certificate grade is issued by the RednBlue server from the uploaded per-test results. It is based on results reported by your machine; RednBlue does not independently re-run them.

---

What is uploaded

Nothing is uploaded unless you pass --submit (or press Submit in the desktop UI). --submit needs at least 10 test images; fewer cannot support a certificate grade. (Audio testing is unlisted for now.)

Session request (HTTPS): your token, a short hash of the model file (or of the provider/endpoint), and a model alias. LLM runs also send the architecture (llm, rag or agent) and the tier. The alias is:

• the file's base name without extension (resnet18);
• the provider's model name (gpt-4o);
• the fixed text llm-endpoint for HTTP endpoints.

--model-name replaces it. An override that looks like a URL, host name, IP address or path is refused, and the default alias is used. A file path, an endpoint URL or its host is never sent. The hash of an endpoint is salted with this installation's secret salt (rnb/core/hasher.py; the salt is never sent), so the server cannot confirm a guessed URL such as http://localhost:8000/chat. Model-file hashes are not salted: they identify the exact weights.

Classifier / YOLO / audio results (encrypted): schema_version, run_type (vision or audio) and the model alias. For each test input, an opaque id (img_000, ...) and, per attack, only allowlisted metrics:

• epsilon, status and success;
• confidence before and after;
• perturbation norms, gradient statistics and iteration counts;
• predicted classes and detection counts;
• device type and library versions.

Any other field is dropped, at any depth. Error messages are truncated to 200 characters. A message is withheld entirely if it contains anything that looks like a path, file name, IP address, host:port or localhost. Never uploaded: model weights, images, audio, transcripts, file paths.

LLM / RAG / Agent results (encrypted): schema_version, run_type: llm and the model alias. Per attack class:

• the class code, status, severity, probe counts (probes_total, probes_errored) and error types;
• per sub-attack, its code, severity, counts and rates;
• recon flags and numbers (for example "system prompt detected: yes");
• whether the health check passed;
• summary counts.

No probe text and no target replies are uploaded by default. rnb llm --share-evidence adds the evidence as recorded, for a human reviewer: probe text, reply snippets of up to 200 characters, the health-check reply, and the recon text (a detected system prompt of up to 250 characters). The desktop UI never shares evidence.

Run attestation (every upload, encrypted): the tool version and the versions of python, torch, torchvision, ultralytics, transformers, torchaudio, numpy and Pillow; the CUDA/cuDNN version and the OS; the SHA-256 of the model file, or of the target script for rnb llm --file; and a SHA-256 of each test input's bytes. An input hash lets the server, or anyone else who already has the same image, confirm that it was used; it reveals nothing about images they do not have. It also records:

• the seed;
• the attack parameters (tier, attacks, epsilons, budgets, preprocessing, selection rule) and a hash of the attack code;
• the device type and target class;
• for LLM runs, the target kind and the provider model (never an endpoint);
• the server session id;
• two hashes of the results: results_sha256 over exactly what was uploaded, and verdicts_sha256 over the pass/fail verdicts only.

Results are encrypted (AES-256) and authenticated with a MAC, using keys derived from your token. That protects them in transit only. RednBlue's server holds the same token, so it can read everything listed above. The MAC is not a signature or a certificate (see the Unreleased changelog).

---

Architecture

---

Links

• Platform: https://dashboard.rednblue.io
• Documentation: https://docs.rednblue.ai
• Website: https://rednblue.io

---

Authors

• Dr. Mahdi Deramgozin — Chief AI Officer
• Dr. Saeid Samizade — Chief Technology Officer

---

License

Proprietary — RednBlue SAS © 2026

Made in France 🇫🇷

---

Changelog

[3.4.0rc1] — 2026-10-13

Release candidate for 3.4.0 (pip install --pre rednblue). 3.3.2 was never published (the latest stable release is 3.3.1); its changes are listed in this section under "From the unpublished 3.3.2". Where this changelog says "3.4.0" it means this release line, starting with 3.4.0rc1.

Hardening work on dev/hardening-333. Grading follows scoring spec 1.2 (below, and docs/SCORING.md).

Added and changed before upload (release candidate)

• rnb --version prints the installed version.
• --submit from a release candidate prints a note before uploading: "pre-release: server-side grading of scoring spec 1.2 is not live yet; the certificate grade may differ from this report". rnb preview --submit and rnb llm --submit show it; the desktop UI does not.
• The tested ultralytics range now includes 8.4.176 (8.4.174 to 8.4.176), the version a plain install resolved on 2026-10-13. Every enterprise YOLO attack gave identical records on 8.4.174 and 8.4.176 (yolov8n and yolo11n, one COCO image, epsilon 0.03), and the real-model YPGD test passes on 8.4.176. rednblue[yolo], the runtime warning, the attestation flag and the CI matrix follow.

Fixed: claims and packaging (release-candidate checks)

• rnb optimize-epsilon did nothing. It printed "Running epsilon optimization..." and "✔ Complete" without running an optimization, and ignored --model and --attack. It now exits with "not implemented yet" and is no longer listed in rnb --help, the banner or this README.
• --tier on rnb preview and rnb llm was described as a manual override. The tier always comes from the token, and a different value was already ignored with a warning; the help text now says so.
• The desktop UI offered "Audio / Speech ASR" and labelled Vision mode "Classifier · YOLO · Audio", although audio attacks are not listed and an audio run only showed "not available yet". The option is disabled and marked "not available yet".
• ISO/IEC 42001, UK DSIT, Canada AIDA and Singapore MAIGF were listed as compliance mappings, and the UI showed an "ISO 42001" badge. The compliance report maps EU AI Act articles and NIST AI RMF functions only; the other names are removed.
• The LLM submit warning in the desktop UI said results "will be saved locally" while the UI submits them. It now says submission needs dashboard v3.1 or later and to keep the local results file if it fails.
• Untranslated UI strings: the visual-explanation panel stayed in English in French mode, and its confidence rows showed the raw keys confidenceBefore / confidenceAfter. The documentation link in the UI footer was malformed (lang=entarget=).
• Python 3.8 and 3.9 were declared but never tested, and current versions of click, huggingface_hub and safetensors need 3.10. requires-python is now >=3.10; CI runs 3.10, 3.11 and 3.12.
• requirements.txt listed langdetect, which nothing imports.
• The first-import welcome banner never worked. import rnb tried to import a function that does not exist and swallowed the error, so in a terminal every import rnb loaded the whole CLI for nothing. The hook is removed; rnb and rnb-welcome still show the banner.

Fixed: YOLO attacks (P0-2)

• The YOLO gradient attacks never ran on ultralytics 8.4. _forward put the model in train mode, where ultralytics 8.4 returns a dict of raw head outputs that the code did not recognise. The loss fell back to summing over the dict's keys and returned the integer 0:
  • YTOG and YNMS crashed with 'int' object has no attribute 'backward'.
  • YNES, YSQA and YGMD crashed with 'int' object has no attribute 'item'.
  • YFGS, YPGD and YDAG caught the same exception and silently replaced the attack with a constant shift or a random-sign walk, flagged gradient_masking_detected. That happened on 20 of 20 records each, on both YOLOv8n and YOLO11n, in the 3.3.1 and 3.3.2 runs, and the results were scored as passes. The YOLO grades in the 3.3.2 before/after table were therefore not measurements of gradient robustness.
• Train mode also updated BatchNorm running statistics on every gradient step. The model being attacked drifted away from the model being measured. Attacks now run in eval mode only.
• The detection loss read channel 4 as objectness. YOLOv8/11 heads are anchor-free and have no objectness channel, so channel 4 is a box coordinate there. The loss now uses the decoded class probabilities: the sum of -log(1 - p) for evasion. The output layout is decided from the head's class count, not from the tensor shape. Anchor-free [B, 4+nc, N] and classic YOLOv5 [B, N, 5+nc] are supported. Everything else is an error with a reason: end-to-end (NMS-free) heads such as YOLOv10/YOLO26 [B, 300, 6], a class-count mismatch, scores that are not probabilities, and NaN or inf output. A raw model without head metadata is also rejected when its output is [B, N, 6], because that could be either a 1-class YOLOv5 or end-to-end rows. A model whose output cannot be read gives errors, even on an image with no detections.
• No more silent fallbacks. A missing, non-finite or zero gradient, a constant black-box loss, an unreadable output format, or a failed detection count now gives an error record ("gradient unavailable, attack degraded"), never a pass.
• An evasion attack on an image with no clean detections is now skipped with a reason. It used to count as a pass, because nothing could be removed.
• YTOG targeted class 0 (person) on every image, so on most images it measured nothing. It now targets the class of the most confident clean detection, and succeeds when detections of that class drop.
• YSQA set the pixel values of each square to ±epsilon, giving near-black squares far outside the epsilon budget. It now re-draws the perturbation's sign inside the square, as in the Square Attack paper.
• The tested ultralytics range is 8.4.174 to 8.4.176 (8.4.176 added in 3.4.0rc1: every enterprise YOLO attack gave identical records on 8.4.174 and 8.4.176 for yolov8n and yolo11n), installed with pip install "rednblue[yolo]". It is not a hard pin; the base install accepts ultralytics>=8.0. Outside the tested range a runtime warning appears, and the attestation records tool_versions.ultralytics and ultralytics_in_tested_range: false. CI (.github/workflows/yolo-pinned.yml) runs the real-model test at both ends. ultralytics 8.3.253 fails with "Inference tensors cannot be saved for backward", which gives error records. A weekly CI job (.github/workflows/yolo-latest.yml) runs the real-model test against the newest ultralytics release.
• Error records no longer carry zero-filled metric fields. They have attack_name, tier, epsilon, success: false (kept for older servers) and error. Skipped records have attack_name, tier, epsilon and skip_reason, and no success key. A server that reads success with a default of false would count them as passes; it must read status (see docs/SCORING.md).
• The YGMD gradient-masking diagnostic compared changes in average confidence. When a white-box attack removed the weak boxes, the survivors' average went up, and the diagnostic reported "GRADIENT_MASKING_HARD" for an attack that had worked. It now compares the fraction of clean detections removed. If the white-box attack removed boxes and the black-box attacks did not, the verdict is NO_MASKING. A zero gradient where nothing else worked either is INCONCLUSIVE. YGMD is diagnostic only and is never scored.
• The .safetensors YOLO loader injected weights with strict=False into a downloaded COCO YOLOv8 and printed "model is still usable". Any mismatched layer silently kept COCO weights, and the results were reported as the customer's model. The load is now strict and fails with a reason.
• YGNI's perturbation_linf is measured on the actual change to the image. Its noise has standard deviation epsilon and no L-inf bound.
• The YDAG, YTOG and YSQA docstrings now say what the code does. YDAG is dense iterative FGSM, inspired by DAG but not DAG. YTOG is a class-restricted vanishing attack with a constant step. YSQA uses a simplified square schedule.
• The desktop UI showed a dimension whose tests were all skipped as "Error". It now shows "Skipped". The explain panel no longer picks skipped or errored records as "worst cases".

Fixed: silent degradation in the other attacks (finding 2)

• LLM: a failed probe was scored as a model reply. Every attack module turned a target exception into the text "ERROR:...", and a None, dict or empty reply into text as well. The scorer then read that text as a reply:

  • As a refusal, giving false passes: a class where every probe failed after the health check reported "not vulnerable".
  • As a finding, giving false fails:
    • SPE-INF counted errors as blocked topics.
    • TM counted an errored baseline as a refusal.
    • BD-STR and EXT-CONS read identical error strings as "consistent" or "deterministic".
    • AGT-PE matched "api key" and "token" in API error messages.

  Now any probe that raised, returned no text or an empty reply, or returned error text is an errored probe. It is left out of both the bypass and the robust counts. The class verdict is:

  • a bypass among completed probes is a fail;
  • no bypass but some probes errored is an error, with probes_errored and probes_total;
  • only a class where every probe completed and none bypassed is a pass.

• LLM: other fixes.

  • A JB-MSJ (many-shot) bypass was left out of the jailbreak total. A many-shot bypass on its own gave success: false.

  • Sub-attacks that compare several replies now need enough completed probes, or they report "not measured" instead of a finding. This applies to SPE-INF (2 of 3 per topic, and at least half the topics), BD-STR (at least half the perturbations), EXT-FEAS (at least half the queries) and EXT-CONS (every repeat). Otherwise one completed reply among errors could decide the result.

  • LPI is a fail only when the injection canary appears. Its system-information keyword check also matches correct refusals that describe the injection, so it is still reported as severity "high" but does not decide the verdict.

  • A failed RAG inject_fn counted as a defended canary. It is now an errored step.

  • RAG-PI flagged any reply containing the word "success". It now looks for the payload's own marker.

  • A target that returned None passed the health check as the text "None". HTTP endpoint targets turned a JSON null or a nested error object into text the same way. Both are now errored probes.

  • Recon probes that fail are recorded as unanswered (probes_errored), not as "ERROR:" text.

  • Error text is recognised by one shared rule:

    • an "ERROR:" prefix;
    • an HTML page (proxy and gateway error pages);
    • a JSON object with an error key (OpenAI, Ollama);
    • a short reply containing a known error keyword.

    Keywords such as "timeout" are only trusted in short replies, so a real answer that mentions them is still scored.

• Classifier: a zero, missing or NaN gradient was a pass. With no usable gradient, FSP, PGD, CW, DEEP and APGD made no change and recorded "success: false". This happens with a saturated softmax or a model that masks gradients. These cases are now errors ("gradient unavailable, attack degraded"). So is NaN model output, and so is a constant SQA score.

• Classifier SQA exceeded its epsilon budget. It added ±epsilon on top of the current image, so squares stacked; we measured 8–14× epsilon on ResNet18, and those results were scored as successes. It now re-draws the perturbation sign inside the square. Its score is now the logit margin rather than -softmax[orig], which rounds to a constant on confident models. If the gradient vanishes only after the attack has already succeeded (e.g. the target class saturates), the attack stops and the finding is kept.

• Classifier: weights were loaded with strict=False. If the detected architecture did not match the checkpoint, the missing layers stayed randomly initialised, with no warning. Loading now fails with the number of mismatched keys.

• Audio:

  • The wav2vec2 SCTC attack passed the audio through numpy, so the gradient never reached it and the audio never changed. The processor's normalisation is now done in torch.
  • WER is a hard-label score. When no sampled perturbation changes the transcript, SNES, SSQA and Whisper SCTC have nothing to follow. They now record skipped with that reason; before, the audio was left unchanged and the test counted as a pass. A finding already made before the signal vanished is kept.
  • An empty clean transcription is now skipped, because no word-error rate can be measured.
  • The audio square attack set waveform values to ±epsilon, silencing segments far outside the budget. It now re-draws the perturbation sign.
  • A listed but missing audio attack is now an error rather than silently dropped.

Fixed: attacks that did not do what their names said (finding 3)

• Every listed attack is now validated. In the tiny-model harness (tests/attack_harness.py, tests/contracts), each one must lower accuracy (classifier) or detections (YOLO) versus clean at its listed parameters. The exceptions are GNI, SHFP and YGNI at Freelancer epsilons, which are strict expected failures; see the open decisions. Adversarial attacks must also be strictly stronger than random-sign noise and a constant ±epsilon shift of the same size, and stay within epsilon. Negative controls check that a do-nothing or shift-everything "attack" fails this.
• DeepFool (DEEP) raised the model's confidence. It stepped up the gradient of the current class's logit; on our test model, confidence went to 1.0 and no prediction ever changed. Its perturbation also ignored epsilon, up to 0.58. It is now untargeted L-inf DeepFool, as in Foolbox: it steps to the nearest linearised boundary over the top 10 classes (counting the clean class) with overshoot 0.02, and is clipped to epsilon. A dev-only test (tests/slow/test_reference_attacks.py, requires requirements-dev.txt) checks that it agrees with Foolbox's LinfDeepFoolAttack on at least 90% of images at 0.005–0.03. In our run it agreed on every one. The test also checks that the size of the unclipped perturbation is within a factor of 1.5 of Foolbox's.
• CW ignored epsilon. The "simplified Carlini-Wagner" used Adam with no projection and produced L-inf ≈ 0.57 whatever epsilon was. Those over-budget successes were scored. It is now PGD on the C&W margin loss within epsilon, and named as such. The original C&W-L2 has no epsilon budget, so it does not fit the scoring.
• APGD stored the post-step point with the pre-step loss, and did not follow the paper's schedule. It now follows Croce & Hein 2020: checkpoints; step halving with restart from the best point when at most 75% of steps increased the loss; momentum 0.75; random start. A dev-only test checks it is at most one image weaker than torchattacks' APGD; in our run the success counts were identical at every epsilon. APGD is always untargeted.
• "Universal Adversarial Patch" (UAP) pasted random noise without any optimisation. It is now a patch optimised per image, and named as such (the code UAP is kept). A test checks that it beats a random patch.
• FSP was described as a "surrogate transfer". It computes FGSM on the model under test, so it is renamed "FGSM (white-box)" and is now a gradient-cap attack (scoring spec 1.1, below).
• Targeted mode: every classifier attack is now scored the same way, as a fail when the prediction changed; target_reached is reported separately. An image whose clean prediction already equals the target is skipped with that reason.
• GNI's perturbation_linf is now measured on the actual change. Its noise has standard deviation epsilon and no L-inf bound.
• YOLO: YPGD, YDAG and YTOG no longer discard a finding when the gradient vanishes after every detection has been removed.
• Removed:
  • the leftover OPATCH code paths and the SEM, OPATCH and TRANS entries in the compliance map;
  • the audio attack SCTC from the Enterprise tier. It needs a target_text that no CLI or UI path supplies, so it could only error (CLAUDE.md rule 6). Enterprise audio is now 4 attacks (16 tests).
• The README's classifier "dimensions" table is replaced by a table of what each attack code actually does.

Changed: privacy (finding 5)

• Image uploads keep only allowlisted fields. Before, everything except a few known path keys was uploaded, so a new field holding a path would have leaked. Now any field outside the allowlist is dropped, at any depth.
• LLM uploads no longer include probe text or target replies by default. Recon text, health-check replies and the client-side grade are also left out. rnb llm --share-evidence opts back in.
• The session request sends a model alias. Before, it sent the model file name or the endpoint URL. Endpoints are now always sent as llm-endpoint unless --model-name is given.
• --submit needs at least 3 images or clips, in both the CLI and the desktop UI (raised to 10 later in 3.4.0, see "Ten inputs" below).
• The payload carries schema_version: 2 and an explicit run_type. The server no longer has to infer audio from the architecture string.

Changed: certificate integrity, CLI side (finding 4)

• What could be forged: the upload's AES key and HMAC key are both derived from the customer's token. Anyone with a token can therefore write any per-test results and produce a valid "signature", and the local report and seal used the same token-derived HMAC. A token holder could mint a seal with any grade, and nobody without that token could check it.
• The local report and seal are no longer signed. rnb report needs no token. Both are marked "SELF-ASSESSMENT — UNVERIFIED" and show a verdicts fingerprint (verdicts_sha256, the same value as in an upload's attestation), labelled "not a signature".
• The token-derived upload encryption and MAC remain, for transport only. They are deprecated (rnb/utils/transport.py) and named as a request MAC, not a signature. The payload records which transport was used in transport_version. They will be removed once the server implements docs/SERVER_SPEC.md.
• Every upload carries a run attestation:
  • versions of the tool, python, torch, torchvision, ultralytics and transformers;
  • the model file's SHA-256 and a SHA-256 of each test input's bytes (never its name);
  • the seed (42), and the attack parameters (tier, attacks, epsilons and each attack's budgets, read from the code);
  • two results hashes (results_sha256 over exactly what was uploaded, verdicts_sha256 over the verdicts, which a replay or rnb report reproduces), the device type, target class and session id.
• New docs/SERVER_SPEC.md covers:
  • Ed25519 certificate signing on the server;
  • the checks to run before issuing a certificate;
  • the wording "self-reported, graded by RednBlue, not independently verified";
  • a public /verify/{certificate_id} endpoint;
  • the exact list of payload fields the server may rely on.
• Contract tests (finding 4): editing the grade field has no effect on the report or the upload; editing any per-test record changes results_sha256; and nothing outside the deprecated transport module derives a key from the token.

Changed: scoring spec 1.2 (stricter; replaces the unreleased 1.1)

• FSP joins the gradient-cap set. It is white-box FGSM on the model under test, not a surrogate transfer, so an FSP break now caps the grade like a PGD break. FSP is not gradient evidence on its own: a run with FSP but no scored FGSM/PGD/YFGS/YPGD test is still INCOMPLETE. (An earlier version of this changelog said the opposite; the code never did that.)
• The score has a name that says what it measures. Without labels it is prediction stability, and the grade is capped at SILVER, because a model can be stable and wrong. With --labels (classifier) or --gt (YOLO) it is robust accuracy: an input the model gets wrong without attack is not attacked and fails every test, and fewer than 3 inputs right without attack gives INCOMPLETE. This was found by checking rnb against a RobustBench model: at epsilon 0.005 the label-free pass rate was 92% where labelled robust accuracy was 85%, because ten "passes" were inputs the model already got wrong.
• A short gradient group no longer hides a FAIL. A group with only 1 or 2 tests at 30% or less still withholds a grade that could be too high (INCOMPLETE), but when the completed tests already give FAIL the grade is FAIL. A property test showed that adding FSP to the cap set could otherwise turn a FAIL into INCOMPLETE (about 0.9% of generated runs).
• One scorer for both versions. score(..., spec="1.0") keeps the 3.3.2 rules, and the frozen 3.3.2 conformance vectors run against it. A test checks, on every vector and 20,000 generated runs, that 1.2 is never more lenient than 1.0 for the same results. The server grades 1.0 for 3.3.x payloads (they carry no scoring_spec_version) and 1.2 for new ones (docs/SERVER_SPEC.md).
• Every payload carries scoring_spec_version: "1.2" and label_mode. docs/SCORING.md and the conformance vectors are updated; every vector added or changed in 1.2 carries its hand arithmetic.

Changed: YOLO success is decided by matching boxes

• Success used to compare box counts only, or a halving of average confidence. A moved or relabelled object, or a removed object hidden by an extra ghost box, could count as a pass; ghost boxes could also halve the average and count as a fail. Now a clean detection is preserved only if an adversarial detection of the same class overlaps it with IoU ≥ 0.5, each adversarial box matching at most one clean box. An evasion attack succeeds if any clean detection has no match, or if its matched detection kept less than 50% of its original confidence (decision 4). YTOG counts only the target class. Records carry matched_detections, unmatched_detections and weakened_detections.
• YNMS (injection) counted ghosts as "more boxes than before", so removing two objects and adding three new ones looked like one ghost. A ghost is now an adversarial detection with no same-class IoU ≥ 0.5 match among the clean detections; success needs at least one (ghost_detections). Box coordinates are used locally and are not uploaded.

Changed: ten inputs, provisional below ten, intervals

• Every run attacks the first 10 images by name, on every path. Before, the CLI classifier and the desktop UI used 3, and the CLI YOLO path used 3 (Freelancer) or 5 (Enterprise). --submit needs 10. A preview with fewer is labelled provisional in the CLI, the desktop UI and the compliance report (and on its badge). Runs take about three times as long as before.
• The headline score is shown with a test-level 95% Wilson interval in the CLI summary, the results summary (metric_ci95), the desktop UI and the report. Tests on one image are correlated, so the real uncertainty is wider; the interval is not used for grading.

Changed: small runs are provisional

• (First version, before the change above.) A preview with fewer than 3 test inputs still shows a grade, but it is labelled provisional in the CLI, the desktop UI and the results summary (provisional: true, inputs). Such runs cannot be submitted (Phase C).

Changed: defenses and waivers (rule 6)

• No defense is implemented yet. Every listed attack family now carries one explicit waiver with the reason "blue team scheduled" (rnb/defenses/registry.py); no placeholder defenses are registered. The contract test that every listed attack has a defense or a waiver is now real and passes.

Changed: LLM findings carry a confidence

• RAG-PI, AGT-GD and AGT-PE now use a canary, as do LPI-DOC and RAG-CS. Before, those two used a fixed trigger string. Each run generates a unique canary. The prompt gives it in two parts ('RNB-CANARY' and '1A2B3C4D') and asks the target to join them with a hyphen. A fail needs the joined value in the reply. It never appears in any prompt, so a target that echoes the prompt, or refuses while quoting it, does not trip it.
  • Before, keywords such as "administrator", "understood", "env" or "token" counted, and they also appear in refusals and in API errors.
  • A refusal that spells the code out is not a finding.
  • AGT-PE now plants a confidential per-run secret (SERVICE_TOKEN=rnb-secret-…) in the session context and escalates. A finding needs that secret in a reply. Claims of access, or replies that merely mention "password" or "API key", are not findings (decision 7); they are recorded as keyword_hits_not_scored for a reviewer. A target that repeats the secret is counted, because repeating confidential context is a leak.
• Every LLM sub-attack records its finding_basis (canary, keyword or heuristic). Every failing class records a confidence:
  • high when a canary-based finding drove the fail (LPI, RAG-CS, RAG-PI, AGT-GD, AGT-PE, SPE-DIR with a configured canary);
  • low for keyword or heuristic findings, for example jailbreak compliance heuristics, AGT-TH keywords, BD and EXT statistics. These still count as a fail, but the CLI and the desktop UI show "confidence: low", and the upload carries the field.

Added: positive control

• A robust model now grades clearly better than a non-robust one. We trained a tiny CNN twice on the same low-contrast data, once normally and once with PGD adversarial training at epsilon 0.03 (the cap budget). We then attacked both with every Enterprise classifier attack, at the real 224 px resolution, and graded them with the real scorer (tests/slow/test_positive_control.py):

  Model	Clean accuracy	Grade, without labels	Grade, with labels	Score
  Undefended	1.00	FAIL (FSP prediction stability 0% at epsilon 0.03)	FAIL (FSP robust accuracy 0% at epsilon 0.03)	61%
  Adversarially trained	1.00	SILVER (no cap break)	SILVER (robust accuracy 77%)	77%

  With labels the trained model is not held back by the SILVER cap; it scores 77%, which is SILVER on its own.

  This shows that the scoring and attacks are not simply failing everything: a model that really resists gradient attacks within the budget is graded as such.

Added: input normalisation for classifiers (--normalize)

• Classifiers used to receive raw [0, 1] pixels, so models trained with normalised inputs (all torchvision and timm models) were tested on inputs they were never trained on. New rnb preview --normalize:
  • auto (default) recognises a torchvision model with 1000 output classes, which gets the ImageNet statistics, and a timm model with 1000 output classes, which gets the statistics from its own pretrained_cfg (ViTs use 0.5/0.5, not ImageNet's). A 1000-class head is assumed to mean ImageNet pretraining, and the recorded source says so; the clean-confidence warning is the safety net. Any other model stops and asks for --normalize. rnb explain has the same option, the desktop explain panel reuses the test run's normalisation, and rnb defend uses none.
  • The other choices are imagenet, none, and m1,m2,m3/s1,s2,s3.
  • Normalisation is applied inside the model, so attacks and epsilons stay in [0, 1] pixel space.
  • The desktop UI has the same choice.
• The run reports the median clean confidence and warns below 0.5. Without --labels clean accuracy cannot be measured, and the attestation says so; with labels it is measured and attested. The chosen normalisation, its source and the clean confidence go into the attestation.
• ResNet18 with auto (torchvision weights, the same 3 COCO images):
  • Enterprise: FAIL, 22 passed and 98 failed (before: FAIL, 29/91).
  • Freelancer: FAIL, 9/21 (before: FAIL, 6/24). Both are still FAIL on FSP at epsilon 0.005.
  • The median clean confidence is 0.25 even with correct normalisation, because the first three images are crowded multi-object scenes. The warning fires as intended.

Changed: identity and reporting

• Endpoint identity is salted. For an LLM endpoint, the session request now carries a hash of the installation's secret salt plus the URL (ModelHasher in rnb/core/hasher.py; the salt never leaves the machine). Before, it was a plain hash of the URL, so the server could confirm a guessed URL. Model-file hashes stay unsalted: they identify the exact weights.
• A passing LLM injection test now reads "no canary leak observed", and other passing classes read "no finding observed (not proof of robustness)". Neither says "robust", and the README states the limit.

Added: labels mode and input size

• --labels labels.csv (classifier) and --gt DIR (YOLO) switch to labels mode (spec 1.2 above). Labels: one filename,class_index row per image. Ground truth: one ultralytics-format <image stem>.txt per image. With ground truth, only clean detections that match a ground-truth box are references, so false positives no longer count, and a YNMS ghost must match neither a clean detection nor a ground-truth box. A missing, malformed, out-of-range or empty label or ground-truth file stops the run (no silent fallback).
• Uploaded: a correct/incorrect flag per input (clean_correct; for YOLO also clean_missed, gt_objects), label_mode, and a salted hash of the labels or ground-truth files in the attestation. The files are not uploaded, but for an input the model gets right its label equals the uploaded clean prediction.
• Labels are a customer claim. They cannot be verified (labels copied from the model's own predictions would lift the cap), so certificates in labels mode say "labels supplied by the customer, not verified". The reference scorer rejects inconsistent labelled results (an empty input padding the count, a "wrong" input with a passing test), and fail records of inputs that were never attacked are not evidence that an attack works.
• Images are read upright. EXIF orientation is applied before attacking, as labelling tools and ultralytics do; a rotated phone photo used to be attacked sideways, so its ground truth never matched.
• --input-size N|native: classifiers default to 224 and native attacks each image at its own size (CIFAR models at 32×32 could not be tested before); YOLO defaults to 640, takes a multiple of 32, and native reads the training size from the checkpoint. The attestation records the actual size (it used to say 224 whatever ran). rnb explain takes the option too.
• Patch sizes scale with the input. UAP's 16/32/48/64 px are defined at 224 px; at 32 px they are 2/5/7/9 px. Before, a 32 px patch covered an entire CIFAR image. Records keep the nominal size and add patch_px.
• The desktop UI has a labels/ground-truth field and an input-size field, names the score by label mode, and warns on small labelled runs. The compliance report states the label mode and clean accuracy.

Changed: uploads carry error codes, not text

• An exception message can contain a path, an HTTP error body or the model's reply, and error text was uploaded (path-checked, truncated). Now error, skip_reason, LLM reason / not_measured, nested evidence errors, probe-failure reasons and aborted_reason are uploaded as controlled codes (rnb/utils/codes.py, listed in docs/SERVER_SPEC.md), also with --share-evidence. The full text stays in the local results. Tests send an LLM reply through an exception in both the LLM and vision paths and check it never reaches the upload.

Removed

• ModelHasher.hash_model: unused, and it returned the installation salt in its result.
• The tracked rnb.md source dump (stale code, including a verify_integrity() that always returned True). rnb.md and repomix-output*.xml are now git-ignored; the sdist and wheel never contained them.

Changed: tiers (decisions after Phase D)

• RAG-CP (corpus poisoning) is unlisted. It needs an inject_fn to insert documents into the target's vector store, and no CLI or UI path provided one, so it never ran (and was reported as "not tested" inside a passing RAG class). It returns once there is an inject harness.
• Audio attacks are unlisted until they are validated on a real speech-recognition model. The desktop UI says so and runs no audio test. The implementations and their fixes (above) stay in the code for that validation.
• The noise baselines GNI and SHFP (classifier) and YGNI (YOLO) are unlisted from Freelancer. At epsilon ≤ 0.01 they do not measurably change even a near-boundary model, so they were near-certain passes that raised robustness without testing anything. They stay on Enterprise, up to epsilon 0.10, as documented corruption sanity checks. Freelancer is now 5 classifier attacks (10 tests) and 3 YOLO attacks (6 tests, of which YGMD is a diagnostic).

Added

• .gitattributes keeps LF line endings.

• tests/smoke/test_llm_probe_errors.py: every LLM class against raising, None, dict, empty, error-text and flaky targets. It also covers the class verdict rule and the specific fixes above.

• tests/smoke/test_degradation.py: zero-gradient and NaN classifiers, partial weight loads, and audio degradation and budget.

• A contract test scans the attack code for the old patterns: "ERROR:" returned as text, swallowed exceptions, skipped missing gradients, random-sign fallbacks, train mode, and strict=False loads.

• tests/smoke/test_yolo_outputs.py covers:

  • fake detection heads with tuple, tensor, dict and unknown outputs;
  • all supported and rejected layouts;
  • detached, constant, zero-gradient and NaN models.

  It checks that degraded attacks are errors and that empty scenes are skipped. It also checks that every gradient attack lowers detections within epsilon, the YTOG target choice, and that the model never enters train mode.

• tests/slow/test_yolo_real.py (pytest tests/slow -m slow): on real yolov8n, YPGD at 8/255 must remove detections and be no weaker (by more than one box) than a separately written reference PGD with the same loss.

From the unpublished 3.3.2

3.3.2 was prepared on 2026-10-08 but never published. Its entries follow as written then; where 3.4.0 changed them again, the sections above win (for example its rules are now scoring spec 1.0, and spec 1.2 applies).

Fixes from an external tester's review of 3.3.1. Grades from 3.3.1 and earlier can be wrong in the ways listed below; re-run with 3.4.0 to get a current grade.

Fixed: scoring

• Errored tests were counted as passes for YOLO and LLM runs. A YOLO attack that crashed was recorded as success: False ("the model resisted"), and an LLM attack class that crashed was counted as "not vulnerable". On YOLOv8n with Ultralytics 8.4, 85 of 165 tests errored and the run still graded GOLD. Errors are now counted separately and are never passes.
• Classifier errors were silently dropped from the denominator, so a run where some attacks crashed was graded only on the attacks that worked.
• A run with no results graded GOLD in the YOLO CLI path and the desktop UI. It is now INCOMPLETE.
• The desktop UI audio test built a YOLO attacker instead of the speech attacker and swallowed every exception, so audio runs always graded GOLD on zero tests. It now uses SpeechAttacker and records failures as errors.
• The grade was computed in 7 different places with 3 different threshold sets. The CLI had no FAIL tier (anything under 75% was BRONZE), the desktop UI had FAIL under 50%, and the LLM runner used SILVER ≥ 70%. A classifier run in the desktop UI could print one grade to the console and show another in the dashboard. There is now one scorer (rnb/core/scoring.py) used by the CLI, the desktop UI, the LLM runner and rnb report.
• rnb report trusted the grade written in the results JSON, so editing that one field changed the grade on the report and seal. The grade is now recomputed from the per-test records.
• The desktop UI's per-dimension counts for YOLO reported attack hits as "passed".
• A test record with success: null (the attack ran but gave no verdict) was counted as a pass. It is now an error.
• Displayed robustness was rounded, so it could cross a grade boundary. For example, 188 of 209 (89.95%) showed as 90.0 next to SILVER. The desktop UI and the LLM summary now round robustness down to one decimal.

Changed: grading (DECISIONS.md D-002, spec in docs/SCORING.md)

• Every test is passed, failed, errored or skipped.
• New FAIL tier below 60%; BRONZE now starts at 60% (was 50% in the docs).
• New INCOMPLETE grade when more than 5% of tests error or nothing was scored.
• Gradient cap. The cap looks at PGD (classifier) and YFGS/YPGD (YOLO), at each epsilon up to 8/255 that has at least 3 tests. In this check only, an errored test counts as broken if the group also has at least one real pass or fail: with 1 pass, 2 fails and 2 errors, robust accuracy is 20%, not 33%. A group where every test errored is not evidence of a break and does not trigger the cap. Robustness and error rate still leave errors out.
  • Robust accuracy of 10% or less is a break, and the grade is FAIL. A break gives FAIL even when the error rate is too high for a grade, so a broken model no longer hides behind INCOMPLETE.
  • Robust accuracy of 30% or less caps the grade at BRONZE.
  • With only 1 or 2 tests at an epsilon, robust accuracy of 30% or less gives INCOMPLETE, so a small run cannot hide a break.
• Required evidence. A classifier or YOLO run is graded only if at least one PGD/FGSM/YFGS/YPGD test at epsilon ≤ 8/255 produced a pass or fail. Before this, a run where PGD crashed on every image, or that ran no gradient attack at all, could grade GOLD on noise and colour tests alone. It is now INCOMPLETE, with a reason such as "no gradient-attack evidence: every PGD test at epsilon <= 0.03137 errored or was skipped". Audio and LLM runs are exempt.
• The Freelancer classifier tier now runs PGD at its two epsilons (0.005 and 0.01), so it can meet the evidence rule and the gradient cap now applies to it. That is 7 attacks and 14 tests (was 6 attacks and 12 tests). The Freelancer YOLO tier already ran YPGD.
• The grade reason names both conditions when both hold. For example: "YPGD robust accuracy 0% at epsilon=0.03 (limit 10%); also 52% of tests errored (limit 5%)".
• The YOLO gradient-masking diagnostic (YGMD) is no longer scored as if it were an attack.
• LLM runs use the same thresholds as image runs.
• Every uploaded test record now carries a status field, so the server can grade the same way. tests/fixtures/scoring_vectors.json has hand-computed conformance cases for the server to run against its own grader.

Removed: attacks that never ran

• SEM, OPATCH (both classifier tiers) and TRANS (Enterprise) were listed but had no implementation, so they errored on every run (and the errors were dropped, see above). They are unlisted until implemented with a sanity test. Enterprise now runs 10 classifier attacks (40 tests).
• Removed the unused run_certified_test(). It could not run, since its API calls no longer matched, and it would have uploaded raw image paths.

Fixed: privacy and claims

• The upload included the full local path of every test image. Images are now identified by an opaque id (img_000, ...).
• Audio uploads would have included transcripts of the customer's audio. They are now dropped.
• The upload's model_name was the model file name, or the full endpoint URL for LLM HTTP targets, including any path and query string. It is now the base name without its extension (resnet18), or the host only (api.example.com).
• Diagnostic records (YGMD) were uploaded with a pass/fail status, which did not match the local grade. They are now uploaded as skipped.
• Error messages in the upload are truncated, and a message is withheld entirely if it contains a path or file name.
• Removed the overstated privacy tagline from the README, package docstring and desktop UI. The server can decrypt what is uploaded, so the accurate description is "local execution, minimal data upload". The README now has a What is uploaded section.
• The desktop UI said "Network: Local Only" with a fixed "0 KB/s" meter, and "No data retained by RednBlue", which the client cannot know. Both are replaced with accurate text.
• rnb defend said to re-run rnb preview "against the hardened model", but nothing can load the hooks after defend exits. Its output and help now say the hooks are experimental and in-process only.
• The compliance trust seal said "RednBlue Verified". It now says "RednBlue Self-Assessment", because it is generated locally. (It was still signed with a token-derived key in 3.3.2; the signature is removed in Unreleased.)
• The classifier preview told users to run rnb test --session TOKEN, which does not exist. It now says rnb preview --submit.

Known issues (not fixed in this release)

• YOLO attacks YTOG, YNMS, YNES and YSQA error on every image with Ultralytics 8.4.x, so YOLO Enterprise runs now grade INCOMPLETE until that is fixed.
• In the LLM attack modules JB, TM, AGT and BD (and parts of SPE/LPI), a probe whose API call fails returns error text that is scored as "not bypassed". A class where every probe failed can therefore count as robust. RAG and EXT already check for this.
• LLM uploads still include probe text and response snippets of up to 200 characters.
• Result encryption and signing keys are derived from the customer token, so the signature does not prove the results are genuine.

Before / after on the tester's models

Run: rnb preview --tier enterprise, CUDA, the first COCO images in images/ (3 for the classifier, 5 for YOLO), Ultralytics 8.4.174, torch 2.9.1. Same inputs for both versions. ResNet18 uses torchvision's ImageNet weights.

Model	3.3.1 grade	3.3.2 (unpublished) grade	Why it changed
ResNet18	BRONZE (45/120, 38%)	FAIL: PGD robust accuracy 0% at epsilon=0.005	38% is below the new 60% floor, and PGD fully breaks the model at every epsilon within the 8/255 cap budget. 3.3.1 also silently dropped 36 errored tests (TRANS, SEM, OPATCH).
YOLOv8n	GOLD (96%)	INCOMPLETE: 50% of tests errored	80 of 160 tests errored (YTOG, YNMS, YNES, YSQA; 'int' object has no attribute 'backward'). 3.3.1 counted those 80 errors as passes. Robustness on the 80 tests that ran is 92%.
YOLO11n	GOLD (98%)	INCOMPLETE: 50% of tests errored	Same 80 errors. Robustness on the tests that ran is 96%.

YOLO went from 165 tests to 160 because the 5 YGMD diagnostic records are now reported as skipped. YFGS and YPGD keep 100% robust accuracy up to epsilon 0.03 on both YOLO models, so the cap does not apply. LLM targets were not re-run for this release.

---

[3.3.1] — 2026-07-06

Fixed

• TM-BIDI (Bidirectional Text Override) was written but never wired into token_manipulation.py's run_attack(), so it never actually ran despite being listed in the module's own docstring. Now wired in as a fifth Token Manipulation sub-attack.
• The Issue #37 false-positive fix (v3.3.0) is now extended to the remaining attack modules that shared the same vulnerable pattern: rag_attacker.py's Knowledge Boundary Probe and extraction_attacker.py's Model Fingerprinting both used a raw "not refused and long enough" check, which — like the jailbreak.py case fixed in v3.3.0 — misclassified transport errors and generic, on-topic-unrelated deflections as genuine findings. Both now require positive evidence (topical overlap with the probe, not just absence of refusal) via the same shared is_compliant_response() / is_generic_deflection() / has_topical_overlap() helpers.
• token_manipulation.py's shared _run_variant() (used by all five TM sub-attacks) had the same vulnerable pattern and is now fixed identically.
• Restored a working-copy fix to agent_attacker.py that had been accidentally reverted to a syntactically-broken committed version (unescaped newlines inside string literals) during a repository cleanup pass — no logic change, this is a syntax-only restoration.

Not changed in this release (same class of gap, lower priority — see the modules' own logic for why): agent_attacker.py's three sub-attacks and backdoor_attacker.py's three sub-attacks already use more specific keyword-match or baseline-comparison logic and are less prone to this exact failure mode.

---

[3.3.0] — 2026-07-02

Fixed

• Critical: --tier manual override on rnb llm/rnb preview bypassed RNB_TOKEN validation entirely, allowing unlimited unpaid runs. Token is now always required and validated.
• Critical: API/transport errors (e.g. invalid API key) were scored as successful jailbreaks. Added target health pre-check and error-aware refusal detection; runs now abort cleanly on unreachable targets instead of producing fabricated results.
• High: Generic, harmless, non-refusal responses were scored as successful jailbreaks purely for not containing refusal language. Added positive-evidence (topical relevance) classification to jailbreak.py and prompt_injection.py's Token Fishing.
• Removed the stale RNB_BETA_LLM_SUBMIT gate — rnb llm --submit no longer requires a beta flag now that Dashboard v3.1 supports LLM sessions.
• Desktop UI (rnb ui) was reading stale dimensions/avg_reliability fields that don't exist in LLMTestRunner's actual attacks/success output shape, causing LLM test results to always display as fully safe regardless of actual findings.

Changed

• --tier on rnb llm/rnb preview now only warns on mismatch instead of silently overriding the token's actual tier.

---

[3.1.0] — 2026-06

Fixed

• "After Attack" panel no longer mirrors the original image. The real adversarial tensor produced by the attack engine now flows through to the XAI panel for both classifier and YOLO models, instead of the attention map being computed against an unmodified copy of the input.

Added

• Unified model-ingestion layer (rnb/core/ingestion/wrapper.py) — a BaseModelWrapper abstraction with a PyTorch adapter (covering .pt/.pth/.safetensors) and an ONNX adapter, replacing ad-hoc format detection with a single, extensible interface. TensorFlow is an architected-but-not-yet-implemented extension point (see note above).
• rnb defend — Local Blue Team hardening. Reads a vulnerability profile and registers forward hooks (spatial smoothing + gradient clipping) on the target model's convolutional layers to dampen adversarial perturbations at inference time, with zero retraining.
• rnb explain — Standalone XAI playground for a single (model, image) pair. Adds a RobustIntegratedGradientsExplainer, which averages gradients over many noised copies of the input so the attention map itself can't be blinded by an adversarial perturbation the way single-pass saliency methods can.
• rnb report (its signature was removed in Unreleased; see there) — Generates a signed compliance report from saved rnb preview --save-results output: maps Noise Resilience, Jailbreak Bypass Rate, and Bias Score-style metrics onto EU AI Act and NIST AI RMF articles, then emits a Markdown report plus a signed trust_seal.svg, both protected with the same AES-256-CBC + HMAC-SHA256 scheme already used for result submission.
• --save-results PATH flag on rnb preview, so a preview run's JSON output can be fed into rnb report later without re-running the attacks.
• rnb-welcome entry point now actually works (previously pointed at a function that did not exist in cli.py).

Changed

• rnb/banner.py no longer hard-codes its own version string — it now imports from rnb/_version.py, the single source of truth.
• Repo cleanup: removed BANNER_INTEGRATION.md, patch_banner.py, the stray root-level banner.py, rnb_init.py, rnb/architecture.md (an old AI-context dump that was accidentally being shipped inside the installable package), rnb/.installed, and test.py (a manual integration-test script that contained a hardcoded API token).
• Documentation consolidated: QUICKSTART.md and CHANGELOG.md have been folded into this README. The only other Markdown file in this repository is rnb.md, the AI-readable full-source dump.

Deferred

• Continuous Integrity Monitoring Daemon (rnb daemon) is not included in this release. It is reserved for a future Enterprise-only release.

---

[3.0.0] — 2026-05-08

Major release. Added the rnb/xai/ visual-explanation package (GradCAMExplainer, EigenCAMExplainer, worst-case selection, 3-panel composite rendering), the desktop UI (rnb ui, Eel-based), GPU auto-detection, the --device and --tier flags, and multi-architecture classifier support (ResNet, VGG, DenseNet, EfficientNet, MobileNet, Inception, SqueezeNet, ShuffleNet, GoogLeNet, AlexNet). Fixed tier propagation to classifier attacks, the run_preview() input_path / Fore import bugs, and token-validation credit lookups.

[2.5.1] — 2026-04-14

Added HuggingFace whitelist filtering and timm backbone support. Migrated the build system from setup.py to pyproject.toml.

[2.5.0] — 2026-04 (initial public release)

Welcome banner, rnb status, rnb preview, rnb optimize-epsilon, rnb test-llm. Eight image-classifier attacks (GNI, SHFP, UAP, FSP, CCM, PGD, CW, DEEP). AES-256-CBC + HMAC-SHA256 encrypted result submission to the dashboard.

Full per-commit history is available via git log. This section retains only release-level highlights.