Skip to main content

LLM Endpoint Red-Team CLI (prow-audit)

Zero-dependency Python 3.10+ command-line utility for endpoint preflight verification, CI/CD release gating, and side-by-side forensic .zip regression diffs paired with Prow // Red-Team.

Benchmark Integrity Notice: To prevent training-corpus contamination, all 41 multi-file and CAD blueprint suites inside Prow // Red-Team remain strictly closed-source. The bundled archives in samples/ use synthetic [SYNTHETIC_SAMPLE_REDACTED] records to demonstrate telemetry schemas and CI exit codes.


Quick Start (Zero Third-Party Dependencies)

Runs on pure Python 3.10+ standard library (urllib.request, zipfile, json, argparse)—no pip packages required in ephemeral CI runners:

git clone https://github.com/ProwLabsIO/llm-endpoint-red-team-cli.git
cd llm-endpoint-red-team-cli

# Optional editable install to expose `prow-audit` on PATH:
pip install -e .

1. Endpoint Preflight (prow-audit preflight)

Before running a full 41-suite multi-file and CAD blueprint sweep on red-team.prowlabs.io, verify that your OpenAI-compatible (/v1/chat/completions) or Ollama (/api/generate) endpoint accepts Bearer auth, handles lossless PNG vision inputs, and does not silently return HTTP 200 OK with empty message.content under reasoning-token exhaustion:

export TARGET_API_KEY="sk-..."

python3 prow_audit.py preflight \
  --endpoint https://api.openai.com/v1 \
  --model gpt-4o \
  --check-vision \
  --cfaa-attest

2. CI/CD Forensic .zip Release Gate (prow-audit gate)

Every completed run on red-team.prowlabs.io outputs a 6-file prow_redteam_{job_id}.zip archive (executive_audit_summary.pdf, audit_report.md, graded_audit.json, audit_scorecard.csv, trap_level_breakdown.csv, audit_summary.txt). Enforce minimum pass and trap-escape rates in GitHub Actions (automatically writes a Markdown scorecard to $GITHUB_STEP_SUMMARY):

python3 prow_audit.py gate samples/prow_redteam_sample_candidate.zip \
  --min-pass-rate 90 \
  --min-trap-escape 95 \
  --max-errors 0 \
  --require-suites CTX-001,EXF-001,HAL-001

3. Side-by-Side Regression Comparator (prow-audit diff)

Diff two forensic .zip bundles across model upgrades, system-prompt changes, or guardrail patches (exit 2 when --fail-on-regression detects any PASS -> FAIL transition):

python3 prow_audit.py diff \
  samples/prow_redteam_sample_baseline.zip \
  samples/prow_redteam_sample_candidate.zip \
  --fail-on-regression

4. Headless Private Core Dispatch (prow-audit run)

For private VPC or X-API-Key orchestrator gateways running api-red-team-core (self-serve $49 Tactical Pass and $149/mo Pro Pass holders run audits via red-team.prowlabs.io):

python3 prow_audit.py run \
  --core-url http://127.0.0.1:8000/api/v1/audit \
  --prow-api-key "$PROW_API_KEY" \
  --target-endpoint https://api.openai.com/v1 \
  --target-model gpt-4o \
  --cfaa-attest

  1. TD-001: Handling Reasoning-Token Exhaustion & Empty Content Payloads in OpenAI-Compatible Audit Pipelines
  2. TD-002: Why Vision LLMs Hallucinate Clearances on Mirrored CAD Blueprints
  3. TD-003: Beyond Jailbreaks: Measuring Executive-Pressure Sycophancy and Statutory Precedence Failures

Authorized Endpoint Policy (18 U.S.C. § 1030 CFAA)

prow-audit and Prow // Red-Team are strictly for evaluating endpoints and API keys you own or hold explicit written authorization to test. Active network commands (preflight, run) enforce --cfaa-attest.

© 2026 Prow Labs · MIT License (CLI wrapper only).

Metadata

Release files for llm-endpoint-red-team-cli 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-endpoint-red-team-cli 1.0.0
File Size Uploaded Provenance Actions
llm_endpoint_red_team_cli-1.0.0.tar.gz 9.7 kB With provenance Details for llm_endpoint_red_team_cli-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llm-endpoint-red-team-cli 1.0.0
File Interpreter ABI Platform Actions
llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl Python 3 none any Details for llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl

Total release size: 20.3 kB

Release files / llm_endpoint_red_team_cli-1.0.0.tar.gz

Download URL llm_endpoint_red_team_cli-1.0.0.tar.gz
Size 9.7 kB
Tags Source
SHA-256 checksum
How to use checksums
8b70a58a9e29b0827e82c954c3924ef2786d3d56afe04c3aec3751b0eff516dd
BLAKE2b-256 checksum
How to use checksums
024a250dcc263480808232837214f92d14415f3ec41c6f24969c18ed5af09f94
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.

Transparency log

Release files / llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl

Download URL llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl
Size 10.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f85cd8f06deb5e472648cc2b2d37b27b5c197bd4f6e4bea792b67ade7ce09c62
BLAKE2b-256 checksum
How to use checksums
4524e93987a1ffafa63cc77d7e61fce8c7d1b93edec9fbeaeb610f0477d72f87
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.

Transparency log

Security for llm-endpoint-red-team-cli 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for llm_endpoint_red_team_cli-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page