LLM Endpoint Red-Team CLI (prow-audit)
Zero-dependency Python 3.10+ command-line utility for endpoint preflight verification, CI/CD release gating, and side-by-side forensic .zip regression diffs paired with Prow // Red-Team.
- Live Self-Serve Console (41 Human-Built Suites): https://red-team.prowlabs.io
- Studio Architecture & Specs: https://prowlabs.io
- Engineering Teardowns: https://prowlabs.io/teardowns/
Benchmark Integrity Notice: To prevent training-corpus contamination, all 41 multi-file and CAD blueprint suites inside Prow // Red-Team remain strictly closed-source. The bundled archives in
samples/use synthetic[SYNTHETIC_SAMPLE_REDACTED]records to demonstrate telemetry schemas and CI exit codes.
Quick Start (Zero Third-Party Dependencies)
Runs on pure Python 3.10+ standard library (urllib.request, zipfile, json, argparse)—no pip packages required in ephemeral CI runners:
git clone https://github.com/ProwLabsIO/llm-endpoint-red-team-cli.git
cd llm-endpoint-red-team-cli
# Optional editable install to expose `prow-audit` on PATH:
pip install -e .
1. Endpoint Preflight (prow-audit preflight)
Before running a full 41-suite multi-file and CAD blueprint sweep on red-team.prowlabs.io, verify that your OpenAI-compatible (/v1/chat/completions) or Ollama (/api/generate) endpoint accepts Bearer auth, handles lossless PNG vision inputs, and does not silently return HTTP 200 OK with empty message.content under reasoning-token exhaustion:
export TARGET_API_KEY="sk-..."
python3 prow_audit.py preflight \
--endpoint https://api.openai.com/v1 \
--model gpt-4o \
--check-vision \
--cfaa-attest
2. CI/CD Forensic .zip Release Gate (prow-audit gate)
Every completed run on red-team.prowlabs.io outputs a 6-file prow_redteam_{job_id}.zip archive (executive_audit_summary.pdf, audit_report.md, graded_audit.json, audit_scorecard.csv, trap_level_breakdown.csv, audit_summary.txt). Enforce minimum pass and trap-escape rates in GitHub Actions (automatically writes a Markdown scorecard to $GITHUB_STEP_SUMMARY):
python3 prow_audit.py gate samples/prow_redteam_sample_candidate.zip \
--min-pass-rate 90 \
--min-trap-escape 95 \
--max-errors 0 \
--require-suites CTX-001,EXF-001,HAL-001
3. Side-by-Side Regression Comparator (prow-audit diff)
Diff two forensic .zip bundles across model upgrades, system-prompt changes, or guardrail patches (exit 2 when --fail-on-regression detects any PASS -> FAIL transition):
python3 prow_audit.py diff \
samples/prow_redteam_sample_baseline.zip \
samples/prow_redteam_sample_candidate.zip \
--fail-on-regression
4. Headless Private Core Dispatch (prow-audit run)
For private VPC or X-API-Key orchestrator gateways running api-red-team-core (self-serve $49 Tactical Pass and $149/mo Pro Pass holders run audits via red-team.prowlabs.io):
python3 prow_audit.py run \
--core-url http://127.0.0.1:8000/api/v1/audit \
--prow-api-key "$PROW_API_KEY" \
--target-endpoint https://api.openai.com/v1 \
--target-model gpt-4o \
--cfaa-attest
Related Engineering Teardowns
- TD-001: Handling Reasoning-Token Exhaustion & Empty Content Payloads in OpenAI-Compatible Audit Pipelines
- TD-002: Why Vision LLMs Hallucinate Clearances on Mirrored CAD Blueprints
- TD-003: Beyond Jailbreaks: Measuring Executive-Pressure Sycophancy and Statutory Precedence Failures
Authorized Endpoint Policy (18 U.S.C. § 1030 CFAA)
prow-audit and Prow // Red-Team are strictly for evaluating endpoints and API keys you own or hold explicit written authorization to test. Active network commands (preflight, run) enforce --cfaa-attest.
© 2026 Prow Labs · MIT License (CLI wrapper only).
Metadata
Release files for llm-endpoint-red-team-cli 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_endpoint_red_team_cli-1.0.0.tar.gz | 9.7 kB | With provenance | Details for llm_endpoint_red_team_cli-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl |
Total release size: 20.3 kB
Release files / llm_endpoint_red_team_cli-1.0.0.tar.gz
| Download URL | llm_endpoint_red_team_cli-1.0.0.tar.gz |
|---|---|
| Size | 9.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8b70a58a9e29b0827e82c954c3924ef2786d3d56afe04c3aec3751b0eff516dd
|
|
BLAKE2b-256 checksum How to use checksums |
024a250dcc263480808232837214f92d14415f3ec41c6f24969c18ed5af09f94
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.
Transparency logRelease files / llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl
| Download URL | llm_endpoint_red_team_cli-1.0.0-py3-none-any.whl |
|---|---|
| Size | 10.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f85cd8f06deb5e472648cc2b2d37b27b5c197bd4f6e4bea792b67ade7ce09c62
|
|
BLAKE2b-256 checksum How to use checksums |
4524e93987a1ffafa63cc77d7e61fce8c7d1b93edec9fbeaeb610f0477d72f87
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.
Transparency logSecurity for llm-endpoint-red-team-cli 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for llm_endpoint_red_team_cli-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.