SkillWitness CI Gate

SkillWitness is a Python 3.12 CLI for producing repeatable security and compliance reports for AI skills, hooks, helper tools, and agent extensions before they are merged into a managed skills repository. It scans local folders or git URLs, applies a policy profile, and emits human and machine-readable reports for CI review.

Copyright 2026 1stIndeXR. Licensed under Apache-2.0; see NOTICE. Status: pre-1.0.

The product requirements are defined in MD/skillwitness-ci-gate-prd.md, the implementation contract in MD/skillwitness-ci-gate-spec.md, and the delivery sequence in PM/skillwitness-roadmap.md.

How This Differs From Other Skill Scanners

Several scanners now detect risky patterns in AI skill and MCP packages. This one is built around the review record, not just the detection list:

• Organization-owned policy, not built-in verdicts. Approval decisions live in your approved-tools.yaml and registry overlay, versioned in your repo. The validator applies them; it does not decide for you what is acceptable.
• Capability evidence survives approval. An approved MCP service or local command no longer creates an approval violation, but it still contributes to the skill's L0-L5 behavioral classification.
• Signed registry overlays. Per-skill classification, ownership, expected license, approval owner, and approval expiry come from an overlay that can be digest- or cosign-signed, so a reviewer can prove which policy produced a result.
• Honest scan coverage. Every scanner reports success, skipped, unavailable, or failed with a reason. A pass with three unavailable scanners is visibly different from a clean pass. Silent skipping is treated as a bug.
• Auditable suppressions. Suppressions are explicit, scoped, and appear in the report while keeping the original evidence count visible.
• Reviewer and machine surfaces. Markdown for humans, plus SARIF, JUnit, JSON, and CycloneDX for code scanning, CI test surfaces, automation, and SBOM consumers.
• Distinguishable exit codes. 0 pass, 1 policy fail, and dedicated non-zero codes for gate misconfiguration, so a broken pipeline never looks like a rejected skill.

Detection rules are the commodity layer here and are deliberately shallow compared to dedicated LLM-assisted scanners. Running this alongside one of those is a reasonable setup: they find, this one records the decision.

Security Review Summary

This tool is intended to run as a CI approval gate for AI skills and agent extensions before they are merged into a managed skills repository. Its purpose is to make review evidence repeatable: it inspects submitted skill folders, applies a named risk profile and dependency allow-list, and produces reports that show whether the submission passed policy, which findings require review, and which scanner/tool coverage was available during the run.

The validator is read-only with respect to submitted skills. It reads the target directory, optional metadata overlays, optional allow-list policy files, and local git state for diff scans; it writes only the explicitly requested report artifacts. It must not create, normalize, or patch files inside third-party or author-submitted skill packages. For local scans, --output, --output-dir, and --evidence-output are rejected when they point inside the scanned target so CI evidence cannot accidentally modify the reviewed package.

The main security checks are:

• structure and metadata validation for SKILL.md
• declared runtime behavior, permissions, hooks, and filesystem access
• local tools, MCP servers, external domains, URLs, scripts, binaries, package manifests, and dependency allow-list violations
• telemetry declarations and disable mechanisms
• prompt-injection, obfuscation, dangerous command, and risky script patterns
• secret scanning when supported secret scanners are installed
• license and dependency-license evidence, including conflict detection
• known-vulnerability checks for package manifests and lockfiles
• upstream health checks when a source URL is declared
• signature and SLSA checks when profile policy and binary inputs make them applicable
• suppressions, registry overlays, scoring, and final pass/fail enforcement

The validator integrates with CI by returning exit code 0 when the scan passes and exit code 1 when policy fails. Policy/configuration problems use dedicated non-zero exit codes so CI can distinguish "the skill failed review" from "the gate was misconfigured." Reports are available as Markdown for reviewers, JSON for automation, SARIF for code-scanning systems, JUnit for CI test surfaces, and CycloneDX for SBOM-oriented consumers.

The tool has limited permissions by design. In the normal local-directory CI path it needs read access to the skills repository, read access to policy files, execute access to optional scanner binaries on PATH, and write access to CI artifact storage. It does not need write access to the submitted skill folders. Network access is not required for purely local rule checks, but it is required when scanning a git URL, running OSV-Scanner, running OSSF Scorecard, or using external scanner installations that fetch their own data. CI operators should make those network egress points explicit in the build environment.

Missing optional tools are reported in scan coverage instead of being hidden. The Markdown report includes a Scan Coverage table with one row per scanner or external tool, its status, finding count, and skip/failure reason. A green tool status means that scanner completed; it does not mean the skill has no findings.

Tools And Integrations

Most policy logic is built into this repository. Third-party tools are used for specialized evidence where mature scanners already exist, such as secrets, license detection, dependency vulnerabilities, static analysis, upstream health, signatures, and SBOM generation. Optional external tools are invoked only when they are applicable and available; missing, skipped, unavailable, and failed tool runs are shown in report scan coverage.

Built-In Scanners And Policy Logic

External Scanner Tools

CI Surfaces And Runtime Libraries

Install

Released versions are published to PyPI. Pin an exact version in CI so the gate cannot change under a build:

python -m pip install "skillwitness==0.1.0"

Every tagged release also attaches the sdist, the wheel, and a SHA256SUMS file to its GitHub Release. Download all three and run sha256sum --check SHA256SUMS when your supply-chain policy requires verifying the artifact you install.

Development

Install the project with development dependencies and run the local checks:

python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev]"
.venv/bin/ruff check .
.venv/bin/mypy
.venv/bin/pytest

The main local verification set is:

.venv/bin/ruff check src tests
.venv/bin/mypy src
.venv/bin/pytest -q

Usage

The scanner is a read-only approval gate for submitted skill packages. It reads the target and writes only the requested report artifacts outside that target. Do not use scan runs to create, normalize, or patch files inside third-party or author-submitted skill directories; remediation changes should happen in the source MR, and reviewer-owned artifacts should live in CI artifact storage or a separate review/policy workspace. For local directory scans, --output and --output-dir are rejected when they point inside the scanned target.

Behavioral gate severity is based on file role. Source files, scripts, manifests, configuration, SKILL.md, and files under an operational references/ directory can provide executable or policy-gated evidence. Ordinary documentation such as README and changelog files contributes LOW inventory findings for URLs, install examples, and telemetry descriptions, but does not create dependency, network, execution, or telemetry blockers by itself. Verified credentials and private keys remain content-based blockers regardless of file type.

Scan a local skill folder:

.venv/bin/skillwitness scan tests/fixtures/skills/well-formed

Scan with the stricter CI profile:

.venv/bin/skillwitness scan ./skills/toolctl --profile company-strict

Scan a git remote directly, optionally pinned to a branch, tag, or commit:

.venv/bin/skillwitness scan https://github.com/example/prompt-skill
.venv/bin/skillwitness scan https://github.com/foo/bar --ref abc123def456

Scan only files changed against a git base ref:

.venv/bin/skillwitness scan ./skills --diff origin/main

Produce one full report with aggregate and per-skill findings:

.venv/bin/skillwitness scan ./skills --output skillwitness-report.md
.venv/bin/skillwitness scan ./skills --format json --output skillwitness-report.json

Supported --format values are markdown, json, sarif, junit, and cyclonedx. JSON report output has a published contract in examples/report-schema.json, a JSON Schema 2020-12 document generated from the report model; validate report files against it, or call report_json_schema() from skillwitness.schemas.report. Use --target-kind auto|skill-package|source-repository to override the profile's target-kind setting for one scan. An explicit CLI value has precedence over the profile. This option does not change normalization.mode.

Normalization mode is profile-only: legacy, shadow, or enforced. CI cannot override it with a command argument. Every report records the resolved mode. Use --evidence-output PATH to write complete raw evidence as stable JSON Lines; the artifact contains identifiers and locations but no descriptions, snippets, or detected values.

Bundled profiles use shadow, so legacy gate decisions remain authoritative while normalized groups and metrics are available for evaluation. For an organization rollout, copy the selected profile into reviewed policy configuration and follow the rollout checklist. Rollback is the same profile-only change from enforced to shadow; use legacy only as a temporary compatibility setting. See the evaluation and release notes for the measured release evidence and counter migration.

With automatic detection, a root SKILL.md and no source-repository marker selects skill-package. Missing or mixed evidence selects the safer source-repository context. JSON reports include the resolved kind and stable reason codes.

Write split Markdown reports for a skills folder:

.venv/bin/skillwitness scan ./skills --output-dir skillwitness-reports

The output directory contains index.md plus one Markdown report per skill.

CI Integration

Templates for five CI systems ship under examples/:

• examples/github-actions/skillwitness.yml
• examples/gitlab-ci/skillwitness.yml
• examples/teamcity/skillwitness.xml
• examples/jenkins/Jenkinsfile
• examples/azure-pipelines/skillwitness.yml

See the portable integration guide for the shared execution contract, container usage, private Git authentication, and tracker-neutral approval evidence.

Attach the skills repository as the VCS source, publish the generated Markdown and SARIF reports as build artifacts, and point the scan at your organization's dependency policy:

skillwitness scan ./skills --profile company-strict --approved-tools ./policies/example-org/approved-tools.yaml --format markdown --output skillwitness-report.md
skillwitness scan ./skills --profile company-strict --approved-tools ./policies/example-org/approved-tools.yaml --format sarif --output skillwitness-report.sarif

policies/example-org/ holds sample policies only. Replace them with your own.

Scan with a skill metadata overlay:

.venv/bin/skillwitness scan ./skills --profile company-strict --registry ./policy/approved-skills.yaml
.venv/bin/skillwitness scan ./skills --registry ./policy/approved-skills.yaml --registry-signature ./policy/approved-skills.sig

Scan with an external dependency policy:

.venv/bin/skillwitness scan ./skills --profile company-strict --approved-tools ./policies/example-org/approved-tools.yaml

Refreshing Reports

Exported reports are static files. Code, rule, profile, or allow-list changes do not update existing .md, .json, SARIF, JUnit, or CycloneDX files automatically. Re-run the scanner whenever you want MR attachments or CI artifacts to reflect the current tool behavior.

Useful refresh commands for local review prep:

.venv/bin/skillwitness scan ./skills/my-skill --profile company-strict --output my-skill-scan-report.md
.venv/bin/skillwitness scan ./skills --profile company-strict --output-dir skillwitness-reports

Generated scan reports should normally be CI artifacts or MR attachments, not committed source files, unless a specific golden fixture is being updated intentionally.

Metadata And Allow-Lists

CI gates need metadata for repeatable trust decisions: source provenance, pinned refs, expected license, risk classification, ownership, runtime behavior, telemetry declarations, and approval status. The scanner is read-only. It reads submitted package files such as SKILL.md, license files, package manifests, scripts, and references; CI-owned trust metadata should come from the registry overlay, not from scanner-generated files inside submitted packages.

Dependency approval is controlled by approved-tools.yaml. The scanner uses --approved-tools PATH when provided, otherwise the nearest approved-tools.yaml found by walking upward from the skill root, or finally the bundled vendor-neutral seed policy in src/skillwitness/rules/approved_tools.yaml. The sample policy used by the CI examples lives at policies/example-org/approved-tools.yaml. A permission-aware v2 policy example lives at policies/example-org/approved-tools-v2.yaml; v2 policy files can use exact strings, wildcard strings, or object entries with match, permissions, approval, and blocked_external blocks.

Version 2 rejects unknown top-level and nested entry fields, validates type-specific match and permission keys, and accepts only approved or review-required approval statuses. Provider IDs and dependency-entry IDs must be unique, and every MCP provider, domain-group, or registry reference must resolve before scanning. MCP entries can opt into deterministic operation authorization with exact canonical allowed_operations and denied_operations tokens. Deny entries take precedence, absent operations fail closed, and service-only or dynamic evidence produces an explicit unknown-operation finding. Every decision identifies the policy entry, provider, service and operation when known, resolution method, and evidence location. Runtime scope declarations remain visible as unverified service-side requirements.

Local entries can opt into command scope authorization with allowed_scopes and denied_scopes. Scope patterns describe command arguments: status:* authorizes status plus further arguments, a pattern without the trailing :* authorizes exactly that argument list, and a pattern that would approve every scope is rejected at policy load. Scope evidence comes from allowed-tools declarations, so Bash(git status:*) is authorized against the allowlist while Bash(git:*) and Bash(git) declare no bounded argument scope and report DEP_COMMAND_SCOPE_UNKNOWN instead of matching any pattern. Denied scopes are applied before allowed scopes, and every decision records the matched entry, pattern, resolved scope, resolution method, and evidence location.

Local entries can also correlate network intent with allowed_methods, allowed_domain_refs, denied_domains, and download_execute. curl and wget invocations in allowed-tools declarations and in scanned scripts are parsed for command, HTTP method, destination host, and whether the download is piped into an execution sink, then classified as read, upload, or download-execute. Referenced external entries act as the domain groups, so an internal mirror is expressed as a normal external entry. Decisions apply in order: unresolved destination, denied domain, download-execute without approval, disallowed method, destination outside the domain groups. A destination assembled at runtime reports DEP_NETWORK_INTENT_UNKNOWN and never matches an allowlist.

Known non-enforcing permission metadata is retained in the parsed policy for audit. The policy_rules section and descriptive MCP provider metadata remain documentary and do not affect gate decisions. The current implementation also emits policy findings for placeholders, MCP wildcard approvals, review-required entries, runtime guardrail requirements, and blocked external dependencies; deeper command/domain permission-context scoring is still being developed. Per-skill registry approved_tools entries override the repo-wide policy for that skill. Unapproved mcp, local, or external dependencies are HIGH by default. A version 2 defaults.unapproved_dependency_severity value takes precedence when supplied; otherwise the profile's dependency_approval.unapproved_severity applies. Operators should keep organization-specific allow-lists outside the generic validator and outside submitted skill packages, either in a fork/internal policy repository or as a CI-managed overlay before running the validator.

External tool integrations are best-effort but explicit. Missing optional tools such as Semgrep, OSV-Scanner, ScanCode, Gitleaks, TruffleHog, or detect-secrets are reported as informational coverage findings instead of silently disappearing. CycloneDX metadata records Syft as successful, unavailable, failed, or skipped.

A profile can pin that tool environment with an optional external_tools block: per-tool approved_versions, install_source, sha256, and timeout_seconds, plus a default_timeout_seconds for declared tools. Verification runs once per scan. A checksum mismatch (TOOL_ENV_CHECKSUM_MISMATCH, BLOCKER), an unapproved version (TOOL_ENV_VERSION_NOT_APPROVED, HIGH), or an unreadable version (TOOL_ENV_UNVERIFIED, HIGH) stops that tool from running and reports failed coverage for the scanner it backs. A tool that simply is not installed stays ordinary unavailable coverage. Observed versions appear in the scan-coverage table. Subprocesses inherit the runner's proxy and certificate environment unchanged; osv-scanner, scorecard, and cosign are the tools that reach the network, and what they disclose is documented in the spec.

An optional performance block bounds one scan: max_scanned_files caps the files scanned per skill and reports the gap as STRUCT_TRAVERSAL_CAPPED instead of truncating silently, max_external_output_bytes refuses oversized tool output instead of parsing it, and upstream_cache_ttl_hours / vulnerability_cache_ttl_hours control the permitted Scorecard and OSV result caches (the vulnerability cache is keyed by a digest of the scanned manifests and lockfiles, and is disabled by default). Scorecard runs once per repository URL per scan, and bundled rule sets are parsed once per process. Per-scanner durations stay out of the report so its bytes remain deterministic; write them with scan --timing-output PATH.

A profile can turn that visible coverage into an enforced contract with an optional coverage block that names required scanners per target kind (skill-package and source-repository) plus the severity for unavailable, failed, and incomplete coverage. Bundled profiles declare no required scanners, so this is opt-in for organization-owned profiles. Required coverage is evaluated per skill: a multi-skill scan whose aggregate row reads success because one skill was covered still fails when another skill's required scanner was skipped or never ran. Unknown scanner identifiers are rejected at profile load with exit code 4; the accepted identifiers are listed in the spec profile section.

Skill Metadata Overlay

For skills submitted to the managed skills repository, use a reviewed metadata overlay as the org-managed source of truth for CI-owned trust metadata. The overlay declares per-skill source_url, source_ref, license_expected, classification, owner_team, approval metadata, optional approval_ref and approval_url evidence, notes, and optional per-skill approved_tools overrides for mcp, local, and external dependencies. It does not fetch or rescan remote upstream repositories.

Keep the metadata overlay and dependency allow-list outside submitted skills: for example in a policy repository, a CI-managed workspace, or an internal fork dedicated to security review. Point the validator at the overlay with --registry and at the org-wide dependency policy with --approved-tools. If the overlay is signed, pass --registry-signature and optionally --registry-cert; invalid signatures exit with policy configuration error code 4.

Verify the same registry independently with the option-based standalone command; --certificate is optional for cosign signatures and any invalid signature exits with code 4:

skillwitness verify \
  --registry ./policy/approved-skills.yaml \
  --signature ./policy/approved-skills.sig \
  --certificate ./policy/approved-skills.pem

Generate a starter overlay from an existing skills tree:

.venv/bin/skillwitness generate-registry-template ./skills > approved-skills.yaml

The template exits 0 and writes v3 YAML to stdout. Operators fill in the TODO metadata, review the dependencies and classifications, then sign and publish the overlay. Downstream tooling can validate overlay shape with examples/registry-schema.json or by calling registry_json_schema() from skillwitness.registry.schema.

Registry Coverage

When --registry is omitted, the validator scans package-local evidence only. It does not expect a separate metadata file in the submitted skill and does not emit missing-metadata findings for one.

When a registry is provided and a skill has no matching entry, the scanner emits SKILL_NO_REGISTRY_ENTRY. Missing entries are HIGH for L3+ skills, MEDIUM for L0-L2 skills, and LOW in the fast-precommit profile. A profile can also require registry coverage without passing --registry by setting registry.require: true.

Install Semgrep outside the project virtualenv, then put its bin directory on PATH before running reports. The Semgrep Python package currently pins click below the version required by ScanCode, so installing both into the same virtualenv can break dependency-license scanning.

python3.12 -m venv /tmp/skillwitness-semgrep
/tmp/skillwitness-semgrep/bin/python -m pip install semgrep
CERT_FILE=$(/tmp/skillwitness-semgrep/bin/python -c 'import certifi; print(certifi.where())')
PATH=/tmp/skillwitness-semgrep/bin:$PWD/.venv/bin:$PATH \
  SSL_CERT_FILE=$CERT_FILE \
  REQUESTS_CA_BUNDLE=$CERT_FILE \
  .venv/bin/skillwitness scan ./skills --profile company-strict --output skillwitness-report.md

Adoption And Triage

Pilots rarely start from zero findings. scan --write-baseline baseline.json records the findings the gate decided on, and scan --baseline baseline.json then gates on new findings only: baselined findings stay in the report and in summary.baselined_findings but no longer decide pass or fail. Baseline keys come from the normalized fingerprint when one exists, so regenerate the baseline after changing a profile's normalization mode.

Suppressions are organization-owned and live in the registry overlay, never in the submitted package. Each entry declares rule_id, path, reason, owner, approved_by, approved_at, expires, and an optional approval_ref. A suppression whose lifetime exceeds the profile's suppressions.max_days suppresses nothing and reports SUPPRESSION_INVALID; expired entries report SUPPRESSION_EXPIRED; BLOCKER findings need allow_blocker_suppressions. Every decision — applied, unused, expired, rejected — is written by scan --suppression-audit-output audit.jsonl.

For review surfaces, --format annotations emits GitHub Actions workflow commands and --format codequality emits GitLab Code Quality JSON. Both cap at 50 findings and state the omitted count in their own output, so a truncated list is never mistaken for a complete one.

scan --metrics-output metrics.json records what the validator can measure during a shadow pilot: findings by severity and disposition, override use, baseline use, and coverage counts. Reviewer agreement and false-positive rate need a reviewer decision per finding, so the artifact names those reviewer-supplied fields and carries the denominators. The pilot formulas and the criteria for promoting a rule set from report-only to blocking are in spec section 11B.

Exit Codes

• 0: scan completed and policy passed
• 1: scan completed and policy failed
• 2: invalid input
• 3: internal error
• 4: policy or registry configuration error

Branch Protection

The default branch should require the CI / Lint, type-check, and test workflow job to pass before merge. The rule should also require pull requests for changes to main; SARIF upload and release automation are intentionally deferred until later roadmap tasks.