llm-code-validator 0.1.0

CLI guardrail for catching stale Python APIs before runtime.

llm-code-validator

Python CLI for detecting stale or version-incompatible third-party API usage in Python source code.

Installation

Install from the repository:

git clone https://github.com/mathew-felix/llm-code-validator
cd llm-code-validator
pip install -e .

Install with test dependencies:

pip install -e ".[dev]"

After the package is published to PyPI:

pip install llm-code-validator

Usage

Check one file:

llm-code-validator check file.py

Check a directory:

llm-code-validator check src/

Check standard input:

llm-code-validator check - < snippet.py

Check staged Git files:

llm-code-validator check --staged

Use an explicit dependency file:

llm-code-validator check --requirements requirements.txt src/

Show low-confidence diagnostics:

llm-code-validator check --show-low-confidence src/

Exit codes:

• 0: no diagnostics
• 1: diagnostics found
• 2: tool error

Output

Text output:

llm-code-validator check src/

JSON output:

llm-code-validator check src/ --format json

GitHub Actions annotation output:

llm-code-validator check src/ --format github

Fixes

Preview fixes:

llm-code-validator fix file.py

Apply safe fixes:

llm-code-validator fix file.py --write

Only rules marked safe_fix are written. Rules marked suggested_fix or no_fix are reported but not changed.

Current rule safety counts:

• safe_fix: 15 rules
• suggested_fix: 51 rules
• no_fix: 2 rules

Signature Database

Validate the rule database:

llm-code-validator validate-signatures

The source rule database is:

data/library_signatures.json

The packaged rule database is:

llm_code_validator/library_signatures.json

Current rule count:

• 68 API-drift rules

Benchmarks

Run the CLI benchmark dataset:

python -m llm_code_validator.benchmark --dataset validation_dataset/cli_benchmark_cases.json

Run the AI-stack benchmark dataset:

python -m llm_code_validator.benchmark --dataset validation_dataset/ai_stack_benchmark_cases.json

Current saved benchmark results:

• CLI dataset: precision 1.0, recall 1.0, p50 0.243ms, p95 6.199ms
• AI-stack dataset: precision 1.0, recall 1.0, p50 0.444ms, p95 4.939ms

Pre-Commit

.pre-commit-hooks.yaml is included:

repos:
  - repo: https://github.com/mathew-felix/llm-code-validator
    rev: v0.1.0
    hooks:
      - id: llm-code-validator

GitHub Actions

Example workflow:

name: API Drift Check

on:
  pull_request:

jobs:
  api-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install llm-code-validator
      - run: llm-code-validator check . --format github

Testing

Run the test suite:

pytest -q

Current local result:

72 passed

Documentation

• docs/demo.md: example check and fix workflow
• docs/rules.md: rule database notes
• docs/release.md: package release steps
• PROJECT_REPORT.md: project report