vallm 0.1.5

A complete toolkit for validating LLM-generated code

vallm

A complete toolkit for validating LLM-generated code.

vallm validates code proposals through a four-tier pipeline — from millisecond syntax checks to LLM-as-judge semantic review — before a single line ships.

Features

• Multi-language AST parsing via tree-sitter (165+ languages)
• Syntax validation with ast.parse (Python) and tree-sitter error detection
• Import resolution checking for Python, JavaScript/TypeScript, Go, Rust, Java, C/C++
• Complexity metrics via radon (Python) and lizard (16 languages)
• Security scanning with language-specific patterns and optional bandit integration
• LLM-as-judge semantic review via Ollama, litellm, or direct HTTP
• Code graph analysis — import/call graph diffing for structural regression detection
• AST similarity scoring with normalized fingerprinting
• Pluggy-based plugin system for custom validators
• Rich CLI with JSON/text output formats

Supported Languages

Language	Syntax	Imports	Complexity	Security
Python	✅ AST + tree-sitter	✅ Full resolution	✅ radon + lizard	✅ bandit + patterns
JavaScript	✅ tree-sitter	✅ Node.js builtins	✅ lizard	✅ XSS, eval patterns
TypeScript	✅ tree-sitter	✅ Node.js builtins	✅ lizard	✅ XSS, eval patterns
Go	✅ tree-sitter	✅ stdlib + modules	✅ lizard	✅ SQL injection, exec
Rust	✅ tree-sitter	✅ crates	✅ lizard	✅ unsafe, unwrap
Java	✅ tree-sitter	✅ stdlib packages	✅ lizard	✅ Runtime.exec, SQL
C/C++	✅ tree-sitter	✅ std headers	✅ lizard	✅ buffer overflow, system
Ruby	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
PHP	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
Swift	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
Kotlin	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited
Scala	✅ tree-sitter	⚠️ Limited	✅ lizard	⚠️ Limited

Installation

pip install vallm

With optional dependencies:

pip install vallm[all]        # Everything
pip install vallm[llm]        # Ollama + litellm for semantic review
pip install vallm[security]   # bandit integration
pip install vallm[semantic]   # CodeBERTScore
pip install vallm[graph]      # NetworkX graph analysis

Quick Start

Python API

from vallm import Proposal, validate, VallmSettings

code = """
def fibonacci(n: int) -> list[int]:
    if n <= 0:
        return []
    fib = [0, 1]
    for i in range(2, n):
        fib.append(fib[i-1] + fib[i-2])
    return fib
"""

proposal = Proposal(code=code, language="python")
result = validate(proposal)
print(f"Verdict: {result.verdict.value}")  # pass / review / fail
print(f"Score: {result.weighted_score:.2f}")

CLI

# Validate a file
vallm validate --file mycode.py

# Quick syntax check
vallm check mycode.py

# With LLM semantic review (requires Ollama)
vallm validate --file mycode.py --semantic --model qwen2.5-coder:7b

# JSON output
vallm validate --file mycode.py --format json

# Show config and available validators
vallm info

With Ollama (LLM-as-judge)

# 1. Install and start Ollama
ollama pull qwen2.5-coder:7b

# 2. Run with semantic review
vallm validate --file mycode.py --semantic

from vallm import Proposal, validate, VallmSettings

settings = VallmSettings(
    enable_semantic=True,
    llm_provider="ollama",
    llm_model="qwen2.5-coder:7b",
)

proposal = Proposal(
    code=new_code,
    language="python",
    reference_code=existing_code,  # optional: compare against reference
)
result = validate(proposal, settings)

Validation Pipeline

Tier	Speed	Validators	What it catches
1	ms	syntax, imports	Parse errors, missing modules
2	seconds	complexity, security	High CC, dangerous patterns
3	seconds	semantic (LLM)	Logic errors, poor practices
4	minutes	regression (tests)	Behavioral regressions

The pipeline fails fast — Tier 1 errors stop execution immediately.

Configuration

Via environment variables (VALLM_*), vallm.toml, or pyproject.toml [tool.vallm]:

# vallm.toml
pass_threshold = 0.8
review_threshold = 0.5
max_cyclomatic_complexity = 15
enable_semantic = true
llm_provider = "ollama"
llm_model = "qwen2.5-coder:7b"

Plugin System

Write custom validators using pluggy:

from vallm.hookspecs import hookimpl
from vallm.scoring import ValidationResult

class MyValidator:
    tier = 2
    name = "custom"
    weight = 1.0

    @hookimpl
    def validate_proposal(self, proposal, context):
        # Your validation logic
        return ValidationResult(validator=self.name, score=1.0, weight=self.weight)

Register via pyproject.toml:

[project.entry-points."vallm.validators"]
custom = "mypackage.validators:MyValidator"

Multi-Language Support

vallm supports 30+ programming languages via tree-sitter parsers:

Auto-Detection

from vallm import detect_language, Language

# Auto-detect from file path
lang = detect_language("main.rs")  # → Language.RUST
print(lang.display_name)  # "Rust"
print(lang.is_compiled)     # True

CLI with Auto-Detection

# Language auto-detected from file extension
vallm validate --file script.py      # → Python
vallm check main.go                   # → Go  
vallm validate --file lib.rs          # → Rust

# Batch validation with mixed languages
vallm batch src/ --recursive --include "*.py,*.js,*.ts,*.go,*.rs"

Supported Languages

Language	Category	Complexity	Syntax
Python	Scripting	✓ radon + lizard	✓ ast + tree-sitter
JavaScript	Web/Scripting	✓ lizard	✓ tree-sitter
TypeScript	Web/Scripting	✓ lizard	✓ tree-sitter
Go	Compiled	✓ lizard	✓ tree-sitter
Rust	Compiled	✓ lizard	✓ tree-sitter
Java	Compiled	✓ lizard	✓ tree-sitter
C/C++	Compiled	✓ lizard	✓ tree-sitter
Ruby	Scripting	✓ lizard	✓ tree-sitter
PHP	Web	✓ lizard	✓ tree-sitter
Swift	Compiled	✓ lizard	✓ tree-sitter
+ 20 more via tree-sitter		✓ tree-sitter	✓ tree-sitter

See examples/07_multi_language/ for a comprehensive demo.

Examples

Each example lives in its own folder with main.py and README.md. Run all at once:

cd examples && ./run.sh

Example	What it demonstrates
01_basic_validation/	Default pipeline — good, bad, and complex code
02_ast_comparison/	AST similarity scoring, tree-sitter multi-language parsing
03_security_check/	Security pattern detection (eval, exec, hardcoded secrets)
04_graph_analysis/	Import/call graph building and structural diffing
05_llm_semantic_review/	Ollama Qwen 2.5 Coder 7B LLM-as-judge review
06_multilang_validation/	JavaScript and C validation via tree-sitter
07_multi_language/	Comprehensive multi-language support — 8+ languages with auto-detection

Architecture

src/vallm/
├── cli.py              # Typer CLI: validate, check, info, batch
├── config.py           # pydantic-settings (VALLM_* env vars)
├── hookspecs.py        # pluggy hook specifications
├── scoring.py          # Weighted scoring + verdict engine
├── core/
│   ├── languages.py    # Language enum, auto-detection, 30+ languages
│   ├── proposal.py     # Proposal model
│   ├── ast_compare.py  # tree-sitter + Python AST similarity
│   ├── graph_builder.py # Import/call graph construction
│   └── graph_diff.py   # Before/after graph comparison
├── validators/
│   ├── syntax.py       # Tier 1: ast.parse + tree-sitter (multi-lang)
│   ├── imports.py      # Tier 1: module resolution (Python)
│   ├── complexity.py   # Tier 2: radon (Python) + lizard (16+ langs)
│   ├── security.py     # Tier 2: patterns + bandit
│   └── semantic.py     # Tier 3: LLM-as-judge
└── sandbox/
    └── runner.py       # subprocess / Docker execution

Roadmap

v0.2 — Completeness

• Wire pluggy plugin manager (entry_point-based validator discovery)
• Add LogicalErrorValidator (pyflakes) and LintValidator (ruff)
• TOML config loading (vallm.toml, [tool.vallm])
• Pre-commit hook integration
• GitHub Actions CI/CD

v0.3 — Depth

• AST edit distance via apted/zss
• CodeBERTScore embedding similarity
• NetworkX cycle detection and centrality in graph analysis
• RegressionValidator (Tier 4) with pytest-json-report
• TypeCheckValidator (mypy/pyright)

v0.4 — Intelligence

• --fix auto-repair mode (LLM-based retry loop)
• hypothesis/crosshair property-based test generation
• E2B cloud sandbox backend
• Streaming LLM output

See TODO.md for the full task breakdown.

License

Apache License 2.0 - see LICENSE for details.

Author

Created by Tom Sapletta - [e-mail]