vallm 0.1.3

A complete toolkit for validating LLM-generated code

vallm

A complete toolkit for validating LLM-generated code.

vallm validates code proposals through a four-tier pipeline — from millisecond syntax checks to LLM-as-judge semantic review — before a single line ships.

Features

• Multi-language AST parsing via tree-sitter (165+ languages)
• Syntax validation with ast.parse and tree-sitter error detection
• Import resolution checking for Python
• Complexity metrics via radon (Python) and lizard (16 languages)
• Security scanning with pattern matching and optional bandit integration
• LLM-as-judge semantic review via Ollama, litellm, or direct HTTP
• Code graph analysis — import/call graph diffing for structural regression detection
• AST similarity scoring with normalized fingerprinting
• Pluggy-based plugin system for custom validators
• Rich CLI with JSON/text output formats

Installation

pip install vallm

With optional dependencies:

pip install vallm[all]        # Everything
pip install vallm[llm]        # Ollama + litellm for semantic review
pip install vallm[security]   # bandit integration
pip install vallm[semantic]   # CodeBERTScore
pip install vallm[graph]      # NetworkX graph analysis

Quick Start

Python API

from vallm import Proposal, validate, VallmSettings

code = """
def fibonacci(n: int) -> list[int]:
    if n <= 0:
        return []
    fib = [0, 1]
    for i in range(2, n):
        fib.append(fib[i-1] + fib[i-2])
    return fib
"""

proposal = Proposal(code=code, language="python")
result = validate(proposal)
print(f"Verdict: {result.verdict.value}")  # pass / review / fail
print(f"Score: {result.weighted_score:.2f}")

CLI

# Validate a file
vallm validate --file mycode.py

# Quick syntax check
vallm check mycode.py

# With LLM semantic review (requires Ollama)
vallm validate --file mycode.py --semantic --model qwen2.5-coder:7b

# JSON output
vallm validate --file mycode.py --format json

# Show config and available validators
vallm info

With Ollama (LLM-as-judge)

# 1. Install and start Ollama
ollama pull qwen2.5-coder:7b

# 2. Run with semantic review
vallm validate --file mycode.py --semantic

from vallm import Proposal, validate, VallmSettings

settings = VallmSettings(
    enable_semantic=True,
    llm_provider="ollama",
    llm_model="qwen2.5-coder:7b",
)

proposal = Proposal(
    code=new_code,
    language="python",
    reference_code=existing_code,  # optional: compare against reference
)
result = validate(proposal, settings)

Validation Pipeline

Tier	Speed	Validators	What it catches
1	ms	syntax, imports	Parse errors, missing modules
2	seconds	complexity, security	High CC, dangerous patterns
3	seconds	semantic (LLM)	Logic errors, poor practices
4	minutes	regression (tests)	Behavioral regressions

The pipeline fails fast — Tier 1 errors stop execution immediately.

Configuration

Via environment variables (VALLM_*), vallm.toml, or pyproject.toml [tool.vallm]:

# vallm.toml
pass_threshold = 0.8
review_threshold = 0.5
max_cyclomatic_complexity = 15
enable_semantic = true
llm_provider = "ollama"
llm_model = "qwen2.5-coder:7b"

Plugin System

Write custom validators using pluggy:

from vallm.hookspecs import hookimpl
from vallm.scoring import ValidationResult

class MyValidator:
    tier = 2
    name = "custom"
    weight = 1.0

    @hookimpl
    def validate_proposal(self, proposal, context):
        # Your validation logic
        return ValidationResult(validator=self.name, score=1.0, weight=self.weight)

Register via pyproject.toml:

[project.entry-points."vallm.validators"]
custom = "mypackage.validators:MyValidator"

Examples

See the examples/ directory:

• 01_basic_validation.py — Default pipeline with good, bad, and complex code
• 02_ast_comparison.py — AST similarity and structural diff
• 03_security_check.py — Security pattern detection
• 04_graph_analysis.py — Import/call graph diffing
• 05_llm_semantic_review.py — Ollama LLM-as-judge review
• 06_multilang_validation.py — JavaScript and C validation

License

Apache License 2.0 - see LICENSE for details.

Author

Created by Tom Sapletta - [e-mail]