skeptic

The independent check on AI-generated Python. Don't trust AI code — verify it.

Your AI wrote a function in two minutes. It looks right. It passes the happy-path test. skeptic finds the input where it quietly does the wrong thing — and hands you the exact line to reproduce it.

We don't ask an LLM "does this look right?" (that's the same blind spot that wrote the bug). We take a function's own stated contract — its name, type hints, and docstring — and use property-based testing to execute it against hundreds of adversarial inputs, surfacing where its real behavior breaks its own promises. Proof, not vibes.

Install

pipx install skepticode     # or:  pip install skepticode

Installs as skepticode, runs as the skeptic command. (The bare name skeptic on PyPI belongs to an unrelated, abandoned 2020 package.)

From source (this repo):

cd product
pip install -e .            # add [dev] for the test suite, [llm] for semantic checks

Use

skeptic check path/to/file.py
skeptic check path/to/file.py --function my_function

Example:

$ skeptic check examples/demo.py

=== median() ===
  intent: Return the median value of a list of numbers.
  !!  likely bug [no_unexpected_crash] confidence=0.80
      raises IndexError: list index out of range
      reproduce with: median(nums=[])

=== discount_price() ===
  intent: Apply a percentage discount to a price.
  OK  no silent bugs found (checks passed with high confidence)

Exit code is 1 when findings exist, so it drops straight into pre-commit or CI.

How it works

1. Read the contract — infer intent from the function's name, type hints, docstring.
2. Cross-examine — Hypothesis fires hundreds of adversarial inputs (empty, zero, negative, huge, unicode) and runs the code.
3. Produce the exhibit — when the code breaks its contract, shrink the failure to the smallest input that triggers it and report a one-line repro.

What it catches today

• Crash-family bugs — ZeroDivisionError, IndexError, KeyError, … on valid typed inputs (the empty list, the zero, the missing key).
• Contract violations — returning None when the signature promises a real value.

Precision is the priority: on a 20-function benchmark it caught 8/10 silent bugs with 0/10 false positives.

Roadmap

• Semantic ("almost right") bugs — wrong formulas, off-by-one logic — via an independent LLM oracle (pip install skeptic[llm], needs ANTHROPIC_API_KEY).
• Editor (VS Code) and GitHub pull-request integrations.
• Beyond pure functions: stateful and I/O code.

Develop

pip install -e .[dev]
pytest                 # unit tests
python eval/run_eval.py    # detection + false-positive rates on the benchmark

Design notes

• The LLM never executes model-written code. It only selects/parameterizes checks from a fixed, safe catalog — immune to prompt-injected code execution.
• Scope is pure functions for now (clear inputs → outputs), where precision is highest.

Working title & early pricing. Python first; more languages to follow.