akshually 0.1.0

A decorator that verifies your code actually does what its docstring says, using LLMs

akshually

A Python decorator that uses LLMs to verify your code actually does what its docstring claims.

DocMismatchError: akshually, 1 function(s) don't do what their docstrings say:

  - calculate_average (math_utils.py:12): The docstring says it returns the average,
    but the code returns the sum without dividing by the length.

---

Installation

# OpenAI
pip install "akshually[openai]"

# Anthropic
pip install "akshually[anthropic]"

# Both
pip install "akshually[all]"

---

Quick start

import akshually

akshually.configure(provider="openai", api_key="sk-...")

@akshually.verify
def double(x: int) -> int:
    """Return x multiplied by two."""
    return x * 2

akshually.check()  # passes — code matches the docstring

If the code doesn't match the docstring:

@akshually.verify
def calculate_average(numbers: list) -> float:
    """Return the average of the list."""
    return sum(numbers)  # oops, forgot to divide

akshually.check()
# DocMismatchError: akshually, 1 function(s) don't do what their docstrings say:
#   - calculate_average (math_utils.py:8): The docstring says it returns the average
#     but the code only returns the sum without dividing by len(numbers).

---

How it works

1. @akshually.verify registers the function in a pending queue. No LLM call yet.
2. akshually.check() makes one LLM call with all registered functions batched together.
3. The LLM compares each function's implementation against its docstring.
4. Any mismatches raise a single DocMismatchError listing every failing function with its file path, line number, and a plain-English explanation.
5. Results are cached by source hash — unchanged functions are skipped on subsequent runs.

---

Examples

Multiple functions — one LLM call

import akshually

akshually.configure(provider="anthropic", api_key="sk-ant-...")

@akshually.verify
def clamp(value: float, lo: float, hi: float) -> float:
    """Return value clamped to the range [lo, hi]."""
    return max(lo, min(hi, value))

@akshually.verify
def is_palindrome(s: str) -> bool:
    """Return True if the string reads the same forwards and backwards."""
    return s == s[::-1]

@akshually.verify
def fahrenheit_to_celsius(f: float) -> float:
    """Convert a Fahrenheit temperature to Celsius."""
    return (f - 32) * 5 / 9

akshually.check()  # one API call checks all three

Warn instead of raise

Use on_mismatch="warn" for functions where a mismatch should be surfaced but not fatal:

@akshually.verify(on_mismatch="warn")
def experimental_sort(items: list) -> list:
    """Return items sorted in ascending order using a novel algorithm."""
    ...

akshually.check()
# UserWarning: akshually: 'experimental_sort' may not match its docstring: ...

You can mix both modes freely — check() will emit warnings for "warn" functions and raise for "raise" functions in the same call.

Class decorator

@akshually.verify_class applies verification to every method on a class that has a docstring. Methods without docstrings are silently skipped.

@akshually.verify_class
class Statistics:
    def median(self, numbers: list) -> float:
        """Return the median value of the list."""
        sorted_nums = sorted(numbers)
        n = len(sorted_nums)
        mid = n // 2
        return sorted_nums[mid] if n % 2 else (sorted_nums[mid - 1] + sorted_nums[mid]) / 2

    def mode(self, numbers: list):
        """Return the most frequently occurring value in the list."""
        from collections import Counter
        return Counter(numbers).most_common(1)[0][0]

    def _private_helper(self):
        # no docstring — skipped automatically
        pass

akshually.check()

Caching

Results are cached at ~/.cache/akshually/results.json keyed by source hash. If a function's source hasn't changed since the last check(), it won't be sent to the LLM again.

akshually.check()               # uses cache (default)
akshually.check(use_cache=False)  # always calls the LLM
akshually.clear_cache()           # wipe all cached results

Using a custom model

akshually.configure(
    provider="openai",
    api_key="sk-...",
    model="gpt-4-turbo",
)

In a test suite

# tests/test_docstrings.py
import os
import akshually
import mypackage  # importing this registers all @akshually.verify functions

def test_docstrings_match_implementation():
    akshually.configure(provider="openai", api_key=os.environ["OPENAI_API_KEY"])
    akshually.check()

Or use the built-in pytest fixture:

# conftest.py
import os
import akshually
import mypackage

akshually.configure(provider="openai", api_key=os.environ["OPENAI_API_KEY"])
pytest_plugins = ["akshually.pytest_plugin"]

# tests/test_docstrings.py
def test_docstrings(akshually_check):
    pass  # the fixture calls akshually.check() automatically

---

CLI

Check Python files or directories without modifying any source code:

# Check a single file
akshually check mypackage/utils.py

# Check an entire directory (recursive)
akshually check mypackage/

# Check multiple targets
akshually check mypackage/utils.py mypackage/models.py

# Specify provider explicitly
akshually check mypackage/ --provider openai --api-key sk-...

# Skip the cache
akshually check mypackage/ --no-cache

The CLI auto-detects your provider from environment variables (OPENAI_API_KEY or ANTHROPIC_API_KEY) so you typically don't need any flags:

export OPENAI_API_KEY=sk-...
akshually check src/

Exits with code 1 if any mismatch is found — suitable for CI pipelines.

---

Configuration

Parameter	Type	Required	Default	Description
provider	"openai" | "anthropic"	Yes	—	LLM provider to use
api_key	str	Yes	—	Your API key
model	str	No	gpt-4o / claude-sonnet-4-6	Override the default model

---

Error reference

Exception	When it's raised
DocMismatchError	One or more "raise" functions don't match their docstrings
AksuallyConfigError	check() called before configure(), or missing API key
ValueError	@akshually.verify applied to a function with no docstring

---

Contributing

git clone https://github.com/example/akshually
cd akshually
pip install -e ".[all,dev]"
pytest