itemwise

LLM-based evaluation of multiple-choice items against the 43 item-writing rules from Haladyna & Downing (1989). Works with any OpenAI-compatible endpoint — hosted APIs (OpenAI, Azure, etc.) or a local server (vLLM, Ollama, LM Studio) for open-weight models.

Installation

pip install itemwise

Requires Python 3.12+.

Quick Start

from itemwise import evaluate

result = evaluate(
    item={
        "stem": "Which of the following is NOT a characteristic of mammals?",
        "options": [
            "They are warm-blooded",
            "They lay eggs",
            "They have hair or fur",
            "They produce milk",
        ],
        "correct": 1,
    },
    model="gpt-5.5",
    api_key="sk-...",  # or set OPENAI_API_KEY
)

print(result.score)            # fraction of rules passed
print(result.violations)       # list of failed RuleResult

Usage

from itemwise import evaluate, evaluate_batch, async_evaluate_batch

# Select specific rules
evaluate(item=item, model="gpt-5.5", rules=[22, 28, 37])

# Batch with progress bar (disable via progress=False)
evaluate_batch(items=items, model="gpt-5.5")

# Async / parallel
await async_evaluate_batch(items=items, model="gpt-5.5")

# Extra kwargs are forwarded to the OpenAI chat.completions API
evaluate(item=item, model="gpt-5.5", reasoning_effort="low")

CLI

itemwise evaluate questions.json --model gpt-5.5
itemwise evaluate questions.json --model gpt-5.5 --rules 22,28,37 --param reasoning_effort=low

Input JSON format:

[{"stem": "...", "options": ["A", "B", "C", "D"], "correct": 0}]

Endpoint Configuration

itemwise sends requests to any OpenAI-compatible endpoint. Point it at one with base_url / api_key (or the OPENAI_BASE_URL / OPENAI_API_KEY environment variables).

Hosted OpenAI (default endpoint):

export OPENAI_API_KEY=sk-...
itemwise evaluate questions.json --model gpt-5.5

Local open-weight model (GLM-5.2) via vLLM — start your own server, then point itemwise at it:

vllm serve zai-org/GLM-5.2  # exposes an OpenAI-compatible API at http://localhost:8000/v1
itemwise evaluate questions.json \
  --model zai-org/GLM-5.2 \
  --base-url http://localhost:8000/v1 \
  --api-key EMPTY

evaluate(
    item=item,
    model="zai-org/GLM-5.2",
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

Any other OpenAI-compatible server (Ollama, LM Studio, llama.cpp, etc.) works the same way — just set base_url to its address.

Item-Writing Rules

43 rules from Haladyna & Downing (1989) across 6 categories:

Rules 11 (item independence) and 36 (answer position distribution) require cross-item analysis and are excluded by default. Pass them explicitly via rules=[11, 36] to include them.

References

• Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 37-50.
• Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-333.

License

MIT