itemwise 0.1.1

LLM-based evaluation of multiple-choice items against item-writing guidelines

itemwise

LLM-based evaluation of multiple-choice items against the 43 item-writing rules from Haladyna & Downing (1989). Works with any LLM provider via litellm.

Installation

pip install itemwise

Requires Python 3.12+.

Quick Start

from itemwise import evaluate

result = evaluate(
    item={
        "stem": "Which of the following is NOT a characteristic of mammals?",
        "options": [
            "They are warm-blooded",
            "They lay eggs",
            "They have hair or fur",
            "They produce milk",
        ],
        "correct": 1,
    },
    model="azure/gpt-5.1-chat",
)

print(result.score)            # fraction of rules passed
print(result.violations)       # list of failed RuleResult
print(result.usage.cost)       # LLM cost in USD

Usage

from itemwise import evaluate, evaluate_batch, async_evaluate_batch

# Select specific rules
evaluate(item=item, model="azure/gpt-5.1-chat", rules=[22, 28, 37])

# Batch with progress bar (disable via progress=False)
evaluate_batch(items=items, model="azure/gpt-5.1-chat")

# Async / parallel
await async_evaluate_batch(items=items, model="azure/gpt-5.1-chat")

# Extra kwargs are forwarded to litellm
evaluate(item=item, model="azure/gpt-5.1-chat", reasoning_effort="low")

CLI

itemwise evaluate questions.json --model azure/gpt-5.1-chat
itemwise evaluate questions.json --model azure/gpt-5.1-chat --rules 22,28,37 --param reasoning_effort=low

Input JSON format:

[{"stem": "...", "options": ["A", "B", "C", "D"], "correct": 0}]

LLM Configuration

Model names and parameters follow litellm conventions. For Azure OpenAI:

export AZURE_API_KEY=...
export AZURE_API_BASE=https://your-resource.cognitiveservices.azure.com/
export AZURE_API_VERSION=2024-12-01-preview

Item-Writing Rules

43 rules from Haladyna & Downing (1989) across 6 categories:

Category	Rules	Description
General (Procedural)	1-7	Format, grammar, readability
General (Content)	8-17	Objectives, vocabulary, higher-order thinking
Stem Construction	18-23	Clarity, positive wording
General Option	24-35	Count, order, homogeneity, length
Correct Option	36-37	Position distribution, uniqueness
Distractor	38-43	Plausibility, common errors

Rules 11 (item independence) and 36 (answer position distribution) require cross-item analysis and are excluded by default. Pass them explicitly via rules=[11, 36] to include them.

References

• Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 37-50.
• Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-333.

License

MIT