Story test

CLI tool that runs tests against a story.

Lets you define a list of checks you expect from your story (e.g. "The hero wins in the end"), and determines whether each one passes or fails.

It does not replace a human editor, but it's a great tool to aid in the editing phase, for both writers and editors:

• Make sure your story's main points are addressed while editing your story.
• Build and grow a repository with standard tests you want to run on manuscripts, and make specific tests for specific genres, etc.

Uses a local Ollama model.

Usage

Create a test file containing assertions about a story:

tests:
	- name: author
		assertion: The story was written by Edgar Allan Poe.
	- name: ending
		assertion: The narrator confesses at the end of the story.

Run the tests against one or more Markdown files:

story-test story.tests.yml story.md
story-test story.tests.yml chapter-1.md chapter-2.md

Failed assertions are reported without failing the process by default. For CI, use strict mode:

story-test story.tests.yml story.md --fail-on-test-failure

🚀 Getting Started

This repo uses direnv, Devbox, Taskfile, and pre-commit for a reproducible dev environment and automatic schema/YAML validation.

Setup

# Automatically enter devbox via direnv (if available)
direnv allow

# Enter dev environment
devbox shell

# Install pre-commit hooks
task pre-commit:install

Tasks

Run task to see a list of available tasks.

Install the development dependencies with:

task environment:dev:install

For a local user installation, run the bootstrap script from this repository:

sh install.sh

This creates an isolated Python environment, installs the story-test command, and pulls the default Ollama model. The installed command can then be used from any directory:

story-test path/to/story.tests.yml path/to/story.md

Set STORY_TEST_MODEL before running the installer to use another model:

STORY_TEST_MODEL=qwen3:30b-a3b sh install.sh

Install Ollama separately, then download the default model through Task:

task environment:ollama:install

To use a model already installed locally:

OLLAMA_MODEL=qwen3:30b-a3b task environment:ollama:install

Run the sample story tests. Failed story assertions are reported but do not fail the task by default:

task test:example-story

The local model is qwen3:8b by default. Set STORY_TEST_MODEL to use another model already installed in Ollama.

The model can be changed with STORY_TEST_MODEL, and the context window can be changed with STORY_TEST_CONTEXT_LENGTH.

To make failed story assertions fail the task, use the strict variant:

task test:example-story:strict

Run the complete local validation suite:

task test:all

This runs schema validation, Python unit tests, and the sample story tests.

The GitHub Actions workflow installs Ollama and pulls qwen3:8b automatically. Qwen open-weight models are Apache 2.0 licensed and Ollama is MIT licensed; always review the license for the exact model tag you deploy.

Run formatting and pre-commit checks with:

task format:check
task pre-commit:run

Before publishing a release, build and validate both distribution formats:

task package:check

This creates the wheel and source archive under dist/ and validates them with Twine. Increment the version in pyproject.toml before building a new release.

Features

• JSON structure validation using jq
• Schema validation using check-jsonschema (temporarily disabled)
• CI/CD integration with GitHub Actions
• Versioning schemas with directories like schemas/v1, schemas/v2
• Documentation with inline schema descriptions
• Code formatting using prettier or jq
• Documentation with README or extended docs folder (pending)
• Schema hosting via $id URLs or SchemaStore (pending)