Compendium Scribe

Compendium Scribe is a Click-driven command line tool and library that builds sourced research compendiums through a bounded OpenAI Agents SDK workflow. It decomposes a topic into planning, web research, verification, and synthesis stages, then renders the final Compendium as a strict Open Knowledge Format (OKF) bundle, XML, HTML, or PDF.

Features

• Agents SDK research workflow - Runs planning, research, repair, verification, and synthesis agents with structured Pydantic outputs.
• Packaged agent definitions - Materializes the complete Agents SDK graph from agent instructions, capability grants, output types, and runtime configuration.
• Generated portable models - Generates the Pydantic models used by the application from the canonical contract types.
• Bounded hosted web search - Enables one medium-context search tool for the research manager, section researcher, and approved targeted researcher. The other agents cannot search.
• Stable renderer contract - Final agent output is validated and passed through the existing Compendium.from_payload() shape.
• Citation ledger - Deduplicates URLs, assigns citation IDs, tracks section usage, and rejects final citations that are not ledger-backed.
• Research trace evidence - Records attempts, provider outcomes, token use, and exact trace checkpoints. It checks controls and workflow stages before rendering.
• Fail-closed capability evidence - Rejects unsupported or unknown provider-hosted response calls and calls outside an agent's declared grants.
• Recoverable sidecars - Writes research state, trace events, trace-closure evidence, usage, and cost data to separate local files.
• Local cost estimates - Uses a checked-in pricing catalog for GPT-5.6 Sol, Terra, and Luna token rates, long-context rates, and built-in tool call prices. It warns when the catalog timestamp is missing, invalid, or more than 90 days old.
• Strict OKF output - Writes typed, linked Markdown concepts with safe YAML frontmatter and deterministic validation.
• Compendium Library publishing - Publishes self-contained OKF compendium trees into a movable, progressively disclosed OKF library.
• Re-rendering - Ingests existing OKF or XML compendiums to generate display or interchange formats without re-running research.
• Offline tests - The workflow uses a runner adapter so tests can stub Agents SDK runs without live API calls.

Quick Start

1. Install

pdm install --dev

Ensure PDM_HOME points to a writable location when developing within a sandboxed environment.

2. Configure credentials

Copy the example environment file, then add your OpenAI credentials:

cp .env.example .env

Set OPENAI_API_KEY in .env. The example selects the bundled production runtime profile. Missing or unknown profile configuration is rejected before cost initialization or research begins.

The research workflow uses the OpenAI Agents SDK. The manager and section researcher can use hosted web search. A targeted researcher can use web search only after explicit approval during recovery.

Cost reports use the local catalog in src/compendiumscribe/research/data/pricing.standard.json. The catalog covers GPT-5.6 Sol, Terra, and Luna token prices, long-context rates, web search calls, and Responses API file search calls. If a model is missing, token usage is still recorded and the USD estimate is unavailable. A stale catalog warning does not stop research because the report is an estimate.

3. Generate a compendium

pdm run compendium create "Lithium-ion battery recycling"

Options:

• --output PATH - Base path/filename for the output. The extension is ignored.
• --format FORMAT - Output format, defaulting to okf. Available: okf, xml, html, pdf. Repeat for multiple outputs.
• --library PATH - Also publish the finished compendium into a Compendium Library directory.

If you pass --output report, Compendium Scribe writes:

• report.okf/ by default, or the requested render formats
• report.research.json
• report.research.trace.jsonl
• report.research.trace-closure.json
• report.costs.json

Without --output, the base name is the slugified topic plus a UTC timestamp.

4. Publish to a Compendium Library

A Compendium Library is itself a strict OKF bundle that agents can scan progressively. Its indexes link to self-contained compendium subtrees:

research-library/
├── index.md
└── compendiums/
    ├── index.md
    └── lithium-ion-battery-recycling/
        ├── index.md
        ├── compendium.md
        ├── sections/
        │   ├── index.md
        │   └── s01.md
        └── sources/
            ├── index.md
            └── c01.md

Creation works the same as usual unless --library is provided. When it is provided, requested outputs are still written normally, and the final compendium is also upserted into the library:

pdm run compendium create "Lithium-ion battery recycling" \
  --output report \
  --format okf \
  --format xml \
  --library research-library

Import an existing XML compendium:

pdm run compendium library import research-library report.xml

Import an existing CompendiumScribe OKF bundle the same way:

pdm run compendium library import research-library report.okf

Validate a Library:

pdm run compendium library validate research-library

Library entries are idempotent by slugified title. Re-publishing the same title replaces its complete OKF subtree. If another title would use the same slug, the new entry gets a numeric suffix such as -2.

5. Recover a research run

Recovery resumes from the next incomplete stage in the sidecar state file:

pdm run compendium recover --input report.research.json

The first verifier can request a repair that uses only the saved evidence. The workflow runs this repair without web search. Verification stays within the approved plan and can remove, narrow, qualify, or move an unsupported claim to the open questions. Missing optional source metadata does not block publication by itself. If an essential in-scope conclusion requires new web evidence, recovery stops before the search and saves the exact issue set. Review that state, then approve one targeted search call with:

pdm run compendium recover \
  --input report.research.json \
  --allow-paid-follow-up

Approval applies only to this recovery command. It is not saved in the state file. The workflow runs at most one final verifier. It does not start another repair cycle if final verification finds more work. The final report keeps the accurate repair, research, or fatal resolution even though the run stops.

The recover command writes outputs using the same base path as the sidecar. For example, report.research.json renders to report.okf/ when the stored format is OKF. Recovery states that request the removed md format are rejected rather than silently reinterpreted. Recovery appends to the matching normalized trace only when its workflow and runtime-configuration digests still match and its closure manifest attests the trace's exact ordered frontier. Any sidecar containing accepted workflow progress or attempted agent work requires a readable, nonempty trace and matching identity-bound trace-closure evidence; only a pristine created sidecar may start without them. Logical invocation IDs remain stable across recovery, while each retry receives a unique, ordered attempt ID linked to its predecessor. A prior attempt is sealed across processes; resumed provider execution always uses the next attempt identity. Paid search agents receive two total attempts. No-search agents receive three total attempts. Undeclared capabilities fail immediately without retry. Successful attempts are selected only after their host stage records are checkpointed. Every completed recovery rechecks its required controls and workflow stages before citation hydration or rendering.

6. Render formats from existing OKF or XML

pdm run compendium render my-topic.okf --format html

Options:

• --format FORMAT - Output format(s) to generate: okf, xml, html, pdf.
• --output PATH - Base path/filename for the output.

Directory outputs remain separate: --output report --format okf writes report.okf/, while --format html writes report/.

Python API Usage

from compendiumscribe import build_compendium, ResearchConfig, DeepResearchError

try:
    compendium = build_compendium(
        "Emerging pathogen surveillance",
        config=ResearchConfig(),
    )
except DeepResearchError:
    raise

xml_payload = compendium.to_xml_string()
okf_files = compendium.to_okf_bundle()
html_files = compendium.to_html_site()
pdf_bytes = compendium.to_pdf_bytes()

Load a canonical OKF bundle with Compendium.from_okf_bundle("report.okf"). XML remains available for interchange, but it does not carry arbitrary OKF extension metadata.

CompendiumScribe OKF Profile

OKF remains Markdown, but the removed md format was a single untyped rendering. The canonical OKF representation is a directory of typed concepts:

report.okf/
├── index.md
├── compendium.md
├── sections/
│   ├── index.md
│   └── s01.md
└── sources/
    ├── index.md
    └── c01.md

Every non-index document contains safe YAML frontmatter with a non-empty type. The root concept declares profile: compendiumscribe and profile_version: "1". Section tags come from key terms; compendium tags are derived deterministically. Relative links connect insights to source concepts, so the same subtree works as a standalone export or inside a Library. Source concepts keep the source publication date in published_at. They keep a commit SHA, tag, or document version in revision, and a timezone-aware access time in accessed_at.

The profile rejects log.md, symlinks, non-Markdown files, path traversal, colliding normalized IDs, malformed frontmatter, and broken required links. Unknown producer-defined frontmatter keys remain valid.

Testing & Quality

• pdm run test - Executes the unit suite. Tests stub Agents SDK runs, so they run offline.
• pdm run lint - Linting.
• pdm run check - Runs tests, linting, and package build.
• pdm run ruff check src tests - Direct lint command.
• pdm build - Produce distributable artifacts.

Before marking implementation work complete, run:

pdm run check

pdm run check runs the full required loop:

pdm run pytest
pdm run ruff check src tests
pdm build

Contributing

1. Fork and clone the repository.
2. Run pdm install --group dev.
3. Make changes following the style guide and update/add tests.
4. Run pdm run check.
5. Raise a pull request with a concise description, verification commands, and representative output samples when user-facing structure changes.