scm-ai-extract

A command-line and GUI tool that extracts upstream supply-chain-management (SCM) sustainability practice/keyword rows from corporate sustainability/ESG report PDFs and writes one tidy CSV per report.

The production pipeline is now three stages:

PDF
  -> Stage 1 parse   (MinerU precision API batch parsing)
  -> Stage 2 compact (cheap text model: paragraph relevance + summaries)
  -> Stage 3 extract (indexed relevant paragraphs -> final CSV)

Stage 1 calls MinerU's precision API and preserves raw paragraph evidence under output/.markdown/.... Stage 2 never repeats relevant paragraph text in model output; it only records paragraph ids, relevance, and summaries for irrelevant paragraphs. Stage 3 sends README.md plus indexed relevant paragraphs and maps returned ids back to the raw evidence text before writing CSV.

Quick Start

# One OpenRouter key powers Stage 2 and Stage 3.
export OPENROUTER_API_KEY=sk-or-...

# MinerU precision API token for Stage 1.
export MINERU_API_KEY=...

# From an installed package:
uvx scm-ai-extract --input input --output output

# From this checkout:
uv run scm-ai-extract --input input --output output

# Pass keys directly instead of using environment variables:
uvx scm-ai-extract --input input --output output \
  --openrouter-api-key 'YOUR_OPENROUTER_KEY' \
  --mineru-api-key 'YOUR_MINERU_KEY'

If config.toml is missing, the CLI asks whether to create it from the packaged template. Before paid API calls, the terminal prints a cost estimate and asks for confirmation. In GUI mode, that confirmation is shown in the window.

Stage 1 uses MinerU's precision batch API with model_version = "vlm". Reports over 200 pages are split into <=200-page PDFs before upload. For faster Stage 1 uploads, pass --cdn-token "$(cat .local-secrets/scm_cdn_token.txt)". That route uploads PDFs to the managed Cloudflare/R2 Worker first, submits short-lived signed URLs to MinerU, and keeps checksum-addressed objects in R2 for up to 7 days so repeat runs can reuse identical PDFs without re-uploading. The R2 bucket lifecycle rule deletes those temporary objects automatically.

Configuration

config.toml is the single source of truth.

Default models:

Stage 1 is not an OpenRouter model; OpenRouter cost estimates cover Stage 2 and Stage 3 only.

CLI

scm-ai-extract [--config PATH] [--input PATH] [--output DIR]
               [--stage {all,parse,compact,extract}]
               [--resume] [--dry-run] [--yes]
               [--max-concurrency N] [--max-pdf-concurrency N]
               [--stage-2-model SLUG] [--stage-3-model SLUG]
               [--temperature F] [--top-p F]
               [--reasoning-effort {none,minimal,low,medium,high,xhigh}]
               [--max-output-tokens N]
               [--openrouter-api-key KEY] [--mineru-api-key KEY]
               [--cdn-token TOKEN] [--gui] [--verbose]

Generation overrides (--temperature, --top-p, --reasoning-effort, --max-output-tokens) apply to Stage 3 only.

Examples:

# A tree of PDFs; output mirrors input subfolders.
uv run scm-ai-extract --input reports/ --output output/

# Estimate cost only.
uv run scm-ai-extract --input reports/ --dry-run

# Run MinerU parse now, then compact/extract later.
uv run scm-ai-extract --input reports/ --stage parse
uv run scm-ai-extract --input reports/ --stage compact --resume
uv run scm-ai-extract --input reports/ --stage extract --resume

# Override models for a single run.
uv run scm-ai-extract --input reports/ \
  --stage-2-model google/gemini-2.5-flash-lite \
  --stage-3-model deepseek/deepseek-v4-flash --yes

# Open the config/progress GUI.
uv run scm-ai-extract --gui

Artifacts

For an input PDF reports/nested/A.pdf and --output output, artifacts are written to:

output/
  nested/A_final_dataset.csv
  .progress/
    nested/A_final_dataset.csv.done
  .markdown/
    nested/
      A.pdf/
        mineru/
        paragraphs.raw.jsonl
        paragraphs.compact.jsonl
        README.md

Resume checks are stage-specific:

Folder runs continue after true per-report failures and mark failed reports in the summary.

Benchmarking Stage 3 Models

scm-ai-extract-benchmark compares final extraction models using completed parse+compact artifacts. It does not run MinerU or Stage 2 compaction.

uv run scm-ai-extract-benchmark --markdown-dir output/.markdown --dry-run

uv run scm-ai-extract-benchmark \
  --markdown-dir output/.markdown \
  --reports NASDAQ_AAPL_2021 NASDAQ_NVDA_2024 NYSE_F_2021 \
  --models deepseek/deepseek-v4-flash deepseek/deepseek-v4-pro \
  --budget-usd 1.00 --yes

Results are written to output/benchmark_stage3/summary.csv and output/benchmark_stage3/summary_by_model.csv.

GUI

--gui opens a pywebview window that edits config.toml, computes the run review/cost estimate, and then shows progress in-process. Progress rows show Stage 1, Stage 2, and Stage 3 bars for each report.

For local GUI development:

pnpm -C frontend install
pnpm -C frontend dev
SCM_AI_EXTRACT_GUI_DEV_URL=http://localhost:5173 uv run scm-ai-extract --gui

Build distributable GUI assets with:

pnpm -C frontend build

Development

uv sync
uv run ruff check src tests
uv run python -m unittest discover -s tests -v
pnpm -C frontend build
git diff --check

Output Columns

The final CSV columns are:

source_report, page, paragraph, keyword, reasoning, scm_category,
scm_subcategory, topic, time_orientation, report_pages, scm_structure

paragraph is always filled from paragraphs.raw.jsonl, not copied from the Stage 3 model response.

Layout