scm-ai-extract 0.1.0

Three-stage MinerU and LLM pipeline for extracting upstream supply-chain-management sustainability practice rows from corporate reports.

scm-ai-extract

A command-line and GUI tool that extracts upstream supply-chain-management (SCM) sustainability practice/keyword rows from corporate sustainability/ESG report PDFs and writes one tidy CSV per report.

The production pipeline is now three stages:

PDF
  -> Stage 1 parse   (MinerU precision API batch parsing)
  -> Stage 2 compact (cheap text model: paragraph relevance + summaries)
  -> Stage 3 extract (indexed relevant paragraphs -> final CSV)

Stage 1 calls MinerU's precision API and preserves raw paragraph evidence under output/.markdown/.... Stage 2 never repeats relevant paragraph text in model output; it only records paragraph ids, relevance, and summaries for irrelevant paragraphs. Stage 3 sends README.md plus indexed relevant paragraphs and maps returned ids back to the raw evidence text before writing CSV.

Quick Start

# One OpenRouter key powers Stage 2 and Stage 3.
export OPENROUTER_API_KEY=sk-or-...

# MinerU precision API token for Stage 1.
export MINERU_API_KEY=...

# From an installed package:
uvx scm-ai-extract --input input --output output

# From this checkout:
uv run scm-ai-extract --input input --output output

# Pass keys directly instead of using environment variables:
uvx scm-ai-extract --input input --output output \
  --openrouter-api-key 'YOUR_OPENROUTER_KEY' \
  --mineru-api-key 'YOUR_MINERU_KEY'

If config.toml is missing, the CLI asks whether to create it from the packaged template. Before paid API calls, the terminal prints a cost estimate and asks for confirmation. In GUI mode, that confirmation is shown in the window.

Stage 1 uses MinerU's precision batch API with model_version = "vlm". Reports over 200 pages are split into <=200-page PDFs before upload. For faster Stage 1 uploads, pass --cdn-token "$(cat .local-secrets/scm_cdn_token.txt)". That route uploads PDFs to the managed Cloudflare/R2 Worker first, submits short-lived signed URLs to MinerU, and keeps checksum-addressed objects in R2 for up to 7 days so repeat runs can reuse identical PDFs without re-uploading. The R2 bucket lifecycle rule deletes those temporary objects automatically.

Configuration

config.toml is the single source of truth.

Section	What it controls
[openrouter]	API base URL and environment variable name for the key.
[stage1]	MinerU precision API endpoint/env var, model version, parser flags, split size, batch size, polling, and timeout.
[stage2]	Paragraph compaction model, batch limits, and compaction/README prompts.
[stage3]	Final extraction model, batch limits, extraction prompt, and time rules.
[output]	format = [{ name, description }, ...]; descriptions define categorical guidance.

Default models:

Stage	Default
Stage 2 compact	google/gemini-2.5-flash-lite
Stage 3 extract	google/gemini-3.1-flash-lite

Stage 1 is not an OpenRouter model; OpenRouter cost estimates cover Stage 2 and Stage 3 only.

CLI

scm-ai-extract [--config PATH] [--input PATH] [--output DIR]
               [--stage {all,parse,compact,extract}]
               [--resume] [--dry-run] [--yes]
               [--max-concurrency N] [--max-pdf-concurrency N]
               [--stage-2-model SLUG] [--stage-3-model SLUG]
               [--temperature F] [--top-p F]
               [--reasoning-effort {none,minimal,low,medium,high,xhigh}]
               [--max-output-tokens N]
               [--openrouter-api-key KEY] [--mineru-api-key KEY]
               [--cdn-token TOKEN] [--gui] [--verbose]

Flag	Default	Meaning
--input	required outside GUI	A PDF file or a folder of PDFs, searched recursively.
--output	output	Output folder; CSVs mirror input subfolders and artifacts go under output/.markdown/.
--stage	all	parse, compact, or extract can run from cached upstream artifacts.
--resume	off	Reuse complete artifacts; otherwise existing outputs are ignored and rebuilt.
--dry-run	off	Discover inputs and estimate Stage 2/3 API cost without calling models.
--yes	off	Skip cost confirmation.
--max-concurrency	32	Shared Stage 2/3 LLM worker pool size.
--max-pdf-concurrency	8	Parallel report orchestration for folder runs.
--stage-2-model	config value	Override compaction model for this run.
--stage-3-model	config value	Override final extraction model for this run.
--openrouter-api-key	env var named by [openrouter].api_key_env	OpenRouter key for Stage 2/3.
--mineru-api-key	env var named by [stage1].api_key_env	MinerU precision API key for Stage 1.
--cdn-token	off	Use the managed Cloudflare/R2 upload path for MinerU Stage 1.

Generation overrides (--temperature, --top-p, --reasoning-effort, --max-output-tokens) apply to Stage 3 only.

Examples:

# A tree of PDFs; output mirrors input subfolders.
uv run scm-ai-extract --input reports/ --output output/

# Estimate cost only.
uv run scm-ai-extract --input reports/ --dry-run

# Run MinerU parse now, then compact/extract later.
uv run scm-ai-extract --input reports/ --stage parse
uv run scm-ai-extract --input reports/ --stage compact --resume
uv run scm-ai-extract --input reports/ --stage extract --resume

# Override models for a single run.
uv run scm-ai-extract --input reports/ \
  --stage-2-model google/gemini-2.5-flash-lite \
  --stage-3-model deepseek/deepseek-v4-flash --yes

# Open the config/progress GUI.
uv run scm-ai-extract --gui

Artifacts

For an input PDF reports/nested/A.pdf and --output output, artifacts are written to:

output/
  nested/A_final_dataset.csv
  .progress/
    nested/A_final_dataset.csv.done
  .markdown/
    nested/
      A.pdf/
        mineru/
        paragraphs.raw.jsonl
        paragraphs.compact.jsonl
        README.md

Resume checks are stage-specific:

Stage	Complete when
Stage 1	MinerU content list exists and paragraphs.raw.jsonl validates.
Stage 2	paragraphs.compact.jsonl covers every raw paragraph id and README.md exists.
Stage 3	CSV has expected columns and the hidden .progress/...csv.done marker validates.

Folder runs continue after true per-report failures and mark failed reports in the summary.

Benchmarking Stage 3 Models

scm-ai-extract-benchmark compares final extraction models using completed parse+compact artifacts. It does not run MinerU or Stage 2 compaction.

uv run scm-ai-extract-benchmark --markdown-dir output/.markdown --dry-run

uv run scm-ai-extract-benchmark \
  --markdown-dir output/.markdown \
  --reports NASDAQ_AAPL_2021 NASDAQ_NVDA_2024 NYSE_F_2021 \
  --models deepseek/deepseek-v4-flash deepseek/deepseek-v4-pro \
  --budget-usd 1.00 --yes

Results are written to output/benchmark_stage3/summary.csv and output/benchmark_stage3/summary_by_model.csv.

GUI

--gui opens a pywebview window that edits config.toml, computes the run review/cost estimate, and then shows progress in-process. Progress rows show Stage 1, Stage 2, and Stage 3 bars for each report.

For local GUI development:

pnpm -C frontend install
pnpm -C frontend dev
SCM_AI_EXTRACT_GUI_DEV_URL=http://localhost:5173 uv run scm-ai-extract --gui

Build distributable GUI assets with:

pnpm -C frontend build

Development

uv sync
uv run ruff check src tests
uv run python -m unittest discover -s tests -v
pnpm -C frontend build
git diff --check

Output Columns

The final CSV columns are:

source_report, page, paragraph, keyword, reasoning, scm_category,
scm_subcategory, topic, time_orientation, report_pages, scm_structure

paragraph is always filled from paragraphs.raw.jsonl, not copied from the Stage 3 model response.

Layout

Path	What
src/scm_ai_extract/	Installable package: CLI, pipeline, stages, GUI backend.
src/scm_ai_extract/resources/config.toml	Packaged default config template.
frontend/	React/Vite source for the GUI window.
eval/	Gold evidence, mention CSVs, review notes, and evaluation material.
small_input/, input/	Sample/report PDFs used during development.