CiteIndex

CiteIndex, ingest sources with proper citation. PDF, URL, media, Office, DJVU.

Evidence-backed citation metadata, Merkle-verified integrity, CJK-first OCR. Optional verification traces accepted metadata corrections to source evidence.

Install

pip install citeindex
# Add only the engines you use, for example:
pip install 'citeindex[ocr-mineru,layout]'

The default install handles digital PDFs and ordinary HTML URLs. Scanned PDFs need ocr-mineru (MinerU's pipeline backend) or ocr-paddle with --ocr-engine glm-ocr. Media transcription with WhisperX needs media; the separate Wenbi environment needs only media-wenbi here. JavaScript-rendered pages need web-browser plus playwright install chromium, while --all-url-article needs crawl. The legacy OCR helpers need ocr-legacy. Enhanced PDF layout extraction uses layout; otherwise it falls back to the built-in heuristic. Install several extras together as needed.

CLI

# Ingest a PDF
citeindex paper.pdf

# Ingest a scanned PDF with the default MinerU backend
citeindex scanned.pdf --ocr-engine mineru

# Opt in to Wenbi raw ASR with an explicit local provider
citeindex lecture.mp4 --media-asr-backend wenbi --wenbi-asr-provider funasr

# Or keep Wenbi dependencies isolated in Wenbi's own environment
citeindex lecture.mp4 --media-asr-backend wenbi --wenbi-asr-provider funasr \
  --wenbi-python ../wenbi/.venv/bin/python

# Use the optional GLM-OCR backend via local Ollama
citeindex scanned.pdf --ocr-engine glm-ocr --ocr-model glm-ocr:latest

# Ingest a URL
citeindex https://example.com/article

# Crawl and ingest all articles from a site
citeindex https://example.com/articles --all-url-article --crawl-depth 2

# Crawl and re-ingest only changed pages
citeindex https://example.com/articles --update-url-article

# Options
citeindex paper.pdf --llm ollama/qwen3 --type thesis --is-primary
citeindex paper.pdf --text-direction vertical --vertical-lang ch
citeindex scanned.pdf --ocr-engine mineru --lang auto --page-range "1-10"
citeindex paper.pdf --no-layout  # disable column/footnote detection
citeindex paper.pdf --force-ocr  # override PDF classification
citeindex -q paper.pdf           # disable default debug logging

# Evidence-backed citation verification (Crossref exact DOI lookup)
citeindex paper.pdf --verify-citations --registry-contact-email you@example.org

# Ask a stronger provider-qualified model only to resolve remaining conflicts
citeindex paper.pdf --verify-citations --citation-verifier-model openai/gpt-5

# Explicit alternative engine for digital PDFs; requires a running GROBID service
citeindex paper.pdf --citation-engine grobid

Verification is opt-in. Crossref receives only an exact DOI, never document text. A model correction is accepted only when it supplies a matching source quote and stable locator; unavailable models leave the record unchanged for review. Use --no-crossref to disable registry lookup or --offline-verification to disable registry and verifier-model requests.

An independent harness audit is available only through an explicit Codex, Claude Code, OpenCode, or Pi skill/command; running citeindex directly does not auto-trigger one.

Benchmarking

The pilot benchmark measures extraction and evidence verification only; it does not evaluate document question answering. Keep reviewed source manifests, predictions, reports, and copied source files local because they may contain private or licensed material.

# First create candidate rows; this does not invent gold labels.
python benchmarks/citations/prepare_manifest.py \
  /home/ajiap/Downloads/Orthodox-China \
  benchmarks/citations/manifest.jsonl --limit 40

# Run all manifest rows serially; every attempted source becomes a prediction row.
python benchmarks/citations/ingest.py benchmarks/citations/manifest.jsonl \
  benchmarks/citations/predictions/dspy.jsonl

# Score host metadata against the human-reviewed labels.
python benchmarks/citations/run.py benchmarks/citations/manifest.jsonl \
  benchmarks/citations/predictions/dspy.jsonl

The 40-source pilot composition, annotation contract, modality-specific controls, and reporting gates are documented in docs/plans/2026-09-14-multimodal-citation-benchmark.md. Validate independent reviewers before scoring with python benchmarks/citations/validate_manifest.py manifest.jsonl; create a reproducible split with python benchmarks/citations/split.py manifest.jsonl split-manifest.jsonl; compare saved reports with python benchmarks/citations/compare.py baseline.json candidate.json. The reviewed 40-source pilot has 36 citations with usable gold metadata (16 digital PDFs, 12 scanned PDFs, 8 URLs). The original 2026-09-24 cloud run scored 6/36 exact records (16.7%) under the earlier scorer, with 3 scanned-PDF timeouts and 4 media failures. A later sampled run completed all 36 PDF/URL sources. Under scorer host-v4-run-accessed-equivalence, which uses the actual retrieval date and rejects redirect-only article snapshots, it scores 3/36 strict records (8.3%); two GCDFL snapshots are redirect-only. These are development results, not a production accuracy estimate.

The repaired 40-source cloud candidate completed all 28 PDFs, 8 URLs and 4 media files. On the 36 reviewed PDF/URL citations, strict record accuracy is 4/36 (11.1%) and separately defined bibliographic equivalence is 9/36 (25.0%). By modality, equivalent record accuracy is 2/16 digital PDFs, 1/12 scanned PDFs and 6/8 URLs. The candidate follows both GCDFL redirects to final articles, so its URL snapshots differ from the old baseline; do not treat the aggregate as a same-snapshot causal comparison. The four media files returned empty transcripts and explicit incomplete citation status without guessed metadata; they remain unscored, as requested. The 90% production target is not met for this multimodal pilot. Predictions and source review are local ignored artifacts under benchmarks/citations/predictions/ and benchmarks/citations/reports/.

Digital PDF two-category core-requirement benchmark (2026-09-26)

A focused 16-source digital-PDF slice was re-benchmarked against the re-reviewed digital-16-2026-09-26-gold-v2 manifest under the two-category citation model (article = cited with page numbers; book = cited without) and a category-aware core-requirement metric with equivalence-normalized matching:

• book core: type, author/editor/translator, title, publisher, issued
• article core: type, author, title, container/journal, number, page range
• fields not printed in the supplied PDF (for example a volume number absent from an article offprint) are marked outside_scope and are not required

Result: 15/16 core records (93.75%), title F1 0.875, type F1 1.0. The single miss is a work whose author and editor are printed only in Latinized form (GREGORII NYSSENI, WERNERUS JAEGER). Trajectory under this metric: 11/16 → 13/16 → 12/16 (one intermediate regression from over-constrained prompt rules) → 15/16, after moving deterministic normalization (OCR numeral repair, CIP author/translator role parsing) out of the prompt and into code.

Scope and caveats: this covers 16 digital PDFs only; the scanned/URL/media slices above are not measured by this metric, so the multimodal pilot statement stands. With n=16 single-run variance is material, and the two confirming runs were replays of the same cached model responses, not independent samples. Reviewed labels, predictions, and reports stay local under benchmarks/citations/predictions/.

Scanned PDF two-category core-requirement benchmark (2026-09-26)

A focused 12-source scanned-PDF slice was re-benchmarked against the re-reviewed scanned-12-2026-09-26-gold-v2 manifest under the same two-category citation model and core-requirement metric as the digital slice. Scanned sources run the full ingestion pipeline; only the page sampler is narrowed (1-10, -3) and page-index/layout are disabled.

Result: 12/12 core records (100%) — every scanned core field recovered. Against the pre-review frozen labels the same run scores 8/12, and every per-source flip from the baseline is forward (no regressions). Four sources needed label re-review where the frozen labels asked for something the scan cannot support (an OCR-omitted title; name forms printed only in catalog or CIP shape); the diff is in docs/2026-09-26-scanned-gold-v2-diff-for-signoff.md.

The work moved scanned title/credit recovery out of prompts and into gated deterministic seeds — heading-based title/subtitle, imprint publisher, and copyright-year branch priority, plus translator/editor credits and Roman-numeral imprint years — each verified against an offline OCR probe before any model run. One deliberately non-obvious case: a publisher that is garbled on the title page is still printable cleanly on a sampled tail page, so the imprint seed reads whichever copy is intact rather than trusting the first occurrence.

Python API

Optional translation plugin

Install Fanyi into the same environment to add on-demand translation of an already ingested PDF corpus. Fanyi may be a private local checkout:

python -m pip install -e /path/to/citeindex -e /path/to/fanyi
citeindex-plugin list

Save translation settings as fanyi-options.json:

{"target_lang": "ZH", "engine": "dspy", "format": "mdx"}

citeindex-plugin run fanyi corpus/Author_Title_hash \
  --options fanyi-options.json --output translations/Author_Title_hash/zh
# Optional: --timeout 7200 for a long document.

The plugin reads the completed corpus without re-ingesting or requiring the original PDF. It writes bilingual MDX, aligned JSON, copied image assets, and plugin_result.json provenance into a separate output directory. The destination must be absent or empty. Partial translations go into <output>.partial and return a nonzero exit status; failed runs do not replace earlier output. Translations are derived text, not original-source quotation evidence.

Plugins register a callable in the citeindex.plugins.v1 Python entry-point group. citeindex.plugins.run_plugin(name, document_path, output_dir=..., options={...}, timeout=3600) is the equivalent Python API. The selected callable receives a version-1 dictionary with document_path, output_dir, and options, and returns protocol_version, status, relative artifacts paths, QA details, errors/warnings, and optional sanitized manifest_options. Only explicitly requested plugins execute, in a child process using the same Python environment. Installed plugins are trusted code; this process boundary is not a sandbox.

python -m pip uninstall fanyi
citeindex-plugin list

Removing Fanyi detaches translation while preserving CiteIndex and existing corpora/output. CiteIndex's base installation has no Fanyi dependency. See the implementation plan for the contract and supported workflow.

Ingestion

from citeindex import ingest, IngestionConfig, IngestionFailure, PipelineResult

# Simple
result = ingest("paper.pdf")
print(result["status"])  # "ok"

# With config
config = IngestionConfig(
    llm_model="ollama/qwen3",
    text_direction="vertical",
    is_primary=True,
)
result = ingest("paper.pdf", corpus_root="my_corpus", config=config)

Code Structure

• citeindex/cli.py maps CLI options into IngestionConfig.
• citeindex/ingestion/master.py detects input types, routes pipelines, verifies metadata, and persists results.
• citeindex/ingestion/pipelines/ contains the digital PDF, scanned PDF, URL, media, GROBID, PageIndex, and metadata-extraction implementations.
• citeindex/ingestion/storage.py and markdown_export.py write corpus artifacts and companion library Markdown.
• .agents/, .claude/, .codex/, .opencode/, and .pi/ expose the optional citation-evidence audit to supported agent harnesses.

Ingestion Pipelines

CiteIndex automatically detects the input type and routes to the correct pipeline:

PDF routing is based on usable existing text, not whether the file was born digital. Searchable scans with readable text layers use the digital pipeline, including invisible OCR text over full-page images. At least 90% of nonblank pages must have usable text; up to 10% may be image-only or sparse pages (such as illustrations or title pages). For example, 91 text pages and 9 image-only pages use the digital pipeline. Those image-only pages are kept in the document but are not OCR'd; text contained only in their images is not extracted. Completely blank pages do not count against the threshold.

Clearly garbled text still triggers OCR. These local quality checks cannot detect every transcription error; use --force-ocr when the existing text is wrong, or --force-digital to explicitly reuse it. Digital layout extraction also disables automatic OCR in PyMuPDF4LLM.

Digital PDF

PDF → PyMuPDF4LLM layout/text (PyMuPDF fallback) + image extraction
    → page-paragraph document structure → DSPy citation extraction
    → PageIndex tree (default) → optional explicit GROBID engine
    → Merkle tree → section_tree + heading injection
    → store document.json and library Markdown

• PyMuPDF4LLM performs layout-aware extraction when enabled; raw PyMuPDF is the fallback
• DSPy is the default citation metadata engine for digital PDFs
• GROBID is an explicitly selected alternative requiring a separate service
• Builds page-based document structure and augments it with PageIndex section headings
• PageIndex builds LLM-driven section hierarchy, persists it to corpus, and feeds library markdown headings

Scanned PDF

PDF → scanned backend selector
    → MinerU (default) OR GLM-OCR + PaddleOCR LayoutDetection
    → normalized content_list / markdown / extracted figures
    → pattern extraction with DSPy-priority metadata
    → document structure → Merkle tree → PageIndex tree (default)
    → store only CiteIndex-native artifacts to corpus/

• MinerU is the default scanned backend
• GLM-OCR is an optional backend that runs through local Ollama using the native /api/generate endpoint
• PaddleOCR LayoutDetection (PP-DocLayout_plus-L) supplies external region proposals for GLM-OCR from the start
• Scanned PDFs do not use GROBID; metadata is extracted from structured backend output via DSPy-backed extraction
• DSPy is allowed to overwrite pattern-extracted metadata fields for scanned documents
• PageIndex runs by default for scanned PDFs, just like digital PDFs
• Only extracted figures / illustrations are exported into the corpus images/ folder; raw backend artifacts are not preserved
• Supports --ocr-engine mineru or --ocr-engine glm-ocr

Scanned PDF Backend Selection

Use the scanned backend flags only for image-based PDFs:

# Default scanned backend
citeindex scanned.pdf --ocr-engine mineru

# Local GLM-OCR through Ollama
citeindex scanned.pdf --ocr-engine glm-ocr --ocr-model glm-ocr:latest

# Custom Ollama host
citeindex scanned.pdf --ocr-engine glm-ocr --ollama-host http://localhost:11434

• mineru is the default and recommended general-purpose backend
• glm-ocr requires a local Ollama model plus PaddleOCR layout-detection dependencies
• --mineru-backend is forwarded directly to the MinerU CLI backend selector

URL Article

URL → Playwright/requests (fetch) → trafilatura/readability (content)
    → Zotero / in-page citation guidance (candidates) → web-specific DSPy + source evidence
    → section-hierarchical paragraphs → PageIndex tree (optional)
    → hashes → Merkle tree → store to corpus/

• Playwright renders JavaScript-heavy pages (fallback to requests)
• trafilatura extracts clean markdown with heading structure (fallback to readability-lxml)
• Zotero extracts citation metadata via translation-server (title, authors, date, DOI)
• Discovers in-page citation guidance: 若要引用 / 引用格式 / Cite this / Zitierweise / Pour citer
• Web-specific DSPy uses original HTML text/metadata, preserves bibliographic subtype and date precision, and requires field quotations.
• Regex/provider candidates remain unverified without evidence; access and revision dates are not publication dates.
• Supports batch crawling with --all-url-article and --update-url-article

Media

Recognized media URL/local file → yt-dlp or local copy → ffmpeg (audio) → WhisperX (transcription)
        → pyannote (diarization, optional) → media-specific DSPy + source evidence → CSL JSON
        → chunking → hashes → Merkle tree → store to corpus/

• yt-dlp downloads from YouTube, Vimeo, podcasts, etc.
• WhisperX transcribes with word-level timestamps
• pyannote speaker diarization (optional)
• Preserves speaker/interviewee, interviewer, host, director and producer roles; uploader is not automatically author, nor platform publisher.
• Recording duration (dimensions) is separate from quotation timestamps. Failed transcription leaves an empty transcript, not invented quotation text.
• Each web/media extraction uses one bounded DSPy call (12,000 source-text characters); unsupported metadata stays missing or unverified. No live accuracy claim follows from this contract.
• Supports audio (.mp3, .wav, .m4a) and video (.mp4, .mkv, .webm)
• Remote media routing recognizes YouTube, Vimeo, podcast, SoundCloud, and youtu.be hosts; other URLs use the article pipeline

Office & DJVU

Office documents (.docx, .doc, .rtf, .odt, .pptx, .ppt, .odp) are converted to PDF via LibreOffice, and DJVU (.djvu) via ddjvu, then routed to the digital or scanned PDF pipeline. Default DJVU conversion is limited to the first 10 pages.

Citation Enrichment Cascade

For digital PDF inputs, CiteIndex uses DSPy citation extraction by default. A separately installed GROBID service is available only when explicitly selected with --citation-engine grobid; it is never an automatic fallback.

Extraction targets the ingested source's own citation metadata—not works cited inside it. DSPy uses original source blocks and validates quote spans; this does not by itself prove correct work/edition attribution. PDF metadata remains a last-resort, unverified identity hint.

Scanned PDFs do not use GROBID. Their structured OCR output is parsed with document-specific patterns, then DSPy values take priority when available.

With --verify-citations, the finalized draft follows this additional evidence-first stage before CSL IDs, hashes, filenames, and Markdown are written:

candidate CSL → DOI extraction → exact Crossref lookup
              → source quote + page/paragraph or URL snapshot evidence
              → deterministic reconciliation → optional conflict model
              → standardized CSL + verification report

Registry-only values are never applied automatically. They must also appear in the original source evidence; otherwise CiteIndex records needs_review.

Configuration Reference

Output

Each ingestion produces a content-addressed corpus folder (for example, corpus/Author_2024_Title_a1b2c3d4e5f6/) and a companion library Markdown file.

Corpus artifacts (corpus/Author_2024_Title_<12-char-hash>/)

Library markdown (library/Author_2024_Title_<12-char-hash>.md)

Human-readable markdown with YAML front-matter, inline citation, page/section/timestamp headers with CSL-level detail, full extracted text, and footnotes. When PageIndex is available, digital PDFs emit section headings into the markdown instead of only flat page labels. Written to library/ (sibling of corpus/).

Agent-harness final audit

Codex, OpenCode, Claude Code, and Pi can load the shared citation-verification skill after ingestion. The audit reads the original source and citation_verification.json, then returns quotation-backed repair recommendations. It never edits persisted csl.json directly.

The CLI itself always performs the core verification when --verify-citations is supplied. The additional agent audit runs only through an explicit harness workflow (for example OpenCode /ingest-verified); a raw citeindex subprocess cannot detect or launch an agent harness.

Online enrichment (opt-in)

citeindex paper.pdf --online-enrich
citeindex paper.pdf --online-enrich --online-enrich-ai-fallback --online-enrich-ai-provider openai

Digital and scanned PDFs share this order: source extraction, optional quotation-backed repair, registry enrichment, then final CSL identifiers, artifact copies and Chicago Markdown rendering. Accepted Crossref/DataCite DOI records (T1) and OpenLibrary ISBN editions (T2) can automatically overwrite existing fields, including source repairs. The audit sidecar online_enrichment.json retains values, identity checks and provenance. Ambiguous matches abstain. Converted non-PDF inputs are outside this first release.

AI discovery requires --online-enrich-ai-fallback and an explicit provider: claude (ANTHROPIC_API_KEY), gemini (GEMINI_API_KEY), or openai (OPENAI_API_KEY). Each uses its native online search tool; search execution and cited identifier evidence are required. The discovered DOI/ISBN must independently resolve in a registry before T4 can fill missing fields. AI cannot overwrite fields. Supported model defaults and allowlisted source domains can be selected with --online-enrich-ai-model and --online-enrich-ai-domains.

Requests share a default 20-second budget, at most 12 attempts, five seconds per request, and a 2 MiB response limit. Registry caching lives in corpus/.registry-cache. --offline-verification blocks verification and enrichment network requests; --no-online-enrich disables automatic discovery. A supplied --enrich-proposal still requires fresh registry validation and is rejected in offline mode. With --online-enrich, validated proposals run first, then discovery considers remaining missing fields. Source repair proposals remain source-only.

Enrichment stays disabled by default until independent held-out release gates pass. See the implementation plan and benchmark instructions. Live provider checks are separate from fixture tests and require CITEINDEX_AI_LIVE_SMOKE=1 plus credentials.

Ingestion log (corpus/ingestion_log.jsonl)

Appended on every ingestion with input_ref, resource_type, csl_id, merkle_root, and ingestion_timestamp.

URL content hashes (corpus/_url_content_hashes.json)

Persisted URL → content-hash mapping used by --update-url-article for change detection.

Return Value

The ingest() function returns a dict:

{
    "schema_version": "1.0.0",
    "status": "ok",                    # "ok" or "blocked"
    "document_path": "corpus/Author_2024_Title",
    "standardized_csl_json": { ... },  # Full CSL-JSON with ci_ extensions
    "sub_pipeline_outputs": { ... },   # Raw pipeline results
    "ingestion_log_entry": { ... },     # Log entry with merkle_root
    "library_md_path": "library/Author_2024_Title.md",
}

# On failure:
{
    "status": "blocked",
    "source_id": "unknown",
    "stage": "detect_resource_type",
    "error_code": "unsupported_input",
    "error_message": "Unsupported input: ...",
    "next_action": "Provide PDF, URL, or media file",
}

Batch URL Ingestion Return

The ingest_all_urls() method (triggered by --all-url-article / --update-url-article) returns:

{
    "status": "ok",
    "root_url": "https://example.com/articles",
    "discovered": 25,      # total article URLs found
    "ingested": 20,        # newly ingested
    "updated": 2,           # re-ingested (content changed)
    "skipped": 3,           # unchanged (--update-url-article only)
    "failed": 0,            # errors
    "results": [            # per-URL status list
        {"url": "...", "status": "ok"},
        {"url": "...", "status": "unchanged"},
        ...
    ]
}

Supported Formats

Citation

If you use CiteIndex in your work, please cite it:

APA:

ajia. (2025). CiteIndex: Ingest sources with proper citation (Version 0.13.4). MIT. https://github.com/ajia/citeindex

BibTeX:

@software{citeindex2025,
  author  = {Yongjia, Yuan},
  title   = {CiteIndex: Ingest sources with proper citation},
  version = {0.13.4},
  year    = {2025},
  license = {MIT},
  url     = {https://github.com/ajia/citeindex},
}

License

MIT