bobine

Standalone PDF / Office / text → Markdown ingestion engine, extracted from the OKFgraph project as a self-contained module. Runs on a single onnxruntime wheel with no CUDA-version coupling — RapidAI family + pdf_oxide for PDFs, pure Python for text documents.

Dual-licensed under the terms of either the MIT License or the Apache License, Version 2.0 — you may choose either (see LICENSE).

Docs

• Quick reference — install, API, config, common tasks
• Architecture — modules, data flow, coordinate spaces, vendoring, testing strategy
• Implementation plan — phases, roadmap, status
• Parity audit — OKFgraph extraction fidelity + drift catalogue

Why bobine?

The ingestion pipeline was entangled with the knowledge-graph project it served. bobine moves the whole pipeline — conversion, image staging, markdown linting, document normalization — into its own package so any consumer (a graph, a CLI, an MCP server, a batch tool) can reuse it without importing a database stack.

Installation

pip install -e .                # core (Pillow only)
pip install -e ".[pdf-ingest]"  # + pdf_oxide + RapidAI ONNX passes
pip install -e ".[formula]"     # + formula OCR (vendored RapidLaTeXOCR)
pip install -e ".[markdown]"    # + mordant linting + frontmatter parsing

Everything is optional: the package imports with zero dependencies and degrades gracefully (no-op fast paths, clear RuntimeErrors when a backend is missing).

Quick start

from bobine import ConverterConfig, RoutingMode, ingest_document

# One PDF → staged, linted markdown in ./out (images → ./out/_assets,
# links rewritten to okf-asset://<id>)
result = ingest_document(
    "paper.pdf",
    "out",
    config=ConverterConfig(
        routing_mode=RoutingMode.SURGICAL,
    ),
)
print(result.md_path, result.image_count, result.page_count)

# Text documents need no native deps at all
doc = ingest_document("notes.txt", "out")

PDF conversion with ONNX heavy passes

HybridConverter routes pages through four modes:

from bobine import HybridConverter, ConverterConfig, RoutingMode

conv = HybridConverter(ConverterConfig(routing_mode=RoutingMode.AUTO))
conv.ensure_models()
md = conv.convert_pdf(
    "paper.pdf", work_dir="work", should_continue=lambda: True, on_page=lambda i, n: None
)
conv.close()

Text-type documents

from bobine import load_markdown_document, wrap_thoughts, lint_markdown

doc = load_markdown_document("note.md")  # frontmatter-aware
thought = wrap_thoughts("raw reasoning…", topic="graphs")
fixed = lint_markdown(doc.body, auto_fix=True)  # mordant, guarded

Module layout

bobine/
├── __init__.py      public API
├── config.py        ConverterConfig, RoutingMode
├── engine.py        OnnxRapidEngine (lazy ONNX model manager)
├── converter.py     HybridConverter (core PDF/Office pipeline)
├── tables.py        HTML table → GFM pipe-table converter
├── assets.py        okf-asset:// staging for extracted images
├── versions.py      RapidAI version pins + runtime check
├── documents.py     Document model, frontmatter, thoughts wrapper
├── markdown.py      mordant linting (guarded, no-op without it)
├── pipeline.py      convert_to_markdown / stage_images / ingest_document
└── _vendor/         third-party code, vendored with licenses intact
    └── rapid_latex_ocr/   formula OCR (MIT (c) 2023 RapidAI; numpy-2 fixed)

Formula OCR (SURGICAL mode)

The LaTeX formula recognizer is vendored (bobine/_vendor/rapid_latex_ocr/, MIT (c) 2023 RapidAI) with the numpy-2 incompatibility fixed upstream never addressed — no external package needed. Runtime deps come from the [formula] extra; the ONNX models (~179 MB) auto-download on first use from github.com/RapidAI/RapidLaTeXOCR/releases/download/v0.0.0/ into bobine/_vendor/rapid_latex_ocr/models/ (git-ignored).

Formula regions come from the text layer (TeX math fonts / unicode math chars), merged line-aware so multi-line display equations become one crop. For text-layer-hostile PDFs (Word/InDesign/OCR output without math fonts), set ConverterConfig(formula_layout_fallback=True) to ask the layout model for equation regions instead (pulls the rapid_layout stack into SURGICAL mode — off by default).

Output contract

ingest_document produces a directory that a graph/import layer can consume:

• <stem>.md — linted markdown with okf-asset://<id> image links
• _assets/<id>.<ext> — staged image bytes (deduped, concept-scoped ids)

bobine never embeds, indexes, or writes to a database. The consumer owns embedding and storage (in OKFgraph that is OKFRouter.import_bundle).

Testing

# unit suite (no native backends needed — fake pdf_oxide objects drive the
# converter's routing/splice/ONNX-assembly paths)
pytest

# integration suite (requires bobine[pdf-ingest] + bobine[formula])
pytest -m integration

# coverage + lint
pytest --cov=bobine --cov-report=term-missing
ruff check . && ruff format --check .

Markers: integration (real pdf_oxide/office_oxide/RapidAI + the PDF corpus) and slow (ONNX runs over real pages)

Test-PDF corpus

tests/fixtures/pdf/ holds trimmed page ranges from three CC BY 4.0 arXiv papers (solitons physics, splitting-methods math, trust-ML tables) plus a generated scanned page — see tests/fixtures/SOURCES.md for provenance and attribution. The full untrimmed PDFs are git-ignored under tests/fixtures/full_pdfs/ for local tests. The scanned page is regenerable:

uv run --with reportlab python tests/fixtures/generate_corpus.py

The integration tests self-skip when backends are missing, so the bare install always stays green. CI (.github/workflows/ci.yml) runs the core suite on Python 3.10–3.13 plus an integration job. 169 tests, 92 % coverage as of 2026-08-09.

Version pinning

RapidAI packages move fast; check_rapid_versions() warns on first import if an installed version drifts from the known-good list. Silence with BOBINE_INGEST_ALLOW_UNPINNED=1 (the legacy OKFGRAPH_INGEST_ALLOW_UNPINNED is still honoured).