bibr 0.0.1

Scientific paper processing pipeline for metadata extraction from PDF/DOCX/LaTeX/etc.

bibr 🦫

Description

Scientific paper processing pipeline, extracts comprehensive metadata from PDF/DOCX/LaTeX/etc. Designed as a preprocessing backend for Metacheck.

Overview

At its core, bibr aims to provide better accuracy than other tools, using more modern approaches and offering additional features. Written in Python.

Features

• easy-to-use Python library, install with uv
• modular, scalable pipeline architecture
• supports PDF, TXT, Markdown, DOCX, HTML, etc. as input
• outputs an Arrow stream with parsed metadata, references, full-text content, etc.
• extracts:
  • paper metadata (title, authors, DOI, etc.) with high accuracy
  • sections with semantic classification of canonical sections (abstract, methods, etc.)
  • sentences (using spaCy tokenization for better accuracy)
  • tables with representation in markdown and JSON (currently fragile)
  • references/bibs
  • (URL) links
• uses a mix of traditional methods (Regex, NLP) and structured LLM-based extraction for better accuracy
• response caching with Redis, with TTL and versioning options
• API key, rate limiting, security features

Dependencies

Not every service is used for every document; the pipeline adapts based on input type and available metadata.

• LightOnOCR 2: Open source OCR model for PDFs, made in EU (https://huggingface.co/lightonai/LightOnOCR-2-1B)
• Pandoc: Converts markdown/docx/LaTeX/etc. documents into a standardized structured tree format (AST)
• LLM API: Helps extract paper metadata with structured output
• Crossref API: Optional - enhances reference metadata, checks for DOIs, etc.

LLM disclaimer

In some places, bibr selectively uses Large Language Models to extract metadata with more accuracy. This is done only where strictly needed - such as with references and citations, where traditional methods (Regex) are not always accurate and sensitive across varying contexts and citation styles.

Supported LLM Providers

bibr supports multiple LLM providers. Set LLM_PROVIDER in your .env file:

• google (default): Google AI Studio (Gemini). Requires GOOGLE_API_KEY.
• openai: OpenAI API or any OpenAI-compatible endpoint. Requires LLM_API_KEY. Use LLM_BASE_URL for custom endpoints (vLLM, LM Studio, etc.).
• ollama: Local Ollama instance. Set OLLAMA_BASE_URL if not using the default http://localhost:11434.

Privacy and security

For LLM providers, we recommend using providers with strict privacy policies or open source models. Please be careful with your API keys in the .env file and if deploying in production, use a Secrets Manager instead.

How to run