GROBID Client Python

A simple, efficient Python client for GROBID REST services that provides concurrent processing capabilities for PDF documents, reference strings, and patents.

📋 Table of Contents

• Features
• Prerequisites
• Installation
• Quick Start
• Usage
  • Command Line Interface
  • Python Library
• Configuration
• Services
• Testing
• Performance
• Development
• License

✨ Features

• Concurrent Processing: Efficiently process multiple documents in parallel
• Flexible Input: Process PDF files, text files with references, and XML patents
• Configurable: Customizable server settings, timeouts, and processing options
• Command Line & Library: Use as a standalone CLI tool or import into your Python projects
• Coordinate Extraction: Optional PDF coordinate extraction for precise element positioning
• Sentence Segmentation: Layout-aware sentence segmentation capabilities
• JSON Output: Convert TEI XML output to structured JSON format with CORD-19-like structure
• Markdown Output: Convert TEI XML output to clean Markdown format with structured sections
• Type Hints: Ships inline type annotations and a py.typed marker (PEP 561) for static type checking
• Archive Streaming: Process files directly from .zip/.tar/.tar.gz archives without fully decompressing them
• S3 Streaming: Read PDFs and zips straight from s3:// (range-streamed, no full download) with the optional [s3] extra
• In-Memory Documents: Send PDFs held as bytes straight to GROBID, without writing them to disk first

📋 Prerequisites

• Python: 3.8 - 3.13 (tested versions)
• GROBID Server: A running GROBID service instance
  • Local installation: GROBID Documentation
  • Docker: docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.8.2
  • Default server: http://localhost:8070
  • Online demo: https://lfoppiano-grobid.hf.space (usage limits apply), more details here.

🚀 Installation

Choose one of the following installation methods:

PyPI (Recommended)

pip install grobid-client-python

# to stream inputs directly from S3 (s3:// URIs), install the optional 's3' extra:
pip install "grobid-client-python[s3]"

Development Version

pip install git+https://github.com/kermitt2/grobid_client_python.git

Local Development

git clone https://github.com/kermitt2/grobid_client_python
cd grobid_client_python
pip install -e .

⚡ Quick Start

Command Line

# Process PDFs in a directory
grobid_client --input ./pdfs --output ./output processFulltextDocument

# Process with custom server
grobid_client --server https://your-grobid-server.com --input ./pdfs processFulltextDocument

Python Library

from grobid_client.grobid_client import GrobidClient

# Create client instance
client = GrobidClient(config_path="./config.json")

# Process documents
client.process("processFulltextDocument", "/path/to/pdfs", n=10)

📖 Usage

Command Line Interface

The client provides a comprehensive CLI with the following syntax:

grobid_client [OPTIONS] SERVICE

Available Services

Common Options

Processing Options

Examples

# Basic fulltext processing
grobid_client --input ~/documents --output ~/results processFulltextDocument

# High concurrency with coordinates
grobid_client --input ~/pdfs --output ~/tei --n 20 --tei_coordinates processFulltextDocument

# Process with JSON output
grobid_client --input ~/pdfs --output ~/results --json processFulltextDocument

# Process with Markdown output
grobid_client --input ~/pdfs --output ~/results --markdown processFulltextDocument

# Process citations with custom server
grobid_client --server https://grobid.example.com --input ~/citations.txt processCitationList

# Force reprocessing with sentence segmentation and JSON output
grobid_client --input ~/docs --force --segment_sentences --json processFulltextDocument

# Resume an interrupted run without retrying the documents that already failed
grobid_client --input ~/docs --output ~/results --skip_errors processFulltextDocument

# Process PDFs directly from a zip or tar.gz archive (streamed, not fully decompressed)
grobid_client --input ~/papers.zip --output ~/results processFulltextDocument
grobid_client --input ~/papers.tar.gz --output ~/results processFulltextDocument

# --input also accepts glob patterns (quote them so the shell does not expand them)
grobid_client --input "~/papers/*.zip"    --output ~/results processFulltextDocument   # many archives
grobid_client --input "~/data/**/*.pdf"   --output ~/results processFulltextDocument   # PDFs in subdirectories

Python Library

Basic Usage

from grobid_client.grobid_client import GrobidClient

# Initialize with default localhost server
client = GrobidClient()

# Initialize with custom server
client = GrobidClient(grobid_server="https://your-server.com")

# Initialize with config file
client = GrobidClient(config_path="./config.json")

# Process documents
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    n=20
)

Advanced Usage

# Process with specific options
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    n=10,
    generate_ids=True,
    consolidate_header=True,
    tei_coordinates=True,
    segment_sentences=True
)

# Process with JSON output
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    json_output=True
)

# Process with Markdown output
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    markdown_output=True
)

# Re-run without retrying the documents that failed before
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    force=False,
    skip_errors=True
)

```python
# Process citation lists
client.process(
    service="processCitationList",
    input_path="/path/to/citations.txt",
    output_path="/path/to/output"
)

Processing a PDF from memory

A PDF that is already in memory - downloaded from an API, read from a database or an object store - can be sent directly, without writing it to a temporary file first. process_pdf takes either a path or the document itself, as bytes or as any binary stream, and returns the TEI as a string:

import io
import requests

pdf = io.BytesIO(requests.get("https://example.org/paper.pdf").content)
pdf.name = "paper.pdf"          # optional, see below

name, status, tei = client.process_pdf(
    service="processFulltextDocument",
    pdf_file=pdf,
    consolidate_header=True,
    tei_coordinates=True
)

if status == 200:
    print(tei)

There is no flag to say where the document comes from: the object itself says it. A document also carries its own name, taken from the name attribute that open() sets on files and that can be set on anything else, io.BytesIO included. The name identifies the document in the request sent to GROBID, in the logs, and as the first element of the result, so documents processed this way stay distinguishable. Bytes passed on their own have nothing to be named after and fall back to document.pdf.

Several documents can be sent concurrently with process_documents, which runs them through the same thread pool the file-based processing uses:

results = client.process_documents(
    service="processFulltextDocument",
    documents=[pdf1, pdf2, "/path/to/paper3.pdf"],
    n=10                            # documents sent concurrently
)

for name, status, tei in results:
    ...

Documents that do not name themselves are named document-1.pdf, document-2.pdf, ... after their position. Results come back in the order the documents were given, not in completion order, so they can be zipped back onto whatever the caller has them keyed by. A document that fails does not stop the others: its own entry carries the error status.

Before an in-memory run starts (this includes archive and s3:// streaming), the client asks the server's /api/health how many engines it actually has and logs a warning when the requested concurrency n exceeds them - the surplus requests would only queue on the server or bounce as 503 - and an info message when engines would sit idle. The check is advisory: a server without the endpoint (older GROBID) never blocks the run.

Standalone Conversion Tools

The library includes standalone scripts to convert TEI XML files to other formats without using the main client or server.

TEI to JSON Converter

Converts TEI XML files to the structured JSON format (similar to --json option).

# Convert a single file
python -m grobid_client.format.TEI2LossyJSON_cli --input path/to/file.tei.xml --output path/to/output.json

# Convert with verbose logging
python -m grobid_client.format.TEI2LossyJSON_cli --input path/to/file.tei.xml --verbose

TEI to Markdown Converter

Converts TEI XML files to Markdown format (similar to --markdown option).

# Convert a single file
python -m grobid_client.format.TEI2Markdown_cli --input path/to/file.tei.xml --output path/to/output.md

⚙️ Configuration

Configuration can be provided via a JSON file. When using the CLI, the --server argument overrides the config file settings.

Default Configuration

{
  "grobid_server": "http://localhost:8070",
  "sleep_time": 5,
  "timeout": 60,
  "coordinates": [
    "persName",
    "figure",
    "ref",
    "biblStruct",
    "formula",
    "s"
  ]
}

Configuration Parameters

Choosing a queue size

queue_size controls how many files are grouped into one processing chunk. It is a memory knob, not a concurrency one: parallelism toward the GROBID server is set by -n, and each chunk is processed n files at a time. When queue_size is not set, the client picks a sensible default per input type (see the table above); set it explicitly only if you need to override that. A few guidelines:

• Never set it below n. Effective parallelism is min(n, queue_size): a queue smaller than the thread pool leaves workers idle. A bit of headroom above n (the default streaming value is 1.2 × n) keeps the pool busy.
• Results are written as they complete. Each output file lands on disk as soon as its document has been processed, whatever the queue size, so an interrupted run only loses the documents that were in flight (already-written results are skipped on re-run unless --force is used).
• Local PDF directories: the queue holds only file paths (each file is opened when it is sent), so large values are essentially free — the default is 1000. At the end of each chunk the pool waits for its slowest document, so a large queue keeps the workers busy.
• Archives (zip/tar) and S3: each chunk is read or downloaded into memory before processing starts, so peak RAM grows with queue_size × average file size. Keep it moderate — the 1.2 × n default is a safe floor; going up to a few multiples of n (e.g. 2–5 ×) trades memory for slightly better throughput around chunk boundaries.

Logging Configuration

The client provides configurable logging with different verbosity levels. By default, only essential statistics and warnings are shown.

Logging Behavior

• Without --verbose: Shows only essential information and warnings/errors
• With --verbose: Shows detailed processing information at INFO level

Always Visible Output

The following information is always displayed regardless of the --verbose flag:

Found 1000 file(s) to process
Processing completed: 950 out of 1000 files processed
Errors: 50 out of 1000 files processed
Processing completed in 120.5 seconds

Verbose Output (--verbose)

When the --verbose flag is used, additional detailed information is displayed:

• Server connection status
• Individual file processing details
• JSON conversion messages
• Detailed error messages
• Processing progress information

Examples

# Clean output - only essential statistics
grobid_client --input pdfs/ processFulltextDocument
# Output:
# Found 1000 file(s) to process
# Processing completed: 950 out of 1000 files processed
# Errors: 50 out of 1000 files processed
# Processing completed in 120.5 seconds

# Verbose output - detailed processing information
grobid_client --input pdfs/ --verbose processFulltextDocument
# Output includes all essential stats PLUS:
# GROBID server http://localhost:8070 is up and running
# JSON file example.json does not exist, generating JSON from existing TEI...
# Successfully created JSON file: example.json
# ... and other detailed processing information

Configuration File Logging

The config file can include logging settings:

{
    "grobid_server": "http://localhost:8070",
    "logging": {
        "level": "WARNING",
        "format": "%(asctime)s - %(levelname)s - %(message)s",
        "console": true,
        "file": null
    }
}

Note: The --verbose command line flag always takes precedence over configuration file logging settings.

🔬 Services

Fulltext Document Processing

Extracts complete document structure including headers, body text, figures, tables, and references.

grobid_client --input pdfs/ --output results/ processFulltextDocument

JSON Output Format

When using the --json flag, the client converts TEI XML output to a structured JSON format similar to CORD-19. This provides:

• Structured Bibliography: Title, authors, DOI, publication date, journal information
• Body Text: Paragraphs and sentences with metadata and reference annotations
• Figures and Tables: Structured JSON format for tables with headers, rows, and metadata
• Reference Information: In-text citations with offsets and targets

JSON Structure

{
  "level": "paragraph",
  "biblio": {
    "title": "Document Title",
    "authors": [
      "Author 1",
      "Author 2"
    ],
    "doi": "10.1000/example",
    "publication_date": "2023-01-01",
    "journal": "Journal Name",
    "abstract": [
      ...
    ]
  },
  "body_text": [
    {
      "id": "p_12345",
      "text": "Paragraph text with citations [1].",
      "head_section": "Introduction",
      "refs": [
        {
          "type": "bibr",
          "target": "b1",
          "text": "[1]",
          "offset_start": 25,
          "offset_end": 28
        }
      ]
    }
  ],
  "figures_and_tables": [
    {
      "id": "table_1",
      "type": "table",
      "label": "Table 1",
      "head": "Sample Data",
      "content": {
        "headers": [
          "Header 1",
          "Header 2"
        ],
        "rows": [
          [
            "Value 1",
            "Value 2"
          ]
        ],
        "metadata": {
          "row_count": 1,
          "column_count": 2,
          "has_headers": true
        }
      }
    }
  ]
}

Usage Examples

# Generate both TEI and JSON outputs
grobid_client --input pdfs/ --output results/ --json processFulltextDocument

# JSON output with coordinates and sentence segmentation
grobid_client --input pdfs/ --output results/ --json --tei_coordinates --segment_sentences processFulltextDocument

# Python library usage
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    json_output=True
)

Markdown Output Format

When using the --markdown flag, the client converts TEI XML output to a clean, readable Markdown format. This provides:

• Structured Sections: Title, Authors, Affiliations, Publication Date, Fulltext, Annex, and References
• Clean Formatting: Human-readable format suitable for documentation and sharing
• Preserved Content: All text content with proper section organization
• Reference Formatting: Bibliographic references in a readable format

Markdown Structure

The generated Markdown follows this structure:

# Document Title

## Authors

- Author Name 1
- Author Name 2

## Affiliations

- Affiliation 1
- Affiliation 2

## Publication Date

January 1, 2023

## Fulltext

### Introduction

Content of the introduction section...

### Methods

Content of the methods section...

## Annex

### Acknowledgements

Acknowledgement text...

### Competing Interests

Competing interests statement...

## References

**[1]** Paper Title. *Author Name*. *Journal Name* (2023).
**[2]** Another Paper. *Author et al.*. *Conference* (2022).

Usage Examples

# Generate both TEI and Markdown outputs
grobid_client --input pdfs/ --output results/ --markdown processFulltextDocument

# Markdown output with coordinates and sentence segmentation
grobid_client --input pdfs/ --output results/ --markdown --tei_coordinates --segment_sentences processFulltextDocument

# Python library usage
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    markdown_output=True
)

Header Document Processing

Extracts only document metadata (title, authors, abstract, etc.).

grobid_client --input pdfs/ --output headers/ processHeaderDocument

Reference Processing

Extracts and structures bibliographic references from documents.

grobid_client --input pdfs/ --output refs/ processReferences

Citation List Processing

Parses raw citation strings from text files.

grobid_client --input citations.txt --output parsed/ processCitationList

🧪 Testing

The project includes comprehensive unit and integration tests using pytest.

Running Tests

# Install development dependencies
pip install -e .[dev]

# Run all tests
pytest

# Run with coverage
pytest --cov=grobid_client

# Run specific test file
pytest tests/test_client.py

# Run with verbose output
pytest -v

Test Structure

• tests/test_client.py - Unit tests for the base API client
• tests/test_grobid_client.py - Unit tests for the GROBID client
• tests/test_integration.py - Integration tests with real GROBID server
• tests/conftest.py - Test configuration and fixtures

Continuous Integration

Tests are automatically run via GitHub Actions on:

• Push to main branch
• Pull requests
• Multiple Python versions (3.8-3.13)

📊 Performance

Benchmark results for processing 136 PDFs (3,443 pages total, ~25 pages per PDF) on Intel Core i7-4790K CPU 4.00GHz:

Additional Benchmarks

• Header processing: 3.74s for 136 PDFs (36 PDF/s) with n=10
• Reference extraction: 26.9s for 136 PDFs (5.1 PDF/s) with n=10
• Citation parsing: 4.3s for 3,500 citations (814 citations/s) with n=10

🛠️ Development

Setting Up Development Environment

# Clone the repository
git clone https://github.com/kermitt2/grobid_client_python
cd grobid_client_python

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode with test dependencies
pip install -e .[dev]

# Install pre-commit hooks (optional)
pre-commit install

Creating a New Release

The project uses bump-my-version for version management:

# Install bump-my-version
pip install bump-my-version

# Bump version (patch, minor, or major)
bump-my-version bump patch

# The release will be automatically published to PyPI

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run the test suite (pytest)
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

📄 License

Distributed under the Apache 2.0 License. See LICENSE for more information.

👥 Authors & Contact

Main Author: Patrice Lopez (patrice.lopez@science-miner.com)
Maintainer: Luca Foppiano (luca@sciencialab.com)

🔗 Links

• GROBID Documentation
• PyPI Package
• GitHub Repository
• Issue Tracker