MinerU RAG

A Python package for MinerU document processing and RAG (Retrieval-Augmented Generation) knowledge base construction.

Features

• 📄 MinerU Integration: Support both online MinerU API and local vLLM backend
• 🤖 RAG Knowledge Base: Easy-to-use RAG system for building knowledge bases
• 🔗 LLM Connection: Seamless integration with LLM APIs
• 🚀 Simple API: Clean and intuitive Python API

Installation

pip install mineru-rag

For RAG functionality, install with extras:

pip install mineru-rag[rag]

📖 详细使用文档请查看 USER_GUIDE.md

Quick Start

1. Process Documents with MinerU

Using Online API

from mineru_rag import MinerUClient

# Initialize client with API token
client = MinerUClient(api_token="your-mineru-api-token")

# Process a single file
result = client.process_file(
    input_path="document.pdf",
    output_path="./output"
)

# Process multiple files
results = client.process_files_batch(
    file_paths=["doc1.pdf", "doc2.pdf"],
    output_dir="./output"
)

Using Local vLLM Backend

from mineru_rag import MinerUClient

# Initialize client for local mode
# Make sure MinerU vLLM backend is running at http://127.0.0.1:30000
client = MinerUClient(use_local=True, local_url="http://127.0.0.1:30000")

# Process files (same API as online mode)
result = client.process_file(
    input_path="document.pdf",
    output_path="./output"
)

2. Build RAG Knowledge Base

from mineru_rag import RAGBuilder
from pathlib import Path

# Initialize RAG builder
rag = RAGBuilder()

# Build from processed markdown files
markdown_files = [
    Path("./output/doc1/full.md"),
    Path("./output/doc2/full.md")
]

rag.build_from_files(
    file_paths=markdown_files,
    library_id="my_library"
)

# Or load existing vector store
rag.load_vector_store(library_id="my_library")

3. Query with LLM

from mineru_rag import LLMClient, RAGBuilder

# Initialize LLM client
llm = LLMClient(
    api_key="your-openai-api-key",
    base_url="http://your-api-server/v1/",
    model="gpt-3.5-turbo"
)

# Initialize RAG builder
rag = RAGBuilder()
rag.load_vector_store(library_id="my_library")

# Query
rag_result = rag.query("What is the main contribution of this paper?", k=4)
answer = llm.query_with_rag(rag_result)

print(answer['answer'])

4. Complete Workflow

from mineru_rag import MinerUClient, RAGBuilder, LLMClient
from pathlib import Path

# 1. Process documents
client = MinerUClient(api_token="your-mineru-api-token")
result = client.process_file("paper.pdf", "./output")

# 2. Build RAG knowledge base
rag = RAGBuilder()
md_file = Path(result['md_file'])
rag.build_from_files([md_file], library_id="papers")

# 3. Query
llm = LLMClient(
    api_key="your-api-key",
    base_url="http://your-api-server/v1/"
)
rag.load_vector_store("papers")
rag_result = rag.query("What are the key findings?", k=4)
answer = llm.query_with_rag(rag_result)
print(answer['answer'])

Configuration

Environment Variables

MinerU Online API

export MINERU_API_TOKEN="your-mineru-api-token"

LLM API

export OPENAI_API_KEY="your-openai-api-key"
export OPENAI_BASE_URL="http://your-api-server/v1/"
export OPENAI_MODEL="gpt-3.5-turbo"
export OPENAI_TEMPERATURE="0.7"

Local MinerU vLLM Backend

To use local MinerU vLLM backend:

1. Install MinerU and start vLLM backend:

# Install MinerU (follow MinerU documentation)
# Start vLLM backend on port 30000

2. Use local mode:

client = MinerUClient(use_local=True, local_url="http://127.0.0.1:30000")

Command Line Usage

Process Documents

# Online mode
mineru-rag process document.pdf -o ./output --api-token your-token

# Local mode
mineru-rag process document.pdf -o ./output --local --local-url http://127.0.0.1:30000

Build RAG Knowledge Base

mineru-rag build doc1.md doc2.md -l my_library

Query RAG

mineru-rag query "What is the main contribution?" -l my_library -k 4

API Reference

MinerUClient

• process_file(input_path, output_path, ...): Process a single file
• process_files_batch(file_paths, output_dir, ...): Process multiple files

RAGBuilder

• build_from_files(file_paths, library_id, ...): Build vector database from files
• load_vector_store(library_id): Load existing vector database
• query(question, k, file_id): Query the knowledge base

LLMClient

• query(question, context): Query LLM with context
• query_with_rag(rag_result): Query LLM with RAG result

📚 文档

• USER_GUIDE.md - 完整用户使用文档（推荐阅读）
• QUICKSTART.md - 快速开始指南
• examples/ - 使用示例代码

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.