pdf2qa 0.1.0

PDF → OCR → sliding-window chunking → multi-question generation via LLM → JSONL dataset for fine-tuning

pdf2qa

End-to-end pipeline: PDF → OCR → sliding-window chunking → multi-question generation → JSONL dataset for LLM fine-tuning.

pip install pdf2qa
pdf2qa book.pdf --api-key sk-...

Features

• OCR — Scanned PDFs via PyMuPDF + Tesseract (optimised for Bengali/Arabic)
• Sliding window — Configurable chunk size/stride ensures no content is missed
• Multi-question — Generates 2–3 distinct questions per chunk, each saved as a separate JSONL object
• Language-agnostic — Default English prompt, override for any language with --system-prompt
• Checkpoint/resume — Survives interruptions and rate-limits
• Retry logic — Primary → fallback API with exponential backoff

Quickstart

# With a PDF (auto-OCR)
pdf2qa book.pdf --api-key sk-... --output ./dataset

# With pre-extracted text
pdf2qa book.txt --skip-ocr --api-key sk-... --output ./dataset

# Custom chunking
pdf2qa book.pdf --window 4 --stride 2 --api-key sk-...

Environment variable

Set PDF_QA_API_KEY instead of passing --api-key every time.

Output format

JSONL (one object per line) in OpenAI fine-tuning format:

{"messages": [{"role": "user", "content": "প্রশ্ন?"}, {"role": "assistant", "content": "উত্তর..."}]}
{"messages": [{"role": "user", "content": "অন্য প্রশ্ন?"}, {"role": "assistant", "content": "উত্তর..."}]}

Options

Argument	Default	Description
--lang	ben+ara	Tesseract languages
--dpi	400	OCR render DPI
--window	3	Pages per chunk
--stride	1	Sliding stride
--output	./output	Output directory
--skip-ocr	–	Input is already .txt
--model	qwen-code-35b	LLM model ID
--api-key	–	API key (or $PDF_QA_API_KEY)
--primary-url	http://...	Primary API URL
--fallback-url	https://...	Fallback API URL
--system-prompt	(built-in)	Custom system prompt ({num_questions} placeholder)
--user-prompt	(built-in)	Custom user prompt ({chunk}, {num_questions})
--question-keywords	What Why How ...	Keywords to identify a question line

Development

git clone https://github.com/yourusername/pdf2qa
cd pdf2qa
pip install -e ".[dev]"
pytest