warraqa 1.0.2

Warraqa (ورّاقة) — Document Scribe Agent. Converts PDFs and Word/PowerPoint files to clean Markdown with self-scoring.

Warraqa (ورّاقة)

The Document Scribe Agent

Named after the Warrāqūn — the master scribes and paper-makers of the Islamic Golden Age.

Warraqa converts PDF, Word, and PowerPoint documents into clean, accurate Markdown — and scores her own work.

---

Why Warraqa?

Most document-to-Markdown tools are one-trick ponies: great at clean PDFs, terrible at scans; great at .docx, blind to .doc; or they silently produce garbage and let you discover it three pipelines later.

Warraqa is a specialist agent. She picks the right engine for each file, falls back gracefully, scores her output from 0–100 with letter grades, and tells you which conversions to trust. She's built to feed RAG pipelines, knowledge bases, and downstream agents — where Markdown quality directly determines retrieval quality.

Features

• Dual-engine architecture — best specialized tool for each format
  • Marker (deep learning) for scanned PDFs: tables, equations, multi-column, OCR
  • PyMuPDF4LLM (fast, CPU-only) for native-text PDFs
  • MarkItDown (Microsoft) for .docx and .pptx
  • MS Office COM auto-converts legacy .doc and .ppt to modern formats first
  • Pandoc fallback for .docx resilience
• Smart triage — every PDF is pre-scanned to detect native vs. scanned content; routing is automatic
• Two-phase batch processing — fast files (native PDFs, Word, PowerPoint) run first; slow OCR work is deferred to a single trailing pass so you don't wait on Marker mid-batch
• Quality scoring — every conversion gets a 0–100 confidence score with an A–F grade across 5 dimensions (completeness, structure, encoding, density, readability)
• Crash-resistant — sanitizes invalid Unicode from upstream engines so a single bad PDF can't kill a 1000-file run
• Folder workflow — input → convert → output + move originals to processed/ or failed/
• Watch mode — continuous monitoring for new files
• Inter-agent API — designed for other agents to call programmatically

---

Quick Start

Step 1 — Install Python 3.11 or 3.12

Important: Warraqa requires Python 3.10 to 3.13. Python 3.14 is not yet supported by some upstream dependencies (Pillow, regex) and will fail during install.

Windows:

1. Go to python.org/downloads and download the Python 3.12 installer (look for "Python 3.12.x" under Stable Releases).
2. Run the installer.
3. On the very first screen, check the box that says "Add python.exe to PATH" — this is the most commonly missed step.
4. Click "Install Now".

Verify — open a new PowerShell window and run:

python --version

You should see Python 3.12.x. If you see an error, close and reopen PowerShell and try again.

---

Step 2 — Install Pandoc (for Word document fallback)

Windows (recommended):

winget install --id JohnMacFarlane.Pandoc -e

Alternative: download the .msi installer from pandoc.org/installing.html.

Verify:

pandoc --version

---

Step 3 — Install Warraqa

Open PowerShell (not the Python prompt — if you see >>>, type exit() first) and run:

pip install warraqa

This downloads Warraqa and all its dependencies (~300–400 MB including PyTorch).

Verify:

warraqa --help

---

Step 4 — Run

warraqa --folder "C:\path\to\your\documents"

The first time you convert a scanned PDF, Marker downloads its deep-learning models (~2–3 GB). This is a one-time download; subsequent runs use the cached models.

---

Option B — Clone + bootstrap script

git clone https://github.com/AALAM-Studio/warraqa.git
cd warraqa
python bootstrap.py        # creates .venv, installs deps, auto-installs Pandoc on Windows
.venv\Scripts\activate     # Linux/macOS: source .venv/bin/activate
python run.py

Option C — Docker (for cloud / headless use)

docker build -t warraqa .
docker run --rm -v "/path/to/docs:/data" warraqa --folder /data

The Docker image is CPU-only and does not include MS Office, so legacy .doc/.ppt will be skipped with a clean error message.

---

Usage

warraqa                              # Manual mode — opens a folder picker dialog
warraqa --folder "C:\path"           # Process a specific folder
warraqa --file path/to/document.pdf  # Convert a single file
warraqa --watch --folder "C:\path"   # Watch mode — continuously monitor
warraqa --folder "C:\path" --no-save --no-move    # Dry run
warraqa --help                       # All options

Output Structure

output/
├── md_files/        # Converted Markdown files
├── processed/       # Successfully converted originals
├── failed/          # Failed conversion originals
├── reports/         # JSON reports with scores and metadata
├── scanned_pdfs/    # Staging area for OCR-bound PDFs (auto-cleaned per run)
└── warraqa.log

Quality Scoring

Every conversion is scored across 5 weighted dimensions:

Dimension	Weight	What It Measures
Text Completeness	30%	Word count vs. expected density for file size
Structure Integrity	25%	Headings, lists, tables, formatting
Encoding Quality	20%	Garbled text, mojibake, Unicode issues
Content Density	15%	Meaningful text vs. noise
Readability	10%	Line length, paragraph structure

Grades: A (90–100) → B (75–89) → C (60–74) → D (40–59) → F (0–39). Files scoring below 40 are moved to output/failed/ automatically.

Inter-Agent API

from warraqa import Warraqa

agent = Warraqa()

# Convert a single file
result = agent.convert_file("document.pdf")
print(result.confidence_score)    # 87
print(result.grade)               # "B"
print(result.markdown_content)    # "# Title\n\n..."
print(result.output_path)         # Path to saved .md file

# Process a folder
results = agent.process_folder("C:/Users/you/Academia")
for r in results:
    print(f"{r.source_file.filename}: {r.grade} ({r.confidence_score}/100)")

Configuration

Edit config.yaml to customize:

• Default mode (manual / watch)
• Output directories
• Engine preferences (primary / fallback per format)
• Scoring thresholds
• Logging level

Supported Formats

Extension	Engine	Notes
.pdf (native text)	PyMuPDF4LLM	Fast, CPU-only
.pdf (scanned)	Marker	Deferred to Phase 2 OCR pass
.docx	MarkItDown → Pandoc	—
.doc	MS Office COM → MarkItDown	Windows + Office required
.pptx	MarkItDown	—
.ppt	MS Office COM → MarkItDown	Windows + Office required

---

Troubleshooting

pip install warraqa fails with "Failed building wheel for Pillow" or "Microsoft Visual C++ required"

Cause: You are running Python 3.14. Warraqa's OCR engine requires Pillow 10.x, which has no pre-built Windows package for Python 3.14.

Fix: Install Python 3.12 from python.org/downloads. You can have multiple Python versions installed. Then run:

py -3.12 -m pip install warraqa

---

pip install warraqa gives SyntaxError: invalid syntax

Cause: You typed pip install warraqa inside the Python REPL (the >>> prompt). pip is a terminal command, not a Python command.

Fix: Type exit() to leave Python, then run pip install warraqa in PowerShell.

---

warraqa is not recognized after install

Cause: Either the install failed, or Python's Scripts folder is not on your PATH.

Check if installed:

pip show warraqa
# If it shows version info, the scripts folder isn't on PATH — run via:
python -m warraqa --help

Permanent PATH fix: search Windows for "Edit the system environment variables" → Environment Variables → User Path → add C:\Users\<YourName>\AppData\Local\Programs\Python\Python312\Scripts.

---

pandoc is not recognized

Reinstall via winget install --id JohnMacFarlane.Pandoc -e and open a new PowerShell window. Warraqa still converts .docx without Pandoc — it just loses the Pandoc fallback if MarkItDown fails.

---

First scanned-PDF conversion is very slow

Normal — Marker downloads ~2–3 GB of model weights on first use. After that, models are cached.

---

Legacy .doc / .ppt files are skipped

These require Microsoft Word / PowerPoint (Windows only). If you see a "COM not available" warning, install Microsoft Office or convert the files to .docx/.pptx format first.

---

License

Warraqa is published under the PolyForm Noncommercial License 1.0.0 — a source-available license that allows free use for:

• Personal projects, research, study, and experimentation
• Academic and educational institutions
• Charitable, public-safety, health, and government organizations
• Internal evaluation by any organization

Commercial use — including using Warraqa as part of a product or service offered to paying customers, internal business operations at a for-profit company, or any revenue-generating workflow — requires a separate commercial license. Contact [e-mail] to discuss licensing.

Note on terminology: PolyForm Noncommercial is source-available, not open source in the OSI sense (which by definition allows commercial use). The full text is in LICENSE.

Versioning Policy

This repository contains the public 1.x line of Warraqa. Future major versions are developed privately and available under commercial license terms. Critical bug fixes are backported to 1.x at AALAM Studio's discretion.

See CHANGELOG.md for the release history.

Citation

If Warraqa contributes to academic research, please cite it. A machine-readable CITATION.cff is provided, or use the GitHub "Cite this repository" button.

Acknowledgements

Warraqa stands on the shoulders of excellent open-source projects:

• Marker — Vik Paruchuri's deep-learning PDF parser
• PyMuPDF4LLM — Artifex's LLM-optimized PDF extraction
• MarkItDown — Microsoft's universal-to-markdown converter
• Pandoc — John MacFarlane's document conversion swiss-army knife
• Rich — Will McGugan's terminal beautifier

Part of Aalam Studio

Warraqa is the first publicly released agent in the AALAM Studio ecosystem. Other agents access her output at a predictable path:

WARRAQA_OUTPUT = "c:/projects/aalam-studio/warraqa/output/"

---

She reads. She transcribes. She scores her own work.

Built with care by AALAM Studio.