weji-gannet

Extract, index, and search any collection of PDF documents.

weji-gannet turns a collection of PDFs into a properly structured, queryable database — the same way a gannet dives into the ocean and surfaces with exactly what it went in for.

Point it at any PDF collection — scanned documents, government disclosures, research papers, contracts, archives — and get clean text extraction with full-text search and optional columnar analytics via weji-goose.

Built by WEJI Northern Technologies Inc. — 100% Indigenous-owned, Qalipu Mi'kmaq First Nation, Bay St. George, NL.

How it works

Your PDFs  →  per-page extraction cascade  →  SQLite FTS5   (full-text search)
                                           →  weji-goose     (analytics, optional)

Each page is processed independently through three passes in order:

1. pdftotext — fast, lossless for digitally-generated PDFs
2. tesseract 250 DPI / PSM 6 — OCR for scanned pages
3. tesseract 300 DPI / PSM 3 — higher-res OCR for complex layouts

A hybrid PDF (typed cover page + scanned attachments) gets complete, correctly ordered text from both sections with no duplication.

Install

pip install weji-gannet

# System dependencies:
sudo apt install poppler-utils tesseract-ocr tesseract-ocr-eng   # Ubuntu/Debian
brew install poppler tesseract                                     # macOS

With weji-goose columnar analytics (optional):

pip install "weji-gannet[goose]"

Quick start — Python

from weji_gannet import Pipeline
from weji_gannet.sources import LocalDirectorySource

p = Pipeline("./output")
p.ingest(LocalDirectorySource("./my_pdfs"))

# Full-text search
results = p.query("search terms here")
for r in results:
    print(r["doc_id"], r["excerpt"])

# Keyword intelligence — define your own categories
report = p.intelligence(targets={
    "contracts":     ["procurement", "tender", "awarded", "vendor"],
    "environmental": ["contamination", "remediation", "PFAS"],
    "complaints":    ["violation", "enforcement", "cease and desist"],
})

print(p.stats())

Quick start — CLI

# Index a directory of PDFs
gannet --data-dir ./output ingest ./my_pdfs

# Search
gannet --data-dir ./output query "search terms"

# Intelligence report (define your targets in a JSON file)
echo '{"contracts": ["procurement", "tender"]}' > targets.json
gannet --data-dir ./output intel --targets targets.json

# Stats
gannet --data-dir ./output stats

Extending to any PDF source

Implement DocumentSource to pull PDFs from anywhere — a web portal, an S3 bucket, a database of URLs, an API:

from weji_gannet.sources import Document, DocumentSource
from pathlib import Path

class MySource(DocumentSource):
    def documents(self):
        for row in my_db.query("SELECT id, title, dept FROM docs"):
            yield Document(
                id=str(row.id),
                title=row.title,
                source=row.dept,
            )

    def download(self, doc, dest_dir):
        pdf_bytes = my_api.fetch(doc.id)
        dest = dest_dir / f"{doc.id}.pdf"
        dest.write_bytes(pdf_bytes)
        return dest

p = Pipeline("./output")
p.ingest(MySource())

Output structure

output/
├── documents.db          — SQLite FTS5 (full-text search index)
├── goose/
│   └── documents/        — weji-goose columnar store (analytics, if installed)
├── files/                — downloaded PDFs
├── report.txt            — intelligence query results (human-readable)
└── report.json           — intelligence query results (structured)

documents table schema

Part of the WEJI stack

License

MIT