Skip to main content

weji-gannet

Extract, index, and search any collection of PDF documents.

License: MIT Python 3.11+

weji-gannet turns a collection of PDFs into a properly structured, queryable database — the same way a gannet dives into the ocean and surfaces with exactly what it went in for.

Point it at any PDF collection — scanned documents, government disclosures, research papers, contracts, archives — and get clean text extraction with full-text search and optional columnar analytics via weji-goose.

Built by WEJI Northern Technologies Inc. — 100% Indigenous-owned, Qalipu Mi'kmaq First Nation, Bay St. George, NL.


How it works

Your PDFs  →  per-page extraction cascade  →  SQLite FTS5   (full-text search)
                                           →  weji-goose     (analytics, optional)

Each page is processed independently through three passes in order:

  1. pdftotext — fast, lossless for digitally-generated PDFs
  2. tesseract 250 DPI / PSM 6 — OCR for scanned pages
  3. tesseract 300 DPI / PSM 3 — higher-res OCR for complex layouts

A hybrid PDF (typed cover page + scanned attachments) gets complete, correctly ordered text from both sections with no duplication.


Install

pip install weji-gannet

# System dependencies:
sudo apt install poppler-utils tesseract-ocr tesseract-ocr-eng   # Ubuntu/Debian
brew install poppler tesseract                                     # macOS

With weji-goose columnar analytics (optional):

pip install "weji-gannet[goose]"

Quick start — Python

from weji_gannet import Pipeline
from weji_gannet.sources import LocalDirectorySource

p = Pipeline("./output")
p.ingest(LocalDirectorySource("./my_pdfs"))

# Full-text search
results = p.query("search terms here")
for r in results:
    print(r["doc_id"], r["excerpt"])

# Keyword intelligence — define your own categories
report = p.intelligence(targets={
    "contracts":     ["procurement", "tender", "awarded", "vendor"],
    "environmental": ["contamination", "remediation", "PFAS"],
    "complaints":    ["violation", "enforcement", "cease and desist"],
})

print(p.stats())

Quick start — CLI

# Index a directory of PDFs
gannet --data-dir ./output ingest ./my_pdfs

# Search
gannet --data-dir ./output query "search terms"

# Intelligence report (define your targets in a JSON file)
echo '{"contracts": ["procurement", "tender"]}' > targets.json
gannet --data-dir ./output intel --targets targets.json

# Stats
gannet --data-dir ./output stats

Extending to any PDF source

Implement DocumentSource to pull PDFs from anywhere — a web portal, an S3 bucket, a database of URLs, an API:

from weji_gannet.sources import Document, DocumentSource
from pathlib import Path

class MySource(DocumentSource):
    def documents(self):
        for row in my_db.query("SELECT id, title, dept FROM docs"):
            yield Document(
                id=str(row.id),
                title=row.title,
                source=row.dept,
            )

    def download(self, doc, dest_dir):
        pdf_bytes = my_api.fetch(doc.id)
        dest = dest_dir / f"{doc.id}.pdf"
        dest.write_bytes(pdf_bytes)
        return dest

p = Pipeline("./output")
p.ingest(MySource())

Output structure

output/
├── documents.db          — SQLite FTS5 (full-text search index)
├── goose/
│   └── documents/        — weji-goose columnar store (analytics, if installed)
├── files/                — downloaded PDFs
├── report.txt            — intelligence query results (human-readable)
└── report.json           — intelligence query results (structured)

documents table schema

Column Description
doc_id unique document identifier
source originating collection or organisation
title short label / document summary
body extracted full text (all pages, joined by \f)
year / month document date
file_path local PDF path
source_url original URL
page_count total pages
ocr_method how text was extracted (pdftotext, tesseract, hybrid:dN/oM/bK, exhausted)

Part of the WEJI stack

Package Role
weji-goose Columnar storage engine — analytics layer
weji-gannet PDF extraction and indexing pipeline

License

MIT

Metadata

Release files for weji-gannet 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for weji-gannet 0.2.0
File Size Uploaded Provenance Actions
weji_gannet-0.2.0.tar.gz 18.2 kB Without provenance Details for weji_gannet-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for weji-gannet 0.2.0
File Interpreter ABI Platform Actions
weji_gannet-0.2.0-py3-none-any.whl Python 3 none any Details for weji_gannet-0.2.0-py3-none-any.whl

Total release size: 39.2 kB

Release files / weji_gannet-0.2.0.tar.gz

Download URL weji_gannet-0.2.0.tar.gz
Size 18.2 kB
Tags Source
SHA-256 checksum
How to use checksums
da699b51c4dac3968dcacbb9a3c005cd283dc862c2ef41a99796b1756a84bd28
BLAKE2b-256 checksum
How to use checksums
b238e0b58d9b90b23f06120318e2133ad0ef24c98d97923860a697cf4d01070c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / weji_gannet-0.2.0-py3-none-any.whl

Download URL weji_gannet-0.2.0-py3-none-any.whl
Size 20.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6cd5ac8ea73fa8270dc0dbe07acc3251cb13dd3ba0ac104bdcf6455a97eb5caa
BLAKE2b-256 checksum
How to use checksums
61179f06e41e28f9f5579dacaf0eeb837b66ade8e0002eefe35333eacba5d58b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for weji-gannet 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page