weji-gannet
Extract, index, and search any collection of PDF documents.
weji-gannet turns a collection of PDFs into a properly structured, queryable database — the same way a gannet dives into the ocean and surfaces with exactly what it went in for.
Point it at any PDF collection — scanned documents, government disclosures, research papers, contracts, archives — and get clean text extraction with full-text search and optional columnar analytics via weji-goose.
Built by WEJI Northern Technologies Inc. — 100% Indigenous-owned, Qalipu Mi'kmaq First Nation, Bay St. George, NL.
How it works
Your PDFs → per-page extraction cascade → SQLite FTS5 (full-text search)
→ weji-goose (analytics, optional)
Each page is processed independently through three passes in order:
- pdftotext — fast, lossless for digitally-generated PDFs
- tesseract 250 DPI / PSM 6 — OCR for scanned pages
- tesseract 300 DPI / PSM 3 — higher-res OCR for complex layouts
A hybrid PDF (typed cover page + scanned attachments) gets complete, correctly ordered text from both sections with no duplication.
Install
pip install weji-gannet
# System dependencies:
sudo apt install poppler-utils tesseract-ocr tesseract-ocr-eng # Ubuntu/Debian
brew install poppler tesseract # macOS
With weji-goose columnar analytics (optional):
pip install "weji-gannet[goose]"
Quick start — Python
from weji_gannet import Pipeline
from weji_gannet.sources import LocalDirectorySource
p = Pipeline("./output")
p.ingest(LocalDirectorySource("./my_pdfs"))
# Full-text search
results = p.query("search terms here")
for r in results:
print(r["doc_id"], r["excerpt"])
# Keyword intelligence — define your own categories
report = p.intelligence(targets={
"contracts": ["procurement", "tender", "awarded", "vendor"],
"environmental": ["contamination", "remediation", "PFAS"],
"complaints": ["violation", "enforcement", "cease and desist"],
})
print(p.stats())
Quick start — CLI
# Index a directory of PDFs
gannet --data-dir ./output ingest ./my_pdfs
# Search
gannet --data-dir ./output query "search terms"
# Intelligence report (define your targets in a JSON file)
echo '{"contracts": ["procurement", "tender"]}' > targets.json
gannet --data-dir ./output intel --targets targets.json
# Stats
gannet --data-dir ./output stats
Extending to any PDF source
Implement DocumentSource to pull PDFs from anywhere — a web portal, an S3
bucket, a database of URLs, an API:
from weji_gannet.sources import Document, DocumentSource
from pathlib import Path
class MySource(DocumentSource):
def documents(self):
for row in my_db.query("SELECT id, title, dept FROM docs"):
yield Document(
id=str(row.id),
title=row.title,
source=row.dept,
)
def download(self, doc, dest_dir):
pdf_bytes = my_api.fetch(doc.id)
dest = dest_dir / f"{doc.id}.pdf"
dest.write_bytes(pdf_bytes)
return dest
p = Pipeline("./output")
p.ingest(MySource())
Output structure
output/
├── documents.db — SQLite FTS5 (full-text search index)
├── goose/
│ └── documents/ — weji-goose columnar store (analytics, if installed)
├── files/ — downloaded PDFs
├── report.txt — intelligence query results (human-readable)
└── report.json — intelligence query results (structured)
documents table schema
| Column | Description |
|---|---|
doc_id |
unique document identifier |
source |
originating collection or organisation |
title |
short label / document summary |
body |
extracted full text (all pages, joined by \f) |
year / month |
document date |
file_path |
local PDF path |
source_url |
original URL |
page_count |
total pages |
ocr_method |
how text was extracted (pdftotext, tesseract, hybrid:dN/oM/bK, exhausted) |
Part of the WEJI stack
| Package | Role |
|---|---|
| weji-goose | Columnar storage engine — analytics layer |
| weji-gannet | PDF extraction and indexing pipeline |
License
MIT
Metadata
Release files for weji-gannet 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| weji_gannet-0.2.0.tar.gz | 18.2 kB | Without provenance | Details for weji_gannet-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| weji_gannet-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for weji_gannet-0.2.0-py3-none-any.whl |
Total release size: 39.2 kB
Release files / weji_gannet-0.2.0.tar.gz
| Download URL | weji_gannet-0.2.0.tar.gz |
|---|---|
| Size | 18.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
da699b51c4dac3968dcacbb9a3c005cd283dc862c2ef41a99796b1756a84bd28
|
|
BLAKE2b-256 checksum How to use checksums |
b238e0b58d9b90b23f06120318e2133ad0ef24c98d97923860a697cf4d01070c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Release files / weji_gannet-0.2.0-py3-none-any.whl
| Download URL | weji_gannet-0.2.0-py3-none-any.whl |
|---|---|
| Size | 20.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6cd5ac8ea73fa8270dc0dbe07acc3251cb13dd3ba0ac104bdcf6455a97eb5caa
|
|
BLAKE2b-256 checksum How to use checksums |
61179f06e41e28f9f5579dacaf0eeb837b66ade8e0002eefe35333eacba5d58b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Security for weji-gannet 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.