Omna

Hybrid search (semantic + keyword), enterprise-grade PII detection & masking, and schema understanding — directly on your Polars DataFrames. No vector database. No API key. Data never leaves your machine.

The problem

# Finding every insurance claim denial — painful
keywords = ["claim denied", "coverage rejected", "policy voided", ...]
pattern  = re.compile("|".join(keywords), re.IGNORECASE)
results  = df[df["text"].str.contains(pattern, na=False)]
# Still misses: "insurer refused to honour the policy"
# Still misses: "claim outcome: not payable"
# Still misses: medical claim rejections using clinical terminology
# ...50+ lines per task. Grows with every edge case. Still wrong.

# With Omna
results = df.omna.search("insurance claim denied", on="text", k=5)
# Finds ALL of them — including docs that never say "denied" literally.
# 9ms. 50,000 documents. Zero cloud.

filtered = df.omna.filter("insurance claim denied", on="text", threshold=0.73)
# Every semantically matching document above the threshold.
# No keyword lists. No guesswork. Pure meaning.

answer = results.omna.ask("What personal data do these documents expose?")
# → "These insurance documents expose SSNs, medical record numbers,
#    dates of birth, health plan numbers, and claimant identifiers."
# Instant. One line.

# Auditing for PII before the data ships — painful
for col in df.columns:
    for i, val in enumerate(df[col].to_list()):
        if re.search(r'\b\d{3}-\d{2}-\d{4}\b', str(val)):   # SSNs only
            print(f"row {i}, {col}: {str(val)[:60]}")
# Catches one pattern. Misses emails, phones, names, IBANs.
# No confidence score. No audit trail. No redaction.

# With Omna
df.omna.pii_report()          # audit — find every leak, every column
df.omna.mask_pii()            # redact — one line, full audit log
df.omna.mask_pii(model=True)  # + on-device AI model for contextual PII
# A six-layer engine: regex + checksum-validated IDs + 220+ secret rules +
# an on-device AI model — the SAME Rust engine as the Omna Mac app & extension.
# Reversible [PERSON_1] tokens; secrets always irreversibly redacted. Local.
# Benchmarked openly on Gretel (core-PII recall 0.84 with the model, up from
# 0.69 on the prior engine): see docs/benchmark.md.

Demo

The Sword — hybrid search (semantic + keyword), filter, and ask across 50,000 documents:

The Shield — PII audit and redaction in one line:

Dataset: Gretel PII Masking Benchmark (Gretel AI, acquired by NVIDIA) — 50,000 synthetic documents (the train split) built to test data privacy tools.

Enterprise-grade PII masking

Most "PII for DataFrames" tools are a regex or a Presidio wrapper. Omna's masking is a six-layer detection engine (pure Rust, fully on-device) — the same engine that powers the Omna Mac app and browser extension:

• L1 — patterns + checksum validators: emails, SSNs, cards (Luhn), IBANs, and 30+ international IDs verified, not just pattern-matched
• L2 — secrets: 220+ rules (AWS keys, GitHub tokens, JWTs, private keys) with entropy checks
• L3 — on-device AI model (mask_pii(model=True)): catches contextual PII no regex can — bare names, addresses, medical context
• L4–L6: entity resolution, reversible [PERSON_1] tokens (secrets always irreversibly redacted), and a full audit trail

No cloud, no API key, no data leaves your machine.

Same Gretel benchmark, same scoring — only the engine changed:

df.omna.pii_report()          # audit — every PII column, with confidence
df.omna.mask_pii()            # redact (L1+L2) — instant, full audit log
df.omna.mask_pii(model=True)  # full L1–L6, model-grade

Full methodology + numbers: docs/benchmark.md.

Install

pip install "omna[all]"

Supported platforms: Python 3.10–3.12 on macOS or Linux. Windows is not supported. The PII engine (omna[pii], and mask_pii(model=True)) runs on Apple Silicon macOS and Linux — it is not available on Intel Macs, because its on-device model runtime (ONNX Runtime) ships no Intel-Mac build; Intel-Mac users can still use search, filter, and schema understanding.

Extras: omna[embed] (search/filter), omna[pii] (PII detection & masking), omna[ask] (LLM queries). A bare pip install omna (just polars + numpy + rich) gives schema understanding (understand_df()); the heavier features are opt-in. PII masking is powered by the compiled omna-pii-mask engine — a self-contained wheel with no heavy Python ML dependencies, pulled in automatically by omna[pii], omna[ask] (which masks rows before the API call), or omna[all]. No API key needed for search, filter, embed, pii_report, mask_pii, or understand. Only ask() requires ANTHROPIC_API_KEY.

Quick start

import polars as pl
import omna

df = pl.read_csv("documents.csv")

# 1 — explore the schema
omna.understand_df(df)

# 2 — audit for PII before anything touches the data
df.omna.pii_report()

# 3 — redact
clean = df.omna.mask_pii()

# 4 — build a search index once
clean.omna.embed("text")

# 5 — search by meaning
results = clean.omna.search("insurance claim denied", on="text", k=5)

# 6 — filter everything above a threshold
flagged = clean.omna.filter("insurance claim denied", on="text", threshold=0.73)

# 7 — ask a question in plain English
results.omna.ask("What personal data do these documents expose?")

What Omna does

API reference

omna.understand_df(df)

 column                dtype    null_pct   label     sample
 uid                   String     0.0%     category  24bb757...
 domain                String     0.0%     category  insurance, healthcare...
 document_type         String     0.0%     category  Invoice, ClaimForm...
 document_description  String     0.0%     text      An insurance claim...
 text                  String     0.0%     text      **Claim ID: 285-14...

df.omna.embed("text")
# → .omna/text.parquet

results = df.omna.search("insurance claim denied", on="text", k=5)

results = df.omna.search("XJ9000", on="parts", k=5)              # exact code → BM25 nails it
results = df.omna.search("claim denied", on="text", hybrid=False) # meaning only

 uid            document_type         domain      text                               _score
 67fccc1e207…   ClaimSummary          insurance   **Claim ID: 285-14-1755, Policy…   0.762
 b8ae088cd21…   ClaimSummary          insurance   **Claim Summary**…                 0.749
 de5bba0a2cc…   Insurance Claim Form  healthcare  **Insurance Claim Form**…          0.748
 ebccdde3b42…   Insurance Claim       healthcare  Insurance Claim for MED74974358…   0.747
 aebb0eb55fb…   ClaimForm             healthcare  **Claim Form** - Patient ID…       0.747

filtered = df.omna.filter("insurance claim denied", on="text", threshold=0.73)
# → N documents matched — all semantically related to claim denials

df.omna.pii_report()

 column    detected types                                    hit rate   flagged
 entities  CREDIT_CARD, EMAIL_ADDRESS, PERSON, PHONE_NUMBER   85.4%    ✓ YES
 text      CREDIT_CARD, EMAIL_ADDRESS, PERSON, PHONE_NUMBER   78.1%    ✓ YES

clean = df.omna.mask_pii()
# → Omna's own six-layer Rust engine (same kernel as the Mac app + browser
#   extension): reversible [PERSON_1]-style tokens; credentials/secrets
#   always irreversibly [REDACTED:KIND]; checksum-validated IDs; 220+ secret
#   rules. No heavy Python ML dependencies.
# → audit log saved to .omna/pii_audit.parquet automatically

clean = df.omna.mask_pii(model=True)
# → adds L3, the on-device AI model, for contextual PII regex can't catch
#   (bare prose names, addresses). Downloads the model (~809 MB) once.
# Requires the omna_pii_mask wheel (built from omna-workspace); see CHANGELOG.md.

# Add the on-device AI layer (L3) for contextual PII regex can't catch —
# bare names, addresses, medical context. Downloads the model once.
clean = df.omna.mask_pii()              # L1+L2 (instant, deterministic)

export ANTHROPIC_API_KEY=sk-ant-...

results.omna.ask("What personal data do these documents expose?")
# → "These insurance documents expose SSNs, medical record numbers,
#    dates of birth, health plan numbers, and claimant identifiers."

# Override model
results.omna.ask("Summarise the key themes", model="claude-sonnet-4-6")

How it works

df.omna.search("insurance claim denied", on="text", k=5)
         │
         ▼
   embedder.py       FastEmbed — nomic-embed-text-v1.5, local ONNX
                     query → [0.12, -0.34, 0.87, ...]  768-dim vector
         │
         ▼
   index.py          loads .omna/text.parquet → Arrow memory, zero-copy
                     50,000 stored vectors in Polars' own allocation
         │
         ▼
   similarity.rs     Rust kernel — cosine similarity over all vectors
                     returns top-k sorted descending, no Python loop
         │
         ▼
   frame.py          slices result rows, attaches _score → pl.DataFrame

The Rust kernel is under 70 lines. Dot products and norms in machine code, no intermediate allocations. 500,000 × 768-dim vectors scored in milliseconds on a single core.

Performance

Benchmarked on MacBook Air M5 with the prior 384-dim model, 10-query median, warm index. The current model (nomic-embed-text-v1.5, 768-dim) roughly doubles the per-query cosine cost — still single-digit-to-low-tens of milliseconds at these sizes.

Omna inherits Polars' Arrow columnar memory. The Rust similarity kernel operates on the same memory — no copy into NumPy, no copy into a C buffer.

FAQ

Roadmap

# Coming in v0.2
matched = transactions.omna.join(regulatory_categories, on="description")
# Match rows between two DataFrames by meaning, not exact key.

Star the repo to follow progress.

Help wanted — scaling toward a billion rows

Omna is fast and fully local at 50k–500k rows. How far can local-first semantic search go — a billion rows, on your own machine, no cloud? That's an open challenge, and the simplest workable idea wins. You don't need to know the internals to pitch in.

• Join the discussion: issue #3
• The three walls + idea directions: docs/scaling-to-a-billion-rows.md
• New contributors welcome — see CONTRIBUTING.md.

What's new

The PII engine was rebuilt from a Presidio + spaCy wrapper into Omna's own six-layer Rust engine (recall up ~2×, see Enterprise-grade PII masking), and the embedding model was upgraded to nomic-embed-text-v1.5. Full history in CHANGELOG.md.

License

omna.dev · PyPI · GitHub