Categorizer

Easily categorize string records into predefined categories with a combination of LLM, regex and keywords.

• Define your categories, feed your input — records come back categorized.
• Nested categories are allowed (any depth).
• For efficiency, regex patterns and keywords can pre-handle records where applicable, and the LLM acts as the fallback — you only pay for what rules can't resolve.

Install

pip install -r categorizer/requirements.txt   # pydantic, pyyaml, langchain-openai, python-dotenv
echo 'OPENAI_API_KEY=sk-...' >> .env           # only needed for the LLM phase

The key is read from a .env file in your project (auto-loaded, no export needed). An already-exported OPENAI_API_KEY environment variable takes precedence over .env.

Quick start

from categorizer import Categorizer, Taxonomy, cat, to_dataframe

taxonomy = Taxonomy(categories=[            # Python builder…
    cat("Food & Dining",
        cat("Coffee", keywords=["KOFEYNYA"]),
        cat("Groceries")),
    cat("Healthcare",
        cat("Medications", keywords=["ECZANE"])),
])
# …or: Taxonomy.from_yaml("categories.yaml") / from_json(...) / from_dict({...})

c = Categorizer(taxonomy)
results = c.categorize([
    "coffee - kofeynya 30 dollars",         # resolved by keyword, free
    "dinner at a small italian place",      # resolved by the LLM
])

for r in results:
    print(r.path, r.method, r.rationale)    # ['Food & Dining', 'Coffee'] keyword …
df = to_dataframe(results)                  # flat table: lvl1, lvl2, method, ok, …

Async: await c.acategorize(records). Inputs: list of strings, dicts, Record objects, or a pandas DataFrame.

Runnable demo: python -m categorizer.example

The three phases

1. Patterns (regex, instant, free) — per-source rules that assign a full category path in one hit; a record's source field selects the rule set.
2. Keywords (substring, instant, free) — per-category trigger words assign the full path to that category.
3. LLM (semantic, costs money) — only the leftovers. Level-by-level walk with structured output, so an out-of-taxonomy answer is impossible. Records sharing a keyword are called once and share the result (method="cache"), and the cache persists across calls.

Each phase can be toggled (use_patterns / use_keywords / use_llm).

Batch mode

batch_size=N (default 1) sends up to N records per LLM call — per tree level, per group — not the whole run in one call:

• Only records that reach the LLM phase batch at all: patterns, keywords, and the cache resolve their records for free first.
• A group = records at the same position in the tree with the same source. Each group is chunked into batches of ≤ N; all batches fly concurrently under max_concurrency.
• The walk stays level-by-level: one call judges a batch against the top-level categories, then records regroup by the parent each one chose and batch again for that parent's children.

Example: 10 records on a 2-level taxonomy = 20 calls per-record, but ~7 batched — 1 call at level 1, then one smaller call per chosen parent at level 2.

Reliability: batch items are id-keyed, so a missing, duplicated, or invalid item in a response automatically re-runs as a single call at that level, and a failed batch call degrades its whole chunk to singles — nothing is silently dropped. allow_no_fit verdicts work inside batches.

When to use it: batching is the cost and rate-limit lever — fewer requests, and the taxonomy tokens amortized across N records. It is not a latency lever: for small runs, N parallel single calls finish faster than one call writing N rationales. Reach for it when record counts vastly outnumber max_concurrency or requests-per-minute limits bind, and keep N modest (≤ ~10–20): per-record accuracy degrades as batches grow.

Config files

The taxonomy is a typed Pydantic model; file formats are just loaders — YAML in, JSON in, dict in, Python builder in, same validated Taxonomy out. Typos, duplicate siblings, and patterns pointing at nonexistent categories fail fast with clear errors.

categories.yaml — the taxonomy (full example: categorizer/examples/categories.yaml):

categories:
  - name: Food & Dining
    rules:                                   # hard constraints shown to the LLM
      - "Supermarket purchases (Migros, BIM, ...) belong here, not in Retail Purchases."
    children:
      - name: Coffee
        keywords: [KOFEYNYA]                 # substring auto-triggers
        description: ""                      # optional context for the LLM

patterns.yaml — per-source regex rules (full example: categorizer/examples/bank_patterns.yaml):

sources:
  QNB Finansbank Enpara:
    - pattern: "Gelen Transfer"
      path: [Incoming P2P Transfers, Incoming Money]

Editor autocomplete / CI validation:

Taxonomy.write_file_schema("categories.schema.json")
Patterns.write_file_schema("patterns.schema.json")
# YAML header:  # yaml-language-server: $schema=categories.schema.json

Steering tip: the LLM decides one level at a time and only sees that level's options. Put rules/description at the level where the wrong turn happens — a helper on a level-2 child cannot fix a level-1 mistake.

What you get back

Each record yields a Result: path (root→leaf names), method (pattern | keyword | llm | cache), per-level rationale, ok flag, and error if unresolved. One bad record never stops the batch.

Records that genuinely don't belong anywhere are signaled, not crammed into the nearest bin: Result.unplaced carries a grounded verdict — no_fit(off_axis | uncovered | under_bar) when no category can be truthfully grounded, or cant_tell when the text is too opaque to identify. The rationale states why. No-fit is an outcome, never a category.

Knobs

Categorizer(
    taxonomy, patterns,
    model="gpt-5.4-nano",        # any OpenAI chat model
    llm=my_langchain_model,      # or bring any LangChain chat model
    use_patterns=True, use_keywords=True, use_llm=True,
    allow_no_fit=True,           # False → forced choice: every record gets a category
    batch_size=1,                # >1 → judge N records per LLM call (cuts requests and
                                 #   amortizes the taxonomy tokens; accuracy degrades as
                                 #   N grows — keep ≤ ~10-20; misses auto-repair as singles)
    cache=True,
    max_concurrency=32,
    on_progress=lambda p: ...,   # Progress(phase, done, total)
    # extra kwargs go to ChatOpenAI, e.g. reasoning_effort="low", timeout=30
)