Pythonaibrain-NLP

Advanced neural language understanding and generation — not rule-based, not transformer-based — both driven by nlg_intents.json, grounded with Retrieval-Augmented Generation (RAG) on the generation side, and aware of multi-turn dialogue context. NLG turns (intent, slots) into text with a Semantically-Controlled LSTM; NLU turns text back into (intent, slots) with a joint BiLSTM intent-classifier/ slot-tagger. Each half works fully standalone, or chained together via NLUNLGPipeline. Part of the Pythonaibrain offline AI toolkit family.

pip install -e .
python -m pythonaibrain_nlp demo

Why this exists

Most "NLG" in the wild is one of two things:

• Rule-based / templated: f"Sure, I can book a flight from {origin} to {destination}." — brittle, doesn't scale past a handful of phrasings, no learning.
• Transformer-based: fine-tuned GPT-style models — powerful, but heavy, hard to run fully offline/from-scratch, and overkill for a closed-domain, slot-driven generation task.

This package is neither. It's a Semantically-Controlled LSTM (SC-LSTM) — a genuinely learned, recurrent neural decoder purpose-built for exactly this problem (intent + slots -> fluent text), implemented as a real torch.nn.Module and trained with torch.autograd + torch.optim.Adam. No attention stack, no pretrained weights, no templates — real gradients through a real (if unfashionable, by 2026 standards) recurrent architecture.

Architecture

nlg_intents.json                  raw text corpus
       │                                │
       ▼                                ▼
 IntentSet (schema.py)          RAGRetriever (retriever.py)
       │                                │
       │            ┌───────────────────┘
       ▼            ▼
 DialogueActVectorizer      ContextManager (context_manager.py)
   (neural/model.py)          - turn history
       │                      - persistent slot memory
       │  d0 (dialogue-act        │
       │   vector)                │ context vector
       ▼                          ▼
            SCLSTMCell (neural/sclstm.py)
     token embedding + context vector -> hidden state
     dialogue-act vector decays via a learned "reading gate"
     as each word is generated
                    │
                    ▼
          output projection -> softmax over vocabulary
                    │
                    ▼
        postprocess.finalize() (relexicalize + cleanup)
                    │
                    ▼
              generated text

Module map

Module	Responsibility
schema.py	Typed data model for nlg_intents.json (Intent, Slot, Example, RetrievalConfig)
intents_loader.py	Load/validate/save nlg_intents.json
intents_builder.py	Build nlg_intents.json from text + RAG + ContextManager (see below)
embeddings.py	Dependency-free TF-IDF + co-occurrence (SVD) embeddings — no transformer
retriever.py	RAG: chunk + TF-IDF index + cosine retrieval, function mapping, save/load
external_sources.py	Load RAG sources from files, directories, or URLs
functions.py	Sandboxed example functions (create/list/read/delete a file) to register via map_function
context_manager.py	Turn history, persistent slot memory, fixed-size context vector
tokenizer.py	Word-level vocab, delexicalization/relexicalization
neural/sclstm.py	The SC-LSTM cell — a torch.nn.Module (forward only; autograd handles backward)
neural/model.py	NLGModel: embedding + context + SC-LSTM + output projection, torch.nn.Module; DialogueActVectorizer; .save()/.load() checkpointing
neural/trainer.py	Teacher-forced training loop with torch.optim.Adam + gradient clipping
generator.py	NLGGenerator — the public, one-call generation API
nlu/tagging.py	BIO tag alignment/extraction, IntentVocabulary, TagVocabulary
nlu/dataset.py	Builds NLU training data from the same nlg_intents.json examples
nlu/model.py	NLUModel — joint BiLSTM intent-classifier + slot-tagger
nlu/trainer.py	NLUTrainer — batched joint training (same large-corpus machinery as NLG)
nlu/parser.py	NLUParser — the public, standalone understanding API
pipeline.py	NLUNLGPipeline — chains NLU + NLG, dispatches mapped functions, RAG fallback
postprocess.py	Relexicalization + text cleanup

nlg_intents.json: what it is and why it's shaped this way

Each intent bundles everything the model needs to condition one kind of response:

{
  "id": "book_flight",
  "description": "user wants to book a flight",
  "dialogue_act": "inform",
  "slots": [
    {"name": "origin", "type": "string", "required": true},
    {"name": "destination", "type": "string", "required": true},
    {"name": "date", "type": "string", "required": false}
  ],
  "context_requirements": ["previous_intent"],
  "retrieval": {"enabled": true, "corpus_ref": "policy", "top_k": 3},
  "examples": [
    {
      "slots": {"origin": "Delhi", "destination": "Mumbai", "date": "tomorrow"},
      "context": {"previous_intent": null},
      "reference_text": [
        "Sure, I can book a flight from Delhi to Mumbai for tomorrow.",
        "Booking your flight from Delhi to Mumbai for tomorrow now.",
        "Got it — a flight from Delhi to Mumbai for tomorrow is on its way."
      ],
      "pattern": [
        "book me a flight from Delhi to Mumbai for tomorrow",
        "I want to fly from Delhi to Mumbai for tomorrow"
      ]
    }
  ]
}

• slots become the dialogue-act vector d0 fed into the SC-LSTM (via DialogueActVectorizer) — this is what the reading gate decays as it "uses up" each slot during generation.
• retrieval points at a named corpus (corpus_ref) so the RAGRetriever knows which text source to search for this intent.
• context_requirements documents which context keys this intent's examples actually depend on (populated for real by ContextManager, not hand-guessed).
• reference_text is a list, not a single string — several valid ways NLG could phrase this same (intent, slots). Training on all of them (see neural/dataset.py) is what lets generation vary its output instead of memorizing one fixed template per slot combination; pair it with NLGGenerator.generate(..., temperature=0.7) (see below) to actually sample a different phrasing across calls, not just train on several. Slot values are delexicalized to placeholders (__destination__) before tokenization so the network learns structure, not memorized entities. A single string is still accepted and auto-wrapped into a one-item list, for files written before this field became a list.
• pattern is the NLU-facing counterpart to reference_text — example user utterances for this same intent+slots ("book me a flight from Delhi to Mumbai", not "Sure, I can book a flight..."), written the way someone would actually ask rather than the way the system would answer. NLU trains on both reference_text and pattern (see nlu/dataset.py); pattern is what actually teaches it to recognize real commands and questions instead of only NLG-style confirmations. Optional — an intent set without it still trains an NLU, just narrower (see honest limitations below).

How to build nlg_intents.json: text + RAG + Context Manager

Hand-writing this file for a real domain doesn't scale. Use IntentsBuilder, which is exactly "some text + RAG + Context Manager" turned into an API:

from pythonaibrain_nlp import IntentsBuilder, save_intents

builder = IntentsBuilder(domain="travel_support")

# 1. "Some text" — raw reference docs become your RAG corpus
builder.add_source_text("policy", open("flight_policy.txt").read())

# 2. Declare an intent + its slots, pointing at that corpus
builder.define_intent(
    "book_flight", "user wants to book a flight", "inform",
    slots=[("origin", "string", True), ("destination", "string", True), ("date", "string", False)],
    corpus_ref="policy",
)

# 3. Add examples in conversational order — a real ContextManager runs
#    underneath and records actual dialogue state into each example's
#    "context" block (not a hand-typed guess). reference_text accepts
#    a single string or a list of phrasing variants; pattern is optional
#    genuine user-query phrasing for the same slots.
builder.add_example(
    "book_flight",
    {"origin": "Delhi", "destination": "Mumbai", "date": "tomorrow"},
    reference_text=[
        "Sure, I can book a flight from Delhi to Mumbai for tomorrow.",
        "Booking your flight from Delhi to Mumbai for tomorrow now.",
    ],
    pattern=["book me a flight from Delhi to Mumbai for tomorrow"],
)

intent_set = builder.build()          # validated IntentSet
save_intents(intent_set, "nlg_intents.json")

See examples/build_intents_demo.py for a runnable version and examples/nlg_intents.json for the file it produces.

External data sources

RAG sources don't have to be inline Python strings — external_sources.py loads them from files, directories, or URLs, feeding into the same {source_name: text} shape RAGRetriever.index() and IntentsBuilder.add_source_text() already expect:

from pythonaibrain_nlp import load_text_file, load_directory, load_url

text = load_text_file("policies/refunds.txt")
sources = load_directory("policies/")          # one source per .txt/.md/.rst file, named by filename stem
web_text = load_url("https://example.com/faq")  # HTML gets tag-stripped automatically

Or via IntentsBuilder directly:

builder.add_source_file("policies/refunds.txt")
builder.add_source_directory("policies/")
builder.add_source_url("https://example.com/faq")

RAGRetriever also supports incremental indexing and persistence, so a large external corpus doesn't need re-chunking on every startup:

retriever.add_document("new_policy", text)     # add one document to an existing index
retriever.save("index.json")                    # persist raw source texts + settings
retriever = RAGRetriever.load("index.json")     # re-derives the TF-IDF index, no re-chunking needed

Stays dependency-free on purpose — stdlib urllib only, no requests/beautifulsoup4 — consistent with the rest of the package. HTML gets a lightweight regex tag-strip, not a full parser; fine for policy pages and docs, not a general-purpose scraper.

Ready-made large corpus

examples/build_large_corpus.py builds a bigger, realistic example: a customer_support domain with 17 intents — 12 transactional (book_flight, cancel_flight, refund_status, track_order, reset_password, update_address, business_hours, file_complaint, cancel_subscription, payment_failed, contact_support, check_balance), each backed by its own RAG policy document, plus 5 general-purpose / chit-chat intents (greeting, farewell, thanks, smalltalk, help) with no RAG grounding — there's nothing factual to retrieve for "hello" or "thanks". 200 examples each (3,400 total). Every example carries several reference_text phrasings and several pattern query phrasings — genuine lexical diversity on both the NLG-output side and the NLU-input side, not one template with values swapped in.

python examples/build_large_corpus.py --n-per-intent 200 --out nlg_intents.json

The output ships pre-built at examples/nlg_intents_large.json (17 intents, 3,400 examples, expanding to 12,440 reference_text variants and 12,912 pattern variants, ~2.3 MB) so you can start training against a real-sized corpus immediately:

from pythonaibrain_nlp import load_intents, NLGGenerator, ContextManager
from pythonaibrain_nlp.retriever import RAGRetriever
from pythonaibrain_nlp.neural.model import DialogueActVectorizer, NLGModel
from pythonaibrain_nlp.neural.trainer import Trainer
from pythonaibrain_nlp.tokenizer import Vocabulary, delexicalize
from examples.build_large_corpus import SOURCES  # the same RAG policy docs used to build it

intent_set = load_intents("examples/nlg_intents_large.json")
retriever = RAGRetriever()
retriever.index(SOURCES)

# every reference_text variant becomes its own training target — see neural/dataset.py
texts = [delexicalize(t, ex.slots) for i in intent_set.intents for ex in i.examples for t in ex.reference_text]
vocab = Vocabulary().build(texts)             # ~221 words, 12,440 training texts
da_vectorizer = DialogueActVectorizer().fit(intent_set)  # 31-dim dialogue-act space (17 unique acts + slot bits)

model = NLGModel(vocab, da_vectorizer, embed_dim=64, hidden_dim=128, context_dim=64)
model.sentence_embedder.fit(texts)

Trainer(model, lr=0.005).fit(intent_set, retriever, epochs=25, batch_size=128)

Verified, not just schema-validated: this was actually trained end to end — 12,440 training texts, 98 batches/epoch, ~3s/epoch on CPU. 25 epochs (loss plateaus ~0.156) generates correctly and slot-faithfully across all 17 intents, no confusion between the new general-purpose intents and the 12 transactional ones:

greeting      : Nice to meet u Divyanshu.
smalltalk     : I'm good thank you what can i do for you.
help          : I can book flights check refunds track orders reset passwords and handle account questions.
book_flight   : Booking your flight from Delhi to Mumbai for tomorrow now.
refund_status : I can confirm the refund for order ORD90210 is in progress.
cancel_flight : Booking BK55221 is now cancelled.
farewell      : Goodbye take care.

The NLU side was trained the same way, on both reference_text and pattern (25,352 combined training texts, 10 epochs, ~90s on CPU — NLU converges much faster than NLG since classification/tagging is an easier task than token-by-token generation) and tested against genuine query-style phrasing it only ever saw via pattern, not reference_text:

"Hi I'm Divyanshu!"                                   -> greeting, {'name': 'Divyanshu'}
"how are you"                                          -> smalltalk, {}
"what can you do"                                       -> help, {}
"book me a flight from Delhi to Mumbai for tomorrow"    -> book_flight, {'origin': 'Delhi', 'destination': 'Mumbai', 'date': 'tomorrow'}
"where is my refund for order ORD90210"                 -> refund_status, {'order_id': 'ORD90210'}
"please cancel booking BK55221"                         -> cancel_flight, {'booking_id': 'BK55221'}
"thanks a lot"                                           -> thanks, {}
"bye"                                                    -> farewell, {}

All correct — no confusion between the 5 new general-purpose intents and the 12 transactional ones, and this is the direct, measured payoff of pattern existing at all: without it, NLU only ever sees reference_text's system-confirmation style, and generalizes noticeably worse to real user commands (see honest limitations below for what that gap looked like before pattern was added).

Two honest, known rough edges visible in that output, both from the word-level tokenizer rather than the SC-LSTM itself: punctuation like em dashes gets dropped (tokenizer only keeps alphanumerics/apostrophes/ underscores), and postprocess.finalize() only capitalizes the first letter of the whole sentence, not a mid-sentence "I". Neither affects slot correctness or intent discrimination — both are straightforward tokenizer-level fixes if you need cleaner surface punctuation.

Training and generating

Small corpus (quick iteration)

from pythonaibrain_nlp import (
    ContextManager, NLGGenerator, load_intents,
)
from pythonaibrain_nlp.neural.model import DialogueActVectorizer, NLGModel
from pythonaibrain_nlp.neural.trainer import Trainer
from pythonaibrain_nlp.tokenizer import Vocabulary, delexicalize
from pythonaibrain_nlp.intents_builder import IntentsBuilder

builder = IntentsBuilder(domain="travel_support")
# ... add_source_text / define_intent / add_example calls ...
intent_set = builder.build()
retriever = builder.retriever

texts = [delexicalize(t, ex.slots)
         for i in intent_set.intents for ex in i.examples for t in ex.reference_text]
vocab = Vocabulary().build(texts)
da_vectorizer = DialogueActVectorizer().fit(intent_set)

model = NLGModel(vocab, da_vectorizer, embed_dim=64, hidden_dim=128, context_dim=64)
model.sentence_embedder.fit(texts)

Trainer(model, lr=0.02).train_intent_set(intent_set, retriever, epochs=200)

gen = NLGGenerator(intent_set, retriever, model, ContextManager())
print(gen.generate("book_flight", {"origin": "Delhi", "destination": "Mumbai"}))

# save / reload the trained model (torch checkpoint + vocab + DA-vectorizer + fitted embedder)
model.save("nlg_model.pt")
from pythonaibrain_nlp.neural.model import NLGModel
reloaded = NLGModel.load("nlg_model.pt")

Runs on GPU for free — pass device="cuda" to NLGModel(...) if one's available; everything else (RAG retrieval, context vectors, DA-vectorizer) stays CPU/numpy since those aren't backprop targets.

Large corpus (thousands–millions of examples)

Use Trainer.fit() directly instead of the train_intent_set alias — it exposes batching, gradient accumulation, mixed precision, and checkpoint/resume:

trainer = Trainer(model, lr=0.01)
history = trainer.fit(
    intent_set, retriever,
    epochs=20,
    batch_size=128,          # real minibatches through the SC-LSTM, not one example at a time
    grad_accum_steps=4,      # effective batch size 512 without needing 512-worth of GPU memory
    num_workers=4,           # parallel batch collation
    checkpoint_path="nlg_checkpoint.pt",
    checkpoint_every=1,      # save after every epoch — long runs can be interrupted safely
    resume=True,             # picks back up from checkpoint_path if it already exists
)

What makes this different from the small-corpus path, and why it matters once the corpus stops being tiny:

• Retrieval and context vectors are computed once, not every epoch. Trainer.fit() calls build_dataset() internally, which walks the intent set through a real ContextManager exactly one time and caches (dialogue-act vector, context vector, target token ids) per example. The original per-example trainer re-ran RAG retrieval and context-vector construction on every epoch — for a large corpus trained over many epochs that dominates total training time. Now it doesn't happen again after the first pass.
• Real minibatches. SCLSTMCell and NLGModel.forward_sequence_batch operate on (B, ...) tensors — the whole point of a GPU.
• Sparse, not dense, retrieval indexing. RAGRetriever / TfidfVectorizer build scipy.sparse matrices, not a dense (n_chunks x vocab) numpy array — a corpus with tens of thousands of chunks and a large vocabulary would otherwise not fit in memory. max_vocab caps pathological vocabulary growth (typos, IDs, noise tokens) to the most frequent terms.
• Truncated SVD, not full dense SVD, for context embeddings. CooccurrenceEmbedder uses scipy.sparse.linalg.svds over a sparse PPMI matrix once the vocabulary passes a few hundred words — the naive dense numpy.linalg.svd over a (V, V) matrix is O(V³) time and O(V²) memory and simply doesn't run on a real large vocabulary. Small vocabularies still get a fast dense fallback, since ARPACK's setup cost isn't worth paying on a toy corpus.
• Mixed precision + gradient accumulation. torch.autocast + GradScaler activate automatically when model.device.type == "cuda" (no-op on CPU); grad_accum_steps lets you reach a large effective batch size on limited GPU memory.
• Checkpoint/resume. Long training runs on a large corpus can get interrupted — checkpoint_path + resume=True saves model + optimizer state and picks back up rather than restarting from scratch.

train_intent_set(...) is kept around as a thin backward-compatible alias (fit(..., batch_size=8)) for existing small scripts and tests.

NLU: understanding text, not just generating it

nlg_intents.json also supervises the reverse direction. Each example's (reference_text, slots) pair is simultaneously "generate this text from these slots" (NLG) and "recover these slots from this text" (NLU) — same facts, opposite direction, so the same intents file trains both sides. No separate annotation schema.

Architecture — a joint BiLSTM intent-classifier + slot-tagger (nlu/model.py), matching the rest of the package's rules: neural end to end (no keyword rules, no regex intent matching), not a transformer (a bidirectional torch.nn.LSTM, no self-attention). One shared encoder feeds two heads: an intent classifier over the concatenated final forward/backward hidden states, and a per-token BIO slot tagger (B-origin, I-origin, O, ...) over every timestep's output. Both train jointly with a combined loss.

Standalone (no NLG involved at all)

from pythonaibrain_nlp import load_intents, NLUParser
from pythonaibrain_nlp.nlu.model import NLUModel
from pythonaibrain_nlp.nlu.tagging import IntentVocabulary, TagVocabulary
from pythonaibrain_nlp.nlu.trainer import NLUTrainer
from pythonaibrain_nlp.tokenizer import Vocabulary

intent_set = load_intents("nlg_intents.json")
# train the NLU vocab on both reference_text and pattern text — pattern is
# what actually teaches recognition of real query phrasing, not just system text
texts = [
    t for i in intent_set.intents for ex in i.examples for t in (*ex.reference_text, *ex.pattern)
]

nlu_vocab = Vocabulary().build(texts, min_freq=2)
intent_vocab = IntentVocabulary().fit(intent_set)
tag_vocab = TagVocabulary().fit(intent_set)

nlu_model = NLUModel(nlu_vocab, intent_vocab, tag_vocab, embed_dim=64, hidden_dim=128)
NLUTrainer(nlu_model, lr=0.005).fit(intent_set, epochs=15, batch_size=128)

parser = NLUParser(nlu_model)
result = parser.parse("please cancel booking BK12345")   # genuine query phrasing, from pattern
print(result.intent_id, result.slots, result.confidence)
# cancel_flight  {'booking_id': 'BK12345'}  1.0

nlu_model.save("nlu_model.pt")   # weights + vocab + intent/tag vocabularies

NLUTrainer.fit() uses the same large-corpus machinery as the NLG side — batching, DataLoader, mixed precision on CUDA, checkpoint/resume (checkpoint_path, resume=True). On the bundled 4,800-example corpus (examples/nlg_intents_large.json), 15 epochs (~30s on CPU) is enough to converge — NLU trains noticeably faster than NLG, since classifying and tagging is an easier task than generating fluent text token by token.

Combined: text in, response out, one call

from pythonaibrain_nlp import NLUNLGPipeline

pipeline = NLUNLGPipeline(nlu_parser, nlg_generator)  # both trained independently, as above
result = pipeline.respond("Sure, I can book a flight from Chennai to Pune.")

print(result.intent_id)       # "book_flight"
print(result.slots)           # {"origin": "Chennai", "destination": "Pune"}
print(result.response_text)   # NLG's generated reply

NLUNLGPipeline is a thin wrapper — NLUParser.parse() and NLGGenerator.generate() don't know it exists, and both remain fully usable on their own. Pass min_confidence to reject low-confidence parses before they reach NLG:

pipeline = NLUNLGPipeline(nlu_parser, nlg_generator, min_confidence=0.6)

Trial AI: a real interactive assistant

examples/trial_ai.py wires everything in this README into one runnable assistant: the bundled 21-intent corpus (12 transactional + 5 general-purpose: greeting, farewell, thanks, smalltalk, help, + 4 function-mapped: create_file, list_files, read_file, delete_file), a trained NLU, a trained NLG, RAG grounding, ContextManager, and NLUNLGPipeline.

python examples/trial_ai.py            # interactive chat
python examples/trial_ai.py --demo     # scripted sample conversation, no typing needed
python examples/trial_ai.py --retrain  # ignore cached checkpoints, retrain from scratch

Trained checkpoints ship pre-built in examples/trial_models/ (NLG 856 KB, NLU 1.4 MB), so a normal run loads instantly — --retrain takes a few minutes on CPU.

You: Hi I'm Divyanshu!
AI : [greeting] Nice to meet you Divyanshu.

You: book me a flight from Delhi to Mumbai for tomorrow
AI : [book_flight] Booking your flight from Delhi to Mumbai for tomorrow now.

You: create a file test.txt
AI : [create_file, ran a real function] I've created the file test.txt.

You: show list of files
AI : [list_files, ran a real function] I found these files test.txt.

You: how many kilograms of baggage can I bring on an economy flight
AI : [from RAG] Flight changes made more than 24 hours before departure are free of charge. Cancellations within 24 hours of booking are fully refundable. Flights cancelled closer to departure are subject to a cancellation fee based on fare type. Baggage allowance is 20 kilograms for economy class and 30 kilograms for business class.

You: I want to book a flight
AI : Sorry, I couldn't help with that — missing required slots for intent 'book_flight': ['origin', 'destination']

You: thanks a lot
AI : [thanks] You're welcome.

create a file test.txt and show list of files are real filesystem operations, sandboxed to examples/trial_workspace/ — see "Function mapping and RAG fallback" below. how many kilograms of baggage... isn't a defined intent at all — it's answered directly from the indexed flight policy text.

Two real bugs this surfaced, both fixed

Building an actual multi-turn, multi-intent assistant — not just single-shot generation calls — surfaced two genuine issues that single-intent testing never would have:

1. Context corruption across an intent switch. build_dataset() trains NLG by walking through examples grouped by intent (every book_flight example, then every cancel_flight example, ...), so the model only ever saw same-intent context during training. A shared ContextManager across a genuine topic switch — book a flight, then ask about a refund — kept generating book_flight-style text for the second turn, driven by leftover context, even with the correct intent+slots passed to generate(). Fixed in NLUNLGPipeline: it now resets the NLG side's conversation whenever NLU's parsed intent differs from the previous turn's (auto_reset_on_intent_change=True by default — see pipeline.py for how to disable it). Covered by test_pipeline_handles_intent_switch_without_context_corruption.

2. A shared dialogue_act collapses the DA vector's discriminating signal. The bundled corpus originally gave every intent the same dialogue_act="inform", so DialogueActVectorizer produced only one shared "act" bit across all 12 intents — the only real intent-discriminating signal came from which slots were filled. For an intent with no required slots and an unfilled optional one (business_hours with no branch mentioned), the DA vector carried almost no information, and generation defaulted to whatever pattern was best-represented in training rather than the actual intent. Fixed by giving each intent a unique dialogue_act (inform_book_flight, inform_refund_status, ...) in build_large_corpus.py, guaranteeing every intent gets its own identifying bit in the DA vector regardless of slot fill state.

Both are the kind of bug that only shows up once you actually chain multiple intents together in one conversation — worth knowing if you build your own multi-intent corpus: give every intent a distinct dialogue_act, and don't assume a shared ContextManager is safe across an intent switch unless your training data was built to teach that.

3. Narrow date-phrasing coverage broke slot extraction on realistic input. book_flight training only ever paired dates with the preposition "for" ("for next Monday") and only ever used Monday/Tuesday as example weekdays — a real user typing "on next sunday" (different preposition, unseen weekday) broke destination extraction outright: "Book a flight from Patna to Rachi on next sunday." returned destination: 'next' instead of the actual city, and that wrong value then correctly-by-design persisted via ContextManager into the next turn's response too, compounding the error. Fixed by training book_flight on both "for" and "on" (where "on" is grammatical — weekday and ordinal dates only, not "on two weeks from now") and the full week of weekday names, not just two of them. Verified: date extraction went from silently dropping "next sunday" entirely to 100% correct regardless of preposition or casing.

   What this fix did not solve, and why: the same test with a misspelled city ("Rachi" instead of "Ranchi", which is in the training vocabulary) still fails to extract a destination at all — correct spelling ("Ranchi") extracts perfectly every time, typo'd spelling doesn't. This is a different, harder problem than phrasing coverage: NLU here is word-level with no character/subword modeling, so a token it's never seen has no path to being recognized as "probably a city name" — its embedding is indistinguishable <unk> from any other unknown word. Broadening training phrasing can't fix this; it would need either fuzzy matching before tagging, or a genuine architecture change (character-level or subword embeddings) to generalize from partial spelling similarity. Worth knowing if your domain has frequently-misspelled proper nouns.

General-purpose / chit-chat intents

The 5 non-transactional intents (greeting, farewell, thanks, smalltalk, help) apply bug #2's lesson directly: each gets its own unique dialogue_act from the start, which matters more here than for the transactional intents — farewell, thanks, smalltalk, and help all have zero slots, so the DA vector's only discriminating signal for any of them is that unique act bit; there's no slot presence to fall back on. greeting is the exception with one required slot (name) — a genuine test of the same single-slot extraction already proven for emails and order IDs, just with a first-name pool instead. ContextManager carries a captured name across turns of the same intent as expected (a repeated "hi" without restating your name still gets addressed by name) — verified this doesn't happen on a fresh conversation with no prior name; NLG correctly raises the missing-required-slot error instead of hallucinating one, which NLUNLGPipeline.respond() surfaces as a ValueError for the caller to handle.

Function mapping and RAG fallback

Two capabilities live in RAGRetriever (not a separate framework) and get used automatically by NLUNLGPipeline.respond():

Function mapping — RAGRetriever.map_function(intent_id, func) registers a plain Python function to run when that intent is recognized, before generation. No parameter schema to declare separately: whatever the function's own keyword arguments are named (filename, content, ...) is exactly what gets filled in from matching slots, via inspect.signature — see functions.py's FileSystemFunctions for a real sandboxed example (create/list/ read/delete a file, with path-traversal rejected via a plain PermissionError before touching disk):

from pythonaibrain_nlp import RAGRetriever, FileSystemFunctions

retriever = RAGRetriever()
retriever.index(SOURCES)
FileSystemFunctions("./workspace").register_all(retriever)  # maps create_file/list_files/read_file/delete_file

# ... build NLGGenerator(intent_set, retriever, nlg_model, ContextManager()), NLUParser as usual ...
pipeline = NLUNLGPipeline(nlu_parser, nlg_gen)
result = pipeline.respond("create a file test.txt")
# result.source == "function" — a real file was created; its output ("status": "created")
# merged into the slots NLG generated the confirmation from

RAG fallback — min_confidence/required-slot checks used to just raise. Now, when a parse looks untrustworthy (low confidence, an unrecognized intent, or required slots NLU couldn't fill), respond() searches the retriever's full text index for the raw query before raising — so a question whose answer lives in an indexed policy doc but was never turned into a defined intent still gets answered:

pipeline.respond("how many kilograms of baggage can I bring on an economy flight")
# no such intent exists — answered directly from the indexed flights policy text
# result.source == "rag_fallback", result.intent_id is None

A ValueError is still raised if RAG has nothing good either — set rag_fallback=False to disable this and always raise immediately, or tune rag_fallback_min_score (default 0.25) if a fallback answer is firing on too-weak a match. That default came from a real measurement, not a guess: testing found "I want to book a flight" (missing required slots, should prompt for them) scoring 0.21 against the flights policy purely from sharing the word "flight", versus 0.37 for a genuine question — without a floor, the weak match would win over a more useful "please give me origin/destination" prompt just for scoring above zero.

Honest limitation: a fallback answer is the entire matching chunk, not just the sentence that answers the question — visible above, where the baggage question's answer comes back wrapped in three unrelated sentences about flight changes and cancellations. There's no LLM here to extract just the relevant part; chunk_text() (see retriever.py) already keeps chunks short (~60 words, sentence-aligned) specifically to limit how much irrelevant text rides along, but it doesn't eliminate it. Chunking your source documents more finely (one fact per chunk) is the practical mitigation if this matters for your domain.

Softmax overconfidence, and why two signals catch it where one doesn't: testing found NLU's confidence score alone isn't a reliable "this doesn't belong to any known intent" signal — genuinely nonsensical input ("asdkj qwerty nonsense gibberish") still scored 0.70-0.97 confidence, always landing on whichever intent had the most open-ended slot (file_complaint's free-text topic), never dropping low enough to trigger any reasonable threshold. Requiring the top intent's required slots to actually be fillable turned out to be a second, more reliable signal — every one of those overconfident misfires also had empty extracted slots for an intent that requires one. respond() checks both together rather than confidence alone.

Honest scope note: none of this is the actual Model Context Protocol (no JSON-RPC 2.0 messages, no stdio/SSE transport, no connecting to external MCP servers) — it's a local function-calling layer living inside the RAG retriever, sized for "a recognized intent triggers a real action," not for connecting to arbitrary external tool servers.

CLI

pythonaibrain-nlp validate path/to/nlg_intents.json
pythonaibrain-nlp demo

Testing

pip install -e ".[dev]"
pytest tests/ -q

Design notes / honest limitations

• pattern closes the "NLU only sees system text" gap, but only when you actually populate it. Before this field existed, NLU had no choice but to train on reference_text alone — confirmations like "Booking X is now cancelled" — which generalizes noticeably worse to real user commands ("please cancel my booking X") than to text in that same confirmation style. pattern gives NLU genuine query-style examples to train on directly (verified: 5/5 correct on held-out query phrasing in the large-corpus test above). The gap doesn't disappear on its own, though — an IntentsBuilder.add_example() call that skips pattern still leaves NLU with only reference_text to learn from for that example, same limitation as before.
• Even batched, the SC-LSTM is a sequential (per-timestep) recurrent decoder — throughput scales with batch size and sequence length, but it's not going to match a fully parallelizable architecture on raw tokens/sec. Batching + gradient accumulation + AMP get you real GPU utilization; they don't remove the recurrence.
• RAGRetriever uses sparse TF-IDF, not dense/transformer embeddings, to stay offline and dependency-minimal; CooccurrenceEmbedder (truncated SVD over a sparse co-occurrence matrix, plain numpy/scipy — not trained via backprop) fills the "dense vector" role instead where one's needed for the context vector.
• The SC-LSTM decodes greedily at inference; beam search would be a straightforward addition to NLGModel.generate() if higher-quality decoding is needed later. Inference is still single-example, not batched — batching matters for training throughput, not for serving one generation request at a time.
• NLGModel.load() / Trainer.load_checkpoint() use weights_only=False since checkpoints intentionally bundle non-tensor state (vocab, fitted embedder, optimizer state) — only load checkpoints from a source you trust.
• max_vocab caps (default 50k for TF-IDF, 20k for the co-occurrence embedder) bound memory on a pathologically large or noisy vocabulary; raise them if your domain genuinely needs a larger vocabulary and you have the memory for it.