aicurt

aicurt is a lightweight, dependency-free Python package for building retrieval and RAG-powered AI applications.

It combines practical capabilities for modern AI systems:

• selective PII detection and redaction through explicit regex or rule registration
• deterministic chunking for downstream model input windows and retrieval pipelines
• lightweight tokenization and token statistics for embedding and search workflows
• retrieval-ready document indexing and search flows for RAG applications

Why AICURT matters for AI search and RAG

AICURT is designed to help teams build the retrieval layer behind AI systems without bringing in a heavy dependency stack.

The package is useful when you want to:

• protect sensitive content before indexing documents
• split large text into clean, reusable chunks
• prepare queryable content for embedding and retrieval pipelines
• search documents by keyword, semantic similarity, or hybrid retrieval patterns
• keep runtime dependencies minimal and production-friendly

Package overview

Area	Capability
PII Engine	Register your own patterns and replace only the matches you want to protect
Chunking Engine	Split text with word, sentence, paragraph, sliding-window, recursive, and token strategies
Tokenizer	Tokenize text and compute token, word, and character statistics
Retrieval	Index chunked text and search it with keyword, semantic, and hybrid retrieval flows
Storage	Use AICurtStore for persistent document storage or AICurtMemoryStore for lightweight testing
CLI	Read from stdin or a file, then detect, redact, chunk, tokenize, or print stats
Packaging	Installable as a standard Python package with an aicurt console script

Retrieval overview

AICURT includes a lightweight retrieval layer built around:

• AICurtEmbedder: provider-neutral embedding abstraction
• AICurtStore: persistent document storage for indexing workflows
• AICurtMemoryStore: in-memory store for testing and quick prototypes
• AICurtRAG: indexing and retrieval workflow for chunked content
• AICurtSearchResult: structured result object with text, score, source, and metadata

This makes it a practical foundation for document search, retrieval pipelines, and RAG-based AI applications without vendor lock-in.

What the current PII model does

The current design is intentionally selective:

• PiiRedactor() starts as a passive redactor with no built-in detection enabled
• you explicitly register the patterns you want to redact by calling register_rule() or register_pattern()
• detect() returns only the matches from the patterns you have registered
• redact() replaces only those matches, leaving unrelated text untouched

That makes the engine safer, more predictable, and easier to embed into production pipelines.

Core API

PiiRedactor

Public methods:

• detect(text) → returns a list of PiiMatch objects
• mask(text) → returns a MaskingResult with original_text, masked_text, matches, and mapping
• redact(text, replacement=None) → same redaction flow, with optional global replacement override
• register_rule(name, pattern, replacement=...) → register a named rule that will be detected and replaced
• register_pattern(name, pattern) → register a custom regex pattern for detection
• addRegex(pattern) / addRegexPatterns(patterns) → register more regexes
• addWord(word, mask_length=None, case_sensitive=None) / addWords(...) → register word-based masking rules
• configureEmailMasking(...) and configurePhoneMasking(...) → tweak the built-in masking settings when needed
• setMaskCharacter(...) and setMaskLength(...) → adjust the mask output style

PiiConfig

PiiConfig is the configuration object that carries replacement, mask style, and masking-policy settings.

Common fields:

• mask_strategy
• mask_char
• mask_visible_prefix
• mask_visible_suffix
• preserve_length
• default_replacement
• masking_policies
• enable_reversible

Installation

This package is available through the PyPI registry.

Before installing, ensure you have Python 3.9 or higher installed. You can download and install Python from python.org.

You can install the package using pip:

pip install aicurt

Editable development install

python -m pip install -e .

Verify the install

aicurt --help
aicurt --version

Quick start

1) Protect sensitive information before indexing

This example shows the simplest and safest pattern: redact email addresses and phone numbers before you store or retrieve documents.

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)
redactor.register_rule(
    "PHONE",
    r"(?<!\w)(?:\+?\d[\d\s().-]{6,}\d)(?!\w)",
    replacement="[PHONE]",
)

text = "Contact alice@example.com or call +1 555 123 4567 for support."
print(redactor.redact(text).text)

Output will look like:

Contact [EMAIL] or call [PHONE] for support.

This is useful when you want to keep private values out of your search index or retrieval store.

2) Chunk long text into smaller retrieval units

Large documents should be split into meaningful chunks before indexing. This improves retrieval quality and keeps context manageable for downstream AI workflows.

from aicurt.chunking import ChunkStrategy, TextChunker

text = """
Artificial Intelligence is transforming the way teams build software.
Retrieval systems improve response quality by finding the right context.
Chunking helps keep each search unit small and relevant.
"""

chunks = TextChunker().chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=35,
)

for chunk in chunks:
    print(f"Chunk {chunk.index}: {chunk.content}")

This produces smaller text blocks that can be indexed individually and searched more accurately.

3) Build a RAG-style search index

The pattern below creates a tiny in-memory index, adds a few document chunks, and then searches them by keyword or semantic similarity.

from aicurt import AICurtEmbedder, AICurtMemoryStore, AICurtRAG


class DemoEmbedder(AICurtEmbedder):
    def __init__(self):
        super().__init__(dimension=3)

    def embed(self, text: str):
        lower = text.lower()
        if "refund" in lower:
            return [1.0, 0.0, 0.0]
        if "shipping" in lower:
            return [0.0, 1.0, 0.0]
        if "support" in lower:
            return [0.0, 0.0, 1.0]
        return [0.4, 0.4, 0.4]

    def embed_many(self, texts):
        return [self.embed(text) for text in texts]


rag = AICurtRAG(
    store=AICurtMemoryStore(),
    embedder=DemoEmbedder(),
    protect=True,
    chunk_size=80,
    overlap=10,
)

rag.index_text(
    "Refunds are available within 30 days for eligible purchases.",
    source="refund_policy.txt",
    metadata={"team": "support"},
)
rag.index_text(
    "Shipping takes three to five business days for domestic delivery.",
    source="shipping_policy.txt",
    metadata={"team": "ops"},
)

results = rag.retrieve("refund policy", mode="keyword", top_k=3)
for item in results:
    print(item.text)
    print(item.score)

This demonstrates the main retrieval flow: index clean text, then quickly fetch the best matching chunks for a user query.

4) Search with semantic, keyword, and hybrid modes

AICURT supports different retrieval styles depending on the search problem.

semantic = rag.retrieve("shipping delivery", mode="semantic", top_k=3)
keyword = rag.retrieve("refund", mode="keyword", top_k=3)
hybrid = rag.retrieve("refund and shipping", mode="hybrid", top_k=3)

print("SEMANTIC:", [item.text for item in semantic])
print("KEYWORD:", [item.text for item in keyword])
print("HYBRID:", [item.text for item in hybrid])

• semantic is best when you want similarity-based matching.
• keyword is best for direct literal term matches.
• hybrid blends both behaviors for more balanced ranking.

5) Filter results by metadata

Metadata filters help narrow retrieval to the right domain or team.

filtered = rag.retrieve(
    "refund",
    mode="keyword",
    top_k=5,
    filters={"team": "support"},
)

for item in filtered:
    print(item.text, item.metadata)

This keeps the results relevant when your corpus contains multiple departments or document categories.

6) Persist data across runs with a local store

For long-lived projects, use a persistent store instead of a memory-only store.

from aicurt import AICurtRAG, AICurtStore

store = AICurtStore("demo_store")
rag = AICurtRAG(store=store, embedder=DemoEmbedder(), protect=True)
rag.index_text("Support can help with billing and refunds.", source="support_guide.txt")
print(rag.retrieve("billing refund", mode="hybrid", top_k=2))
rag.close()

This makes it easier to reuse indexed knowledge between runs without needing a third-party vector database.

Python API

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact("Contact alice@example.com now")
print(result.text)

Custom replacement callback

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact(
    "alice@example.com",
    replacement=lambda match: f"<{match.entity_type}>",
)
print(result.text)

Selective partial masking policy

from aicurt.pii import PiiConfig, PiiRedactor

config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)

redactor = PiiRedactor(config)
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)
redactor.register_rule(
    "PHONE",
    r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b",
    replacement="[PHONE]",
)

result = redactor.redact("Email: john@example.com, Phone: 9876543210")
print(result.text)

This keeps labels like Email: and Phone: intact while only replacing the registered sensitive values.

End-to-end examples

Example 1: Detect a custom email rule

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_pattern("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b")

matches = redactor.detect("Contact alice@example.com now")
for match in matches:
    print(match.entity_type, match.value, match.start, match.end)

Example 2: Redact a custom organization name

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule("ORG", r"Acme", replacement="[COMPANY]")

result = redactor.redact("Acme is here")
print(result.text)

Example 3: Register a custom word and mask it with a specific length

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.addWord("secret-token", mask_length=6)

result = redactor.redact("The secret-token value must be hidden")
print(result.text)

Example 4: Chunk text

from aicurt.chunking import ChunkStrategy, TextChunker

text = "Paragraph one. Paragraph two."
chunks = TextChunker().chunk(text, strategy=ChunkStrategy.SENTENCE, chunk_size=20)
for chunk in chunks:
    print(chunk.index, chunk.content)

Example 5: Tokenize and compute stats

from aicurt.tokenizer import SimpleTokenizer

text = "hello world"
tokenizer = SimpleTokenizer()
print(tokenizer.tokenize(text))
print(tokenizer.count_tokens(text))
print(tokenizer.stats(text))

Example 6: Embedding-ready payload

import json

result = redactor.redact("Contact alice@example.com now")
stats = SimpleTokenizer().stats("Contact alice@example.com now")
payload = {
    "masked_text": result.text,
    "token_count": stats.token_count,
    "word_count": stats.word_count,
    "character_count": stats.character_count,
    "matched_entities": [
        {"entity_type": match.entity_type, "value": match.value}
        for match in result.matches
    ],
}

print(json.dumps(payload, ensure_ascii=False, indent=2))

PII Examples

Refer to the PII examples below and use them as a guide when implementing the PII masking in your code.

from aicurt.pii import PiiConfig, PiiRedactor

paragraph_text = """Customer Information Report

Krishna Tadi is a product manager based in Bengaluru, Karnataka. His work email is krishna.t@example.com and his contact number is +91 90000000000.
During onboarding, Krishna shared his Aadhaar number 234567891234, PAN number ABCDE1234F, and passport number N1234567. The same profile also included a driver's license number DL-0420110012345.
The account team reviewed the customer's credit card number 4111 1111 1111 1111 and bank account number 123456789012. The date of birth listed in the record was 15-08-1995, and the latest login IP address was 192.168.1.105.
The account API key used for integration testing was sk_live_51N8example123456789, and the username associated with the account was krishna_t_95.
This document should remain confidential and should only be shared with authorized personnel for secure verification steps.
"""

mask_config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)
redactor = PiiRedactor(mask_config)
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b", replacement="[PHONE]")
redactor.register_rule("AADHAAR", r"\b\d{12}\b", replacement="[AADHAAR]")
redactor.register_rule("PAN", r"\b[A-Z]{5}[0-9]{4}[A-Z]\b", replacement="[PAN]")
redactor.register_rule("BANK_ACCOUNT", r"\b\d{9,18}\b", replacement="[BANK_ACCOUNT]")
redactor.register_rule("DOB", r"\b(?:0?[1-9]|[12]\d|3[01])[-/](?:0?[1-9]|1[0-2])[-/](?:\d{4})\b", replacement="[DOB]")
redactor.register_rule("IP", r"\b(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}\b", replacement="[IP]")

redactor.addWord("KRISHNA")
redactor.addWord("4111 1111 1111 1111")

result = redactor.redact(paragraph_text)
matches = redactor.detect(paragraph_text)

print("Masked paragraph preview:")
print(result.text)
print("\nDetected matches:")
for match in matches[:6]:
    print(f"- {match.entity_type}: {match.value}")

Chunking Examples

Refer to the chunking examples below and use them as a guide when implementing the chunking strategy in your code.

text = """
Artificial Intelligence is transforming the way developers build applications.
Large Language Models can understand and generate human-like text.
Retrieval Augmented Generation combines search with AI models.
Chunking is an important step because large documents need to be split into smaller pieces.
Good chunking improves embeddings, retrieval accuracy, and response quality.
This library provides deterministic text preprocessing utilities for AI workflows.
"""

chunker = TextChunker()

def print_chunks(title, chunks):
    print("\n")
    print("=" * 80)
    print(title)
    print("=" * 80)

    for chunk in chunks:
        print("\nChunk Index:", chunk.index)
        print("Content:")
        print(chunk.content)
        print("Start:", chunk.start)
        print("End:", chunk.end)
        print("Characters:", chunk.character_count)
        print("Words:", chunk.word_count)
        print("Tokens:", chunk.token_count)


# ============================================================
# 1. STANDARD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.STANDARD,
    chunk_size=100
)

print_chunks(
    "STANDARD CHUNKING",
    chunks
)


# ============================================================
# 2. CHARACTER CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.CHARACTER,
    chunk_size=80
)

print_chunks(
    "CHARACTER CHUNKING",
    chunks
)


# ============================================================
# 3. WORD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=20
)

print_chunks(
    "WORD CHUNKING",
    chunks
)


# ============================================================
# 4. SENTENCE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=150
)

print_chunks(
    "SENTENCE CHUNKING",
    chunks
)


# ============================================================
# 5. PARAGRAPH CHUNKING
# ============================================================

paragraph_text = """
Artificial Intelligence is transforming applications.

Large Language Models are powerful AI systems.

Chunking improves retrieval performance.
"""


chunks = chunker.chunk(
    paragraph_text,
    strategy=ChunkStrategy.PARAGRAPH,
    chunk_size=100
)

print_chunks(
    "PARAGRAPH CHUNKING",
    chunks
)


# ============================================================
# 6. SLIDING WINDOW CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SLIDING_WINDOW,
    window_size=100,
    stride=50
)

print_chunks(
    "SLIDING WINDOW CHUNKING",
    chunks
)


# ============================================================
# 7. RECURSIVE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.RECURSIVE,
    chunk_size=120
)

print_chunks(
    "RECURSIVE CHUNKING",
    chunks
)


# ============================================================
# 8. TOKEN CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.TOKEN,
    chunk_size=30
)

print_chunks(
    "TOKEN CHUNKING",
    chunks
)


# ============================================================
# 9. WORD CHUNK WITH OVERLAP
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=15,
    overlap=5
)

print_chunks(
    "WORD CHUNKING WITH OVERLAP",
    chunks
)


# ============================================================
# 10. CUSTOM SEPARATOR TEST
# ============================================================

custom_text = """
AI|Machine Learning|Deep Learning|Generative AI
"""


chunks = chunker.chunk(
    custom_text,
    strategy=ChunkStrategy.WORD,
    chunk_size=2,
    separators=["|"]
)

print_chunks(
    "CUSTOM SEPARATOR CHUNKING",
    chunks
)


# ============================================================
# 11. DISABLE SMALL CHUNK MERGING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=200,
    merge_small=False
)

print_chunks(
    "SENTENCE CHUNK WITHOUT MERGING",
    chunks
)


# ============================================================
# 12. INVALID INPUT TESTS
# ============================================================

print("\n")
print("=" * 80)
print("ERROR HANDLING TESTS")
print("=" * 80)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        chunk_size=0
    )

except Exception as e:
    print("Chunk size error:")
    print(type(e).__name__, e)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        overlap=-1
    )

except Exception as e:
    print("Overlap error:")
    print(type(e).__name__, e)


# ============================================================
# 13. ENUM TEST
# ============================================================

print("\n")
print("=" * 80)
print("SUPPORTED STRATEGIES")
print("=" * 80)


for strategy in ChunkStrategy:
    print(strategy.value)

RAG and retrieval examples

The sections below add the retrieval and RAG patterns used in real-world workflows. They are designed to be added on top of the existing preprocessing features without changing the rest of the README.

Example 1: Protect text before indexing

Use PII redaction before storing documents in a retrieval layer so private values do not remain in your index.

from aicurt.pii import PiiRedactor

text = "Contact alice@example.com for refund policy questions and call +1 555 123 4567."
redactor = PiiRedactor()
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"(?<!\w)(?:\+?\d[\d\s().-]{6,}\d)(?!\w)", replacement="[PHONE]")

protected_text = redactor.redact(text).text
print(protected_text)

Example output:

Contact [EMAIL] for refund policy questions and call [PHONE].

This keeps your knowledge base safe before search and retrieval are performed.

Example 2: Index policy documents with metadata

This shows a realistic workflow where each document is stored with metadata such as department and language.

from aicurt import AICurtEmbedder, AICurtRAG, AICurtStore


class MyEmbedder(AICurtEmbedder):
    def __init__(self):
        super().__init__(dimension=3)

    def embed(self, text):
        lower = text.lower()
        if "refund" in lower:
            return [1.0, 0.0, 0.0]
        if "policy" in lower:
            return [0.0, 1.0, 0.0]
        if "shipping" in lower:
            return [0.0, 0.0, 1.0]
        return [0.1, 0.1, 0.1]

    def embed_many(self, texts):
        return [self.embed(text) for text in texts]


store = AICurtStore("aicurtstore")
rag = AICurtRAG(store=store, embedder=MyEmbedder(), protect=True)

rag.index_text(
    "Contact alice@example.com for refund policy questions and call +1 555 123 4567.",
    source="customer_support.txt",
    metadata={"department": "support", "lang": "en"},
)
rag.index_text(
    "Shipping updates and delivery status are available in the order policy guide.",
    source="shipping_guide.txt",
    metadata={"department": "ops", "lang": "en"},
)

print(rag.store.count())
print(rag.store.list_ids())

This is the basic indexing pattern for RAG applications: store chunked text with useful metadata, then query it later.

Example 3: Run semantic, keyword, and hybrid retrieval

Each retrieval mode serves a different purpose.

semantic = rag.retrieve("refund", mode="semantic", top_k=5)
keyword = rag.retrieve("shipping status", mode="keyword", top_k=5)
hybrid = rag.retrieve("refund policy", mode="hybrid", top_k=5, filters={"department": "support"})

print("SEMANTIC:")
for item in semantic:
    print({
        "source": item.source,
        "text": item.text,
        "score": item.score,
        "metadata": item.metadata,
    })

print("KEYWORD:")
for item in keyword:
    print({
        "source": item.source,
        "text": item.text,
        "score": item.score,
        "metadata": item.metadata,
    })

print("HYBRID:")
for item in hybrid:
    print({
        "source": item.source,
        "text": item.text,
        "score": item.score,
        "metadata": item.metadata,
    })

What each mode is best for:

• semantic: similarity-based retrieval when wording differs from the source text
• keyword: direct match retrieval when exact terms matter most
• hybrid: balanced results that combine similarity and direct term relevance

Example 4: Update an indexed record

Sometimes the content or metadata changes after indexing. AICURT allows updating a stored chunk record inline.

first_id = rag.store.list_ids()[0]
record = rag.store.get(first_id)
print(record["text"])

rag.store.update(first_id, {
    "text": "Updated support policy: contact support@example.com for refund assistance.",
    "metadata": {"department": "support", "lang": "en", "status": "updated"},
})

updated = rag.store.get(first_id)
print(updated["text"])
print(updated["metadata"])

This is useful when a document changes and you want the retrieval index to reflect the new version.

Example 5: Delete a record from the index

This pattern is useful for housekeeping when a document is no longer valid or should be removed from search.

first_id = rag.store.list_ids()[0]
rag.store.delete(first_id)
print(rag.store.list_ids())
print(rag.store.count())

This keeps the index in sync with your actual source documents.

Example 6: Close the store cleanly

On Windows and in long-lived apps, it is good practice to close the SQLite-backed store when you finish with it.

rag.store.close()
print("Database closed successfully.")

This helps avoid stale file locks when deleting temporary directories or rerunning tests.

Example 7: Full end-to-end RAG pattern

This is the full, realistic pattern used in real applications: redact content, index documents, search them, and narrow the results with metadata.

from aicurt import AICurtEmbedder, AICurtRAG, AICurtStore
from aicurt.pii import PiiRedactor


class MyEmbedder(AICurtEmbedder):
    def __init__(self):
        super().__init__(dimension=3)

    def embed(self, text):
        lower = text.lower()
        if "refund" in lower:
            return [1.0, 0.0, 0.0]
        if "policy" in lower:
            return [0.0, 1.0, 0.0]
        if "shipping" in lower:
            return [0.0, 0.0, 1.0]
        return [0.1, 0.1, 0.1]

    def embed_many(self, texts):
        return [self.embed(text) for text in texts]


store = AICurtStore("aicurtstore")
rag = AICurtRAG(store=store, embedder=MyEmbedder(), protect=True)

redactor = PiiRedactor()
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"(?<!\w)(?:\+?\d[\d\s().-]{6,}\d)(?!\w)", replacement="[PHONE]")

policy_text = "Contact alice@example.com for refund policy questions and call +1 555 123 4567."
protected_text = redactor.redact(policy_text).text

rag.index_text(protected_text, source="customer_support.txt", metadata={"department": "support", "lang": "en"})
rag.index_text(
    "Shipping updates and delivery status are available in the order policy guide.",
    source="shipping_guide.txt",
    metadata={"department": "ops", "lang": "en"},
)

results = rag.retrieve("refund policy", mode="hybrid", top_k=5, filters={"department": "support"})
for item in results:
    print(item.text)
    print(item.score)

rag.close()

Summary

aicurt is a secure, lightweight, dependency-free toolkit for AI text preprocessing. The PII engine intentionally favors explicit, user-controlled matching and replacement so that only the patterns the caller chooses are masked. That makes it suitable for privacy-sensitive AI pipelines, embedded retrieval systems, and deterministic text cleaning workflows.

CLI usage

The package installs one console entry point named aicurt.

CLI command table

Command	Purpose
aicurt --help	Show CLI help
aicurt --version	Show the package version
aicurt detect <file>	Detect registered entities from a file or stdin
aicurt redact <file>	Redact registered entities from a file or stdin
aicurt chunk <file>	Chunk the input text
aicurt tokenize <file>	Tokenize the input text
aicurt stats <file>	Show token, word, and character statistics
aicurt rag index --input file.txt --store rag.db	Index text into the local SQLite-backed RAG store
aicurt rag search --query "refund policy" --store rag.db --mode hybrid --top-k 5	Search the indexed content

CLI examples

aicurt detect sample.txt
aicurt redact sample.txt --output redacted.txt
aicurt chunk sample.txt --strategy word --chunk-size 50
aicurt tokenize sample.txt
aicurt rag index --input sample.txt --store rag.db --source support.txt
aicurt rag search --query "refund policy" --store rag.db --mode hybrid --top-k 5

RAG CLI details

The built-in CLI includes a lightweight retrieval flow that does not require any third-party dependencies.

# Index plain text from stdin into a local SQLite store
printf "Refund policy allows returns within 30 days." | aicurt rag index --input - --store rag.db --source refund_policy.txt

# Search by keyword, semantic similarity, or hybrid ranking
 aicurt rag search --query "return refund" --store rag.db --mode hybrid --top-k 3

The search command prints JSON output like:

{
  "query": "return refund",
  "mode": "hybrid",
  "top_k": 3,
  "results": [
    {
      "text": "Refund policy allows returns within 30 days.",
      "score": 0.95,
      "source": "refund_policy.txt",
      "document_id": "refund_policy.txt",
      "chunk_id": "...",
      "metadata": {}
    }
  ]
}

Required inputs

The caller should provide:

Workflow	Required input
PII detect	a text string, file path, or stdin stream
PII redact	a text string, file path, or stdin stream plus a rule or regex to register
Chunking	a text string, file path, or stdin stream plus a chunk strategy and chunk size
Tokenization	a text string, file path, or stdin stream
Custom masking	a regex pattern, a rule name, and a replacement token

Contributions

Contributions are welcome through the normal repository maintainer process.

For formal contribution or review requests:

1. open a pull request or review request through the repository workflow
2. keep changes aligned with the package’s security, data-privacy, and dependency-free design goals
3. preserve the selective, explicit registration model for PII redaction

For more details on contribution process please vist - Contribution Guidelines

Code of Conduct

Please review our Code of Conduct before contributing to this project.

Testing

This project uses Python's built-in unittest framework. All test cases are located inside the tests/ directory.

Security

Security-sensitive workflows should keep all processing local to the runtime environment.

Please review SECURITY.md for vulnerability disclosure guidance.

License

This package is distributed under a restrictive proprietary all-rights-reserved license. See LICENSE and NOTICE for the exact legal terms.

Best practices

• Use PiiRedactor with explicit register_rule() and register_pattern() calls for sensitive text handling.
• Use mask() when you want a structured MaskingResult object.
• Use redact() when you want a simple text replacement flow.
• Use SimpleTokenizer for lightweight token-aware chunking and embedding payload generation.
• Keep chunk sizes deterministic for downstream model input limits.
• Prefer explicit user-controlled redaction over implicit broad masking.