aicurt 0.1.0

`AICurt` is a lightweight, Python AI toolkit for AI text preprocessing that provides customizable PII redaction, deterministic text chunking, and token analysis utilities for LLM, embedding, and retrieval workflows.

aicurt

aicurt is a lightweight, dependency-free Python toolkit for AI text preprocessing.

It focuses on three things:

• selective PII detection and redaction through explicit regex or rule registration
• deterministic chunking for downstream model input windows
• lightweight tokenization and token statistics for embedding and retrieval workflows

Package overview

Area	Capability
PII Engine	Register your own patterns and replace only the matches you want to protect
Chunking Engine	Split text with word, sentence, paragraph, sliding-window, recursive, and token strategies
Tokenizer	Tokenize text and compute token, word, and character statistics
CLI	Read from stdin or a file, then detect, redact, chunk, tokenize, or print stats
Packaging	Installable as a standard Python package with an aicurt console script

What the current PII model does

The current design is intentionally selective:

• PiiRedactor() starts as a passive redactor with no built-in detection enabled
• you explicitly register the patterns you want to redact by calling register_rule() or register_pattern()
• detect() returns only the matches from the patterns you have registered
• redact() replaces only those matches, leaving unrelated text untouched

That makes the engine safer, more predictable, and easier to embed into production pipelines.

Core API

PiiRedactor

Public methods:

• detect(text) → returns a list of PiiMatch objects
• mask(text) → returns a MaskingResult with original_text, masked_text, matches, and mapping
• redact(text, replacement=None) → same redaction flow, with optional global replacement override
• register_rule(name, pattern, replacement=...) → register a named rule that will be detected and replaced
• register_pattern(name, pattern) → register a custom regex pattern for detection
• addRegex(pattern) / addRegexPatterns(patterns) → register more regexes
• addWord(word, mask_length=None, case_sensitive=None) / addWords(...) → register word-based masking rules
• configureEmailMasking(...) and configurePhoneMasking(...) → tweak the built-in masking settings when needed
• setMaskCharacter(...) and setMaskLength(...) → adjust the mask output style

PiiConfig

PiiConfig is the configuration object that carries replacement, mask style, and masking-policy settings.

Common fields:

• mask_strategy
• mask_char
• mask_visible_prefix
• mask_visible_suffix
• preserve_length
• default_replacement
• masking_policies
• enable_reversible

Installation

This package is available through the PyPI registry.

Before installing, ensure you have Python 3.9 or higher installed. You can download and install Python from python.org.

You can install the package using pip:

pip install aicurt

Editable development install

python -m pip install -e .

Verify the install

aicurt --help
aicurt --version

Quick start

Python API

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact("Contact [e-mail] now")
print(result.text)

Custom replacement callback

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)

result = redactor.redact(
    "[e-mail]",
    replacement=lambda match: f"<{match.entity_type}>",
)
print(result.text)

Selective partial masking policy

from aicurt.pii import PiiConfig, PiiRedactor

config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)

redactor = PiiRedactor(config)
redactor.register_rule(
    "EMAIL",
    r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b",
    replacement="[EMAIL]",
)
redactor.register_rule(
    "PHONE",
    r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b",
    replacement="[PHONE]",
)

result = redactor.redact("Email: [e-mail], Phone: [phone]")
print(result.text)

This keeps labels like Email: and Phone: intact while only replacing the registered sensitive values.

End-to-end examples

Example 1: Detect a custom email rule

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_pattern("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b")

matches = redactor.detect("Contact [e-mail] now")
for match in matches:
    print(match.entity_type, match.value, match.start, match.end)

Example 2: Redact a custom organization name

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.register_rule("ORG", r"Acme", replacement="[COMPANY]")

result = redactor.redact("Acme is here")
print(result.text)

Example 3: Register a custom word and mask it with a specific length

from aicurt.pii import PiiRedactor

redactor = PiiRedactor()
redactor.addWord("secret-token", mask_length=6)

result = redactor.redact("The secret-token value must be hidden")
print(result.text)

Example 4: Chunk text

from aicurt.chunking import ChunkStrategy, TextChunker

text = "Paragraph one. Paragraph two."
chunks = TextChunker().chunk(text, strategy=ChunkStrategy.SENTENCE, chunk_size=20)
for chunk in chunks:
    print(chunk.index, chunk.content)

Example 5: Tokenize and compute stats

from aicurt.tokenizer import SimpleTokenizer

text = "hello world"
tokenizer = SimpleTokenizer()
print(tokenizer.tokenize(text))
print(tokenizer.count_tokens(text))
print(tokenizer.stats(text))

Example 6: Embedding-ready payload

import json

result = redactor.redact("Contact [e-mail] now")
stats = SimpleTokenizer().stats("Contact [e-mail] now")
payload = {
    "masked_text": result.text,
    "token_count": stats.token_count,
    "word_count": stats.word_count,
    "character_count": stats.character_count,
    "matched_entities": [
        {"entity_type": match.entity_type, "value": match.value}
        for match in result.matches
    ],
}

print(json.dumps(payload, ensure_ascii=False, indent=2))

PII Examples

Refer to the PII examples below and use them as a guide when implementing the PII masking in your code.

from aicurt.pii import PiiConfig, PiiRedactor

paragraph_text = """Customer Information Report

Krishna Tadi is a product manager based in Bengaluru, Karnataka. His work email is [e-mail] and his contact number is [phone].
During onboarding, Krishna shared his Aadhaar number 234567891234, PAN number ABCDE1234F, and passport number N1234567. The same profile also included a driver's license number DL-0420110012345.
The account team reviewed the customer's credit card number 4111 1111 1111 1111 and bank account number 123456789012. The date of birth listed in the record was 15-08-1995, and the latest login IP address was 192.168.1.105.
The account API key used for integration testing was sk_live_51N8example123456789, and the username associated with the account was krishna_t_95.
This document should remain confidential and should only be shared with authorized personnel for secure verification steps.
"""

mask_config = PiiConfig(
    mask_strategy="partial",
    mask_char="*",
    mask_visible_prefix=1,
    mask_visible_suffix=1,
    preserve_length=True,
)
redactor = PiiRedactor(mask_config)
redactor.register_rule("EMAIL", r"\b[\w.%+-]+@[\w.-]+\.[A-Za-z]{2,}\b", replacement="[EMAIL]")
redactor.register_rule("PHONE", r"\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})\b", replacement="[PHONE]")
redactor.register_rule("AADHAAR", r"\b\d{12}\b", replacement="[AADHAAR]")
redactor.register_rule("PAN", r"\b[A-Z]{5}[0-9]{4}[A-Z]\b", replacement="[PAN]")
redactor.register_rule("BANK_ACCOUNT", r"\b\d{9,18}\b", replacement="[BANK_ACCOUNT]")
redactor.register_rule("DOB", r"\b(?:0?[1-9]|[12]\d|3[01])[-/](?:0?[1-9]|1[0-2])[-/](?:\d{4})\b", replacement="[DOB]")
redactor.register_rule("IP", r"\b(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}\b", replacement="[IP]")

redactor.addWord("KRISHNA")
redactor.addWord("4111 1111 1111 1111")

result = redactor.redact(paragraph_text)
matches = redactor.detect(paragraph_text)

print("Masked paragraph preview:")
print(result.text)
print("\nDetected matches:")
for match in matches[:6]:
    print(f"- {match.entity_type}: {match.value}")

Chunking Examples

Refer to the chunking examples below and use them as a guide when implementing the chunking strategy in your code.

text = """
Artificial Intelligence is transforming the way developers build applications.
Large Language Models can understand and generate human-like text.
Retrieval Augmented Generation combines search with AI models.
Chunking is an important step because large documents need to be split into smaller pieces.
Good chunking improves embeddings, retrieval accuracy, and response quality.
This library provides deterministic text preprocessing utilities for AI workflows.
"""

chunker = TextChunker()

def print_chunks(title, chunks):
    print("\n")
    print("=" * 80)
    print(title)
    print("=" * 80)

    for chunk in chunks:
        print("\nChunk Index:", chunk.index)
        print("Content:")
        print(chunk.content)
        print("Start:", chunk.start)
        print("End:", chunk.end)
        print("Characters:", chunk.character_count)
        print("Words:", chunk.word_count)
        print("Tokens:", chunk.token_count)


# ============================================================
# 1. STANDARD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.STANDARD,
    chunk_size=100
)

print_chunks(
    "STANDARD CHUNKING",
    chunks
)


# ============================================================
# 2. CHARACTER CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.CHARACTER,
    chunk_size=80
)

print_chunks(
    "CHARACTER CHUNKING",
    chunks
)


# ============================================================
# 3. WORD CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=20
)

print_chunks(
    "WORD CHUNKING",
    chunks
)


# ============================================================
# 4. SENTENCE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=150
)

print_chunks(
    "SENTENCE CHUNKING",
    chunks
)


# ============================================================
# 5. PARAGRAPH CHUNKING
# ============================================================

paragraph_text = """
Artificial Intelligence is transforming applications.

Large Language Models are powerful AI systems.

Chunking improves retrieval performance.
"""


chunks = chunker.chunk(
    paragraph_text,
    strategy=ChunkStrategy.PARAGRAPH,
    chunk_size=100
)

print_chunks(
    "PARAGRAPH CHUNKING",
    chunks
)


# ============================================================
# 6. SLIDING WINDOW CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SLIDING_WINDOW,
    window_size=100,
    stride=50
)

print_chunks(
    "SLIDING WINDOW CHUNKING",
    chunks
)


# ============================================================
# 7. RECURSIVE CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.RECURSIVE,
    chunk_size=120
)

print_chunks(
    "RECURSIVE CHUNKING",
    chunks
)


# ============================================================
# 8. TOKEN CHUNKING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.TOKEN,
    chunk_size=30
)

print_chunks(
    "TOKEN CHUNKING",
    chunks
)


# ============================================================
# 9. WORD CHUNK WITH OVERLAP
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.WORD,
    chunk_size=15,
    overlap=5
)

print_chunks(
    "WORD CHUNKING WITH OVERLAP",
    chunks
)


# ============================================================
# 10. CUSTOM SEPARATOR TEST
# ============================================================

custom_text = """
AI|Machine Learning|Deep Learning|Generative AI
"""


chunks = chunker.chunk(
    custom_text,
    strategy=ChunkStrategy.WORD,
    chunk_size=2,
    separators=["|"]
)

print_chunks(
    "CUSTOM SEPARATOR CHUNKING",
    chunks
)


# ============================================================
# 11. DISABLE SMALL CHUNK MERGING
# ============================================================

chunks = chunker.chunk(
    text,
    strategy=ChunkStrategy.SENTENCE,
    chunk_size=200,
    merge_small=False
)

print_chunks(
    "SENTENCE CHUNK WITHOUT MERGING",
    chunks
)


# ============================================================
# 12. INVALID INPUT TESTS
# ============================================================

print("\n")
print("=" * 80)
print("ERROR HANDLING TESTS")
print("=" * 80)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        chunk_size=0
    )

except Exception as e:
    print("Chunk size error:")
    print(type(e).__name__, e)


try:
    chunker.chunk(
        text,
        strategy=ChunkStrategy.WORD,
        overlap=-1
    )

except Exception as e:
    print("Overlap error:")
    print(type(e).__name__, e)


# ============================================================
# 13. ENUM TEST
# ============================================================

print("\n")
print("=" * 80)
print("SUPPORTED STRATEGIES")
print("=" * 80)


for strategy in ChunkStrategy:
    print(strategy.value)

CLI usage

The package installs one console entry point named aicurt.

CLI command table

Command	Purpose
aicurt --help	Show CLI help
aicurt --version	Show the package version
aicurt detect <file>	Detect registered entities from a file or stdin
aicurt redact <file>	Redact registered entities from a file or stdin
aicurt chunk <file>	Chunk the input text
aicurt tokenize <file>	Tokenize the input text
aicurt stats <file>	Show token, word, and character statistics

CLI examples

aicurt detect sample.txt
aicurt redact sample.txt --output redacted.txt
aicurt chunk sample.txt --strategy word --chunk-size 50
aicurt tokenize sample.txt

Required inputs

The caller should provide:

Workflow	Required input
PII detect	a text string, file path, or stdin stream
PII redact	a text string, file path, or stdin stream plus a rule or regex to register
Chunking	a text string, file path, or stdin stream plus a chunk strategy and chunk size
Tokenization	a text string, file path, or stdin stream
Custom masking	a regex pattern, a rule name, and a replacement token

Contributions

Contributions are welcome through the normal repository maintainer process.

For formal contribution or review requests:

1. open a pull request or review request through the repository workflow
2. keep changes aligned with the package’s security, data-privacy, and dependency-free design goals
3. preserve the selective, explicit registration model for PII redaction

For more details on contribution process please vist - Contribution Guidelines

Code of Conduct

Please review our Code of Conduct before contributing to this project.

Security

Security-sensitive workflows should keep all processing local to the runtime environment.

Please review SECURITY.md for vulnerability disclosure guidance.

License

This package is distributed under a restrictive proprietary all-rights-reserved license. See LICENSE and NOTICE for the exact legal terms.

Best practices

• Use PiiRedactor with explicit register_rule() and register_pattern() calls for sensitive text handling.
• Use mask() when you want a structured MaskingResult object.
• Use redact() when you want a simple text replacement flow.
• Use SimpleTokenizer for lightweight token-aware chunking and embedding payload generation.
• Keep chunk sizes deterministic for downstream model input limits.
• Prefer explicit user-controlled redaction over implicit broad masking.

Summary

aicurt is a secure, lightweight, dependency-free toolkit for AI text preprocessing. The PII engine intentionally favors explicit, user-controlled matching and replacement so that only the patterns the caller chooses are masked. That makes it suitable for privacy-sensitive AI pipelines, embedded retrieval systems, and deterministic text cleaning workflows.