aitextutils

Text cleaning, chunking, and prompt helpers for AI pipelines.

This package is the text-only counterpart to aitextkit. It has no OS installer and no system.py.

Install

pip install aitextutils

Local development:

cd aitextutils
pip install -e ".[dev]"

Usage

from aitextutils import (
    build_chat_messages,
    build_prompt,
    clean_text,
    chunk_text,
    estimate_tokens,
    extract_code_block,
    extract_json_block,
    redact_secrets,
    truncate_to_tokens,
)

clean_text("Hello   <b>world</b>")
# "Hello world"

estimate_tokens("abcd")
# 1

build_prompt("Summarize this.", system="You are concise.", context="A long article.")
build_chat_messages("Hello", system="Be brief.")
extract_code_block("```python\nprint(1)\n```", "python")
extract_json_block('Answer: {"ok": true}')
redact_secrets("contact me@site.com")
truncate_to_tokens(long_text, 200)

API

• clean_text(value, strip_html=True, strip_urls=False)
• chunk_text(value, max_chars=1000, overlap_chars=100)
• estimate_tokens(value)
• truncate_to_tokens(value, max_tokens)
• build_prompt(instruction, system="", context="", max_context_tokens=None)
• build_chat_messages(user, system="", context="")
• extract_code_block(value, language=None)
• extract_json_block(value)
• redact_secrets(value)

License

MIT