aitextkit

Text cleaning, chunking, and token estimation for AI pipelines.

Install

pip install aitextkit

For local development:

pip install -e .

Usage

from aitextkit import (
    build_chat_messages,
    build_prompt,
    clean_text,
    extract_code_block,
    extract_json_block,
    redact_secrets,
    truncate_to_tokens,
)

clean_text("Hello   <b>world</b>")
# "Hello world"

prompt = build_prompt(
    "Summarize this.",
    system="You are concise.",
    context="A long article.",
    max_context_tokens=500,
)

messages = build_chat_messages("Hello", system="Be brief.")
extract_code_block("```python\nprint(1)\n```", "python")
extract_json_block('Answer: {"ok": true}')
redact_secrets("contact me@site.com")
truncate_to_tokens(long_text, 200)

API

• clean_text(value, strip_html=True, strip_urls=False)
• chunk_text(value, max_chars=1000, overlap_chars=100)
• estimate_tokens(value)
• truncate_to_tokens(value, max_tokens)
• build_prompt(instruction, system="", context="", max_context_tokens=None)
• build_chat_messages(user, system="", context="")
• extract_code_block(value, language=None)
• extract_json_block(value)
• redact_secrets(value)
• get_os_version() — returns windows, mac, or linux in name

License

MIT