tokens-lab 0.5.4

LLM utilities with agents, file parsing, and preprocessing built in.

tokens-lab

LLM utilities with agents, file parsing, and preprocessing built in.

Features

• Agent Framework: LangGraph-based agents with OpenAI integration
• File Text I/O: Parse PDFs, DOCX, PPTX, and images with ease
• Preprocessing: Text normalization and cleaning utilities
• Azure Integration: Azure Storage and service utilities
• ML Tools: String similarity and text analysis
• Utilities: Date, Excel, JSON, and Pandas helpers
• Logging: Structured logging support
• Exception Handling: Custom exception hierarchy

Installation

pip install tokens-lab

Quick Start

LLM Agent

from llm_lab.agent import Agent
from llm_lab.agent.litellm_client import LiteLLMClient

client = LiteLLMClient(model="gpt-4", api_key="your-api-key")
agent = Agent(client=client)
response = agent.process("Hello, how can you help me?")

File Parsing

from llm_lab.filetextio import parsers

content = parsers.parse_document("document.pdf")

Text Preprocessing

from llm_lab.preprocessing import text_normalization

clean_text = text_normalization.normalize_text("messy   text  ")

Documentation

For complete documentation, visit: https://pwc-me-adv-strategyand.github.io/LLM_Lab/

Requirements

• Python >= 3.10

License

MIT