chunkifyr 0.1.2

Your ultimate toolkit for text chunking.

Chunkifyr 📜🔪

Chunkifyr is a powerful and flexible text chunking library designed to split and chunk text into meaningful segments. Whether you're processing large documents, preparing data for NLP tasks, or simply need to manage text in manageable chunks, Chunkifyr provides a range of customizable chunking strategies.

Features ✨

• Language Model Chunking: Leverage the context understanding of languag models to chunk text based on semantic understanding.
• Syntactic Chunking: Break down text into syntactically meaningful segments, preserving grammatical structures.
• Spacy-Enhanced Chunking: Integrate with SpaCy for both semantic and syntactic chunking, taking advantage of SpaCy's powerful NLP pipeline.
• Customizable Settings: Easily adjust chunk sizes, overlap percentages, and more to fit your specific needs.
• Robust and Fast: Efficiently handles large texts, ensuring your workflow remains smooth and responsive.

Installation 🛠️

Install Chunkifyr via pip:

pip install chunkifyr

Note: Python 3.8+ is required.

Usage 🚀

Here’s a quick example using LMChunker to get you started:

from chunkifyr import LMChunker
from openai import OpenAI

# this creds can be replaced with your local oai server creds, if your running local OAI server. (llama_cpp, llamafile, ollama)
client = OpenAI(api_key="YOUR_API_KEY", base_url="DEPLOYMENT_URL") 

chunker = LMChunker(model="gpt-3.5-turbo-0125", client=client)
chunks = chunker.from_file('path_to_your_text_file.txt')

for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}: {chunk.text}")
    print(f"Chunk {i} description: {chunk.meta.description}")
    print()

This example demonstrates how to use the LMChunker with an OpenAI client to break a text file into meaningful chunks. Each chunk also includes a description generated by the model. (which can be further used as metadata when embedding)

Available Chunkers

• LMChunker: Utilizes pre-trained language models for contextual chunking.
• SimpleSemanticChunker: Groups similar splits together for basic semantic chunking.
• SimpleSyntacticChunker: Simple syntactic chunking with desired chunk size, overlap and seperator. (very similar to langchain character splitter)
• SemanticChunker: Groups text semantically using the Adjacent Sentence Clustering process with a configurable similarity threshold.
• SyntacticChunker: Chunks text based on syntactic structures using customizable tokenization (supports hf_tokenizer, tiktoken). More soon... (Regex based, etc)

Contributing 🤝

Contributions are welcome! If you have ideas for improving Chunkifyr or encounter any issues, feel free to submit a pull request or open an issue.

License 📄

This project is licensed under the MIT License - see the LICENSE file for details.