Tixent: A Text Splitting Tool

PyPI

CI/CD

Build System

Code

Docstrings

Installation

pip install tixent

Example

Suppose we have a function template that generates a string from a list of texts. Additionally, suppose we have a large list of texts. When you apply that list of texts to the function, it generates a long string.

Tixent can split the string generated by the template function so that the return value of counter for each element is less than a certain number.

Here, counter is a function that maps a string to an integer. Examples of such functions are len, which measures the length of a string, or tiktoken_counter("text-davinci-003"), which measures the number of tokens in a string

from typing import List

from tixent import split, tiktoken_counter


def summarization_template(texts: List[str]) -> str:
    text = " ".join(texts)
    t = "Summarize the following text.\n"
    t += f'Text: """{text}"""'
    return t


texts = [
    "Lorem ipsum dolor sit amet",
    "consectetur adipiscing elit",
    "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua",
    "Ut enim ad minim veniam",
    "quis nostrud exercitation ullamco laboris nisi",
    "ut aliquip ex ea commodo consequat",
    "Duis aute irure dolor in reprehenderit in voluptate velit",
    "esse cillum dolore eu fugiat nulla pariatur",
    "Excepteur sint occaecat cupidatat non proident",
    "sunt in culpa qui officia deserunt mollit anim id est laborum",
]
counter = tiktoken_counter("text-davinci-003")
max_count = 60

split_texts = split(texts, summarization_template, counter, max_count)
for text in split_texts:
    count = counter(text)
    assert count <= max_count
    print(f"count: {count}")
    print(text)
    print()

count: 60
Summarize the following text.
Text: """Lorem ipsum dolor sit amet consectetur adipiscing elit sed do eiusmod tempor incididunt ut labore et dolore magna aliqua Ut enim ad minim veniam"""

count: 58
Summarize the following text.
Text: """quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat Duis aute irure dolor in reprehenderit in voluptate velit"""

count: 43
Summarize the following text.
Text: """esse cillum dolore eu fugiat nulla pariatur Excepteur sint occaecat cupidatat non proident"""

count: 31
Summarize the following text.
Text: """sunt in culpa qui officia deserunt mollit anim id est laborum"""