delos-unichunking 1.4.0

Universal chunking functions to extract LLM-friendly chunks from any file type.

Unichunking

Extract LLM-friendly chunks from any file type.

Supported file types are :

• DOCX & DOCX-like (DOC, ODT)
• PPTX & PPTX-like (PPT, ODP)
• XLSX & XLSX-like (XLS, ODS)
• TXT, MD, CSV
• IPYNB

Installation

To install, run the following command:

python3 -m pip install delos-unichunking

How to use

The main functions are :

• extract_subchunks returns a list of all the text particles in the file.
• split_chunks_with_overlap transforms a list of subchunks on a given page into a list of chunks following default or specified parameters for minimum/maximum token size and overlap.
• build_chunked_pages returns a list of "pages", which are lists of formated chunks, following the structure of the document.
• compute_pages approximates the pagination of a file that does not have a native pagination system (such as DOCX) by comparing it to a PDF version.

Specificities

Please note that the package requires a LibreOffice installation to run soffice commands, used during file conversions : for instance, DOC/ODT are first converted to DOCX format and processed as such.

The page numbers computed for DOCX files are an approximation and can be off by a few pages for large files.

Artifical page numbers are used for page-less structures such as TXT files, or to split large XLSX sheets into multiple pages, to follow a tokens-per-page limit.