Unichunking
Extract LLM-friendly chunks from any file type.
Supported file types are :
- DOCX & DOCX-like (DOC, ODT)
- PPTX & PPTX-like (PPT, ODP)
- XLSX & XLSX-like (XLS, ODS)
- TXT, MD, CSV
- IPYNB
Installation
To install, run the following command:
python3 -m pip install delos-unichunking
How to use
The main functions are :
extract_subchunksreturns a list of all the text particles in the file.split_chunks_with_overlaptransforms a list of subchunks on a given page into a list of chunks following default or specified parameters for minimum/maximum token size and overlap.build_chunked_pagesreturns a list of "pages", which are lists of formated chunks, following the structure of the document.compute_pagesapproximates the pagination of a file that does not have a native pagination system (such as DOCX) by comparing it to a PDF version.
Specificities
Please note that the package requires a LibreOffice installation to run soffice commands, used during file conversions : for instance, DOC/ODT are first converted to DOCX format and processed as such.
The page numbers computed for DOCX files are an approximation and can be off by a few pages for large files.
Artifical page numbers are used for page-less structures such as TXT files, or to split large XLSX sheets into multiple pages, to follow a tokens-per-page limit.
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file delos_unichunking-1.5.0.tar.gz.
File metadata
- Download URL: delos_unichunking-1.5.0.tar.gz
- Upload date:
- Size: 223.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"EndeavourOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9923fd69c62c59b5c15994f42d58430691c6247aa867b1f2bdd3229a58d32058
|
|
| MD5 |
eaa8ee6b9f7ea883abc94e8d2832c9a9
|
|
| BLAKE2b-256 |
45cf66f971af3cfb4418890966534cc0ae444578af011ef9319008ddafd1e8d2
|
File details
Details for the file delos_unichunking-1.5.0-py3-none-any.whl.
File metadata
- Download URL: delos_unichunking-1.5.0-py3-none-any.whl
- Upload date:
- Size: 43.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"EndeavourOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
193f28a680e736cd4df6e53d90039e0447953b62835972840f6d27d2201bd61a
|
|
| MD5 |
77c9f05908a6855b1bf92a4d514ac609
|
|
| BLAKE2b-256 |
d9457a4b672bc2010915a94aba1f0651180323ea81c579d461199fe0d1f3c21f
|
