Unichunking
Extract LLM-friendly chunks from any file type.
Supported file types are :
- DOCX & DOCX-like (DOC, ODT)
- PPTX & PPTX-like (PPT, ODP)
- XLSX & XLSX-like (XLS, ODS)
- TXT, MD, CSV
- IPYNB
Installation
To install, run the following command:
python3 -m pip install delos-unichunking
How to use
The main functions are :
extract_subchunksreturns a list of all the text particles in the file.split_chunks_with_overlaptransforms a list of subchunks on a given page into a list of chunks following default or specified parameters for minimum/maximum token size and overlap.build_chunked_pagesreturns a list of "pages", which are lists of formated chunks, following the structure of the document.compute_pagesapproximates the pagination of a file that does not have a native pagination system (such as DOCX) by comparing it to a PDF version.
Specificities
Please note that the package requires a LibreOffice installation to run soffice commands, used during file conversions : for instance, DOC/ODT are first converted to DOCX format and processed as such.
The page numbers computed for DOCX files are an approximation and can be off by a few pages for large files.
Artifical page numbers are used for page-less structures such as TXT files, or to split large XLSX sheets into multiple pages, to follow a tokens-per-page limit.
2 maintainers
Metadata
2 maintainers
Release files for delos-unichunking 1.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| delos_unichunking-1.6.1.tar.gz | 226.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| delos_unichunking-1.6.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 270.9 kB
Release files / delos_unichunking-1.6.1.tar.gz
| Download URL | delos_unichunking-1.6.1.tar.gz |
|---|---|
| Size | 226.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5ae529e7b78a4f435efc21910a1d06aca2e5a85275ad466a49f6ce251acdaac6
|
|
BLAKE2b-256 checksum How to use checksums |
5931056f750d66078654607dc4f9f9f0328b57d006c59828b11f45f65e7b474d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.20 {"installer":{"name":"uv","version":"0.12.20","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / delos_unichunking-1.6.1-py3-none-any.whl
| Download URL | delos_unichunking-1.6.1-py3-none-any.whl |
|---|---|
| Size | 44.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
01db5e9a92d4068a939cc24468ecbe8dc4896e898ccf7bf51b5e9a1952a35ad3
|
|
BLAKE2b-256 checksum How to use checksums |
ce3ada5f0d399551b6222050603bbd6131a3587fae5c553912d7fb002cf6be71
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.20 {"installer":{"name":"uv","version":"0.12.20","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
