sayou-chunking
Overview
The Structure-Aware Splitter for Sayou Fabric.
sayou-chunking splits large texts into smaller, semantically meaningful units called Chunks. Unlike traditional splitters that blindly cut text by character count, this library understands the syntax structure of the data.
It focuses on preserving the integrity of code blocks, tables, and JSON objects, ensuring that Retrieval (RAG) systems fetch complete and executable contexts.
1. Architecture & Role
The Chunking engine takes raw text (from Refinery) and applies a Syntax-Aware Strategy to produce atomic chunks.
graph LR
Text[Refined Text] --> Pipeline[Chunking Pipeline]
subgraph Strategies
MD[Markdown Header]
Code[Code AST]
JSON[JSON Object]
end
Pipeline -->|Config Routing| Strategies
Strategies --> Chunks[Atomic Chunks]
1.1. Core Features
- Syntax Awareness: Never splits in the middle of a code block or a markdown table.
- Hierarchy Preservation: Attaches metadata about the parent section (e.g., Header Path, Class Name) to every chunk.
- Atomic Integrity: Ensures that a JSON object or a Python function remains a single unit.
2. Available Strategies
sayou-chunking prioritizes deterministic structural splitting over probabilistic methods.
| Strategy Key | Target Format | Description |
|---|---|---|
markdown |
Markdown, Text | Splits by Headers (#, ##). Preserves Tables and Code Blocks as atomic units. |
code |
Python, JS, Java | Uses AST (Abstract Syntax Tree) to split by Class and Function definitions. |
json |
JSON, JSONL | Splits large JSON arrays into individual records or sub-trees. |
3. Installation
pip install sayou-chunking
4. Usage
The ChunkingPipeline is the entry point. It accepts a ChunkingRequest containing content and metadata.
Case A: Markdown Splitting (RAG Standard)
Ideal for documentation. It splits by headers while keeping sections together.
from sayou.chunking import ChunkingPipeline
text_content = """
# Section 1
Introduction text...
## Subsection 1.1
- Item A
- Item B
"""
chunks = ChunkingPipeline.process(
data={"content": text_content, "metadata": {"source": "doc.md"}},
strategy="markdown"
)
# 4. Result
for chunk in chunks:
print(f"[{chunk.metadata['type']}] {chunk.content[:20]}...")
# Output: [heading] # Section 1...
# Output: [text] Introduction text...
Case B: Code Splitting (Python AST)
Ideal for code analysis. It splits by logical units (Functions/Classes).
from sayou.chunking import ChunkingPipeline
code_content = """
class MyClass:
def method_a(self):
print("hello")
def global_func():
pass
"""
chunks = ChunkingPipeline.process(
data={"content": code_content, "metadata": {"language": "python"}},
strategy="code"
)
# Result: 2 Chunks (1 Class block, 1 Function block)
print(f"Generated {len(chunks)} logic blocks.")
Case C: JSON Splitting
Ideal for processing large data logs or API responses.
from sayou.chunking import ChunkingPipeline
json_content = '[{"id": 1, "val": "A"}, {"id": 2, "val": "B"}]'
chunks = ChunkingPipeline.process(
data={"content": json_content, "metadata": {}},
strategy="json"
)
# Result: 2 Chunks (Each object is a separate chunk)
5. Configuration Keys
Customize the behavior of each splitter via the config dictionary.
markdown:header_depth(1-6),strip_headers(bool).code:language(python),chunk_lines(min/max lines).json:jq_query(filter pattern),max_size.
6. License
Apache 2.0 License © 2026 Sayouzone
7. Plugin List
| Plugin | Example | Description |
|---|---|---|
Json Chunking |
▶ | |
Markdown Chunking |
▶ | |
Python Chunking |
▶ | |
Java Chunking |
▶ | |
Javascript Chunking |
▶ |
1 maintainer
Metadata
1 maintainer
Release files for sayou-chunking 0.5.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| sayou_chunking-0.5.1.tar.gz | 76.0 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sayou_chunking-0.5.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 120.5 kB
Release files / sayou_chunking-0.5.1.tar.gz
| Download URL | sayou_chunking-0.5.1.tar.gz |
|---|---|
| Size | 76.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ea7fd05c458c6a82b692b49f063226a554e2aa744faa0b44dc35a4b1b8571af8
|
|
BLAKE2b-256 checksum How to use checksums |
7f4e71621e7a2ca3447d86ee83491524a5c503f4871211c5de154a391d01ce03
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / sayou_chunking-0.5.1-py3-none-any.whl
| Download URL | sayou_chunking-0.5.1-py3-none-any.whl |
|---|---|
| Size | 44.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e18496617df0e34c39a6ba175e332318e188e2301a0833b7415f1309d95a6d19
|
|
BLAKE2b-256 checksum How to use checksums |
99915ddbd49f668736ebb5b20f128333f01e34cc419f3de4a553c4cf16728606
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Security for sayou-chunking 0.5.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
