sayou-refinery
Overview
The Universal Data Cleaning & Normalization Engine for Sayou Fabric.
sayou-refinery acts as the "Cleaning Plant" in your data pipeline. It transforms heterogeneous raw data (JSON Documents, HTML, DB Records) into a standardized stream of SayouBlocks.
It ensures that downstream components (like Chunkers or LLMs) receive clean, uniform data regardless of whether the source was a messy web scrape or a structured database row.
1. Architecture & Role
Refinery operates in two distinct stages to guarantee data quality: Normalization (Shape Shifting) and Processing (Hygiene).
graph LR
Raw[Raw Input] --> Pipeline[Refinery Pipeline]
subgraph Stage1 [Normalization]
Doc[Doc Normalizer]
Html[Html Normalizer]
Json[Json Normalizer]
end
subgraph Stage2 [Processing Chain]
Space[Whitespace]
PII[PII Masker]
Link[Link Extractor]
end
Pipeline --> Stage1
Stage1 --> Stage2
Stage2 --> Blocks[Clean SayouBlocks]
1.1. Core Features
- Normalization: Flattens complex structures (Nested JSON, HTML Trees) into a linear list of blocks.
- Hygiene: Removes invisible characters, normalizes Unicode, and fixes broken encoding.
- Safety: Automatically masks sensitive information (PII) like emails or phone numbers before they reach the LLM.
2. Available Strategies
sayou-refinery provides strategies tailored to specific input formats.
| Strategy Key | Target Format | Description |
|---|---|---|
standard_doc |
Sayou Document | [Default] Converts parsed document dictionaries into Markdown blocks. Applies standard text cleaning. |
html |
Web Pages | Strips HTML tags, extracts links, and converts the DOM tree into readable text blocks. |
json |
API/DB Records | Flattens JSON objects into key-value pairs or text representations. |
3. Installation
pip install sayou-refinery
4. Usage
The RefineryPipeline orchestrates the normalization and processing chain.
Case A: Document Cleaning (Standard)
Cleans messy OCR output or parsed document text.
from sayou.refinery import RefineryPipeline
raw_doc = {
"metadata": {"title": "Test Doc"},
"pages": [{
"elements": [
{"type": "text", "text": "Contact: admin@sayou.ai "},
{"type": "text", "text": "Generic Whitespace Error"}
]
}]
}
blocks = RefineryPipeline.process(
data=raw_doc,
strategy="standard_doc"
)
for block in blocks:
print(f"[{block.type}] {block.content}")
# Output: [text] Contact: [EMAIL]
# Output: [text] Generic Whitespace Error
Case B: HTML Processing
Converts web content into clean text while preserving hyperlinks.
from sayou.refinery import RefineryPipeline
raw_html = """
<html>
<body>
<h1>Welcome</h1>
<p>Click <a href='https://sayou.ai'>here</a>.</p>
</body>
</html>
"""
blocks = RefineryPipeline.process(
data=raw_html,
strategy="html"
)
# Result:
# [heading] Welcome
# [text] Click here (Link: https://sayou.ai)
5. Configuration Keys
Customize the cleaning processors via the config dictionary.
mask_pii: (bool) Mask emails, phone numbers, and IP addresses.normalize_whitespace: (bool) Collapse multiple spaces and trim lines.extract_links: (bool) Extract<a>tags or markdown links into metadata.remove_stopwords: (bool) Filter out common stopwords (optional).
6. License
Apache 2.0 License © 2026 Sayouzone
7. Plugin List
| Plugin | Example | Description |
|---|---|---|
Doc Refinery |
▶ | |
HTML Refinery |
▶ | |
Json Refinery |
▶ | |
Record Refinery |
▶ |
1 maintainer
Metadata
1 maintainer
Release files for sayou-refinery 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| sayou_refinery-0.5.0.tar.gz | 42.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sayou_refinery-0.5.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 71.7 kB
Release files / sayou_refinery-0.5.0.tar.gz
| Download URL | sayou_refinery-0.5.0.tar.gz |
|---|---|
| Size | 42.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a83f4177da451df48a2a2f89c188436798d3767f0a320addf6690f033268f715
|
|
BLAKE2b-256 checksum How to use checksums |
bf47ed5acb8a9b7f67d8ae0d5d621f9623fd40e962b29f0b645e405445b65285
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / sayou_refinery-0.5.0-py3-none-any.whl
| Download URL | sayou_refinery-0.5.0-py3-none-any.whl |
|---|---|
| Size | 29.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0764fc944436a0fa87b7e6e525f252f2ab054f7688de5a08a2162c9a59f3941e
|
|
BLAKE2b-256 checksum How to use checksums |
0358c532ef0bf86fcf2f17b1469c35e1fec1c15cdeb411c87b27484e8416b095
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for sayou-refinery 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
