milky 0.1.2

Minimal dataset builder for User/Assistant ML training data

🥛 milky

Minimal dataset toolkit for building and cleaning User/Assistant .mlk training data.

Designed for small language model pipelines like Author.

---

📌 Overview

milky is a lightweight utility for:

• appending training pairs
• reading .mlk datasets
• validating dataset format
• sanitizing corrupted samples
• repairing broken entries
• dataset statistics
• dataset quality scoring
• automatic formatting (multiline-safe)

It is intentionally simple and file-based.

---

⚙️ Initialization

When imported, milky prints:

[YYYY-MM-DD] Milky initialized.

---

📄 Dataset Format (.mlk)

Each line follows:

User: <prompt> Assistant: <response><eos>

---

Example

User: What is AI? Assistant: AI is the study of intelligent systems.<eos>
User: Write a poem Assistant: The sea is calm tonight...<eos>

---

🧠 Multiline Handling

milky automatically converts real newlines into safe escaped format:

\n → \\n

So all samples remain single-line safe.

---

🚀 API Reference

---

append()

append(prompt: str, response: str, target_file: str) -> None

Appends a User/Assistant pair to a .mlk dataset.

Behavior:

• validates .mlk extension
• automatically formats input (cleaning + newline escaping)
• ensures single-line storage
• appends <eos> token

---

read()

read(target_file: str) -> list

Reads dataset file.

Returns:

• raw list of dataset lines

Notes:

• validates .mlk extension
• no parsing or decoding applied

---

check()

check(file_path: str) -> list

Validates dataset structure.

Detects:

• empty lines
• missing User:
• missing Assistant:
• missing <eos>

Output:

• colored CLI diagnostics
• returns (line_number, error_type) list

---

sanitize()

sanitize(file_path: str, apply: bool = False)

Removes invalid dataset entries.

Removes:

• empty lines
• malformed samples

Behavior:

• apply=False → preview only
• apply=True → overwrites file

---

repair()

repair(file_path: str, apply: bool = False)

Attempts to fix malformed entries instead of deleting them.

Fixes:

• adds missing <eos>
• normalizes spacing
• drops unfixable lines

Behavior:

• apply=False → preview mode
• apply=True → writes file

---

stats()

stats(file_path: str) -> dict

Computes dataset metrics.

Measures:

• total lines
• valid samples
• empty lines
• malformed lines
• missing <eos>
• average prompt length
• average response length

Notes:

• automatically decodes escaped newlines (\\n → \n)
• used for dataset inspection before training

---

score()

score(file_path: str) -> float

Returns a 0–100 dataset quality score.

What it evaluates:

• ratio of valid samples
• malformed lines penalty
• missing <eos> penalty
• empty line penalty

Interpretation:

• 90–100 → clean dataset (train-ready)
• 70–89 → good but slightly noisy
• 40–69 → needs cleaning
• <40 → unsafe / unstable dataset

---

🧠 Design Philosophy

milky is:

• minimal
• dependency-free
• file-based
• strict format enforcement
• optimized for small LLM pipelines

It does NOT:

• support JSON / Parquet datasets
• perform tokenization
• handle streaming data
• manage multi-file corpora

---

⚠️ Important Notes

• .mlk format is strict
• each sample must be single-line
• <eos> is required for training stability
• sanitize is destructive (use carefully)
• repair is heuristic-based
• formatting is automatic in append()

---

🥛 Why “milky”?

Because it turns raw text into structured training pairs — cleaning noise into model-ready data.

---

📦 Typical Workflow

append → check → stats → score → sanitize → repair → train

---

🚀 Example Workflow

append("Hello", "Hi!", "data.mlk")

errors = check("data.mlk")

info = stats("data.mlk")

quality = score("data.mlk")

cleaned, removed = sanitize("data.mlk", apply=True)

fixed, changes = repair("data.mlk", apply=True)