omg-data

Automatic HuggingFace dataset pipeline for OMGFormer — download, clean, tokenize.

One call handles everything: finding the right datasets for your language & task, downloading them, cleaning the text, and producing a ready-to-train OMGDataset.

Installation

pip install omg-data
# with language detection support
pip install omg-data[langdetect]

Quick Start

from omg_data import DataPipeline

# Türkçe, 5 GB, GPT-2 tokenizer
pipe = DataPipeline(
    language="tr",
    size_gb=5,
    tokenizer="gpt2",
)
dataset = pipe.build()

trainer.fit(dataset)  # OMGFormer Trainer ile doğrudan uyumlu

Examples

Task-specific pipeline

pipe = DataPipeline(
    language="en",
    task="chat",               # "text" | "chat" | "instruction" | "qa" | "code" | ...
    size_gb=10,
    tokenizer="meta-llama/Llama-2-7b-hf",
    seq_len=2048,
)
dataset = pipe.build()

Custom datasets

pipe = DataPipeline(
    language="en",
    tokenizer="gpt2",
    custom_datasets=["wikitext", "openwebtext"],
)
dataset = pipe.build()

Cleaning disabled

pipe = DataPipeline(language="en", tokenizer="gpt2", clean=False)
dataset = pipe.build()

Fine-grained cleaning control

pipe = DataPipeline(
    language="tr",
    tokenizer="gpt2",
    clean=True,
    clean_options={
        "dedup": True,
        "min_chars": 50,
        "remove_urls": True,
        "remove_html": True,
        "lang_filter": True,
    },
)

Raw text (no tokenizer)

pipe = DataPipeline(language="de", size_gb=2)
hf_dataset = pipe.build()   # returns HuggingFace Dataset

Supported Languages

tr en de fr es ar ru ja zh ko pt it nl pl sv

Supported Tasks

text · lm · chat · conversation · instruction · instruct · qa · summarization · classification · code

Pipeline Steps

1. Search — Finds suitable HuggingFace datasets for your language & task
2. Download — Streams & caches datasets via HuggingFace datasets
3. Clean — Removes HTML, URLs, duplicates, fixes Unicode, filters by length
4. Tokenize — Chunks text into fixed-length token windows
5. Return — OMGDataset (PyTorch Dataset) ready for trainer.fit()

OMGDataset API

from omg_data import OMGDataset

# Compatible with any PyTorch DataLoader
loader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)

# Info
print(dataset.info())
# {'num_sequences': 125000, 'seq_len': 512, 'total_tokens': 64000000, 'approx_size_gb': 0.128}

License

Apache-2.0