Shog AI

Streaming dataset engineering for foundation model training.

Build multilingual AI datasets from Wikipedia, OPUS, Common Crawl and more with a unified schema — designed for large-scale LLM pretraining.

Why Shog AI?

Preparing multilingual, large-scale datasets for foundation model pretraining usually means gluing together one-off scripts for every source — Wikipedia dumps, OPUS corpora, Common Crawl shards — each with a different schema, different cleaning rules, and no consistent output format.

Shog AI standardizes that process. It gives you a single pipeline that ingests data from multiple sources, normalizes it into a unified schema, validates and deduplicates it, and exports it in a form ready for training or publishing.

Features

• Unified schema — every source is normalized into the same record format
• Streaming pipeline — process datasets without loading everything into memory
• Multi-source ingestion — combine Wikipedia, OPUS, and other sources in a single run
• Built-in validation & deduplication
• One-line publishing — push finished datasets straight to the Hugging Face Hub or local storage
• Reproducible outputs — every run produces metadata, statistics, and a manifest alongside the data

Installation

pip install shog-ai

Quick Start

from datasets import load_dataset, load_from_disk

from parsers.wikipedia import WikipediaParser
from pipeline.config import PipelineConfig
from pipeline.pipeline import Pipeline

# yo = load_from_disk("data/wikipedia/yo")
# yo = load_dataset("data/wikipedia/yo")
yo_10 = load_dataset(
    "wikimedia/wikipedia",
    "20231101.yo",
    split="train[:10]"
)

wiki = WikipediaParser(
    language="yo_10"
)

config = PipelineConfig(
    output_dir="output",
    dataset_name="shog_alpha",
)

pipeline = Pipeline(config)

stats = pipeline.process(
    (wiki, yo_10),
)

pipeline.summary()

Processing multiple sources at once

stats = pipeline.process(
    (wiki, wiki_dataset),
    (opus, opus_dataset),
    (books, books_dataset),
)

pipeline.summary(stats)

Architecture

Shog AI is built around three concepts:

• Parsers — source-specific adapters (WikipediaParser, OpusParser, ...) that read raw data and emit standardized records
• Pipeline — orchestrates one or more parsers, applies validation and deduplication, and writes output
• Publishers — take a finished dataset and ship it somewhere (Hugging Face Hub, local storage, ...)

Source Dataset → Parser → Pipeline → Output Package → Publisher

Supported Datasets

• ✅ Wikipedia
• ✅ OPUS-100

Planned:

• FLORES-200
• OSCAR
• Common Crawl
• CC100
• WikiMatrix
• The Stack
• FineWeb
• Community datasets

Output Format

A pipeline run produces a self-contained dataset package:

output/

    shog_alpha/

        data.jsonl
        metadata.json
        statistics.json
        manifest.json
        README.md
        LICENSE

Standardized record

Every record, regardless of source, follows the same schema:

{
  "id": "...",
  "source": "...",
  "language": "...",
  "text": "..."
}

Publishing

Hugging Face Hub

Usage 1 — with a token

from shog.publishers import HuggingFacePublisher

publisher = HuggingFacePublisher(
    token="hf_xxxxxxxxx"
)

publisher.publish(
    dataset_path="output/shog_alpha/data.jsonl",
    repo_id="GodsaveKawurem/shog-alpha"
)

Usage 2 — CLI login (no token in code)

huggingface-cli login

publisher = HuggingFacePublisher()

publisher.publish(
    dataset_path="output/shog_alpha/data.jsonl",
    repo_id="GodsaveKawurem/shog-alpha"
)

Local storage

from publishers.local import LocalStorage

LocalStorage.save(
    dataset_path="output/shog_alpha/data.jsonl",
    output_dir="datasets/shog_alpha",
)

Documentation

[Unreleased]

Full documentation lives in docs/:

• Getting Started
• Parsers
• Pipeline
• Exporters
• Publishers
• Storage
• Architecture

Roadmap

• Common Crawl / CC100 support
• FLORES-200 and OSCAR parsers
• Configurable deduplication strategies
• Streaming exporters for very large corpora
• Expanded publisher targets

Contributing

Contributions are welcome. Please open an issue to discuss significant changes before submitting a pull request.

License

Released under the MIT License.

Citation

If you use Shog AI in your research, please cite:

@software{shog_ai,
  title  = {Shog AI: Streaming Dataset Engineering for Foundation Model Training},
  author = {Godsave Kawurem},
  year   = {2026},
  url    = {git@github.com:Perfect-Aimers-Enterprise/shog-ai-data}
}

About Shog AI

Shog AI is an open AI research initiative focused on building multilingual foundation models and research infrastructure, with an emphasis on African languages. This library provides the data engineering pipeline used to construct large-scale training corpora for foundation models.