Auralith Data Pipeline

Production-grade multimodal data processing pipeline for training RT-DLM and large-scale AI systems.

Overview

Auralith Data Pipeline ingests raw text, images, audio, video, and code, applies production-quality curation (perplexity filtering, LLM-as-Judge scoring, FAISS deduplication, PII scrubbing, license detection), tokenizes everything through BPE + Vector Quantization, and outputs SafeTensors shards ready for distributed model training.

Pipeline Stages

Installation

# Core text pipeline
pip install auralith-data-pipeline

# With all extras (multimodal, cloud, distributed, dev tools)
pip install "auralith-data-pipeline[all]"

# Pick only what you need
pip install "auralith-data-pipeline[quality]"        # + perplexity filter + FAISS dedup
pip install "auralith-data-pipeline[distributed]"    # + Ray
pip install "auralith-data-pipeline[cloud,pdf]"      # + S3/GCS/Azure + PDF extraction
pip install "auralith-data-pipeline[multimodal]"     # + video/image/audio (PyTorch)
pip install "auralith-data-pipeline[tracking]"       # + MLflow + W&B

Quick Start

CLI

# List available datasets
auralith-pipeline list-datasets

# Process Wikipedia dataset
auralith-pipeline collect \
  --dataset wikipedia \
  --output ./data/shards \
  --max-samples 100000 \
  --preset production

End-to-End Workflow

# 1. Train tokenizers (BPE + VQ codebooks)
auralith-pipeline train-tokenizer all \
  --corpus  data/corpus/ \
  --images  data/images/ \
  --audio   data/audio/ \
  --videos  data/videos/ \
  --output  tokenizers/ \
  --vocab-size 32000 \
  --codebook-size 1024

# 2. Process raw data into SafeTensors shards
auralith-pipeline process \
  --input  data/raw/ \
  --output shards/ \
  --tokenizers tokenizers/ \
  --max-seq-len 4096 \
  --shard-size 10000

# 3. Upload to cloud storage or HuggingFace Hub
auralith-pipeline upload --source shards/ --dest s3://my-bucket/training-data/

Python API

from auralith_pipeline import Pipeline, PipelineConfig
from auralith_pipeline.sources import create_source

config = PipelineConfig.from_preset("production")
pipeline = Pipeline(config)
source = create_source("wikipedia", streaming=True, max_samples=1_000_000)
pipeline.add_source(source)

stats = pipeline.run()
print(stats.summary())

Key Features

Data Processing

• Multi-source ingestion (HuggingFace, Common Crawl, local files, video)
• Weighted round-robin interleaving across multiple sources
• MinHash + FAISS embedding deduplication
• Quality filtering (length, language, perplexity, LLM-as-Judge)
• PII removal (multi-jurisdiction, 15+ countries)
• License compliance scanning for code data
• Document extraction (PDF, DOCX, HTML, Markdown)
• SafeTensors sharding with Zstd compression and SHA-256 checksums
• Streaming checkpointing with seeded reproducibility

Tokenization

• Custom BPE tokenizer with 16 special tokens and byte-level fallback
• Vector quantization for images, audio, and video
• Multimodal token fusion with encode_with_mask()
• Configurable vocab size (32k-128k)

Distributed Processing

• Embedded mode — in-process coordinator + workers (no Redis needed)
• External mode — multi-machine with Redis state store
• Worker failure detection + automatic task requeue
• Linear scaling up to 64+ workers

Observability & Compliance

• MLflow / Weights & Biases experiment tracking
• Per-sample lineage (source to shard provenance)
• Auto-generated data cards (HuggingFace-compatible)
• Full audit logging (JSONL) for accept/reject decisions
• Credential and secret sanitization

SafeTensors Schema (v2)

Every output shard is directly compatible with RT-DLM training.

Special Tokens

Available Datasets

Performance

Configuration

# configs/production.yaml
pipeline:
  name: production-pipeline
  output_dir: ./data/shards
  deduplicate: true
  quality_filter: true
  remove_pii: true
  seed: 42
  checkpoint_every: 10000

advanced_quality:
  enabled: true
  perplexity_filter: true
  max_perplexity: 1500.0

Documentation

For the full documentation, architecture diagrams, distributed processing guide, and contributor guide, visit the GitHub repository.

• Architecture
• Contributing
• Distributed Processing
• Changelog

License

Apache License 2.0 — see LICENSE.

Built by AuralithAI for RT-DLM.