Olaverse Documentation

Welcome to the official developer documentation for the Olaverse SDK.

Olaverse is an open-source multilingual AI infrastructure toolkit for building NLP, speech, retrieval, and language systems for underrepresented languages.

30-Second Quick Start

pip install olaverse

from olaverse.nlp import Diacritizer

d = Diacritizer(model="auto")       # detects the language, routes to the right model
d.restore("Ojo lo si oja lana")     # → 'Òjó lọ sí ọjà lana'

# 10 languages via diactag-1.0 (pip install olaverse[deeplearning]):
d = Diacritizer(model="diactag-1.0", lang="yo")
d.restore("se eranko naa si gbo o?")   # → 'ṣé ẹranko náà sì gbọ́ ọ?'

New in v0.4.1 — language identification for 608 languages, African-first: LIDLite608 (fastText, 37 MB, CPU) and LIDNeural608 (mmBERT, the most accurate on short text). Both have a coverage mode (every language equally likely, for corpus building) and a traffic mode (shifted by real-world frequency, for user input). Labels carry the script, e.g. yor_Latn. This release also fixes model downloads: a dropped connection can no longer leave a broken file in the cache (downloads resume and are size-checked), and a failed download is no longer reported as a missing file.

from olaverse import LIDLite608, LIDNeural608

LIDLite608().predict("Ẹ kú àárọ̀, ṣé dáadáa ni?")                      # → 'yor_Latn'
LIDNeural608(mode="traffic").predict_batch(["Good morning", "Habari za asubuhi"])
# → ['eng_Latn', 'swh_Latn']

New in v0.4.0 — Arabic, and the 2.0 diacritizers. diactag-2.0 adds Arabic to the per-character tagger, diacnet-2.0 / diacnet-mini-2.0 are ByT5 text-to-text models with meaning hints and <auto> language detection. Existing diactag-1.0 and diacnet-1.0/1.1 code is unchanged.

# Arabic; case_endings=False leaves word-final vowels off (shadda kept)
Diacritizer(model="diactag-2.0", lang="ar", case_endings=False).restore("ذهب الطالب إلى المدرسة")

# diacnet-2.0: output is aligned onto your text by default, so only marks change
# (aligned=False returns the raw generation, which can also fix typos)
d = Diacritizer(model="diacnet-2.0", lang="vie", device="cuda")    # bf16 on GPU
d.restore("Chi ay chi that su ranh vao nhung buoi toi sau khi da cho con ngu say.",
          hints={"ranh": "free (time)"})              # meaning hint → 'rảnh', not 'rành'
Diacritizer(model="diacnet-2.0").restore("El nino esta en la casa")   # no lang= → <auto>
d.restore_batch(texts)                                # sorted, padded batches

New in v0.3.0 — diactag-1.0, diacritization as per-character tagging. It copies every base character through and only predicts the marks, so the output is guaranteed to be your input with accents added and nothing else — no dropped words, no rewritten clauses. 38 MB on CPU, and best-in-class on 7 of its 10 languages:

d = Diacritizer(model="diactag-1.0")           # no lang= → it detects the language
d.restore("Co ay rat dam dang")                # → 'Cô ấy rất đảm đang'

# Per-character confidence, for routing uncertain output to human review
text, details = d.restore("se eranko naa", lang="yo", return_details=True)
review = [c for c in details if c.confidence < 0.9]

New in v0.2.0 — title and question generation (pip install olaverse[deeplearning]):

from olaverse import MISTTitleGenerator, MISTQuestionGenerator

MISTTitleGenerator().generate("My laptop keeps freezing when I open too many tabs, why?")
# → 'Laptop Freezing Impact'

MISTQuestionGenerator().generate(passage, n=3, language="eng")
# → ['What causes ocean tides?', 'Does the sun affect tides?', ...]

📚 Full API Documentation: https://Olaverse-Labs.github.io/olaverse/ 📦 PyPI: https://pypi.org/project/olaverse/

Key Capabilities

• 🗣️ Natural Language Processing: Diacritization for 10+ languages (Yoruba, Igbo, Hausa, Vietnamese, Polish, Turkish, Portuguese, Spanish, French, Italian via diactag-1.0 or diacnet-1.0/1.1, plus Arabic via diactag-2.0 or diacnet-2.0), Language Detection from 5 to 608 languages (LIDLite5/LIDNeural5, LIDLite25/LIDNeural25, LIDLite608/LIDNeural608, and the Nigerian-only LIDNeural5_1), Byte-Level BPE tokenization (Nigerian languages plus Swahili/Kinyarwanda/merged families), PII masking, and TTS text normalization.
• ⚡ MIST Model Family: Unified interface for the MIST LLM family (8B, 70B, 140B, Thinking). Supports local inference via transformers and hosted inference via Featherless or any OpenAI-compatible endpoint. Correct stop tokens and generation defaults per variant are baked in. Plus two task-specific models: MISTTitleGenerator (short chat titles from a user's first message) and MISTQuestionGenerator (search-style question generation from a passage, across 25 languages).
• 🧠 Domain LLMs: LegalPeace — memory-efficient 4-bit inference for legal contract reasoning (fine-tuned Mistral-7B-v0.3).
• 🔎 Retrieval: Reranker (cross-encoder, RAG/search second stage) and Embedder (cross-lingual Hausa/Yoruba/Igbo sentence embeddings).
• 🖼️ Vision — Prism: PrismUpscaler (2x/4x/arbitrary-resolution super-resolution), PrismDenoiser (noise/blur/compression removal), and PrismSteganography (hide/recover short messages in images).
• 📊 Datasets: load_dataset / list_datasets — direct access to every public olaverse dataset on Hugging Face (reranker training pairs, multilingual QG passages, DiacBench, and more).
• 🎙️ Speech Architecture (Roadmap / Experimental): TTS pipeline architecture connecting normalization, diacritization, acoustic model, and vocoder. The NLP front-end is production-ready; acoustic synthesis is in development.
• 🌍 Global Utilities: Currency formatters, generic constants, and .wav audio I/O tools.

Quick Install

# Core (NLP, tokenizer, lightweight LID)
pip install olaverse

# Neural models (LIDNeural5/25/608/5_1, diactag-1.0/2.0, diacnet-1.0/1.1/2.0, MIST local inference)
pip install olaverse[deeplearning]

# int8 CPU backend for diactag
pip install olaverse[onnx]

# Lightweight 25- and 608-language LID (fastText, CPU-only)
pip install olaverse[lid]

# Retrieval (Reranker, Embedder)
pip install olaverse[retrieval]

# Vision (PrismUpscaler, PrismDenoiser, PrismSteganography)
pip install olaverse[vision]

# Hosted inference (MIST via Featherless, Modal, etc.)
pip install olaverse[hosted]

# Legal reasoning (LegalPeace)
pip install olaverse[legal]

# Datasets (load_dataset — reranker pairs, QG passages, DiacBench, ...)
pip install olaverse[data]

Navigation

• Models: Product pages for every model family — DiacNet, LID, OTK-BPE, Retrieval, MIST, LegalPeace, Prism — with comparison tables.
• Benchmarks: All published numbers in one place.
• Solutions: Worked pipelines — Speech AI, OCR, search, education, support, translation.
• NLP & Tokenization: Tokenizer, Language Detection, Diacritization, Retrieval (Reranker/Embedder), PII masking, TTS normalizer.
• Language Models: MIST model family, MISTTitleGenerator, MISTQuestionGenerator, LegalPeace, LIDNeural5.
• Vision: PrismUpscaler, PrismDenoiser, PrismSteganography.
• Datasets: load_dataset, list_datasets, dataset_info — all public olaverse datasets.
• Speech Architecture: TTSPipeline and base classes (experimental — roadmap).
• Global Utilities: Constants and audio utilities.
• Enterprise: Commercial support — fine-tuning, custom datasets, deployment.
• Roadmap: What's shipped and what's next.