gherbal 1.0.1

FastText-based multilingual language identification with HuggingFace integration

Gherbal

FastText-based multilingual language identification with HuggingFace Hub integration.

Supports 200+ languages including fine-grained Arabic dialect detection.

Installation

pip install gherbal

Quick Start

from gherbal import Gherbal

# Load from HuggingFace Hub
model = Gherbal.from_pretrained("omarkamali/gherbal")

# Predict language
model.predict("Hello, how are you?")
# => [('eng_Latn', 0.99)]

model.predict("مرحبا كيف حالك")
# => [('arb_Arab', 0.95)]

Loading a Local Model

model = Gherbal.from_pretrained("./path/to/model")

Training

import pandas as pd
from gherbal import Gherbal

df = pd.DataFrame({"text": [...], "label": [...]})
model = Gherbal.train(df, save_path="./my_model")

Preprocessing

from gherbal import preprocess_text, create_clean_script_function

text = preprocess_text("Hello @user https://example.com 🎉")
# => "hello"

# Script-aware cleaning
clean = create_clean_script_function()
clean("Latn", "Hello World 123")
# => "Hello World"

Pushing to HuggingFace Hub

model.push_to_hub("username/my-gherbal-model")

License

MIT