language-detection-nlp 0.0.0

Tiny examples and utilities for language detection

language-detection-nlp

Small examples and utilities for simple language detection tasks. The repository contains:

• A tiny wrapper using langdetect in src/langdetector/.
• An example sklearn-based detector in examples/language_detection.py (toy dataset).
• Tests in tests/ that exercise the langdetector package.

Quick start

1. Create and activate a virtual environment (optional):

python3 -m venv .venv
source .venv/bin/activate

2. Install dependencies:

python3 -m pip install -r requirements.txt

3. Install the package (editable) so langdetector imports work:

python3 -m pip install -e .

Run tests

python3 -m pytest -q

Run the sklearn example

python3 examples/language_detection.py

Run the langdetect CLI

python3 -m langdetector.cli --text "This is a test"

Notes

• The sklearn example uses a toy dataset for demonstration. For production tasks, use larger labeled corpora and proper preprocessing.
• Consider using pycountry or similar to map language codes to full names if needed.

Sample data - sentences with their language labels

texts = [ "Hello, how are you?", # English "What is your name?", # English "Bonjour, comment ça va?", # French "Quel est ton nom?", # French "Hola, ¿cómo estás?", # Spanish "¿Cuál es tu nombre?", # Spanish "Hallo, wie geht's dir?", # German "Wie heißt du?", # German "Ciao, come stai?", # Italian "Come ti chiami?", # Italian ] labels = [ "English", "English", "French", "French", "Spanish", "Spanish", "German", "German", "Italian", "Italian" ]

Split data

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.3, random_state=42)

Feature extraction

vectorizer = TfidfVectorizer() X_train_vec = vectorizer.fit_transform(X_train) X_test_vec = vectorizer.transform(X_test)

Train classifier

model = MultinomialNB() model.fit(X_train_vec, y_train)

Test accuracy

y_pred = model.predict(X_test_vec) accuracy = accuracy_score(y_test, y_pred) print(f"Language detection accuracy: {accuracy:.2f}")

Optional: Predict language for new sentences

sample_sentences = ["Wie geht es Ihnen?", "Good morning!", "¿Dónde está la estación?"] sample_vec = vectorizer.transform(sample_sentences) predictions = model.predict(sample_vec) for i, sentence in enumerate(sample_sentences): print(f"'{sentence}' predicted as: {predictions[i]}")