multilang-probe

A Python package for analyzing multilingual text.

Overview

multilang-probe is a toolkit designed to classify character sets, detect languages in text files, and extract specific multilingual passages. It supports character detection for a wide range of writing systems using Unicode script properties (e.g., Latin, Japanese, Cyrillic, Arabic, Devanagari, and more). Additionally, it leverages the FastText model for robust language detection. The Zenodo DOI for this release is 10.5281/zenodo.14520665.

Features

Character Set Classification:

• Detect and calculate proportions of character types (e.g., Latin, Japanese, Cyrillic, Arabic, Devanagari) in text.
• Uses regex with Unicode script properties (\p{Script}) for more accurate classification.
• Special handling for Japanese vs Chinese characters (Han script).

Example: Character Detection

from multilang_probe.character_detection import classify_text_with_proportions

# Sample text with multiple languages/scripts
text = "これは日本語です。Привет мир! Ελληνικά και हिन्दी।"

# Classify the text
proportions = classify_text_with_proportions(text)

# Print the proportions
print("Character script proportions:")
print(proportions)

Expected outcome:

Character script proportions:
{'japanese': 19.51, 'cyrillic': 21.95, 'greek': 26.83, 'devanagari': 14.63, 'other': 17.07}

Explanation:

• If the text contains Hiragana/Katakana, Han characters are considered Japanese Kanji.
• Otherwise, Han characters are considered Chinese.

Language Detection:

• Identify top languages in text using Facebook's FastText pre-trained model.
• Install the package from PyPI:

pip install multilang-probe

• Download the model once (example command):

curl -L -o lid.176.bin https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin

• Place lid.176.bin in your working directory, set MULTILANG_PROBE_MODEL_PATH, or pass a model_path argument (the model file is not bundled with the PyPI package).

Example: Language Detection

from multilang_probe.language_detection import detect_language_fasttext

text = "Ceci est un texte en français."
languages = detect_language_fasttext(text, model_path="path/to/lid.176.bin")
print(languages)
# Output example: "fr: 99.2%, en: 0.8%"

Mathematical Language Detection:

• Identify text with a high density of mathematical symbols.

Example: Mathematical Detection

from multilang_probe.math_detection import detect_mathematical_language

text = "Let f(x) = ∫_0^∞ e^{-x} dx and x^2 + y^2 = r^2."
result = detect_mathematical_language(text, threshold=1.0)
print(result)

Visualizations:

• Generate simple bar charts for script and language proportions.

Example: Script Proportions Plot

from multilang_probe.character_detection import classify_text_with_proportions
from multilang_probe.visualization import plot_script_proportions

text = "これは日本語です。Привет мир!"
proportions = classify_text_with_proportions(text)
plot_script_proportions(proportions, output_path="scripts.png")

Example: Language Proportions Plot

from multilang_probe.corpus_analysis import analyze_corpus_with_fasttext, calculate_language_proportions
from multilang_probe.visualization import plot_language_proportions

results = analyze_corpus_with_fasttext("path/to/corpus", model_path="path/to/lid.176.bin")
proportions = calculate_language_proportions(results)["example.txt"]
plot_language_proportions(proportions, output_path="languages.png")

Command-Line Interface (CLI)

Install the package and run commands via the multilang-probe entry point:

multilang-probe classify-text --text "これは日本語です。Привет мир!"

multilang-probe detect-language --text "Ceci est un texte en français." --model-path path/to/lid.176.bin

multilang-probe detect-math --text "x^2 + y^2 = r^2" --threshold 1.0

multilang-probe visualize-scripts --text "これは日本語です。Привет мир!" --output scripts.png

multilang-probe visualize-languages --folder path/to/corpus --output languages.png --model-path path/to/lid.176.bin

multilang-probe analyze-corpus --folder path/to/corpus --model-path path/to/lid.176.bin

multilang-probe filter-by-language --folder path/to/corpus --languages fr,en --threshold 70 --model-path path/to/lid.176.bin

You can also tune false-positive behavior by aggregating multiple lines into blocks and enforcing a minimum margin between top-1 and top-2 predictions:

multilang-probe analyze-corpus \
  --folder path/to/corpus \
  --block-size 3 \
  --min-length 120 \
  --model-path path/to/lid.176.bin

multilang-probe filter-by-language \
  --folder path/to/corpus \
  --languages fr,en \
  --threshold 85 \
  --min-margin 15 \
  --model-path path/to/lid.176.bin

Corpus Analysis:

• Analyze all .txt files in a folder to detect multilingual passages and language distributions.
• Character-based filtering: Identify and filter text lines containing specific character sets (e.g., Japanese, Cyrillic, Arabic).
• Language-based filtering: Extract passages in a specific language, with customizable confidence thresholds (e.g., 70%).
• Targeted extraction: Extract lines of text meeting both minimum length requirements and language detection accuracy.
• Calculate language proportions: Aggregate detected languages across files and calculate their proportions.

Example: Analyze and Detect Multilingual Passages

from multilang_probe.corpus_analysis import analyze_corpus_with_fasttext

folder_path = "path/to/corpus/"
results = analyze_corpus_with_fasttext(folder_path)
for filename, langs in results.items():
    print(filename, langs)

Example: Filter Passages by Character Types

Example: Extract Passages by Language with Threshold

from multilang_probe.corpus_analysis import filter_passages_by_language

folder_path = "path/to/corpus/"
target_languages = ["fr", "en"]
threshold = 70
filtered = filter_passages_by_language(results, target_languages, folder_path, threshold)
for filename, passages in filtered.items():
    print(filename, passages)

Supported Character Sets

• Japanese (Hiragana, Katakana)
• Han (Kanji; considered Japanese if Hiragana/Katakana present, else Chinese)
• Korean (Hangul)
• Cyrillic (for languages like Russian, Bulgarian, etc.)
• Arabic
• Hebrew
• Greek
• Latin (basic and extended)
• Devanagari (e.g., Hindi, Sanskrit)
• Tamil, Bengali, Thai
• Extendable via Unicode scripts
• "other" category for characters not belonging to known scripts

Dependencies

• Python 3.7+
• FastText
• Matplotlib (for visualizations)
• Regex (for Unicode script classification)

License

This project is licensed under the MIT License. While the MIT License allows unrestricted use, modification, and distribution of this software, I kindly request that proper credit be given when this project is used in academic, research, or published work. For citation purposes, please refer to the following:

CAFIERO Florian, 'multilang-probe', 2024, [https://github.com/floriancafiero/multilang-probe].

Contributing

Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.

Author

Florian Cafiero
GitHub: floriancafiero
Email: florian.cafiero@chartes.psl.eu

Future Features

• Support for other pre-trained language models (e.g., spaCy).
• CLI (Command-Line Interface) for easy usage without writing code.