langsegment-backup 0.3.5.post1

Multilingual text segmentation tool for zh/ja/en/ko and more. Unofficial backup of LangSegment 0.3.5.

LangSegment (Unofficial Backup)

⚠️ This is an unofficial backup of LangSegment 0.3.5. The original repository has been removed or made private.

A multilingual text segmentation tool that automatically identifies and splits text by language. Particularly useful for TTS (Text-to-Speech) processing with mixed-language content.

Attribution & History

• Original Author: sunnyboxs (juntaosun)
• Original Repository: https://github.com/juntaosun/LangSegment (now removed)
• First Backup: https://github.com/chameleon-ai/LangSegment-0.3.5-backup
• This Fork: https://github.com/MiniXC/LangSegment-0.3.5-backup (PyPI release)

This package is published to PyPI to preserve access to this useful tool after the original repository was removed. All credit for the original work goes to the original author.

Installation

pip install langsegment-backup

Supported Languages

Primary support:

• 🇨🇳 Chinese (zh)
• 🇯🇵 Japanese (ja)
• 🇬🇧 English (en)
• 🇰🇷 Korean (ko)

Experimental support:

• 🇫🇷 French (fr)
• 🇻🇳 Vietnamese (vi)
• 🇷🇺 Russian (ru)
• 🇹🇭 Thai (th)

The tool can actually support up to 97 different languages through the underlying py3langid library.

Quick Start

from LangSegment import LangSegment

# Basic usage - segment mixed language text
text = "你好世界！Hello World! こんにちは！안녕하세요!"
results = LangSegment.getTexts(text)

for item in results:
    print(f"[{item['lang']}] {item['text']}")

Output:

[zh] 你好世界！
[en] Hello World! 
[ja] こんにちは！
[ko] 안녕하세요!

Features

Language Filtering

You can specify which languages to detect and in what priority order:

from LangSegment import LangSegment

# Set language filter (priority order: left = highest)
LangSegment.setfilters(["zh", "ja", "en", "ko"])

# Or for Chinese-English only
LangSegment.setfilters(["zh", "en"])

Language Statistics

Get statistics about the languages in your text:

from LangSegment import LangSegment

text = "你好世界！Hello World! こんにちは！"
LangSegment.getTexts(text)

# Get language counts (sorted by character count, descending)
counts = LangSegment.getCounts()
print(counts)  # [('zh', 10), ('en', 12), ('ja', 6)]

# Get the primary language
primary_lang, char_count = counts[0]
print(f"Primary language: {primary_lang}")

Manual Language Tags

You can manually specify language regions using tags:

text = "这是中文<ja>これは日本語です</ja>这又是中文"
results = LangSegment.getTexts(text)

SSML Support (Chinese)

The tool includes SSML-like tags for Chinese number/date processing:

# Number reading
text = "<number>12345</number>"  # → 一二三四五

# Phone number
text = "<telephone>13812345678</telephone>"  # → 幺三八幺二三四五六七八

# Currency
text = "<currency>12345</currency>"  # → 一万二千三百四十五

# Date
text = "<date>2024-08-24</date>"  # → 二零二四年八月二十四日

Configuration Options

from LangSegment import LangSegment

# Set Chinese/Japanese priority threshold (0-1, default: 0.89)
LangSegment.setPriorityThreshold(0.89)

# Enable/disable result merging
LangSegment.setLangMerge(True)

# Keep Chinese pinyin format
LangSegment.setKeepPinyin(False)

# Enable preview features (French, Vietnamese support)
LangSegment.setEnablePreview(True)

API Reference

Main Functions

Function	Description
LangSegment.getTexts(text)	Segment text and return list of {lang, text, score} dicts
LangSegment.classify(text)	Alias for getTexts()
LangSegment.getCounts()	Get language statistics as list of (lang, count) tuples
LangSegment.setfilters(list)	Set language filter/priority list
LangSegment.getfilters()	Get current language filters

Configuration Functions

Function	Description
setPriorityThreshold(float)	Set zh/ja disambiguation threshold (0-1)
getPriorityThreshold()	Get current threshold
setLangMerge(bool)	Enable/disable merging adjacent same-language segments
getLangMerge()	Get merge setting
setKeepPinyin(bool)	Keep Chinese pinyin format in parentheses
getKeepPinyin()	Get pinyin setting
setEnablePreview(bool)	Enable experimental language support
getEnablePreview()	Get preview setting

Dependencies

• numpy >= 1.19.5
• py3langid >= 0.2.2

License

BSD 3-Clause License

Copyright (c) 2024 juntaosun. All rights reserved.

See LICENSE for full license text.

Contributing

Since this is a backup/preservation fork, major feature additions are not planned. However, bug fixes and compatibility updates are welcome. Please open an issue or pull request at https://github.com/MiniXC/LangSegment-0.3.5-backup.

Acknowledgments

• Original author: sunnyboxs (juntaosun) for creating LangSegment
• chameleon-ai for creating the first backup at https://github.com/chameleon-ai/LangSegment-0.3.5-backup
• The py3langid library by Adrien Barbaresi
• The original langid.py by Marco Lui