Skip to main content

A package for splitting sentences by language (concatenating over-split substrings based on their language)

Project description

1. split-lang

PyPI version Downloads Downloads

Open Source Love

wakatime

Splitting sentences by concatenating over-split substrings based on their language

powered by wtpsplit and fast-langdetect and langdetect

Idea

Stage 1: rule-based split using punctuation

  • hello, how are you -> hello | , | how are you

Stage 2: then, over-split text to substrings by wtpsplit

  • 你喜欢看アニメ吗 -> | 喜欢 | | アニメ |

Stage 3: concatenate substrings based on their languages using fast-langdetect and langdetect

  • | 喜欢 | | アニメ | -> 你喜欢看 | アニメ |

2. Motivation

  1. TTS (Text-To-Speech) model often fails on multi-language sentence, separate sentence based on language will bring better result
  2. Existed NLP toolkit (e.g. SpaCy) is helpful for parsing text in one language, however when it comes to multi-language texts like below are hard to deal with:
你喜欢看アニメ吗?
你最近好吗、最近どうですか?요즘 어떻게 지내요?sky is clear and sunny。
Vielen Dank merci beaucoup for your help.

3. Usage

3.1. Installation

You can install the package using pip:

pip install split-lang

3.2. Basic

3.2.1. split_by_lang

from langsplit import split_by_lang

texts = [
    "你喜欢看アニメ吗?",
]

for text in texts:
    substr = split_by_lang(
        text=text,
        threshold=4.9e-5,
        default_lang="en",
    )
    for index, item in enumerate(substr):
        print(f"{index}|{item.lang}:{item.text}")
    print("----------------------")
0|zh:你喜欢看
1|ja:アニメ
2|zh:吗
3|punctuation:?

3.3. Advanced

threshold

the threshold used in wtpsplit, default to 1e-4, the smaller the more substring you will get in wtpsplit stage

[!NOTE] Check GitHub Repo tests/split_acc.py to find best threshold for your use case

3.3.1. usage of lang_map (for better result)

[!IMPORTANT] Add lang code for your usecase if other languages are needed

  • default lang_map looks like below
    • if langdetect or fasttext or any other language detector detect the language that is NOT included in lang_map will be set to 'x'
    • every 'x' would be merge to the near substring
  • default default_lang is 'en'
LANG_MAP = {
    "zh": "zh",
    "zh-cn": "zh",
    "zh-tw": "x",
    "ko": "ko",
    "ja": "ja",
    "de": "de",
    "fr": "fr",
    "en": "en",
}
DEFAULT_LANG = "en"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

split_lang-0.4.0.tar.gz (8.3 kB view hashes)

Uploaded Source

Built Distribution

split_lang-0.4.0-py3-none-any.whl (10.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page