uniscriptipa 0.1.0

Selective IPA transliteration for multilingual text

UniScriptIPA

UniScriptIPA provides script-selective IPA transliteration for multilingual text.

It uses Phonemizer's eSpeak backend to generate IPA, but only replaces tokens that contain characters from a user-specified Unicode script. Digits, whitespace, punctuation, symbols, and text written in other writing systems are otherwise preserved in their original positions.

Public API

UniScriptIPA provides three main functions:

Function	Purpose
tokenize()	Losslessly splits one string into tokens
str_to_ipa()	Converts one string and returns its IPA text and token-level annotations
list_to_ipa()	Converts multiple strings together and returns their IPA texts and token-level annotations

Two more descriptive aliases are also provided:

Alias	Equivalent function
text_to_ipa()	str_to_ipa()
texts_to_ipa()	list_to_ipa()

They can be imported directly from the package:

from uniscriptipa import (
    tokenize,
    str_to_ipa,
    list_to_ipa,
    text_to_ipa,
    texts_to_ipa,
)

text_to_ipa and str_to_ipa are the same function. Likewise, texts_to_ipa and list_to_ipa are the same function.

Installation

Install UniScriptIPA

Install UniScriptIPA and its Python dependencies from PyPI:

pip install uniscriptipa

Install eSpeak NG

UniScriptIPA uses Phonemizer's eSpeak backend. Therefore, eSpeak NG must also be installed on the system.

macOS with Homebrew

brew install espeak-ng

Debian or Ubuntu

sudo apt-get update
sudo apt-get install espeak-ng

Fedora

sudo dnf install espeak-ng

Windows

Install the appropriate eSpeak NG .msi package for the system architecture. Most modern Windows computers should use the x64 installer.

Configure the eSpeak NG library path when necessary

Phonemizer can normally locate the eSpeak NG shared library automatically. No additional configuration is needed when automatic detection succeeds.

If UniScriptIPA raises an error such as:

RuntimeError: espeak not installed on your system

set the PHONEMIZER_ESPEAK_LIBRARY environment variable to the absolute path of the eSpeak NG shared library.

macOS with Homebrew

A portable Homebrew-based configuration is:

export PHONEMIZER_ESPEAK_LIBRARY="$(brew --prefix espeak-ng)/lib/libespeak-ng.dylib"

On an Apple Silicon Mac, the path may also be:

export PHONEMIZER_ESPEAK_LIBRARY=/opt/homebrew/lib/libespeak-ng.dylib

On an Intel Mac, it may be located under:

/usr/local/lib/libespeak-ng.dylib

Linux

First locate the shared library:

ldconfig -p | grep libespeak-ng

Then set the environment variable using the absolute path reported by the system:

export PHONEMIZER_ESPEAK_LIBRARY=/absolute/path/to/libespeak-ng.so

Windows PowerShell

$env:PHONEMIZER_ESPEAK_LIBRARY = "C:\Program Files\eSpeak NG\libespeak-ng.dll"

The exact path may differ depending on where eSpeak NG was installed.

Configure the path from Python

The environment variable may also be set inside a Python program. It must be set before importing UniScriptIPA:

import os

os.environ["PHONEMIZER_ESPEAK_LIBRARY"] = (
    "/opt/homebrew/lib/libespeak-ng.dylib"
)

from uniscriptipa import text_to_ipa

Alternatively, the library may be configured through Phonemizer directly:

from phonemizer.backend.espeak.wrapper import EspeakWrapper

EspeakWrapper.set_library(
    "/absolute/path/to/libespeak-ng"
)

from uniscriptipa import text_to_ipa

Quick Start

Convert one string

from uniscriptipa import text_to_ipa

ipa_text, token_notes = text_to_ipa(
    "en-us",
    "Latin",
    input_str_text=(
        "Hello, world! This is a simple test "
        "and let's try the string iPhone17Pro."
    ),
)

print(ipa_text)
print(token_notes)

The converted text preserves the punctuations and the digits in iPhone17Pro:

həlo͡ʊ, wɜːld! ðɪs ɪz e͡ɪ sɪmpə͡l tɛst ænd lɛts tɹa͡ɪ ðə stɹɪŋ a͡ɪfo͡ʊn17pɹo͡ʊ.

The token-level annotations indicate how each output token was produced:

[
    ("həlo͡ʊ", "ipa"),
    (",", ""),
    (" ", ""),
    ("wɜːld", "ipa"),
    ("!", ""),
    # ...
    ("a͡ɪfo͡ʊn", "ipa"),
    ("17", ""),
    ("pɹo͡ʊ", "ipa"),
    (".", ""),
]

Convert multiple strings

from uniscriptipa import texts_to_ipa

ipa_texts, token_notes = texts_to_ipa(
    "es",
    "Latin",
    input_list_texts=[
        "Buenos días.",
        "Me gusta escuchar música.",
        "¿Dónde está el restaurante?",
        "Tenemos dos perros.",
    ],
)

print(ipa_texts)
print(token_notes)

The first returned value is a list containing one converted string for each input string:

[
    "bwenos dias.",
    "me ɡusta eskut͡ʃaɾ musika.",
    "¿donde esta el resta͡ʊɾante?",
    "tenemos dos peros.",
]

The second returned value contains the corresponding token-level annotations for every string:

[
	[('bwenos', 'ipa'), (' ', ''), ('dias', 'ipa'), ('.', '')], 
	[('me', 'ipa'), (' ', ''), ('ɡusta', 'ipa'), (' ', ''), ('eskut͡ʃaɾ', 'ipa'), (' ', ''), ('musika', 'ipa'), ('.', '')], 
	[('¿', ''), ('donde', 'ipa'), (' ', ''), ('esta', 'ipa'), (' ', ''), ('el', 'ipa'), (' ', ''), ('resta͡ʊɾante', 'ipa'), ('?', '')], 
	[('tenemos', 'ipa'), (' ', ''), ('dos', 'ipa'), (' ', ''), ('peros', 'ipa'), ('.', '')]
]

tokenize()

tokenize(text)

Losslessly divides one input string into tokens.

Tokenization is performed in two stages. First, the text is segmented with ICU's word BreakIterator using the root locale, following Unicode word-boundary rules. Each token produced by ICU is then further subdivided whenever the character class changes between:

1. Unicode letters (L*), combining marks (M*), or characters with the Unicode Quotation_Mark property; and
2. all other Unicode characters.

Consecutive characters belonging to the same class remain in the same token. This additional subdivision separates letters from adjacent digits and other non-letter content. For example, iPhone17Pro is divided into "iPhone", "17", and "Pro". The second stage only subdivides the tokens produced by ICU and never merges separate ICU tokens.

Parameter

Parameter	Type	Description
text	str	The text to tokenize

Return value

list[str]

Returns the input as a list of tokens.

For example:

from uniscriptipa import tokenize

tokens = tokenize('This is iPhone17Pro')

Result:

['This', ' ', 'is', ' ', 'iPhone', '17', 'Pro']

Tokenization does not delete, replace, normalize, or reorder any character. The following condition is always enforced:

"".join(tokenize(text)) == text

If the tokens cannot be joined to reproduce the original input exactly, tokenize() raises a RuntimeError.

str_to_ipa() / text_to_ipa()

str_to_ipa(
    lang_code,
    script,
    *,
    input_str_text,
    norm_arab_revers_punc=True,
    njobs=16,
    with_stress=False,
    tie=True,
)

Converts one input string using the selected eSpeak language and Unicode script.

Parameters

Parameter	Type	Default	Description
lang_code	str	Required	The language code passed to Phonemizer as its language parameter, such as "en-us", "es", "ru", or "ug"
script	str	Required	The target Unicode script, such as "Latin", "Cyrillic", "Arabic", or "Greek"
input_str_text	str	Required	The single input string to convert
norm_arab_revers_punc	bool	True	Whether eligible Arabic or reversed punctuation characters should be normalized through Uroman
njobs	int	16	The number of parallel jobs passed to Phonemizer
with_stress	bool	False	Whether Phonemizer should include stress marks in its output
tie	bool	True	Whether Phonemizer should use tie characters for multi-character phones

input_str_text, norm_arab_revers_punc, njobs, with_stress, and tie are keyword-only and must therefore be written explicitly:

str_to_ipa(
    "en-us",
    "Latin",
    input_str_text="Hello, world!",
)

Return value

tuple[
    str,
    list[tuple[str, str]],
]

The function returns two values:

ipa_text, token_notes = str_to_ipa(...)

1. ipa_text

A string reconstructed from all converted and preserved tokens in their original order.

2. token_notes

A list of two-item tuples:

(output_token, conversion_type)

The conversion type has one of three values:

Value	Meaning
"ipa"	The token was replaced with the IPA output produced by Phonemizer
"uroman"	The punctuation token was normalized by Uroman
""	The token was preserved without IPA or Uroman conversion

For example:

[
    ("həlo͡ʊ", "ipa"),
    (",", ""),
    (" ", ""),
    ("wɜːld", "ipa"),
    ("!", ""),
]

The first item in each tuple is the token appearing in the final output, not necessarily the original input token.

list_to_ipa() / texts_to_ipa()

list_to_ipa(
    lang_code,
    script,
    *,
    input_list_texts,
    norm_arab_revers_punc=True,
    njobs=16,
    with_stress=False,
    tie=True,
)

Converts multiple input strings using one shared Phonemizer call over the unique tokens collected from the complete input list.

This function is intended for batch processing and large text corpora.

Parameters

Parameter	Type	Default	Description
lang_code	str	Required	The language code passed to Phonemizer as its language parameter
script	str	Required	The target Unicode script
input_list_texts	list[str]	Required	A list of input strings to convert
norm_arab_revers_punc	bool	True	Whether eligible Arabic or reversed punctuation characters should be normalized through Uroman
njobs	int	16	The number of parallel jobs passed to Phonemizer
with_stress	bool	False	Whether Phonemizer should include stress marks
tie	bool	True	Whether Phonemizer should use tie characters for multi-character phones

input_list_texts, norm_arab_revers_punc, njobs, with_stress, and tie are keyword-only:

list_to_ipa(
    "es",
    "Latin",
    input_list_texts=[
        "Buenos días.",
        "Tenemos dos perros.",
    ],
)

Return value

tuple[
    list[str],
    list[list[tuple[str, str]]],
]

The function returns two values:

ipa_texts, token_notes = list_to_ipa(...)

1. ipa_texts

A list of converted strings.

The output list preserves the order of input_list_texts, and each input string produces exactly one output string.

[
    "bwenos dias.",
    "tenemos dos peros.",
]

2. token_notes

A nested list containing one token-annotation list for each converted string:

[
    [
        ("bwenos", "ipa"),
        (" ", ""),
        ("dias", "ipa"),
        (".", ""),
    ],
    [
        ("tenemos", "ipa"),
        (" ", ""),
        ("dos", "ipa"),
        (" ", ""),
        ("peros", "ipa"),
        (".", ""),
    ],
]

The outer list corresponds to the input strings. Each inner list records how the output tokens of that string were produced.

How Tokens Are Selected for IPA Conversion

A token (tokenized by tokenizer()) is replaced with Phonemizer's IPA result only when both of the following conditions are satisfied:

1. It contains at least one character that is a Unicode letter, or combining mark, or quotation-mark character.
2. It contains at least one character belonging to the Unicode script specified by script.

When a token satisfies these conditions, the entire token is replaced with the IPA result.

This script-based selection provides an approximate way to restrict IPA transliteration to words written in the expected form of the target language. It is intentionally not a strict language-identification system.

For example:

text_to_ipa(
    "en-us",
    "Latin",
    input_str_text="iPhone17Pro",
)

The input is first divided into:

["iPhone", "17", "Pro"]

The Latin-script tokens are converted, while the number is preserved:

a͡ɪfo͡ʊn17pɹo͡ʊ

The script parameter selects which writing system is eligible for conversion. The lang_code parameter selects which eSpeak language interprets the eligible tokens.

This is script-based selection, not strict language identification. For example, Spanish and English words both use the Latin script and may therefore both be eligible when script="Latin".

Parallel Processing

The njobs parameter is forwarded directly to Phonemizer:

texts_to_ipa(
    "es",
    "Latin",
    input_list_texts=my_large_corpus,
    njobs=16,
)

It controls how many parallel jobs are used during phonemization. Parallel processing can substantially accelerate conversion when processing large corpora containing many unique tokens.

Motivation

Recent studies suggest that converting multilingual orthographies into the International Phonetic Alphabet (IPA) as a shared phonemic representation can benefit natural language processing, including cross-lingual transfer, low-resource named entity recognition, multilingual tokenization, multilingual cognitive modeling, and phoneme-level language modeling (Imperial, 2022; Jung et al., 2024; Sohn et al., 2024; Goriely & Buttery, 2025; Miletić et al., 2026).

Grapheme-to-phoneme conversion is therefore useful not only for speech processing, but also as a preprocessing strategy for multilingual and low-resource NLP.

Why UniScriptIPA?

When converting multilingual corpora into a shared IPA representation, we may want the conversion to behave like selective machine transliteration, rather than complete text-to-speech normalization.

In such a workflow, only text associated with the target language and grapheme should be converted to IPA. Other parts of the source text—such as punctuation, digits, whitespace, symbols, and obvious foreign words written in different writing systems—may carry useful information and should remain in their original positions.

Existing G2P systems make different trade-offs in this respect.

[Epitran] uses language-specific mapping tables together with preprocessing and postprocessing rules. When an input character cannot be matched by its mapping table, Epitran preserves that character in the output rather than discarding it. This produces useful transliteration-like behavior in which mapped orthographic content is converted while unmapped information is retained.

However, Epitran provides fewer built-in language–script configurations than eSpeak NG. The eSpeak NG backend used by Phonemizer supports a considerably broader range of languages, including languages such as Belarusian, Icelandic, and Latin.

Direct use of the Phonemizer eSpeak backend, however, is not always ideal for preservation-oriented corpus transliteration. eSpeak NG is primarily a text-to-speech system, so it attempts to assign pronunciations to the complete input. As a result:

• digits may be expanded into their spoken forms;
• foreign or mixed-language words may receive unexpected pronunciations across different target language settings;
• punctuation preservation may depend on the backend configuration and the configured punctuation inventory.

UniScriptIPA provides a selective, structure-preserving layer on top of Phonemizer's eSpeak backend. It uses eSpeak NG for its broad language coverage, while restricting phonemization to tokens in the target language's form containing characters from a user-specified target language's Unicode script.

For each token:

• if it contains at least one character belonging to the target language's Unicode script, the entire token is passed to the selected eSpeak language;
• otherwise, the token is copied to the output without phonemization.

This makes UniScriptIPA suitable for producing IPA-based multilingual NLP corpora while retaining non-target content and the overall structure of the original text.

Lossless Tokenization

UniScriptIPA applies a two-stage tokenization procedure before selective phonemization.

Stage 1: ICU word-boundary segmentation

The input is first segmented with ICU's word BreakIterator using the root locale. ICU word-boundary analysis is based on Unicode Standard Annex #29, Unicode Text Segmentation.

Under the default UAX #29 word-boundary rules, adjacent letters and numbers are not necessarily separated. For example, strings such as:

iPhone17Pro
Year2027

may initially be treated as single word tokens.

Passing such a token directly to eSpeak NG could cause the complete token—including its digits—to be converted into a pronunciation.

Stage 2: Unicode-category subdivision

Each token produced by ICU is therefore subdivided into two character classes.

Class A

• Unicode General Category Letter (L*)
• Unicode General Category Mark (M*)
• Unicode binary property Quotation_Mark=Yes

Class B

• every other Unicode character

A new boundary is inserted whenever the character class changes within an ICU token. Consecutive characters belonging to the same class remain together.

For example:

iPhone17Pro

is segmented as:

["iPhone", "17", "Pro"]

and:

Year2027

is segmented as:

["Year", "2027"]

The second stage only subdivides tokens produced by ICU; it never merges separate ICU tokens.

Tokenization is fully lossless: no character is deleted, replaced, normalized, or reordered. The following invariant is always enforced:

"".join(tokenize(text)) == text

If reconstruction does not exactly reproduce the original input, tokenize() raises a RuntimeError.

Selective IPA Conversion

After tokenization, UniScriptIPA collects all unique tokens from the input text.

For str_to_ipa(), the input text is provided through the input_str_text parameter. For list_to_ipa(), the list of multiple input texts is provided through the input_list_texts parameter, and unique tokens are collected across the entire input.

These unique tokens are passed to Phonemizer's phonemize() function using the eSpeak backend. The following UniScriptIPA parameters in str_to_ipa() and list_to_ipa() are forwarded to Phonemizer:

UniScriptIPA parameter	Phonemizer parameter	Purpose
lang_code	language	Selects the eSpeak language
with_stress	with_stress	Controls whether stress marks are included
tie	tie	Controls whether multi-character phones are joined with tie characters
njobs	njobs	Controls the number of parallel phonemization jobs

The njobs parameter allows Phonemizer to process tokens in parallel. Increasing this value can considerably accelerate the conversion of large multilingual corpora.

Script-Based Token Selection

Although all unique tokens are passed through Phonemizer, UniScriptIPA only inserts the resulting IPA transliteration into the final output when a token satisfies both of the following conditions:

1. The token contains at least one character that is a Unicode letter, or combining mark, or character with the Unicode Quotation_Mark property.
2. The token contains at least one character belonging to the Unicode script specified by the user through the script parameter.

Examples of valid script names include:

"Latin"
"Cyrillic"
"Arabic"
"Greek"

When both conditions are satisfied, the complete token is replaced with the IPA result returned by Phonemizer.

This script-based selection provides an approximate way to restrict IPA transliteration to words written in the expected form of the target language. It is intentionally not a strict language-identification system.

For example, when using:

lang_code="en-gb"
script="Latin"

a Spanish word such as:

Cuándo

will also be replaced by the IPA result returned by Phonemizer because its characters belong to the Unicode Latin script.

Similarly, when using:

lang_code="ru"
script="Cyrillic"

Kazakh words written in Cyrillic will also be replaced by the IPA result returned by Phonemizer.

The script parameter therefore controls the writing system that is eligible for transliteration, while lang_code controls the eSpeak language used to interpret eligible tokens.

Optional Arabic and Reversed Punctuation Normalization

Tokens that do not satisfy the IPA-conversion conditions are normally preserved unchanged.

An optional normalization step is available through the norm_arab_revers_punc parameter, which is enabled by default:

norm_arab_revers_punc=True

When this option is enabled, a non-phonemized token is passed to Uroman for punctuation normalization only when all of the following conditions are satisfied:

1. The token contains exactly one character.
2. The character belongs to a Unicode punctuation category (P*).
3. The Unicode character name contains either ARABIC or REVERSED.

Such characters are converted by Uroman into their corresponding ASCII-style punctuation forms.

When the option is disabled:

norm_arab_revers_punc=False

Arabic and reversed punctuation characters are preserved exactly as they appear in the input.

Lossless Reconstruction

After selective IPA replacement and optional punctuation normalization, all tokens are joined in their original order.

Every token that does not satisfy an enabled conversion rule is preserved without deletion, reordering, or modification. This retains the structure of the original text, including whitespace, digits, symbols, non-target scripts, and other unconverted content.

Current Scope and Future Work

UniScriptIPA was initially designed primarily with languages from the Indo-European, Turkic, and Austronesian language families in mind. Languages of the Sinosphere and the Mainland Southeast Asia linguistic area may also be supported in principle, but they have not yet been systematically tested and were not the primary focus of the current version. In particular, mixed-script Japanese and Korean text—such as Japanese text combining Kanji and Kana, or Korean text combining Hangul and Hanja—can be difficult to handle effectively because it involves multiple Unicode scripts. We will continue exploring selective IPA transcription for languages of the Sinosphere and Mainland Southeast Asia, either through future UniScriptIPA updates or through separate libraries. Follow us to receive the latest feature updates and announcements about new libraries. ✨

References

• Imperial, J. M. (2022). NU HLT at CMCL 2022 Shared Task: Multilingual and Crosslingual Prediction of Human Reading Behavior in Universal Language Space. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, 108–113. Association for Computational Linguistics.

• Jung, H., Oh, C., Kang, J., Sohn, J., Song, K., Kim, J., & Mortensen, D. R. (2024). Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), 200–211. Association for Computational Linguistics.

• Sohn, J., Jung, H., Cheng, A., Kang, J., Du, Y., & Mortensen, D. R. (2024). Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource Languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 13595–13602. Association for Computational Linguistics.

• Goriely, Z., & Buttery, P. (2025). IPA CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling. In Proceedings of the 29th Conference on Computational Natural Language Learning, 502–521. Association for Computational Linguistics.

• Miletić, M., Kallini, J., & Shutova, E. (2026). Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 40323–40349. Association for Computational Linguistics.

License

UniScriptIPA is licensed under the GNU General Public License version 3 or any later version.

See the LICENSE file for details.