Skip to main content

A python package to help avoid pitfalls when using unicode for linguistic data.

Project description

Multicode: Miscelleneous functions for handling unicode in transcriptions

Build Status PyPI

General information

This repository provides code and data to normalize datasets, to avoid pitfalls of unicode when creating linguistic data, and to infer general information about digital data when dealing with it in linguistic contexts.

One major concern is the normalization of phonetic transcriptions. There are many lookalikes in unicode which linguists use without knowing that they are not what they initially want to write. Thus the character used to transcribe an alveolar voiceless fricative, for example, has three candidates in Unicode which look extremely similar: ʃ ∫ ꭍ, although only the first one is the character that should be used when writing phonetic transcriptions. Our goal is to provide ways to detect and normalize those cases which result from the confusion of Unicode characters when transcribing languages phonetically.

Other cases include specific writing systems, like CJK systems, where we want to offer services to more quickly handle them. These cases include the possibility to detect the Unicode code points of very rare characters by naming only the parts, but also simple Python functions that allow to determine whether a given character is a Chinese character or not.

CLI

Installing this package will also install the multicode command.

Normalizing data

The main purpose of this package is data normalization, i.e. replacing lookalikes in strings with the canonical characters. This can be done using the recode subcommand, either passing a string as argument:

$ multicode recode "ʃ ∫ ꭍ"
ʃ ʃ ʃ

or using pipes, i.e. having recode read from stdin:

$ echo "ʃ ∫ ꭍ" | multicode recode 
ʃ ʃ ʃ

Structure of the Repository

Our repository contains data and code. Data is represented in a custom CSV dialect, namely tab-separated, with | as optional secondary separator for field content. Unicode characters can be represented either as UTF-8 encoded strings, or using notation like U+0020.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

multicode-0.2.1.tar.gz (148.6 kB view hashes)

Uploaded Source

Built Distribution

multicode-0.2.1-py2.py3-none-any.whl (149.4 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page