Skip to main content

A python package to help avoid pitfalls when using unicode for linguistic data.

Project description

Multicode: Miscelleneous functions for handling unicode in transcriptions

Build Status codecov PyPI

General information

This repository provides code and data to normalize datasets, to avoid pitfalls of unicode when creating linguistic data, and to infer general information about digital data when dealing with it in linguistic contexts.

One major concern is the normalization of phonetic transcriptions. There are many lookalikes in unicode which linguists use without knowing that they are not what they initially want to write. Thus the character used to transcribe an alveolar voiceless fricative, for example, has three candidates in Unicode which look extremely similar: ʃ ∫ ꭍ, although only the first one is the character that should be used when writing phonetic transcriptions. Our goal is to provide ways to detect and normalize those cases which result from the confusion of Unicode characters when transcribing languages phonetically.

Other cases include specific writing systems, like CJK systems, where we want to offer services to more quickly handle them. These cases include the possibility to detect the Unicode code points of very rare characters by naming only the parts, but also simple Python functions that allow to determine whether a given character is a Chinese character or not.

Structure of the Repository

Our repository contains data and code. Data is represented in a custom CSV dialect, namely tab-separated, with | as optional secondary separator for field content. Unicode characters can be represented either as UTF-8 encoded strings, or using notation like U+0020.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for multicode, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size multicode-0.2.0-py2.py3-none-any.whl (151.4 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size multicode-0.2.0.tar.gz (148.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page