chartoken-vp 1.1.0

Character-level tokenizer and morphological feature encoder for NLP pipelines (UniMorph, NFKC, padding, serialization)

chartoken-vp

Character-level tokenizer and morphological feature encoder for NLP pipelines.

Part of the MorphFormer project by Voluntas Progressus.

Installation

pip install chartoken-vp

Requires Python >= 3.14 and PyTorch >= 2.0.

Features

• CharVocab — character-level tokenizer with NFKC normalization, SOS/EOS/PAD special tokens
• FeatureVocab — morphological feature encoder for UniMorph tag sets with padding masks
• Encode text to padded tensors or plain ID lists
• Serialize/deserialize vocabularies via to_dict() / from_dict() for checkpoint compatibility
• Dynamic vocabulary expansion from new texts

Quick Start

from chartoken import CharVocab, FeatureVocab

# Build vocab from texts
vocab = CharVocab.from_texts(["hello", "world"])
ids = vocab.encode("hello", max_len=32)
print(vocab.decode(ids.tolist()))  # "hello"

# Feature vocab for morphological tags
feat_vocab = FeatureVocab.from_tags([["V", "IND", "PRS"], ["N", "SG"]])
feat_ids, feat_mask = feat_vocab.encode(["V", "IND"], max_features=12)

API

Class / Constant	Description
CharVocab	Character vocabulary with encode/decode/from_texts/to_dict
FeatureVocab	UniMorph feature vocabulary with encode/to_dict
PAD, SOS, EOS	Special token strings
FEATURE_PAD	Padding ID for feature sequences
normalize_text	NFKC text normalization
SPECIAL_TOKENS	Set of all special tokens

License

MIT