perovskite_tokenizer

A small chemical-aware BPE encoder for perovskite formulas.

Install

pip install .

Use

from perovskite_tokenizer import PerovskiteTokenizer

tokenizer = PerovskiteTokenizer()

encoded = tokenizer.encode_formula("CsPbBr3")
batch = tokenizer.encode_formula(["CsPbBr3", "FAPbI3"])

token_ids = encoded.token_ids
features = encoded.output_tokens
payload = encoded.to_dict()

raw = tokenizer.encode_formula("CsPbBr3", add_special_tokens=False)

encode_formula accepts either one formula or a list of formulas and returns an EncodedFormula or a list of them. encode is an alias with the same behavior. The packaged model adds [CLS] and [END] by default. Unsupported spans become [UNK], empty formulas raise ValueError, and an empty list returns [].

output_tokens keeps non-element tokens as vocabulary IDs. Each element symbol is followed by its atomic number, atomic mass, electronegativity, atomic radius, ionization energy, and average ionic radius. Before tokenization, MA, FA, and DMA expand to CH3NH3, CH(NH2)2, and (CH3)2NH2, respectively. Missing or non-finite numeric properties are represented by 0.0.

To use another compatible model directory:

tokenizer = PerovskiteTokenizer("path/to/model-directory")