Implements lexical analyzers as iterators yielding tokens.
Project description
tokenizertools
Text file tokenizers supporting multiple start states and lexical tracking.
tokenizertools news
Release history.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tokenizertools-0.1a.tar.gz
(10.1 kB
view hashes)