tokenize-output 0.4.3

Get identifiers, names, paths, URLs and words from the command output.

Get identifiers, names, paths, URLs and words from the command output.
The xontrib-output-search for xonsh shell is using this library.

If you like the idea click ⭐ on the repo and stay tuned by watching releases.

Install

pip install -U tokenize-output

Usage

Words tokenizing

$ echo "Try https://github.com/xxh/xxh" | tokenize-output -p
Try
https://github.com/xxh/xxh

JSON, Python dict and JavaScript object tokenizing

$ echo '{"Try": "xonsh shell"}' | tokenize-output -p
Try
shell
xonsh
xonsh shell

env tokenizing

$  echo 'PATH=/one/two:/three/four' | tokenize-output -p
/one/two
/one/two:/three/four
/three/four
PATH

Development

Tokenizers

Tokenizer is a functions which extract tokens from the text.

Priority	Tokenizer	Text	Tokens
1	dict	{"key": "val as str"}	['key', 'val as str']
2	env	PATH=/bin:/etc	['PATH', '/bin:/etc', '/bin', '/etc']
3	split	Split me \n now!	['Split', 'me', 'now!']
4	strip	{Hello}	['Hello']

You can create your tokenizer and add it to tokenizers_all in tokenize_output.py.

Tokenizing is a recursive process where every tokenizer returns final and new tokens. The final tokens directly go to the result list of tokens. The new tokens go to all tokenizers again to find new tokens. As result if there is a mix of json and env data in the output it will be found and tokenized in appropriate way.

Test and debug

Run tests:

cd ~
git clone https://github.com/tokenizer/tokenize-output
cd tokenize-output
python -m pytest tests/

To debug the tokenizer:

echo "Hello world" | ./tokenize_outupt -p

Related projects

• xontrib-output-search for xonsh shell