https://pypi.org/project/tokeniser-py/ Recent updates to the Python Package Index for tokeniser-py en https://pypi.org/project/tokeniser-py/0.1.4/ A custom tokeniser with a 131,072-token vocabulary derived from 0.5B (val) and 1B (val+test) tokens in SlimPajama. Uses a novel token generation algorithm and a dynamic programming-based segmentation method for fast, interpretable tokenisation, which can also be used for tokeniation on custom token maps. keshavtibrewal2@gmail.com Thu, 03 Apr 2025 11:14:59 GMT https://pypi.org/project/tokeniser-py/0.1.3/ A custom tokeniser with a 131,072-token vocabulary derived from 0.5B (val) and 1B (val+test) tokens in SlimPajama. Uses a novel token generation algorithm and a dynamic programming-based segmentation method for fast, interpretable tokenisation, which can also be used for tokeniation on custom token maps. keshavtibrewal2@gmail.com Thu, 03 Apr 2025 10:30:41 GMT https://pypi.org/project/tokeniser-py/0.1.2/ A custom tokeniser with a 131,072-token vocabulary derived from 0.5B (val) and 1B (val+test) tokens in SlimPajama. Uses a novel token generation algorithm and a dynamic programming-based segmentation method for fast, interpretable tokenisation, which can also be used for tokeniation on custom token maps. keshavtibrewal2@gmail.com Sat, 22 Mar 2025 18:24:26 GMT https://pypi.org/project/tokeniser-py/0.1.1/ A custom tokeniser with a 131,072-token vocabulary derived from 0.5B (val) and 1B (val+test) tokens in SlimPajama. Uses a novel token generation algorithm and a dynamic programming-based segmentation method for fast, interpretable tokenisation, which can also be used for tokeniation on custom token maps. keshavtibrewal2@gmail.com Sat, 22 Mar 2025 18:12:57 GMT https://pypi.org/project/tokeniser-py/0.1.0/ A custom tokeniser with a 131,072-token vocabulary derived from 0.5B (val) and 1B (val+test) tokens in SlimPajama. Uses a novel token generation algorithm and a dynamic programming-based segmentation method for fast, interpretable tokenisation, which can also be used for tokeniation on custom token maps. keshavtibrewal2@gmail.com Sat, 22 Mar 2025 18:03:33 GMT