count-tokens 0.1.0

count number of tokens in the text file using toktoken tokenizer from OpenAI

Count tokens

Simple tools that have one purpose - count tokens in a text file.

Requirements

This package is using tiktoken library for tokenization.

## Installation
For usage from comman line install the package in isolated environement with pipx:

```sh
$ pipx install count_tokens

or install it in your current environment with pip.

Usage

Open terminal and run:

$ count-tokens document.txt

You should see something like this:

File: document.txt
Encoding: cl100k_base
Number of tokens: 67

if you want to see just the tokens count run:

$ count-tokens document.txt --quiet

and the output will be:

67

Related Projects

• tiktoken - tokenization library used by this package

Credits

Thanks to the authors of the tiktoken library for open sourcing their work.

License

MIT © Krystian Safjan.