gradio-tokenizertextbox 0.0.1

Textbox tokenizer

---

tags: [gradio-custom-component, TextBox, textbox] title: gradio_tokenizertextbox short_description: Textbox tokenizer colorFrom: blue colorTo: yellow sdk: gradio pinned: false app_file: space.py

gradio_tokenizertextbox

Textbox tokenizer

Installation

pip install gradio_tokenizertextbox

Usage

#
# demo/app.py
#
import gradio as gr
from gradio_tokenizertextbox import TokenizerTextBox 
import json



TOKENIZER_OPTIONS = {
    "Xenova/clip-vit-large-patch14": "CLIP ViT-L/14",
    "Xenova/gpt-4": "gpt-4 / gpt-3.5-turbo / text-embedding-ada-002",
    "Xenova/text-davinci-003": "text-davinci-003 / text-davinci-002",
    "Xenova/gpt-3": "gpt-3",
    "Xenova/grok-1-tokenizer": "Grok-1",
    "Xenova/claude-tokenizer": "Claude",
    "Xenova/mistral-tokenizer-v3": "Mistral v3",
    "Xenova/mistral-tokenizer-v1": "Mistral v1",
    "Xenova/gemma-tokenizer": "Gemma",
    "Xenova/llama-3-tokenizer": "Llama 3",
    "Xenova/llama-tokenizer": "LLaMA / Llama 2",
    "Xenova/c4ai-command-r-v01-tokenizer": "Cohere Command-R",
    "Xenova/t5-small": "T5",
    "Xenova/bert-base-cased": "bert-base-cased",
  
}

# 2. Prepare the choices for the gr.Dropdown component
# The format is a list of tuples: [(display_name, internal_value)]
dropdown_choices = [
    (display_name, model_name) 
    for model_name, display_name in TOKENIZER_OPTIONS.items()
]

def process_output(tokenization_data):
    """
    This function receives the full dictionary from the component.
    """
    if not tokenization_data:
        return {"status": "Waiting for input..."}
    return tokenization_data

# --- Gradio Application ---
with gr.Blocks() as demo:
    gr.Markdown("# TokenizerTextBox Component Demo")
    gr.Markdown("# Component idea taken from the original example application on [Xenova Tokenizer Playground](https://github.com/huggingface/transformers.js-examples/tree/main/the-tokenizer-playground) ")
    gr.Markdown("## Select a tokenizer from the dropdown menu to see how it processes your text in real-time.")
    gr.Markdown("## For more models, check out the [Xenova Transformers Models](https://huggingface.co/Xenova/models) page.")
    
    with gr.Row():
        # 3. Create the Dropdown for model selection
        model_selector = gr.Dropdown(
            label="Select a Tokenizer",
            choices=dropdown_choices,
            value="Xenova/clip-vit-large-patch14", # Set a default value
        )
        
        display_mode_radio = gr.Radio(
            ["text", "token_ids", "hidden"],
            label="Display Mode",
            value="text"
        )
    
    # 4. Initialize the component with a default model
    tokenizer_input = TokenizerTextBox(
        label="Type your text here",
        value="Gradio is an awesome tool for building ML demos!",
        model="Xenova/clip-vit-large-patch14", # Must match the dropdown's default value
        display_mode="text",
    )
    
    output_info = gr.JSON(label="Component Output (from preprocess)")

    # --- Event Listeners ---

    # A. When the tokenizer component changes, update the JSON output
    tokenizer_input.change(
        fn=process_output, 
        inputs=tokenizer_input, 
        outputs=output_info
    )

    # B. When the dropdown value changes, update the 'model' prop of our component
    def update_tokenizer_model(selected_model):
        return gr.update(model=selected_model)

    model_selector.change(
        fn=update_tokenizer_model,
        inputs=model_selector,
        outputs=tokenizer_input
    )

    # C. When the radio button value changes, update the 'display_mode' prop
    def update_display_mode(mode):
        return gr.update(display_mode=mode)

    display_mode_radio.change(
        fn=update_display_mode,
        inputs=display_mode_radio,
        outputs=tokenizer_input
    )

if __name__ == '__main__':
    demo.launch()

TokenizerTextBox

Initialization

name	type	default	description
value	typing.Union[str, dict, typing.Callable, NoneType][ str, dict, Callable, None ]	None	The initial value. Can be a string to initialize the text, or a dictionary for full state. If a function is provided, it will be called when the app loads to set the initial value.
model	str	"Xenova/gpt-3"	The name of a Hugging Face tokenizer to use (must be compatible with Transformers.js). Defaults to "Xenova/gpt-2".
display_mode	"text" | "token_ids" | "hidden"	"text"	Controls the content of the token visualization panel. Can be 'text' (default), 'token_ids', or 'hidden'.
lines	int	2	The minimum number of line rows for the textarea.
max_lines	int | None	None	The maximum number of line rows for the textarea.
placeholder	str | None	None	A placeholder hint to display in the textarea when it is empty.
autofocus	bool	False	If True, will focus on the textbox when the page loads.
autoscroll	bool	True	If True, will automatically scroll to the bottom of the textbox when the value changes.
text_align	typing.Optional[typing.Literal["left", "right"]][ "left" | "right", None ]	None	How to align the text in the textbox, can be: "left" or "right".
rtl	bool	False	If True, sets the direction of the text to right-to-left.
show_copy_button	bool	False	If True, a copy button will be shown.
max_length	int | None	None	The maximum number of characters allowed in the textbox.
label	str | None	None	The label for this component, displayed above the component.
info	str | None	None	Additional component description, displayed below the label.
every	float | None	None	If `value` is a callable, this sets a timer to run the function repeatedly.
show_label	bool	True	If False, the label is not displayed.
container	bool	True	If False, the component will not be wrapped in a container.
scale	int | None	None	The relative size of the component compared to others in a `gr.Row` or `gr.Column`.
min_width	int	160	The minimum-width of the component in pixels.
interactive	bool | None	None	If False, the user will not be able to edit the text.
visible	bool	True	If False, the component will be hidden.
elem_id	str | None	None	An optional string that is assigned as the id of this component in the HTML DOM.
elem_classes	list[str] | str | None	None	An optional list of strings that are assigned as the classes of this component in the HTML DOM.

Events

name	description
change	Triggered when the value of the TokenizerTextBox changes either because of user input (e.g. a user types in a textbox) OR because of a function update (e.g. an image receives a value from the output of an event trigger). See .input() for a listener that is only triggered by user input.
input	This listener is triggered when the user changes the value of the TokenizerTextBox.
submit	This listener is triggered when the user presses the Enter key while the TokenizerTextBox is focused.
blur	This listener is triggered when the TokenizerTextBox is unfocused/blurred.
select	Event listener for when the user selects or deselects the TokenizerTextBox. Uses event data gradio.SelectData to carry value referring to the label of the TokenizerTextBox, and selected to refer to state of the TokenizerTextBox. See EventData documentation on how to use this event data

User function

The impact on the users predict function varies depending on whether the component is used as an input or output for an event (or both).

• When used as an Input, the component only impacts the input signature of the user function.
• When used as an output, the component only impacts the return signature of the user function.

The code snippet below is accurate in cases where the component is used as both an input and an output.

• As output: Is passed, a dictionary enriched with 'char_count' and 'token_count'.
• As input: Should return, the value to set for the component, can be a string or a dictionary.

def predict(
    value: dict | None
) -> str | dict | None:
    return value