visuals-tokenization 0.1.1

Library to visualize different tokenization methords

English | Español | Français | Deutsch | 中文 | Türkçe | 日本語 | 한국어

train_test_sim

A library to create quick simulation of optimal train-test size you can keep

Developed by Marcel Tino (c) 2024

Examples of How To Use the library

You can use this to alter according to your requirements

##syntax
from visual_tokenization import view_tokenize

view_tokenize(text)

text="Strong economic growth in the first quarter of FY23 helped India overcome the UK to become the fifth-largest economy after it recovered from the COVID-19 pandemic shock. Nominal GDP or GDP at Current Prices in the year 2023-24 is estimated at Rs. 295.36 lakh crores (US$ 3.54 trillion), against the First Revised Estimates (FRE) of GDP for the year 2022-23 of Rs. 269.50 lakh crores (US$ 3.23 trillion). The growth in nominal GDP during 2023-24 is estimated at 9.6% as compared to 14.2% in 2022-23. Strong domestic demand for consumption and investment, along with Government’s continued emphasis on capital expenditure are seen as among the key driver of the GDP in the second half of FY24. During the period April-June 2025, India’s exports stood at US$ 109.11 billion, with Engineering Goods (25.35%), Petroleum Products (18.33%) and electronic goods (7.73%) being the top three exported commodity. Rising employment and increasing private consumption, supported by rising consumer sentiment, will support GDP growth in the coming months."

from visual_tokenization import view_tokenize

view_tokenize(text)

• Share visual_tokenizations on these social media platforms if you like it!