multi-tokenizer 0.1.1

No project description provided

Multi-Tokenizer

Tokenization of Multilingual Texts using Language-Specific Tokenizers

Overview

Multi-Tokenizer is a Python package that provides tokenization of multilingual texts using language-specific tokenizers. The package is designed to be used in a variety of applications, including natural language processing, machine learning, and data analysis. Behind the scenes, the package uses lingua library to detect the language of the text segments, the tokenizers library to create language-specific tokenizers, and then tokenizes the text segments using the appropriate tokenizer. Multi-tokenizer introduces additional special tokens to handle the language-specific tokenization, which can be used to reconstruct the original text segments after tokenization and allows the models to differentiate between the languages in the text segments.

Development Setup

Prerequisites

• Use the VSCode Dev Containers for easy setup (Recommended)
• Install dev dependencies

  pip install poetry
  poetry install
  

Linting, Formatting and Type Checking

• Add the directory to safe.directory

  git config --global --add safe.directory /workspaces/multi-tokenizer
  

• Run the following command to lint and format the code

  pre-commit run --all-files
  

• To install pre-commit hooks, run the following command (Recommended)

  pre-commit install
  

Running the tests

Run the tests using the following command

pytest -n "auto"

Approaches

1. Approach 1: Individual tokenizers for each language
2. Approach 2: Unified tokenization approach across languages using utf-8 encondings

Evaluation

• Evaluation Methodologies
• Data Collection and Analysis
• Comparative Analysis
• Implementation Plan
• Future Expansion

Contributors

• Rob Neuhaus - ⛴👨🏻‍✈️
• Chandra Irugalbandara
• Alvanli
• Vishnu Vardhan
• Anthony Susevski