multi-tokenizer 0.1.0

No project description provided

Tokenization of Multilingual Texts using Language-Specific Tokenizers

Approaches

1. Approach 1: Individual tokenizers for each language
2. Approach 2: Unified tokenization approach across languages using utf-8 encondings

Evaluation

• Evaluation Methodologies
• Data Collection and Analysis
• Comparative Analysis
• Implementation Plan
• Future Expansion

Development Setup

Prerequisites

• Use the Dev Container for easy setup
• Install dev dependencies

  pip install poetry
  poetry install
  

Linting, Formatting and Type Checking

• Add the directory to safe.directory

  git config --global --add safe.directory /workspaces/multi-tokenizer
  

• Run the following command to lint and format the code

  pre-commit run --all-files
  

• To install pre-commit hooks, run the following command (Recommended)

  pre-commit install
  

Running the tests

Run the tests using the following command

pytest -n "auto"