Skip to main content

Luau Tokenizer for LLM and Luau Code Analysis.

Project description

Luau Tokenizer for LLM and Code Analysis

In simple terms, a Tokenizer Created specifically for LUAU.

Installation:

pip install luau-tokenizer

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

luau-tokenizer-1.1.1.tar.gz (2.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

luau_tokenizer-1.1.1-py3-none-any.whl (2.8 kB view details)

Uploaded Python 3

File details

Details for the file luau-tokenizer-1.1.1.tar.gz.

File metadata

  • Download URL: luau-tokenizer-1.1.1.tar.gz
  • Upload date:
  • Size: 2.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.0.1 CPython/3.11.0

File hashes

Hashes for luau-tokenizer-1.1.1.tar.gz
Algorithm Hash digest
SHA256 efc931c0e6bdb56c49fb0261b00bef8489efda4e48b095417af660076ab4c50b
MD5 4e3b788b03845078254ca6a446bcde77
BLAKE2b-256 c385ffe2f0c94ad14969d022f6105e2f148ccf9a8dea2a938bcf9d61c0b4cc2c

See more details on using hashes here.

File details

Details for the file luau_tokenizer-1.1.1-py3-none-any.whl.

File metadata

  • Download URL: luau_tokenizer-1.1.1-py3-none-any.whl
  • Upload date:
  • Size: 2.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.0.1 CPython/3.11.0

File hashes

Hashes for luau_tokenizer-1.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 a8a9331610b22cefa7ea62cbaf059102738d45ef61f5217ce8e2aa012ae12b9e
MD5 24ef95f03971d42ffaf03a318e40cdf5
BLAKE2b-256 478998be385ebbdd77a77eab136205979f8fac04b45a00592c47157f46a9c676

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page