Skip to main content

Prune unused tokens from LLM models

Project description

Remove all non-English tokens from the model

pip install prunetokens

prunetokens /path/to/model_directory

Model Before After Saved Vocab
Qwen3.5-9B-AWQ-4bit 8.5 GB 7.0 GB ~18% 248k → 146k
MiniCPM5-1B 2.1 GB 1.9 GB ~10% 130k → 94k

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

prunetokens-0.1.2.tar.gz (5.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

prunetokens-0.1.2-py3-none-any.whl (5.3 kB view details)

Uploaded Python 3

File details

Details for the file prunetokens-0.1.2.tar.gz.

File metadata

  • Download URL: prunetokens-0.1.2.tar.gz
  • Upload date:
  • Size: 5.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.12.3

File hashes

Hashes for prunetokens-0.1.2.tar.gz
Algorithm Hash digest
SHA256 2788af1a65c374114b8c372194c0fa432a02bfa9fef9f51e6459d03c9b58cbda
MD5 4a3a954dc235f5d5e59948e9dd1c5d0a
BLAKE2b-256 d784d62550008bdf7af6846f859f97257e16ade3d5ec576f98fc478349792c20

See more details on using hashes here.

File details

Details for the file prunetokens-0.1.2-py3-none-any.whl.

File metadata

  • Download URL: prunetokens-0.1.2-py3-none-any.whl
  • Upload date:
  • Size: 5.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.12.3

File hashes

Hashes for prunetokens-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 8c0530bac87c73622bb467b1c748951c139815d9a23624caa693e9a3670ffbbe
MD5 3a7edba4e95a0b555dbaa69c34475aca
BLAKE2b-256 3dacbdd5a10cce29d9328d02980591b981cd7d5a5261af9a0699ceb629c603c0

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page