DOM-aware tokenizers for Hugging Face language models
Project description
DOM tokenizers
DOM-aware tokenizers for 🤗 Hugging Face language models.
Installation
With PIP
pip install dom-tokenizers[train]
From sources
git clone https://github.com/gbenson/dom-tokenizers.git
cd dom-tokenizers
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev,train]
Train a tokenizer
train-tokenizer gbenson/interesting-dom-snapshots -n 10000
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dom_tokenizers-0.0.2.tar.gz
(12.0 kB
view hashes)
Built Distribution
Close
Hashes for dom_tokenizers-0.0.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4992d3e63415b2414307beb02a23f74669e3fe6f749de1212931d611abea6e50 |
|
| MD5 | e44119b7ad51532115de303c8d777b89 |
|
| BLAKE2b-256 | 58afc3e45018e3a1296be6ea9efd1e8c5016d882ee056405eeab4bf3f418f8a8 |
