HTML DOM-aware tokenizers for Hugging Face language models
Project description
DOM tokenizers
HTML DOM-aware tokenizers for Hugging Face language models.
Setup for development
git clone https://github.com/gbenson/dom-tokenizers.git
cd dom-tokenizers
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev,train]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dom_tokenizers-0.0.1.tar.gz
(15.7 kB
view hashes)
Built Distribution
Close
Hashes for dom_tokenizers-0.0.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3afc5cb9d92fdf1b76b341acf4e66bb6e6a24e5cd545160add16510254ebefe2 |
|
| MD5 | 0cab8d69fe637cd9eb2d4e26241ac62e |
|
| BLAKE2b-256 | ef9cf66bebfde5938737a1724a1414fc280eb0057c739c7dfe7dbcd0e5173d5d |
