Skip to main content

DOM-aware tokenization for 🤗 Hugging Face language models

Project description

version badge license badge

DOM tokenizers

DOM-aware tokenization for Hugging Face language models.



    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width">
    <title>Hello world</title>
    document.getElementById("demo").innerHTML = "Hello JavaScript!";




With PIP

pip install dom-tokenizers[train]

From sources

git clone
cd dom-tokenizers
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -e .[dev,train]

Train a tokenizer

On the command line

Check everything's working using a small dataset of around 300 examples:

train-tokenizer gbenson/interesting-dom-snapshots

Train a tokenizer with a 10,000-token vocabulary using a dataset of 4,536 examples and upload it to the Hub:

train-tokenizer gbenson/webui-dom-snapshots -n 10000 -N 4536
huggingface-cli login
huggingface-cli upload dom-tokenizer-10k

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dom_tokenizers-0.0.17.tar.gz (85.1 kB view hashes)

Uploaded Source

Built Distribution

dom_tokenizers-0.0.17-py3-none-any.whl (28.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page