Add your description here
Project description
bs4-token-ext
Add token counting capabilities to BeautifulSoup tags for LLM applications.
LLM（大規模言語モデル）向けに、BeautifulSoup のタグにトークン数をカウントする機能を追加する。
Usage
from bs4_token_ext.main import TokenAwareBeautifulSoup
html = "<div><p>Hello, world!</p></div>"
soup = TokenAwareBeautifulSoup(html, 'html.parser')
div = soup.find('div')
p = soup.find('p')
print(p.token_count)
print(p.token_count_with_html)
print(div.token_count)
print(div.token_count_with_html)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bs4_token_ext-0.0.1.tar.gz
(38.7 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file bs4_token_ext-0.0.1.tar.gz.
File metadata
- Download URL: bs4_token_ext-0.0.1.tar.gz
- Upload date:
- Size: 38.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ca9158f9eb27056c30cfcaf3e5e14243a9e40bf167bf4061ba508c74dfba7fd3
|
|
| MD5 |
b53ded9a78099d1bbd7eb432a918dec9
|
|
| BLAKE2b-256 |
faf72c7c21b62a069b2af2990a8e29e2448c162f7b029b52ce32cb951b505697
|
File details
Details for the file bs4_token_ext-0.0.1-py3-none-any.whl.
File metadata
- Download URL: bs4_token_ext-0.0.1-py3-none-any.whl
- Upload date:
- Size: 2.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ce3b19dd15116bb08b787398ff9cca7a1ab7a12279fc018b8a9783b6d57b770a
|
|
| MD5 |
a28a1ebf583ad0a65e76127c9dcf6901
|
|
| BLAKE2b-256 |
ae1bac9e6d66a3bb176b31d831570747bfdd96a47bf9925f0d0c26ebf6d4f0de
|
