Add your description here
Project description
bs4-token-ext
Add token counting capabilities to BeautifulSoup tags for LLM applications.
LLM（大規模言語モデル）向けに、BeautifulSoup のタグにトークン数をカウントする機能を追加する。
Usage
from bs4_token_ext import TokenAwareBeautifulSoup
html = "<div><p>Hello, world!</p></div>"
soup = TokenAwareBeautifulSoup(html, 'html.parser')
div = soup.find('div')
p = soup.find('p')
print(p.token_count)
print(p.token_count_with_html)
print(div.token_count)
print(div.token_count_with_html)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bs4_token_ext-0.0.2.tar.gz
(38.7 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file bs4_token_ext-0.0.2.tar.gz.
File metadata
- Download URL: bs4_token_ext-0.0.2.tar.gz
- Upload date:
- Size: 38.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
565e0578230cd4c059f1623161542fbd80aaeb524d5ef180495b4b5be707e0be
|
|
| MD5 |
3872befef5bc809cc9c4b713093d66fb
|
|
| BLAKE2b-256 |
8c33e8b29456b8b082200fdcd4ca4e4e3429d1eed884cda74d9bd887399e1ba9
|
File details
Details for the file bs4_token_ext-0.0.2-py3-none-any.whl.
File metadata
- Download URL: bs4_token_ext-0.0.2-py3-none-any.whl
- Upload date:
- Size: 3.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.12.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c8b3bda2d59fcd847c0c7fbba5be917e5213005eaa0a37868c1106ca73949e12
|
|
| MD5 |
ffcf4c661728982772ce9ad2ab2be342
|
|
| BLAKE2b-256 |
b10170f1e07203620c5f4c93a55a6b7d8c97b15b9c84c15132b63d5f801d21c4
|
