langchain-jtoken
A LangChain document transformer that compresses
JSON-shaped Document.page_content with jtoken
so retrieved documents (Elasticsearch hits, MongoDB JSON, API responses) occupy
fewer tokens in the LLM context window — losslessly, so any agent can decode
them back with jtoken.decode_document when needed.
Measurable wins (tiktoken, cl100k_base): −11% Elasticsearch hits, −19% MongoDB extended JSON, −13% nested API events on the reproducible benchmark.
Install
pip install langchain-jtoken
Usage
from langchain_jtoken import JSONTokenDocumentTransformer
transformer = JSONTokenDocumentTransformer()
compressed_docs = transformer.transform_documents(docs)
Non-JSON content passes through unchanged (strict=True raises instead).
Test
pytest
MIT — © Hermann Samimi
1 maintainer
Release files for langchain-jtoken 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| langchain_jtoken-0.1.0.tar.gz | 2.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| langchain_jtoken-0.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 5.9 kB
Release files / langchain_jtoken-0.1.0.tar.gz
| Download URL | langchain_jtoken-0.1.0.tar.gz |
|---|---|
| Size | 2.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
167bd0c9ac13f25baf25f12f924779afd1dd0a3870032050062ff72d36f81234
|
|
BLAKE2b-256 checksum How to use checksums |
bebab993a31a7b1099c136bf96ce77e1a7a51009c76f082fa8cce3f994627056
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
Release files / langchain_jtoken-0.1.0-py3-none-any.whl
| Download URL | langchain_jtoken-0.1.0-py3-none-any.whl |
|---|---|
| Size | 3.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
894e0e88706e2084dd72116e6d3c434c28394962bc03e7d7fa6b10d007ba27f7
|
|
BLAKE2b-256 checksum How to use checksums |
3226b58b03662c38f9a43ba495831cbbd4e51b9fa83d60f149d5c5282fa6d4dc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
