Pocket Dimension provides a memory-efficient, dense, random projection of sparse vectors. This random projection is used to take records {“id”: str, “features”: List[bytes], “counts”: List[int]}, convert them into sparse random vectors using scikit-learn’s FeatureHasher, and then project them down to lower dimensional dense vectors.
Pocket Dimension includes three vectorizer classes for this workflow:
TFVectorizer for term-frequency vectors
TFIDFVectorizer for term-frequency inverse-document-frequency vectors
BM25Vectorizer for BM25-style scoring and ranking-friendly vector weights
The vectorizers accept a count-min sketch via the cms_file argument, which may be a path to a saved sketch or an already instantiated sketch object. This keeps older code paths working while allowing in-memory reuse of a shared sketch.
When the very large sparse universe becomes too inhospitable, escape into a cozy pocket dimension.
Quick example
from sketchnu.countmin import CountMin
from pocket_dimension.vectorizer import BM25Vectorizer
records = [
{"id": "one", "features": [b"apple", b"banana"], "counts": [2, 1]},
{"id": "two", "features": [b"apple", b"cherry"], "counts": [1, 3]},
]
cms = CountMin("linear", width=200)
for rec in records:
for feature, count in zip(rec["features"], rec["counts"]):
cms.add(feature) # BM25 uses the number of documents a term appears in, not the total count
cms.n_added_records[1] += 1 # increment the number of 'documents' added to the sketch
embedder = BM25Vectorizer(128, cms_file=cms)
X, ids = embedder(records)
Documentation
Documentation for the API and theoretical foundations of the algorithms can be found at https://mhendrey.github.io/pocket_dimension
Installation
Pocket Dimension may be install using pip:
pip install pocket_dimension
I’m working on a conda-forge version, but this uses pybloomfiltermmap3 which is currently only on PyPi.
1 maintainer
Metadata
1 maintainer
Release files for pocket_dimension 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pocket_dimension-0.3.1.tar.gz | 30.8 kB | Without provenance | Details for pocket_dimension-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pocket_dimension-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for pocket_dimension-0.3.1-py3-none-any.whl |
Total release size: 58.1 kB
Release files / pocket_dimension-0.3.1.tar.gz
| Download URL | pocket_dimension-0.3.1.tar.gz |
|---|---|
| Size | 30.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a0dd04f1a356d4fedc56650aab2468ff4c30f55830fcd1933acdfaa294a339cd
|
|
BLAKE2b-256 checksum How to use checksums |
edfb43d8715cd3ce307fc3f45139fed6bf28fd42d00d089aae160e0313aea6e2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.7 {"installer":{"name":"uv","version":"0.11.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Pop!_OS","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pocket_dimension-0.3.1-py3-none-any.whl
| Download URL | pocket_dimension-0.3.1-py3-none-any.whl |
|---|---|
| Size | 27.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d79a76882dc3885a231fa43b98d4d0139deeb214303584953ad6ef479c7d0203
|
|
BLAKE2b-256 checksum How to use checksums |
82f63890da0e28e6434a0acc5da170c906bf02bfc75401d1b9c870de2580ecac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.7 {"installer":{"name":"uv","version":"0.11.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Pop!_OS","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pocket_dimension 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
