dirsql-plugin-embeddings

A first-party dirsql plugin for semantic search over files.

Installing the plugin loads the sqlite-vec extension (for vec_distance_cosine() and friends) and declares an embed() SQL scalar function that turns TEXT or BLOB values into embedding vectors:

uvx --with dirsql-plugin-embeddings dirsql "
  SELECT path
  FROM (SELECT path, embed(content ->> 'abstract') AS emb
        FROM 'arxiv-firehose/data/**/metadata.json')
  ORDER BY vec_distance_cosine(emb, embed('local private models'))
  LIMIT 10"

embed() is inert until a query calls it: no worker process is spawned and no model is loaded for queries that never use it. On the first call, dirsql spawns the plugin's worker process (dirsql-plugin-embeddings worker), which serves every call of the invocation over stdin/stdout.

Model

Embeddings come from model2vec (static embeddings — numpy + tokenizers, no torch), defaulting to minishlab/potion-retrieval-32M. The model downloads to the standard Hugging Face cache on the first ever run, with progress on stderr when stderr is a TTY.

An optional second argument overrides the model per call — the id must be model2vec-loadable (sentence-transformers/torch models are out of scope):

SELECT embed('some text', 'minishlab/potion-base-8M')

Vector cache

Computed vectors are cached at ~/.cache/dirsql/embeddings/ (or $XDG_CACHE_HOME/dirsql/embeddings/ when XDG_CACHE_HOME is set), keyed by the SHA-256 of the value bytes plus the model identifier — changing either recomputes; switching models never serves stale vectors. There is no eviction: the directory is safe to wipe at any time; the only cost is re-embedding. The cache never lives inside a queried tree — the worker receives values, not paths, and writes nothing anywhere else.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.1.18)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thekevinscott from gravatar.com thekevinscott

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dirsql_plugin_embeddings-0.1.12.tar.gz (10.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dirsql_plugin_embeddings-0.1.12-py3-none-any.whl (9.8 kB view details)

Uploaded Python 3

File details

Details for the file dirsql_plugin_embeddings-0.1.12.tar.gz.

File metadata

File hashes

Hashes for dirsql_plugin_embeddings-0.1.12.tar.gz
Algorithm Hash digest
SHA256 ea83424a6ff614e82b4013ca61f0f4132d0947b19f25fe31ec6380a79d8d558c
MD5 d403a1850d276bc79f26c79548c9914d
BLAKE2b-256 d4f2772327dd9c029349a2f1e1c2ec574d6326f094baa9426880281605fbb76a

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql_plugin_embeddings-0.1.12.tar.gz:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql_plugin_embeddings-0.1.12-py3-none-any.whl.

File metadata

File hashes

Hashes for dirsql_plugin_embeddings-0.1.12-py3-none-any.whl
Algorithm Hash digest
SHA256 fba5a12f3f9b06e607d13600aa9470fb77143f4fbfc153958e611c17909590cb
MD5 a74305d93755f836bf30afec35ee1b3c
BLAKE2b-256 8329f2f499f9d762cc43b97464935f19676dfc581adc761487de5f13d341e60f

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql_plugin_embeddings-0.1.12-py3-none-any.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

This release

0.1.12 This release

2 files

0.1.11

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.7

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page