dirsql (Python SDK)

Ephemeral SQL index over a local directory. dirsql watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.

Documentation

Also available as dirsql on crates.io and dirsql on npm.

Installation

pip install dirsql

Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.

Quick start

DirSQL is async by default: the constructor returns immediately, scanning runs in a background thread, and you await db.ready() before querying. Each table is a (ddl, glob, extract) triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and extract turns a matched file into a list of row dicts. dirsql does not read file contents -- if extract needs the file body it reads path itself; return an empty list to skip a file.

import asyncio
import json
from dirsql import DirSQL, Table

async def main():
    db = DirSQL(
        "./my-blog",
        tables=[
            Table(
                ddl="CREATE TABLE posts (title TEXT, author TEXT)",
                glob="posts/*.json",
                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
            ),
        ],
    )
    await db.ready()

    posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
    print(posts)

asyncio.run(main())

Multiple tables and joins

db = DirSQL(
    "./my-blog",
    tables=[
        Table(
            ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
            glob="posts/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
        Table(
            ddl="CREATE TABLE authors (id TEXT, name TEXT)",
            glob="authors/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
    ],
)
await db.ready()

results = await db.query("""
    SELECT posts.title, authors.name
    FROM posts JOIN authors ON posts.author_id = authors.id
""")

Ignoring files

Pass ignore patterns to skip files during scanning and watching:

db = DirSQL(
    "./my-blog",
    ignore=["**/drafts/**", "**/.git/**"],
    tables=[...],
)

Loading SQLite extensions

Pass extensions to load SQLite extension shared libraries onto the connection at startup (before any CREATE TABLE). Each entry is a dict with a path and an optional entrypoint init-symbol override:

db = DirSQL(
    "./my-blog",
    tables=[...],
    extensions=[
        {"path": "./ext/vec0.dylib", "entrypoint": "sqlite3_vec_init"},
        {"path": "./ext/myext.so"},  # entrypoint derived from the filename
    ],
)
await db.ready()

# The extension's functions are now callable in queries:
rows = await db.query("SELECT vec_version() AS v")

dirsql enables extension loading only while loading the configured libraries, then disables it again, so the SQL load_extension() function is never exposed to your queries. Programmatic entries load first, followed by any [[dirsql.extension]] entries declared in a config file. See the config reference.

Watching for changes

db.watch() returns an async iterator of row-level change events as files change on disk:

async for event in db.watch():
    print(f"{event.action} on {event.table}: {event.row}")
    if event.action == "error":
        print(f"  error: {event.error}")

Each event has .action ("insert", "update", "delete", or "error"), .table, .row (the new row, or the deleted row on delete), .old_row (the previous row, on update), .file_path, and .error (on error).

CLI

pip install dirsql also installs a dirsql console script that runs an HTTP server exposing the SDK over HTTP: POST /query for SQL and GET /events for a Server-Sent Events change stream. Run dirsql (or uvx dirsql) to start it. See the CLI reference.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.35)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thekevinscott from gravatar.com thekevinscott

Credits

Author: Kevin Scott

License expression

MIT
View SPDX License List

Requires

Python >=3.11

Tags

sql filesystem directory sqlite index

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dirsql-0.3.118.tar.gz (338.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dirsql-0.3.118-cp311-abi3-win_amd64.whl (5.3 MB view details)

Uploaded CPython 3.11+Windows x86-64

dirsql-0.3.118-cp311-abi3-manylinux_2_39_x86_64.whl (6.4 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.39+ x86-64

dirsql-0.3.118-cp311-abi3-manylinux_2_39_aarch64.whl (6.2 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.39+ ARM64

dirsql-0.3.118-cp311-abi3-macosx_11_0_arm64.whl (5.5 MB view details)

Uploaded CPython 3.11+macOS 11.0+ ARM64

dirsql-0.3.118-cp311-abi3-macosx_10_12_x86_64.whl (5.7 MB view details)

Uploaded CPython 3.11+macOS 10.12+ x86-64

File details

Details for the file dirsql-0.3.118.tar.gz.

File metadata

  • Download URL: dirsql-0.3.118.tar.gz
  • Upload date:
  • Size: 338.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.3.118.tar.gz
Algorithm Hash digest
SHA256 d6f99b4719ba33a7911d8a18f227f37fab51ee40b805e77b63402c9356aeec70
MD5 10816f512e1c61b1e4799fcb499415ba
BLAKE2b-256 b4cfcd647e5048342618f2360db03780266d26a859a3d9acda18d4d48b0d569f

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.118.tar.gz:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.118-cp311-abi3-win_amd64.whl.

File metadata

  • Download URL: dirsql-0.3.118-cp311-abi3-win_amd64.whl
  • Upload date:
  • Size: 5.3 MB
  • Tags: CPython 3.11+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.3.118-cp311-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 b7e26c89caf459aaded114d44039cf527535978d477ff016764d31fcc1a48282
MD5 62d31d1300d7cbc480e63aedc78c5446
BLAKE2b-256 dd5233b8d46c5ffe3e24994c89e536517cc6bc66e694ece6ccb67df6f912b35d

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.118-cp311-abi3-win_amd64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.118-cp311-abi3-manylinux_2_39_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.118-cp311-abi3-manylinux_2_39_x86_64.whl
Algorithm Hash digest
SHA256 b61b10bb67e4437048457a5a7b08cc33492614e74c2d1f8241fd804a44297548
MD5 ba09a86ac27cd7bc42368430ea4e5d50
BLAKE2b-256 649178697b289de99f6d4d22dd204d1b330290edb88cc24f8dd0416bf3ea08e1

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.118-cp311-abi3-manylinux_2_39_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.118-cp311-abi3-manylinux_2_39_aarch64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.118-cp311-abi3-manylinux_2_39_aarch64.whl
Algorithm Hash digest
SHA256 088cc1bdafad49b8475a2d3473a570d4a57f0e31cb1dd4e60d1e7161a1b1fac0
MD5 fd429452ef71ef6160ab46df57d370c7
BLAKE2b-256 fbe868e93dabd1f9fd9ddeb5022aa65661f331a2b7e20765c61c293caf6f068e

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.118-cp311-abi3-manylinux_2_39_aarch64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.118-cp311-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.118-cp311-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c73b573a3d59c5c98b28d7fd84be2b4fa2449dd8a74f20d628ac08ce2b77b830
MD5 552548f17dc1755189a5155a27feb263
BLAKE2b-256 08bf0b1aeb0b0d0d6fea729459ba3e3dfdebe01a30e58c3aeaefdcdff6d51cc4

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.118-cp311-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.118-cp311-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.118-cp311-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 d954f66f13d9faff7b176a09d90f8f09bbd4c54bd8f4fbb322f6b1dcbdeefd8f
MD5 db38ae4f32042d9ca629ec983c9693c2
BLAKE2b-256 e26347de1f4990b22053d5ae19653eae7582aba89bd5bd23e54701037fd8fbc8

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.118-cp311-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.4.35

6 files

0.4.34

6 files

0.4.33

6 files

0.4.32

6 files

0.4.31

6 files

0.4.30

6 files

0.4.29

6 files

0.4.28

6 files

0.4.27

6 files

0.4.26

6 files

0.4.25

6 files

0.4.24

6 files

0.4.23

6 files

0.4.22

6 files

0.4.21

6 files

0.4.20

6 files

0.4.19

6 files

0.4.18

6 files

0.4.17

6 files

0.4.16

6 files

0.4.15

6 files

0.4.14

6 files

0.4.13

6 files

0.4.12

6 files

0.4.11

6 files

0.4.10

6 files

0.4.9

6 files

0.4.8

6 files

0.4.7

6 files

0.4.6

6 files

0.4.5

6 files

0.4.4

6 files

0.4.3

6 files

0.4.2

6 files

0.4.1

6 files

0.4.0

6 files

0.3.127

6 files

0.3.126

6 files

0.3.125

6 files

0.3.124

6 files

0.3.123

6 files

0.3.122

6 files

0.3.121

6 files

0.3.120

6 files

0.3.119

6 files

This release

0.3.118 This release

6 files

0.3.101

6 files

0.3.100

6 files

0.3.99

6 files

0.3.98

6 files

0.3.97

6 files

0.3.96

6 files

0.3.95

6 files

0.3.94

6 files

0.3.93

6 files

0.3.92

6 files

0.3.91

6 files

0.3.90

6 files

0.3.89

6 files

0.3.88

6 files

0.3.87

6 files

0.3.86

6 files

0.3.85

6 files

0.3.84

6 files

0.3.83

6 files

0.3.82

6 files

0.3.81

21 files

0.3.80

21 files

0.3.79

21 files

0.3.78

21 files

0.3.77

21 files

0.3.76

21 files

0.3.75

21 files

0.3.74

21 files

0.3.73

21 files

0.3.72

21 files

0.3.71

21 files

0.3.70

21 files

0.3.69

21 files

0.3.67

21 files

0.3.66

21 files

0.3.65

21 files

0.3.64

21 files

0.3.63

21 files

0.3.62

21 files

0.3.61

21 files

0.3.60

21 files

0.3.59

21 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page