dirsql (Python SDK)

Ephemeral SQL index over a local directory. dirsql watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.

Documentation

Also available as dirsql on crates.io and dirsql on npm.

Installation

pip install dirsql

Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.

Quick start

DirSQL is async by default: the constructor returns immediately, scanning runs in a background thread, and you await db.ready() before querying. Each table is a (ddl, glob, extract) triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and extract turns a matched file into a list of row dicts. dirsql does not read file contents -- if extract needs the file body it reads path itself; return an empty list to skip a file.

import asyncio
import json
from dirsql import DirSQL, Table

async def main():
    db = DirSQL(
        "./my-blog",
        tables=[
            Table(
                ddl="CREATE TABLE posts (title TEXT, author TEXT)",
                glob="posts/*.json",
                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
            ),
        ],
    )
    await db.ready()

    posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
    print(posts)

asyncio.run(main())

Multiple tables and joins

db = DirSQL(
    "./my-blog",
    tables=[
        Table(
            ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
            glob="posts/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
        Table(
            ddl="CREATE TABLE authors (id TEXT, name TEXT)",
            glob="authors/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
    ],
)
await db.ready()

results = await db.query("""
    SELECT posts.title, authors.name
    FROM posts JOIN authors ON posts.author_id = authors.id
""")

Ignoring files

Pass ignore patterns to skip files during scanning and watching:

db = DirSQL(
    "./my-blog",
    ignore=["**/drafts/**", "**/.git/**"],
    tables=[...],
)

Loading SQLite extensions

Pass extensions to load SQLite extension shared libraries onto the connection at startup (before any CREATE TABLE). Each entry is a dict with a path and an optional entrypoint init-symbol override:

db = DirSQL(
    "./my-blog",
    tables=[...],
    extensions=[
        {"path": "./ext/vec0.dylib", "entrypoint": "sqlite3_vec_init"},
        {"path": "./ext/myext.so"},  # entrypoint derived from the filename
    ],
)
await db.ready()

# The extension's functions are now callable in queries:
rows = await db.query("SELECT vec_version() AS v")

dirsql enables extension loading only while loading the configured libraries, then disables it again, so the SQL load_extension() function is never exposed to your queries. Programmatic entries load first, followed by any [[dirsql.extension]] entries declared in a config file. See the config reference.

Watching for changes

db.watch() returns an async iterator of row-level change events as files change on disk:

async for event in db.watch():
    print(f"{event.action} on {event.table}: {event.row}")
    if event.action == "error":
        print(f"  error: {event.error}")

Each event has .action ("insert", "update", "delete", or "error"), .table, .row (the new row, or the deleted row on delete), .old_row (the previous row, on update), .file_path, and .error (on error).

CLI

pip install dirsql also installs a dirsql console script that runs an HTTP server exposing the SDK over HTTP: POST /query for SQL and GET /events for a Server-Sent Events change stream. Run dirsql (or uvx dirsql) to start it. See the CLI reference.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.35)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thekevinscott from gravatar.com thekevinscott

Credits

Author: Kevin Scott

License expression

MIT
View SPDX License List

Requires

Python >=3.11

Tags

sql filesystem directory sqlite index

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dirsql-0.3.122.tar.gz (351.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dirsql-0.3.122-cp311-abi3-win_amd64.whl (5.4 MB view details)

Uploaded CPython 3.11+Windows x86-64

dirsql-0.3.122-cp311-abi3-manylinux_2_39_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.39+ x86-64

dirsql-0.3.122-cp311-abi3-manylinux_2_39_aarch64.whl (6.3 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.39+ ARM64

dirsql-0.3.122-cp311-abi3-macosx_11_0_arm64.whl (5.5 MB view details)

Uploaded CPython 3.11+macOS 11.0+ ARM64

dirsql-0.3.122-cp311-abi3-macosx_10_12_x86_64.whl (5.8 MB view details)

Uploaded CPython 3.11+macOS 10.12+ x86-64

File details

Details for the file dirsql-0.3.122.tar.gz.

File metadata

  • Download URL: dirsql-0.3.122.tar.gz
  • Upload date:
  • Size: 351.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.3.122.tar.gz
Algorithm Hash digest
SHA256 367fc990574429a7dedc97c0254ef568b99c4fbdd1b436dbc09f82f68f926c0d
MD5 04e1adaf886f2742eec0f7716dfbfb78
BLAKE2b-256 4a92beaa9d27a9f181d70f0bb8d3266be8379d7bb1240a259d0f67877fbe49a9

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.122.tar.gz:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.122-cp311-abi3-win_amd64.whl.

File metadata

  • Download URL: dirsql-0.3.122-cp311-abi3-win_amd64.whl
  • Upload date:
  • Size: 5.4 MB
  • Tags: CPython 3.11+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.3.122-cp311-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 89f9acae242b57e1df16744e177755f5805f4702da16314d2fe987dd26b10daf
MD5 3a593abb841885c9c941f8dfc1da0d91
BLAKE2b-256 6c208956003d1ad0b4c32234120fa87b66344d8ee7d52afac8b5fdfce68d38d4

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.122-cp311-abi3-win_amd64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.122-cp311-abi3-manylinux_2_39_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.122-cp311-abi3-manylinux_2_39_x86_64.whl
Algorithm Hash digest
SHA256 fd286c9e6422c57125b7767183cb2ed142c11abd9e51cb3d0decbe6c4f0423e8
MD5 1613fc543c31089bcab8fd28d3e42fe2
BLAKE2b-256 f0287a1ecf5aef7ada93cba4a427b2cf167d2cb3d10134c293aba50275b9d561

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.122-cp311-abi3-manylinux_2_39_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.122-cp311-abi3-manylinux_2_39_aarch64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.122-cp311-abi3-manylinux_2_39_aarch64.whl
Algorithm Hash digest
SHA256 d561c2ada7158c69408ea605376b6f337bce341fd8981c8f18a5ddd7a42ab246
MD5 2c7bac3346d5f1a6cdbba92570a2f7a1
BLAKE2b-256 67c8182d8f819ffcb032fa1b5a652f0a03da46a8cdb85d99f15a09acedd4037d

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.122-cp311-abi3-manylinux_2_39_aarch64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.122-cp311-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.122-cp311-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 375b11cf738c66aa043a89460ea2bac78b1be204a565f4b97676a5d9173201b5
MD5 716ac33d13cc452c034f884e71bf3dbf
BLAKE2b-256 1ddbf181caeb23ab202cf0afa6a142651175926ecdef7a275e727782152a8b6c

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.122-cp311-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.122-cp311-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.122-cp311-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 6a9be5a93d0d851a7f1b52faf07dabdcfb492b5506e540491ed46d5a11d5d6dd
MD5 123049ce70f608f05fd464388534ffdc
BLAKE2b-256 9a00c3f7b60521757760773d6984cf5115d8bd742a44c69c1a1d6cc61a7a4492

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.122-cp311-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.4.35

6 files

0.4.34

6 files

0.4.33

6 files

0.4.32

6 files

0.4.31

6 files

0.4.30

6 files

0.4.29

6 files

0.4.28

6 files

0.4.27

6 files

0.4.26

6 files

0.4.25

6 files

0.4.24

6 files

0.4.23

6 files

0.4.22

6 files

0.4.21

6 files

0.4.20

6 files

0.4.19

6 files

0.4.18

6 files

0.4.17

6 files

0.4.16

6 files

0.4.15

6 files

0.4.14

6 files

0.4.13

6 files

0.4.12

6 files

0.4.11

6 files

0.4.10

6 files

0.4.9

6 files

0.4.8

6 files

0.4.7

6 files

0.4.6

6 files

0.4.5

6 files

0.4.4

6 files

0.4.3

6 files

0.4.2

6 files

0.4.1

6 files

0.4.0

6 files

0.3.127

6 files

0.3.126

6 files

0.3.125

6 files

0.3.124

6 files

0.3.123

6 files

This release

0.3.122 This release

6 files

0.3.121

6 files

0.3.120

6 files

0.3.119

6 files

0.3.118

6 files

0.3.101

6 files

0.3.100

6 files

0.3.99

6 files

0.3.98

6 files

0.3.97

6 files

0.3.96

6 files

0.3.95

6 files

0.3.94

6 files

0.3.93

6 files

0.3.92

6 files

0.3.91

6 files

0.3.90

6 files

0.3.89

6 files

0.3.88

6 files

0.3.87

6 files

0.3.86

6 files

0.3.85

6 files

0.3.84

6 files

0.3.83

6 files

0.3.82

6 files

0.3.81

21 files

0.3.80

21 files

0.3.79

21 files

0.3.78

21 files

0.3.77

21 files

0.3.76

21 files

0.3.75

21 files

0.3.74

21 files

0.3.73

21 files

0.3.72

21 files

0.3.71

21 files

0.3.70

21 files

0.3.69

21 files

0.3.67

21 files

0.3.66

21 files

0.3.65

21 files

0.3.64

21 files

0.3.63

21 files

0.3.62

21 files

0.3.61

21 files

0.3.60

21 files

0.3.59

21 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page