dirsql (Python SDK)

Ephemeral SQL index over a local directory. dirsql watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.

Documentation

Also available as dirsql on crates.io and dirsql on npm.

Installation

pip install dirsql

Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.

Quick start

DirSQL is async by default: the constructor returns immediately, scanning runs in a background thread, and you await db.ready() before querying. Each table is a (ddl, glob, extract) triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and extract turns a matched file into a list of row dicts. dirsql does not read file contents -- if extract needs the file body it reads path itself; return an empty list to skip a file.

import asyncio
import json
from dirsql import DirSQL, Table

async def main():
    db = DirSQL(
        "./my-blog",
        tables=[
            Table(
                ddl="CREATE TABLE posts (title TEXT, author TEXT)",
                glob="posts/*.json",
                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
            ),
        ],
    )
    await db.ready()

    posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
    print(posts)

asyncio.run(main())

Multiple tables and joins

db = DirSQL(
    "./my-blog",
    tables=[
        Table(
            ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
            glob="posts/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
        Table(
            ddl="CREATE TABLE authors (id TEXT, name TEXT)",
            glob="authors/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
    ],
)
await db.ready()

results = await db.query("""
    SELECT posts.title, authors.name
    FROM posts JOIN authors ON posts.author_id = authors.id
""")

Ignoring files

Pass ignore patterns to skip files during scanning and watching:

db = DirSQL(
    "./my-blog",
    ignore=["**/drafts/**", "**/.git/**"],
    tables=[...],
)

Loading SQLite extensions

Pass extensions to load SQLite extension shared libraries onto the connection at startup (before any CREATE TABLE). Each entry is a dict with a path and an optional entrypoint init-symbol override:

db = DirSQL(
    "./my-blog",
    tables=[...],
    extensions=[
        {"path": "./ext/vec0.dylib", "entrypoint": "sqlite3_vec_init"},
        {"path": "./ext/myext.so"},  # entrypoint derived from the filename
    ],
)
await db.ready()

# The extension's functions are now callable in queries:
rows = await db.query("SELECT vec_version() AS v")

dirsql enables extension loading only while loading the configured libraries, then disables it again, so the SQL load_extension() function is never exposed to your queries. Programmatic entries load first, followed by any [[dirsql.extension]] entries declared in a config file. See the config reference.

Watching for changes

db.watch() returns an async iterator of row-level change events as files change on disk:

async for event in db.watch():
    print(f"{event.action} on {event.table}: {event.row}")
    if event.action == "error":
        print(f"  error: {event.error}")

Each event has .action ("insert", "update", "delete", or "error"), .table, .row (the new row, or the deleted row on delete), .old_row (the previous row, on update), .file_path, and .error (on error).

CLI

pip install dirsql also installs a dirsql console script. dirsql "<sql>" (or uvx dirsql "<sql>") runs one query and prints the rows as JSON — the default. dirsql server starts an HTTP server exposing the SDK over HTTP: POST /query for SQL and GET /events for a Server-Sent Events change stream. See the CLI reference.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.35)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thekevinscott from gravatar.com thekevinscott

Credits

Author: Kevin Scott

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Tags

sql filesystem directory sqlite index

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dirsql-0.4.1.tar.gz (363.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dirsql-0.4.1-cp310-abi3-win_amd64.whl (5.4 MB view details)

Uploaded CPython 3.10+Windows x86-64

dirsql-0.4.1-cp310-abi3-manylinux_2_39_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.39+ x86-64

dirsql-0.4.1-cp310-abi3-manylinux_2_39_aarch64.whl (6.3 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.39+ ARM64

dirsql-0.4.1-cp310-abi3-macosx_11_0_arm64.whl (5.5 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

dirsql-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl (5.8 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file dirsql-0.4.1.tar.gz.

File metadata

  • Download URL: dirsql-0.4.1.tar.gz
  • Upload date:
  • Size: 363.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.4.1.tar.gz
Algorithm Hash digest
SHA256 f695842cebc9e5ec68b228d2c079f80505b4964a16250b7aec31fe103e91f468
MD5 4dbf53f0775349478116bb3a21f5be91
BLAKE2b-256 d5a7c8f7295022da9bf1b60c2f6d6284015fd2026aff8e35ae6e4f943f5cf811

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.4.1.tar.gz:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.4.1-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dirsql-0.4.1-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 5.4 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.4.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 774480939f4ea04e80434052c9e235e3e28a98fa9db152ddebec605fdb730ecb
MD5 9cf479ef12948737d3e5789ce0c2875d
BLAKE2b-256 484bc6e0be7b96ac43a34cf6eb2b56d7b3522568cb155bb2505f06d5381e75b9

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.4.1-cp310-abi3-win_amd64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.4.1-cp310-abi3-manylinux_2_39_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.4.1-cp310-abi3-manylinux_2_39_x86_64.whl
Algorithm Hash digest
SHA256 6b4590f4652478c91280a0ec40c89ca2d05aa95038173b6d64e9db996927d7f1
MD5 95dc1b06971a35b0490f2540097394ac
BLAKE2b-256 97b23fca3922fc09436bf5e85bc0802184e4cf9fbd499c9d2bba18a7e9e7488a

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.4.1-cp310-abi3-manylinux_2_39_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.4.1-cp310-abi3-manylinux_2_39_aarch64.whl.

File metadata

File hashes

Hashes for dirsql-0.4.1-cp310-abi3-manylinux_2_39_aarch64.whl
Algorithm Hash digest
SHA256 94d720cda8485d5e251a17ccb5a24cf675243b6b6de15e9544aeef0817a8ed1f
MD5 d0ea7ac5b43e2b80d46c2d4d680c0560
BLAKE2b-256 1e53c17ba1697855d060f32ed0e3d5808dc5ffd7c60ca09eae18cb2d81af97f2

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.4.1-cp310-abi3-manylinux_2_39_aarch64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.4.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dirsql-0.4.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3cd3a0d8c4a3936c1a4faa6c6bea1f545d42c629ff1f9932789f5579fa0cb972
MD5 966d4ce3cd8c276e978f407b0048485a
BLAKE2b-256 f503588d4ec5e144b801c62e6884fb7e455b5b9923914b4377fa667d86baa791

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.4.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 562caf98dc35fa002ae3f1ac9cd14ad74d3a165efd7c38b9cefb0e3dbdbc6f24
MD5 385da47724c06fbabbe6f27250eb38ce
BLAKE2b-256 6db0334bf309bfe5fabbfb32339607a901002ce96647dafaa253dcf64efbff77

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.4.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.4.35

6 files

0.4.34

6 files

0.4.33

6 files

0.4.32

6 files

0.4.31

6 files

0.4.30

6 files

0.4.29

6 files

0.4.28

6 files

0.4.27

6 files

0.4.26

6 files

0.4.25

6 files

0.4.24

6 files

0.4.23

6 files

0.4.22

6 files

0.4.21

6 files

0.4.20

6 files

0.4.19

6 files

0.4.18

6 files

0.4.17

6 files

0.4.16

6 files

0.4.15

6 files

0.4.14

6 files

0.4.13

6 files

0.4.12

6 files

0.4.11

6 files

0.4.10

6 files

0.4.9

6 files

0.4.8

6 files

0.4.7

6 files

0.4.6

6 files

0.4.5

6 files

0.4.4

6 files

0.4.3

6 files

0.4.2

6 files

This release

0.4.1 This release

6 files

0.4.0

6 files

0.3.127

6 files

0.3.126

6 files

0.3.125

6 files

0.3.124

6 files

0.3.123

6 files

0.3.122

6 files

0.3.121

6 files

0.3.120

6 files

0.3.119

6 files

0.3.118

6 files

0.3.101

6 files

0.3.100

6 files

0.3.99

6 files

0.3.98

6 files

0.3.97

6 files

0.3.96

6 files

0.3.95

6 files

0.3.94

6 files

0.3.93

6 files

0.3.92

6 files

0.3.91

6 files

0.3.90

6 files

0.3.89

6 files

0.3.88

6 files

0.3.87

6 files

0.3.86

6 files

0.3.85

6 files

0.3.84

6 files

0.3.83

6 files

0.3.82

6 files

0.3.81

21 files

0.3.80

21 files

0.3.79

21 files

0.3.78

21 files

0.3.77

21 files

0.3.76

21 files

0.3.75

21 files

0.3.74

21 files

0.3.73

21 files

0.3.72

21 files

0.3.71

21 files

0.3.70

21 files

0.3.69

21 files

0.3.67

21 files

0.3.66

21 files

0.3.65

21 files

0.3.64

21 files

0.3.63

21 files

0.3.62

21 files

0.3.61

21 files

0.3.60

21 files

0.3.59

21 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page