dirsql (Python SDK)

Ephemeral SQL index over a local directory. dirsql watches a filesystem, ingests structured files into an in-memory SQLite database, and exposes a SQL query interface -- the filesystem is always the source of truth.

Documentation

Also available as dirsql on crates.io and dirsql on npm.

Installation

pip install dirsql

Requires Python >= 3.12. Ships as a native extension (Rust via PyO3); prebuilt binary wheels are provided for common platforms.

Quick start

DirSQL is async by default: the constructor returns immediately, scanning runs in a background thread, and you await db.ready() before querying. Each table is a (ddl, glob, extract) triple: the DDL defines the SQLite schema, the glob selects files (relative to the root), and extract turns a matched file into a list of row dicts. dirsql does not read file contents -- if extract needs the file body it reads path itself; return an empty list to skip a file.

import asyncio
import json
from dirsql import DirSQL, Table

async def main():
    db = DirSQL(
        "./my-blog",
        tables=[
            Table(
                ddl="CREATE TABLE posts (title TEXT, author TEXT)",
                glob="posts/*.json",
                extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
            ),
        ],
    )
    await db.ready()

    posts = await db.query("SELECT * FROM posts WHERE author = 'alice'")
    print(posts)

asyncio.run(main())

Multiple tables and joins

db = DirSQL(
    "./my-blog",
    tables=[
        Table(
            ddl="CREATE TABLE posts (title TEXT, author_id TEXT)",
            glob="posts/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
        Table(
            ddl="CREATE TABLE authors (id TEXT, name TEXT)",
            glob="authors/*.json",
            extract=lambda path: [json.loads(open(path, encoding="utf-8").read())],
        ),
    ],
)
await db.ready()

results = await db.query("""
    SELECT posts.title, authors.name
    FROM posts JOIN authors ON posts.author_id = authors.id
""")

Ignoring files

Pass ignore patterns to skip files during scanning and watching:

db = DirSQL(
    "./my-blog",
    ignore=["**/drafts/**", "**/.git/**"],
    tables=[...],
)

Loading SQLite extensions

Pass extensions to load SQLite extension shared libraries onto the connection at startup (before any CREATE TABLE). Each entry is a dict with a path and an optional entrypoint init-symbol override:

db = DirSQL(
    "./my-blog",
    tables=[...],
    extensions=[
        {"path": "./ext/vec0.dylib", "entrypoint": "sqlite3_vec_init"},
        {"path": "./ext/myext.so"},  # entrypoint derived from the filename
    ],
)
await db.ready()

# The extension's functions are now callable in queries:
rows = await db.query("SELECT vec_version() AS v")

dirsql enables extension loading only while loading the configured libraries, then disables it again, so the SQL load_extension() function is never exposed to your queries. Programmatic entries load first, followed by any [[dirsql.extension]] entries declared in a config file. See the config reference.

Watching for changes

db.watch() returns an async iterator of row-level change events as files change on disk:

async for event in db.watch():
    print(f"{event.action} on {event.table}: {event.row}")
    if event.action == "error":
        print(f"  error: {event.error}")

Each event has .action ("insert", "update", "delete", or "error"), .table, .row (the new row, or the deleted row on delete), .old_row (the previous row, on update), .file_path, and .error (on error).

CLI

pip install dirsql also installs a dirsql console script that runs an HTTP server exposing the SDK over HTTP: POST /query for SQL and GET /events for a Server-Sent Events change stream. Run dirsql (or uvx dirsql) to start it. See the CLI reference.

License

MIT

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.35)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thekevinscott from gravatar.com thekevinscott

Credits

Author: Kevin Scott

License expression

MIT
View SPDX License List

Requires

Python >=3.11

Tags

sql filesystem directory sqlite index

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dirsql-0.3.123.tar.gz (351.5 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dirsql-0.3.123-cp311-abi3-win_amd64.whl (5.4 MB view details)

Uploaded CPython 3.11+Windows x86-64

dirsql-0.3.123-cp311-abi3-manylinux_2_39_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.39+ x86-64

dirsql-0.3.123-cp311-abi3-manylinux_2_39_aarch64.whl (6.3 MB view details)

Uploaded CPython 3.11+manylinux: glibc 2.39+ ARM64

dirsql-0.3.123-cp311-abi3-macosx_11_0_arm64.whl (5.5 MB view details)

Uploaded CPython 3.11+macOS 11.0+ ARM64

dirsql-0.3.123-cp311-abi3-macosx_10_12_x86_64.whl (5.8 MB view details)

Uploaded CPython 3.11+macOS 10.12+ x86-64

File details

Details for the file dirsql-0.3.123.tar.gz.

File metadata

  • Download URL: dirsql-0.3.123.tar.gz
  • Upload date:
  • Size: 351.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.3.123.tar.gz
Algorithm Hash digest
SHA256 433516f8820f48a0c51955270ab67a1c1672298f5d5ea73b4533c7e9fed0b253
MD5 3d90c21fa68906269768dce040abac7d
BLAKE2b-256 cbf997674abbe73e9a9c515903674d3cded66175a59071a5c5825c42b7676458

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.123.tar.gz:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.123-cp311-abi3-win_amd64.whl.

File metadata

  • Download URL: dirsql-0.3.123-cp311-abi3-win_amd64.whl
  • Upload date:
  • Size: 5.4 MB
  • Tags: CPython 3.11+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dirsql-0.3.123-cp311-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 15a824136953b617b0ce6c2fa7e2ce28aebaeae7d83e32132cfb2e09feaf70ba
MD5 a0eef1164152e6301a19e9d482e14981
BLAKE2b-256 fc61df9f96269862519aefa3c499ad37b916028e941ceab0cdd9434b276276de

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.123-cp311-abi3-win_amd64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.123-cp311-abi3-manylinux_2_39_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.123-cp311-abi3-manylinux_2_39_x86_64.whl
Algorithm Hash digest
SHA256 db23dc066bbf3026833d030ded7cce2230bc368f9d8a34a95abed0d24b8ffae0
MD5 cf6003382e062a625164cc6b5d34551e
BLAKE2b-256 b8cf4e71c5c17637029be3f1d5224b305b304abe5b0d6d375b98e8ae9b1c382c

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.123-cp311-abi3-manylinux_2_39_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.123-cp311-abi3-manylinux_2_39_aarch64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.123-cp311-abi3-manylinux_2_39_aarch64.whl
Algorithm Hash digest
SHA256 2cdf6714a19c01d325768c01f458ab678f77191a4d31be4016158a2c648da235
MD5 799cb2a770e1fa82f53450e0c8a402cc
BLAKE2b-256 6c39ec24ad4ba54bd78644b944d4c0e6ecc1a959ebaa8abceb4297fc5b457c75

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.123-cp311-abi3-manylinux_2_39_aarch64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.123-cp311-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.123-cp311-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ebaa72a54e4c85ef188c508bbf329148d5eac2550634f70ab9abe1e00e41ac1d
MD5 e32f156a1c0e318149a6bd0ed9bdedc6
BLAKE2b-256 3134387a483c217badc9444af9fcba3814d34ae2dfaabea66be609c15037f6c1

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.123-cp311-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dirsql-0.3.123-cp311-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for dirsql-0.3.123-cp311-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 53e1a7660800c5d824732a012a39aad2548dd27d347700bcb844414189452680
MD5 f96389135f3060e92c4e8219ba147ebe
BLAKE2b-256 01da00878362bde66b844a2c74bf3f0d9aebd4f4e5ea8a4a413ab28ec14e6b54

See more details on using hashes here.

Provenance

The following attestation bundles were made for dirsql-0.3.123-cp311-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on thekevinscott/dirsql

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.4.35

6 files

0.4.34

6 files

0.4.33

6 files

0.4.32

6 files

0.4.31

6 files

0.4.30

6 files

0.4.29

6 files

0.4.28

6 files

0.4.27

6 files

0.4.26

6 files

0.4.25

6 files

0.4.24

6 files

0.4.23

6 files

0.4.22

6 files

0.4.21

6 files

0.4.20

6 files

0.4.19

6 files

0.4.18

6 files

0.4.17

6 files

0.4.16

6 files

0.4.15

6 files

0.4.14

6 files

0.4.13

6 files

0.4.12

6 files

0.4.11

6 files

0.4.10

6 files

0.4.9

6 files

0.4.8

6 files

0.4.7

6 files

0.4.6

6 files

0.4.5

6 files

0.4.4

6 files

0.4.3

6 files

0.4.2

6 files

0.4.1

6 files

0.4.0

6 files

0.3.127

6 files

0.3.126

6 files

0.3.125

6 files

0.3.124

6 files

This release

0.3.123 This release

6 files

0.3.122

6 files

0.3.121

6 files

0.3.120

6 files

0.3.119

6 files

0.3.118

6 files

0.3.101

6 files

0.3.100

6 files

0.3.99

6 files

0.3.98

6 files

0.3.97

6 files

0.3.96

6 files

0.3.95

6 files

0.3.94

6 files

0.3.93

6 files

0.3.92

6 files

0.3.91

6 files

0.3.90

6 files

0.3.89

6 files

0.3.88

6 files

0.3.87

6 files

0.3.86

6 files

0.3.85

6 files

0.3.84

6 files

0.3.83

6 files

0.3.82

6 files

0.3.81

21 files

0.3.80

21 files

0.3.79

21 files

0.3.78

21 files

0.3.77

21 files

0.3.76

21 files

0.3.75

21 files

0.3.74

21 files

0.3.73

21 files

0.3.72

21 files

0.3.71

21 files

0.3.70

21 files

0.3.69

21 files

0.3.67

21 files

0.3.66

21 files

0.3.65

21 files

0.3.64

21 files

0.3.63

21 files

0.3.62

21 files

0.3.61

21 files

0.3.60

21 files

0.3.59

21 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page