Skip to main content

basalt-vault — the Basalt Protocol v1 in Python

An independent implementation of the Basalt Protocol, held to the same language-neutral fixture corpus as the TypeScript reference in conformance/.

Conformance: L0–L4 (§14.1) — Transport, Content, Index, Security, Gatekeeper. L5 (history) is not implemented.

The load-bearing claim is in §0 and §13: two independent implementations given the same content MUST emit byte-identical index objects. This package reproduces every file in conformance/vaults/minimal/expected/ — generated by the TypeScript implementation — byte for byte, and the reverse holds too: the TypeScript check command confirms that indexes written by this package reproduce exactly.

$ basalt conformance-regen --vault ../conformance/vaults/minimal/vault --out /tmp/regen
$ diff -r /tmp/regen ../conformance/vaults/minimal/expected/   # no output

Install

pip install basalt-vault

This installs the basalt package and the basalt command. boto3 is an optional extra, so the core protocol — indexing, link resolution, ACL evaluation — runs with no cloud dependency installed:

pip install 'basalt-vault[s3]'

Topic search is a second extra, because gensim brings numpy and scipy with it:

pip install 'basalt-vault[topics]'

From a checkout, uv sync (with --extra s3 or --extra topics) sets up the same thing for development.

Dependencies are deliberately few: ruamel.yaml, because §13.4 requires comments be preserved when an implementation edits YAML it did not author, and regex, because §9.6's inline-tag grammar and §10.5's tokenizer are written in Unicode property classes (\p{L}, \p{N}) that the standard re module does not implement. The gatekeeper is http.server.

The self-check, §14.3

§14.3 makes this command required rather than optional, "because without it the answer is unknowable":

basalt check --vault ../conformance/vaults/minimal/vault \
             --principal u:reader --path "Notes/Columns.md"
vault 01JBQ7X8Z4N3VYK9M2T5HRWC6P Minimal Conformance Vault
audience=gated bypass_traverse=false

integrity  0 errors, 1 warning(s)
  warn  UNRESOLVED_LINK Notes/Hexes.md "Does Not Exist"

index      fresh
rebuild    reproduces stored indexes byte for byte

access     u:reader on Notes/Columns.md
  deny  read            by deny:g:staff
  allow list            by allow:everyone
  allow traverse        by allow:everyone
  …

It reports all four things §14.3 asks for: integrity (§10.7), index freshness (§10.6), whether a rebuild reproduces the stored indexes byte for byte (§13), and the ACL evaluation of a named principal against a named path (§8.7) — naming the ACE that decided each right, which is what makes a band-ordering bug visible.

Commands

basalt check       self-check (§14.3)
basalt index       rebuild every index from content alone; --write to persist
basalt mint        mint node ids for unbound objects (§4.3)
basalt add         store a note, image or file as a NEW object (§12.7)
basalt gist        export a folder to a GitHub gist, or import a gist into a vault
basalt serve       run the gatekeeper (§12)
basalt effective   effective rights with the deciding ACE per right (§8.7)
basalt tree        the visible tree for a principal
basalt digest      the content digest and index freshness (§10.6)
basalt find        find notes by topic (optional, not protocol; needs the topics extra)

Every command takes --vault DIR, --git DIR (a folder in a git working tree: each write is one commit, --remote NAME to pull and push), --gist ID (a vault inside a GitHub gist: text only, at most 300 files), or an S3-compatible store:

basalt check --bucket my-vault --endpoint https://….r2.cloudflarestorage.com --flavor r2
basalt check --bucket my-vault --endpoint http://localhost:9000 --flavor minio

The flavors exist because the three stores disagree in ways §1 pushes into the adapter: R2 requires region auto and rejects the integrity headers newer SDKs send by default, MinIO needs path-style addressing, and no two of them compute an etag for a multipart upload the same way — which is exactly why §10.1 records content_hash separately and treats it as authoritative.

Layout

src/basalt/
  order.py        §13.3  the one comparator all ordering goes through
  keys.py         §1.1, §2  key validation, path algebra, collisions
  canonical.py    §13.1  canonical JSON; json.dumps is wrong four ways
  yamlio.py       §13.4  YAML in, YAML out, comments intact
  frontmatter.py  §5     parsing with error recovery, kinds
  links.py        §9     extraction, exclusion zones, the six resolution steps
  acl.py          §7, §8 principals, roles, the four-band evaluation
  security.py     §8.5, §8.6  descriptor chains and the traverse composition
  indexer.py      §10    every index, plus the merkle digest
  vault.py        §3     the facade: load, scan, reindex, read, mint
  store.py        §12.7  creating objects: notes, images, any file
  gatekeeper.py   §12    the HTTP read surface
  cli.py          §14.3  the self-check and friends
  topics.py       (none) topic search; optional and not protocol, see below
  gist.py         (none) `basalt gist export` and `import`
  transport/      §1     the five operations: local, memory, git, gist, S3
tests/
  test_conformance.py  the L0-L2 corpus
  test_acl.py          the L3 corpus
  test_expected.py     byte-exact index reproduction
  test_gatekeeper.py   L4, including the §12.5 cache-leak test
  test_store_conformance.py  the §12.7 corpus, shared with ts/ and rust/
  test_git.py          the git store, against repositories on disk
  test_gist.py         the gist name corpus, export, import and the gist store,
                       against a local stand-in for the gist API
  …

Topic search (optional, not protocol)

basalt find "volcanic rock" --vault ../conformance/vaults/demo/vault
basalt find --like "Geology/Basalt.md" --vault ../conformance/vaults/demo/vault
1.00  Basalt  Geology/Basalt.md
0.88  Columnar Jointing  Geology/Columnar Jointing.md
0.71  Columns  Start Here.md

search.json matches words: a query for "volcanic stone" finds only the notes that contain them. basalt find fits a latent semantic model (gensim: TF-IDF, then a truncated SVD) and ranks notes by how close they are in topic, so a note about hexagonal basalt columns answers a query about volcanic stone without sharing a word with it. From Python it is basalt.topics.TopicSearch, whose find and related take the principal to ask as.

It is not part of the protocol, and these are the places that shows:

  • Python only. The TypeScript and Rust implementations have no equivalent and the gatekeeper has no route for it; §12.3 is the protocol's read surface and this is not on it.
  • Nothing is written to the vault. A fitted model is floating point that moves with library versions, so it is not an index (§0) and cannot be held to byte-identical output (§13). It is fitted on demand and kept in memory.
  • A model never learns from a note its requester cannot read. It is fitted per principal over only the notes that principal may read, because a model's vocabulary and loadings carry the text it was fitted to (§8.8), and cached by which notes those are rather than by who asked (§12.5). The cost is one fit per distinct visibility set; 300 notes fit in about a fifth of a second.
  • It needs a few notes to generalise from: at least three readable ones, and basalt find says so rather than returning nothing. The number of topics defaults to about the square root of the note count, since a model with as many dimensions as notes is a keyword match; --topics overrides it.
  • Scores below 0.3 are dropped, a cutoff chosen on a 300-article corpus where the lower scores were mostly noise. min_score changes it from Python.

Tests

uv run pytest

312 tests, of which 15 need the topics extra and are skipped without it. The 157 that come from conformance/ define nothing themselves — every expectation is loaded from the corpus, because §14.2 is explicit that no implementation defines its own answers for behavior the spec covers. The rest cover the conditions §4–§12 describe in prose that no single fixture exercises end to end: duplicate-id repair, orphaned sidecars, protected folders, the traverse chain, inline ACLs replacing folder entries, §8.8's search leak rule in both directions, and §12.3's 404/403 split. The topic tests are about the same thing for a model: that its vocabulary never holds a private note's words, and that a model warmed by the owner is not served to the reader.

Python-specific notes

Three places where the obvious Python is the wrong Python:

  • json.dumps cannot produce §13.1. sort_keys sorts by code point rather than by the UTF-8 byte sequence §13.3 specifies, ensure_ascii escapes exactly what §13.1 forbids escaping, and U+2028/U+2029 are left literal. The serializer in canonical.py is written out.
  • bool is a subclass of int. An isinstance(value, int) test in a JSON writer renders True as 1. Checked in that order, with a test.
  • UTF-8 is order-preserving, so Python's own str ordering happens to agree with §13.3 — the ordering that differs is UTF-16 code unit order, which a JavaScript implementation has to correct for. This package still sorts on s.encode("utf-8"), because §13.3 specifies bytes and sorting the bytes cannot drift.

See the implementation report for the spec ambiguities found while writing this.

Metadata

Release files for basalt-vault 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for basalt-vault 1.0.0
File Size Uploaded Provenance Actions
basalt_vault-1.0.0.tar.gz 92.2 kB With provenance Details for basalt_vault-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for basalt-vault 1.0.0
File Interpreter ABI Platform Actions
basalt_vault-1.0.0-py3-none-any.whl Python 3 none any Details for basalt_vault-1.0.0-py3-none-any.whl

Total release size: 199.9 kB

Release files / basalt_vault-1.0.0.tar.gz

Download URL basalt_vault-1.0.0.tar.gz
Size 92.2 kB
Tags Source
SHA-256 checksum
How to use checksums
62d529eaf793286b8ef2a39d6bc7acd3e7426f3d2b77d13e282749808b199e88
BLAKE2b-256 checksum
How to use checksums
c99eda23ae46838257599e06996b85a526da74ec337d0f25376649dd883e9966
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / basalt_vault-1.0.0-py3-none-any.whl

Download URL basalt_vault-1.0.0-py3-none-any.whl
Size 107.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4ae63c45f369a6ae2b71c35ad97196ae76ce01c73cb16d10bfeca8060313407a
BLAKE2b-256 checksum
How to use checksums
765573491ef29376234f3ac1ba8507186d97e8197d8efbef2e11a85b416f528f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for basalt-vault 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for basalt_vault-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page