dsviper — Digital Substrate Viper Runtime

dsviper is the Python extension module for Viper, a C++ runtime built around metadata-driven type and value systems. It provides a seamless bidirectional bridge between Python and Viper's strong-typed data model.

Key Features

• Type & Value System — Dynamically construct types and instantiate strong-typed values with automatic Python ↔ C++ bridging.
• Definitions — Define data models programmatically: concepts, structures, enumerations, attachments.
• Two stores, and a way between them — CommitDatabase keeps every mutation as a commit; Database keeps only the current state and writes inside a transaction. Neither is a mode of the other, and DatabaseToCommitDatabaseConverter, CommitDatabaseToDatabaseConverter, DatabaseCopier and CommitDatabaseFlattener move documents and blobs between them, both ways.
• Blob System — BlobArray reads a sealed blob as a flat array, and numpy.asarray() over it is a read-only view on the blob's own bytes: the dtype follows the layout with nothing to declare, and reshape(-1, blob_layout.components()) gives the elements, still without a copy. BlobArrayBuilder fills one the same way — copy(buffer) in a single call, or a writable view for in-place work. BlobPack carries structured binary data, and BlobStream writes a blob past 2 GB — read_blob reads one back in pieces, since blob() refuses them.
• Codecs — Codec resolves the three stream codecs by name and says which to take, and what each keeps. Value also encodes to and from JSON and XML as text, and a structure to and from BSON.
• Services — ServiceRemote calls the functions a service exposes, over TCP or a local socket. It is the client half only: the service is a C++ host that registers its function pools, run outside this package.
• Showing it — Html renders a value, a type or a whole document as syntax-coloured HTML, and Html.style() supplies the stylesheet those classes need. DocumentNode presents a document as a tree of labelled rows that say their own type, which is what a viewer or an editor binds to.
• Typed — ships its type hints (PEP 561): mypy and Pyright check your code against the classes, protocols and returns the binding actually has.

Quick Start

from dsviper import *

# Strong-typed value system
names = Value.create(TypeVector(Type.STRING))
names.extend(["alice", "bob"])
print(names)  # ['alice', 'bob']
print(names.type())  # vector<string>

names.append(42)  # ViperError: expected 'str', got 'int'

Commit Database

A complete example — define a model, create a versioned database, commit and query:

from dsviper import *

# Define a model dynamically
defs = Definitions()
# The namespace uuid is the model's lasting identity, so it is a literal:
# every type takes its runtime_id from it, and that is what lets a later
# run read what this one wrote.
ns = NameSpace(ValueUUId.create("6f9619ff-8b86-d011-b42d-00cf4fc964ff"), "App")
t_User = defs.create_concept(ns, "User")
d_Profile = TypeStructureDescriptor("Profile")
d_Profile.add_field("name", Type.STRING)
t_Profile = defs.create_structure(ns, d_Profile)
a_profile = defs.create_attachment(ns, "profile", t_User, t_Profile)

# Create a versioned database and commit data
db = CommitDatabase.create_in_memory()
db.extend_definitions(defs.const())

key = a_profile.create_key()
doc = Value.create(t_Profile)
doc.name = "Alice"

# initial_state starts the line; carry on from the commit_id each commit returns
state = CommitStateBuilder.initial_state(db)
mutable = CommitMutableState(state)
mutable.attachment_mutating().set(a_profile, key, doc)
commit_id = db.commit_mutations("Add Alice", mutable)

# The next change continues the same line
state = CommitStateBuilder.state(db, commit_id)
mutable = CommitMutableState(state)
mutable.attachment_mutating().set(a_profile, a_profile.create_key(), doc)
commit_id = db.commit_mutations("Add another", mutable)

# Query
got = CommitStateBuilder.state(db, commit_id).attachment_getting().get(a_profile, key)
# Optional({name='Alice'})

# Read a field. The optional holds a_profile.document_type(), which a type-checker cannot
# follow — the attachment is a runtime value — so the cast is what tells it. got.unwrap().name
# runs too, and mypy refuses it.
ValueStructure.cast(got.unwrap(encoded=False)).at("name")
# 'Alice'

Going back to initial_state for the second change would branch from the root instead, and the database would carry two heads — the first change is still there, on the other one, and nothing is raised. CommitStore holds that thread for you, and adds undo, redo and notification over it.

On disk

create_in_memory() above keeps the example self-contained. A database that outlives the process is the same object, opened differently:

db = CommitDatabase.create("project.cdb")     # refuses to overwrite an existing file
db.extend_definitions(defs.const())           # the schema is stored with the data
db.close()

db = CommitDatabase.open("project.cdb")       # raises when the file is not there
db = CommitDatabase.open("project.cdb", readonly=True)

Database takes the same three, and writes inside a transaction, keeping only the current state:

db = Database.create("project.db")
db.extend_definitions(defs.const())
db.begin_transaction()                        # 'Deferred' unless a mode is given
db.set(a_profile, key, doc)                   # the document is copied in
db.commit()
db.get(a_profile, key)                        # a copy: Optional({name='Alice'})
db.close()

The name and the extension are yours — the runtime reads neither, and applications pick their own. A file carries its own definitions, so reopening one needs nothing else: db.definitions() answers what it was written with, and extend_definitions adds to that.

Handed a file you did not write, ask before opening: CommitDatabase.is_compatible(path) and Database.is_compatible(path) answer whether it is an SQLite3 file carrying that store's tables. Both answer False rather than raising — for a missing file, for something that is not a database at all, and for the other store's file.

Undo and redo

CommitStore holds the thread of commits for you — it follows the head, so the chaining above is done on your behalf — and adds undo, redo and notification over it. Changes go through the store — its dispatch_* methods, or its own commit_mutations — and the undo stack tracks every commit the store makes; one made on the CommitDatabase directly is not on it.

from dsviper import *

defs = Definitions()
ns = NameSpace(ValueUUId.create("6f9619ff-8b86-d011-b42d-00cf4fc964ff"), "App")
t_User = defs.create_concept(ns, "User")
d_Profile = TypeStructureDescriptor("Profile")
d_Profile.add_field("name", Type.STRING)
t_Profile = defs.create_structure(ns, d_Profile)
a_profile = defs.create_attachment(ns, "profile", t_User, t_Profile)

db = CommitDatabase.create_in_memory()
db.extend_definitions(defs.const())
key = a_profile.create_key()

store = CommitStore()
store.use(db)                       # both attaches the database and moves onto its head

store.dispatch_set("Set Alice", a_profile, key, Value.create(t_Profile, {"name": "Alice"}))
store.dispatch_set("Set Bob", a_profile, key, Value.create(t_Profile, {"name": "Bob"}))

store.undo()          # back to Alice
store.undo()          # back to nothing
store.redo()          # Alice again
store.can_undo(), store.can_redo()

state = store.state()   # None only before use(db), or after store.close()
assert state is not None
state.attachment_getting().get(a_profile, key)   # Optional({name='Alice'})

Undo is not a rollback: nothing is removed, and the history keeps growing forward — each undo is itself a commit, labelled Undo [Set Bob], and each redo Redo [Set Bob]. Both answer Disable to commit_header(commit_id).commit_type(), where a change answers Mutations: the label says which call was undone, the type says what kind of commit it is. The whole DAG stays readable with CommitStateBuilder.state(db, commit_id).

DSM — the model as text

DSM is the human form of what Definitions, Type and Attachment build through the API. A namespace carries a name and a uuid, and holds the declarations:

namespace App {6f9619ff-8b86-d011-b42d-00cf4fc964ff} {
    concept User;
    struct Profile { string name; optional<uint32> age; };
    attachment<User, Profile> profile;
};

DSMBuilder().append("app.dsm", source) then parse() returns a report and the definitions — the same objects the API builds. A declaration is reached by name:

builder = DSMBuilder()
builder.append("app.dsm", source)
report, dsm_definitions, defs = builder.parse()
assert defs is not None, report.errors()             # None when the text has errors
a_profile = DefinitionsInspector(defs).check_attachment("App::User.profile")

That is the shape, not the whole language: a concept specialises another with is a, and clubs, memberships, enumerations, field defaults, doc comments and function pools — the functions a model declares — are written the same way.

You do not have to guess the rest. definitions.const().to_dsm_definitions().to_dsm() renders definitions you already hold back as DSM source, so the package shows you the syntax for your own model. The language is specified at docs.digitalsubstrate.io/dsm.

Installation

pip install dsviper

Requires Python ≥ 3.10. Pre-built wheels are provided for all supported platforms.

Supported Platforms

Documentation

Full documentation at docs.digitalsubstrate.io/dsviper-python

License

Proprietary — Commercial license required. Contact Digital Substrate for licensing information.