Silwright

Silwright is a tool to work with programming language independent definitions of node tree structures. The goal is, to be able to focus tree structure definitions to the very essence of these trees rather than having to deal with programming language specific details. The definitions and the generated code are intended for use in compilers as the foundation of AST and IR structures. However other use cases might be feasible as well.

The current implementation also features a converter to turns such definitions into C++ code.

Installation

Install the command-line tool from PyPI:

pip install silwright

Alternatively, run it in an isolated environment without installing it permanently:

uvx silwright path/to/module.ndef

Silwright requires Python 3.14 or newer. Generated C++ currently requires C++20; in particular, the generated model and dump support use std::variant and C++20 constraints.

Project status

Silwright is beta software. Its current language and C++ generation workflow are usable, but the project is still pre-1.0, has limited external production validation, and may make compatibility-affecting changes while its contracts are finalized.

Development setup

Prerequisites:

• Git
• uv
• Python 3.14 or newer; uv can install the requested interpreter when necessary

Clone the repository and create the locked development environment:

uv sync

Run all project checks:

uv run pytest
uv run ruff check .
uv run mypy

The package uses a src layout. Runtime dependencies and development tools are declared in pyproject.toml; exact resolved versions are committed in uv.lock. Add dependencies with uv add <package> and development dependencies with uv add --dev <package>.

VS Code extension development

The VS Code extension lives in vscode/silwright-ndef-language-support. Open that directory in Visual Studio Code and press F5 to launch an Extension Development Host. Open an .ndef file in that window to exercise the extension.

The extension has no build step or runtime dependencies. Run its unit tests with:

cd vscode/silwright-ndef-language-support
npm test

The Marketplace icon is committed as icon.png; icon.svg is the editable source for future icon iterations and is excluded from the packaged VSIX through .vscodeignore.

To package it after installing @vscode/vsce:

cd vscode/silwright-ndef-language-support
vsce package

Usage

Generate C++ from the example node definition:

uv run silwright examples/simple_lang.ndef

The command automatically loads the required backend_cpp.map from the same directory, then writes the model .hpp/.cpp, dump _dump.hpp/_dump.ipp/_dump.cpp, visitor _visitor.hpp/_visitor.cpp, and transformer _transformer.hpp/_transformer.cpp files beside simple_lang.ndef. Existing generated output files are replaced. Syntax errors include their source location.

Library users can call silwright.parse_definition_file(path) for the same combined lookup behavior. silwright.parse_definitions(text) and silwright.parse_type_mappings(text) are available when parsing already-loaded content independently.

silwright.analyze(parsed) performs centralized semantic validation and returns a ValidatedModel containing resolved declarations, backend mappings, flattened trait fields, dependency order, and visitor reachability. The model, dump, visitor, and transformer generators all accept this validated model; passing raw parsed input remains supported and invokes the same analysis boundary automatically.

The complete current syntax and generation contract are documented in docs/language-reference.md.

Documentation

The project documentation can be previewed locally with:

uv sync --group docs
uv run mkdocs serve

The GitHub Pages workflow publishes the documentation from master when its source files change. The built site/ directory is local output and is not version controlled.

Releasing

Python package releases use v<version> tags and are independent of VS Code extension releases. Before creating a release, update the version in pyproject.toml, move the relevant entries in CHANGELOG.md into a matching version section, and commit both changes. Then create and push the matching tag; for version 0.1.0 this is:

git tag v0.1.0
git push origin v0.1.0

The release workflow verifies that the tag, project version, and changelog agree before it builds and publishes the distributions to PyPI.

VS Code extension releases use vscode-v<version> tags and are independent of Python package releases. Before creating a release, update the version in vscode/silwright-ndef-language-support/package.json, move the relevant entries in vscode/silwright-ndef-language-support/CHANGELOG.md into a matching version section, and commit those changes. Then create and push the matching tag; for version 0.7.0 this is:

git tag vscode-v0.7.0
git push origin vscode-v0.7.0

The VS Code extension release workflow runs the tests, packages the .vsix, and attaches it to a dedicated GitHub Release. The workflow rejects a tag when its version does not match package.json.

Definition language

One .ndef file describes one module. Its required first declaration is module <name>; the module name is intended to become the generated C++ namespace.

module expressions

node Variable
    name: identifier
end

node Number
    value: number
end

choice Expr
    Variable | Number
end

enum Op
    Add | Subtract | Multiply | Divide | Modulus
end

node BinaryExpression
    op: Op
    left: Expr
    right: Expr
end

node FunctionDefinition
    head: value FunctionHead
    code: *Expr
end

• node declares a future C++ struct. Each field is written as <name>: <type>.
• trait declares reusable fields. A node applies one or more traits with node FunctionHead with Location, Documented; generated traits are public base structs.
• A choice applies traits to all transitively reachable node alternatives with choice Expr allwith Location, Attributes. This changes those generated node types globally, including uses outside the choice; overlapping applications are deduplicated.
• Prefixing a field type with *, as in code: *Expr, makes it a repeated field.
• Prefixing a repeated field with fixed, as in arguments: fixed *Expr, preserves the field's element count in generated transformers. It uses the same C++ storage as *Expr.
• Prefixing a field type with ?, as in label: ?identifier, makes it optional. Built-in, enum, and value fields use std::optional; pointer-backed node and choice fields remain std::unique_ptr because they already represent absence. * and ? cannot be combined.
• Prefixing a field type with value, as in head: value FunctionHead, embeds node or choice types directly instead of using std::unique_ptr. The modifiers compose as parameters: *value Parameter for std::vector<parameter>.
• Prefixing a field type with transient, as in function_scope: transient scope, emits the C++ member but excludes it from structural dump and visitor generation.
• choice declares a sum type. Its alternatives refer to node types or other declared types.
• Choice alternatives may span lines. The | separator may be placed after the preceding option or at the start of the following option's line, and these layouts may be mixed within one choice definition.
• enum declares a set of new enumerator names rather than references to node types.
• Enum entries support the same inline, multiline, and mixed | layouts as choice alternatives.
• end closes every node, trait, choice, and enum declaration.
• Names currently use ASCII letters, digits, and underscores and cannot start with a digit.
• // starts a comment that continues to the end of the line. Comments may occupy a complete line or follow regular definition code. Multiline comments are not supported. Blank lines are accepted around declarations; blank lines inside declaration bodies are not currently supported. Horizontal spacing is insignificant.

C++ backend types are mapped separately in backend_cpp.map:

@include <cstddef>
@include "project/symbol.hpp"

identifier: std::string
number: long
index: std::size_t
type: project::type_id

Each mapping has a Silwright type on the left and its C++ type spelling on the right. @include accepts either a system header in <...> or a project header in "..."; these headers are deduplicated and emitted into the generated model header. For a definition such as simple_lang.ndef, the parser always looks for backend_cpp.map in the same directory. See examples/simple_lang.ndef and examples/backend_cpp.map for complete examples.

End-to-end C++ example

examples/simple_lang_main.cpp constructs a function definition containing several expression nodes and writes its YAML-like dump to std::cout. examples/CMakeLists.txt defines the simple_lang_example C++20 executable from the sample and generated translation units. The example files are currently provided as a proof of concept; automated C++ compilation is not yet part of the project checks.

Requirements and decisions

This section is maintained as requirements are discovered or changed during development.

• The project requires Python 3.14 or newer and uses uv for environments, dependencies, locking, and command execution.
• A .ndef file contains exactly one module and must start with a module declaration.
• .ndef comments start with // and continue through the end of the current line; there is no multiline comment syntax.
• A module may contain node, trait, choice, and enum declarations in source order.
• Node fields and choice alternatives reference types by name. Enums introduce values by name. This distinction prevents enum values from being mistaken for type references.
• Backend types and their C++ spellings live in the required backend_cpp.map file beside each parsed .ndef file. C++ is currently selected statically; no dynamic backend selection is supported yet.
• backend_cpp.map may declare generated model-header dependencies with @include <header> and @include "header". Includes preserve declaration order and duplicates are removed.
• Generated .hpp and .cpp files use the .ndef basename and are written beside it. The module name is used directly as the C++ namespace.
• Definition names are converted from CamelCase to snake_case. Enum names additionally use a _t suffix. Enum fields are scalar; node and choice fields use std::unique_ptr; built-in fields use their mapped C++ spelling.
• Traits generate public base structs and do not participate in visitor traversal. Choices can propagate traits transitively to their node alternatives with allwith. Trait fields are flattened into node dump output. Repeated applications are deduplicated; unknown traits and inherited/local field-name collisions are rejected before C++ is emitted.
• Transient fields hold tooling or later-stage state that belongs on the in-memory node but is not part of its structural representation. They are generated normally but omitted from dump output and visitor child traversal.
• Repeated fields use std::vector around the otherwise generated field type. For example, *identifier becomes std::vector<std::string> and *Expr becomes the generated expr_list alias for std::vector<std::unique_ptr<expr>>.
• Fixed repeated fields use the same C++ storage as repeated fields, but generated transformers preserve one output slot per input slot. Null elements remain null; non-null elements must produce exactly one replacement.
• Optional fields use std::optional around the otherwise generated value type. Missing optional values and null node pointers are both emitted as null by generated dumpers.
• Value fields require complete C++ types. The generator dependency-orders affected structs and rejects recursive value-member cycles. Built-in and enum fields are already values, so applying value to them has no additional effect.
• Choices are std::variant aliases. Node structs are forward-declared before choice aliases, then defined afterward. Choice-to-choice dependencies are ordered, and cycles between choice aliases are rejected because C++ aliases cannot be forward-declared.
• Parsing currently validates syntax only. Duplicate declarations, unresolved references, naming rules for generated C++, mapping conflicts, and recursive type constraints remain future semantic-validation decisions.
• Generated headers currently include <memory>, <optional>, <string>, <variant>, and <vector>, plus dependencies declared by the backend map.
• Dump support is generated separately as _dump.hpp, _dump.ipp, and _dump.cpp. The IPP contains template implementations and is included at the end of the dump header.
• Generated dump overloads emit YAML-like objects with four-space nesting, an artificial _type field, .ndef field names, lists for repeated fields, and null for null pointers. Strings are quoted and escaped, and enums use their original .ndef entry names.
• Scalar values pass through an inline generic dump_value customization point. Its default uses operator<<; context-sensitive custom types can provide a more-specific overload in the value type's namespace so it is found through argument-dependent lookup. A mapped type that is neither streamable nor customized produces a targeted compile-time diagnostic.
• Mutable visitor support is generated as _visitor.hpp and _visitor.cpp. Public visit overloads accept every node and choice as an entry point. Node traversal calls protected virtual enter and leave hooks around pointer-backed children; scalars and value fields are deliberately skipped. Definitions referenced exclusively through value fields are omitted from the visitor API, while unreferenced definitions remain possible entry points. Derived visitors can override only the hooks they need. Visitors also provide protected replace_<choice> helpers for pointer-backed choice children; replacements are consumed by the parent traversal immediately after each child visit.
• Mutable transformer support is generated as _transformer.hpp and _transformer.cpp. Public rewrite overloads perform bottom-up rewrites of pointer-backed node and choice fields. Concrete node hooks are split into visit_single and visit_multiple; repeated choice rewrites use generated <choice>_list aliases.