Skip to main content

tree-sitter-usd

This library parses USD ASCII files using tree-sitter to produce a light-weight grammar of the file.

For those who don't know what tree-sitter is and why you'd care to use it, see Why Tree-sitter?. For install / usage instructions, see below.

Disclaimer

This repository's parsing rules are subject to change.

Building + Using

Neovim

Make sure you include the following somewhere in your init.lua file.

require("nvim-treesitter.configs").setup {
    ensure_installed = {"usd"},
    parser_install_dir = installation_directory,
    highlight = { enable = true },

    -- More stuff
}

Python

pip install tree-sitter-usda
import tree_sitter_usda
from tree_sitter import Language, Parser

parser = Parser(Language(tree_sitter_usda.language()))
tree = parser.parse(b'def Xform "root"\n{\n    custom int value = 10\n}\n')

print(tree.root_node)

The bundled highlights query is available as tree_sitter_usda.HIGHLIGHTS_QUERY.

Why Tree-sitter?

In the beginning, Tree-sitter was made to give text editors better syntax highlighting.

Most text editors today create syntax highlighting with regex patterns. On large files with long line counts, this approach is slow and error-prone.

In contrast to regex, Tree-sitter actually knows about your file. It can convert a USD file like:

#usda 1.0

def "root"
{
    custom uniform int value = 10
}

Into a tokenized tree like this:

(prim_definition) ; [3:1 - 2:5]
 (prim_type) ; [3:1 - 4:2]
 (string) ; [3:5 - 11:2]
 (block) ; [4:1 - 2:5]
  (attribute_assignment) ; [5:5 - 34:4]
   (custom) ; [5:5 - 11:4]
   (uniform) ; [5:12 - 19:4]
   (attribute_type) ; [5:20 - 23:4]
   (identifier) ; [5:24 - 29:4]
   (integer) ; [5:32 - 34:4]

That tree is built sparsely, interactively, and even works with WIP files where you may be missing a } or two. Tree-sitter is accurate, fast, and getting better all the time.

Having this tree is really powerful. It became clear very quickly to others that Tree-sitter can be used for a lot more than just syntax highlighting. Here's some of the fun plug-ins showing off what you can do using this USD parser.

Neovim

aerial.nvim - Navigate USD Files Effortlessly

aerial.nvim

Display And Move Through A Prim Tree

Effortlessly move in, out, or around any USD Prim, no matter how large it is.

https://user-images.githubusercontent.com/10103049/235325105-1490fb62-4c95-46bf-a170-50df4c7409ff.mp4

Prim Tree Based On Your Current Position

Many times I find myself thinking "I'm in a nested Prim but I actually need to go one down, and over". This aerial.nvim view is super good at moving around.

https://user-images.githubusercontent.com/10103049/235325115-a74d68c6-8f2d-40dd-a7ff-d58240f9b1cd.mp4

Syntax Highlighting

Tree-sitter is an incremental parser. That means

  • Parsing is lightning quick
  • Making edits to the file doesn't require a full re-parse of the file
  • WIP files with syntax errors still parse

usd_treesitter_syntax_highlighting

And the results are pretty good. My Neovim theme is hybrid2.nvim. If you desire even more colors (e.g. coloring uniform as blue, instead of white), there's already an out-of-box highlight group for that over at nvim-treesitter-highlights-usd. In the future, this might get upstreamed to nvim-treesitter, maybe.

Maintain The Current Prim Context

https://user-images.githubusercontent.com/10103049/235326266-93c8e868-ed7f-47a7-bda9-238f02979e82.mp4

nvim-treesitter-context

Have you ever been viewing a huge USD file and, in the middle of viewing some Prim, forget the name / tree of the Prim that you're viewing? This fun plug-in keeps the Prim name pinned as you scroll so you never lose your place.

Prim Statusline

nvim-gps + winbar.nvim

https://user-images.githubusercontent.com/10103049/235326401-64be269b-5e96-4483-b6ee-995392603ef9.mp4

The top bar tracks your location in the file.

Auto-Folding

https://user-images.githubusercontent.com/10103049/235326728-076f14d8-63fc-4c0c-b3c8-e29065bb2917.mp4

nvim-treesitter

Text Objects

nvim-treesitter-textobjects

Select, move, delete, comment, edit anything easily, using whatever mappings you desire.

In truth, most people don't have much need to edit USD files directly. But it's a testiment to tree-sitter that the same mappings do as you expect across all languages.

Qt

examples/qt is a runnable USD layer viewer - a QLineEdit which takes a path on-disk plus a read-only, syntax highlighted QPlainTextEdit.

uv run --no-editable --extra example examples/qt/usda_viewer.py /path/to/some_layer.usda
Image

The example uses Qt.py, so the same code runs on PySide6, PySide2, PyQt5, or PyQt6. Only its Usda-prefixed classes and its layer reader know about USD - everything else works for any tree-sitter grammar.

Integrating Tree-sitter With Qt

Qt colors text with QSyntaxHighlighter. It calls highlightBlock once per block (one line, in a QPlainTextEdit) and you answer with setFormat calls. tree-sitter parses whole files and answers with captured nodes. Bridging the two is mostly a matter of translating coordinates:

from Qt import QtGui
from tree_sitter import Language, Parser, Query, QueryCursor

import tree_sitter_usda


class Highlighter(QtGui.QSyntaxHighlighter):
    def __init__(self, parent=None):
        super().__init__(parent)

        language = Language(tree_sitter_usda.language())
        self._parser = Parser(language)
        self._cursor = QueryCursor(Query(language, tree_sitter_usda.HIGHLIGHTS_QUERY))
        self._formats = {"string": _make_format("#98c379")}  # And so on, per capture

    def highlightBlock(self, text):
        # NOTE: Real code caches this parse. See examples/qt for how + why.
        source = self.document().toPlainText().encode("utf-8")
        tree = self._parser.parse(source)

        start_byte = _get_block_start_byte(source, self.currentBlock().blockNumber())
        end_byte = start_byte + len(text.encode("utf-8"))
        self._cursor.set_byte_range(start_byte, end_byte)

        for _, captures in self._cursor.matches(tree.root_node):
            for capture, nodes in captures.items():
                format_ = self._formats.get(capture)  # e.g. @spell is not a color

                if format_ is None:
                    continue

                for node in nodes:
                    # NOTE: Byte offsets are Qt offsets only while the line is ASCII
                    start = max(node.start_byte, start_byte) - start_byte
                    end = min(node.end_byte, end_byte) - start_byte
                    self.setFormat(start, end - start, format_)

The parts which that sketch glosses over, and which examples/qt handles:

  • Offsets - tree-sitter counts UTF-8 bytes, Qt counts UTF-16 code units. They agree until a line contains a é (2 bytes, 1 unit) or a 🙂 (4 bytes, 2 units), and then every color on that line slides sideways.
  • Priority - a highlights query captures the same text more than once on purpose. (comment) @spell @comment and (attribute_type) @type + @type.builtin both do. tree-sitter 0.25+ gives the last-written pattern priority, so sort the captures by pattern order and paint the low priority ones first. Qt's setFormat is last-write-wins, which does the rest.
  • Speed - do not re-parse per block. Re-parse once per edit, hand the old tree to Parser.parse so tree-sitter re-uses the subtrees which did not change, and give each block a QueryCursor.set_byte_range so it is not querying the whole document.
  • Repaints - Qt only re-highlights the blocks which the user typed in, which is not enough for multi-line constructs. Deleting the """ which opened a docstring re-interprets every line below it. Tree.changed_ranges says exactly which bytes changed meaning, so those blocks can be repainted.

See examples/qt/README.md for the details.

Need A Parser? Look No Further

USD of course has parsing capabilities but, at the time of writing, most of the parsing classes and functions are private. On top of that, it's a multi-million like repository written in C++.

In contrast, tree-sitter

Tree-sitter is easy to embed and extend, making it very attractive for plug-in authors.

Future Improvements

Plug-Ins

There's a bunch of open-source momentum behind tree-sitter. New tools and plug-ins may come out that further expands upon the list of reasons above.

Some other plug-ins that could be useful in the future

And others

Neovim 0.10+

I spotted a couple Neovim roadmap items that seem to want to make tree-sitter faster and more async. It's already fast but more speed is definitely welcome on larger USD files. Needless to say I'll be keeping an eye on those!

Testing

Unittests

cd {root}
tree-sitter test

All tests should pass.

Highlighting

If everything worked correctly, you should be able to highlight any USD file from the tree-sitter CLI like so:

tree-sitter highlight /path/to/file.usda

You should see something like this

tree-sitter_example_usd_hightlighting

And the next time you run tree-sitter test, highlighting information will be in the output.

syntax highlighting:
  ✓ payload.usda (N assertions)
  ✓ references.usda (N assertions)
  ✓ relationship.usda (N assertions)
  ✓ specializes.usda (N assertions)
  ✓ string.usda (N assertions)

  ...

Actual USD Files

The best way to test tree-sitter-usd is to parse USD files in-action.

The basic steps are

  • Download from any of the links above
  • Install the tree-sitter-cli
  • Find + parse the files. e.g.
find /path/to/your/root/usd_files/folder -name "*.usda" -type f | xargs tree-sitter parse

tree-sitter-usd parses all of the files, everywhere, without errors.

Contributing

If you find a bug in a USD file, please submit an issue or pull request specifying the expected parse and the actual results.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tree_sitter_usda-0.8.0.tar.gz (94.1 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tree_sitter_usda-0.8.0-cp310-abi3-win_amd64.whl (50.9 kB view details)

Uploaded CPython 3.10+Windows x86-64

tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl (57.5 kB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl (57.9 kB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

tree_sitter_usda-0.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (58.3 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64manylinux: glibc 2.28+ ARM64

tree_sitter_usda-0.8.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (57.5 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ x86-64manylinux: glibc 2.5+ x86-64

tree_sitter_usda-0.8.0-cp310-abi3-macosx_11_0_arm64.whl (49.7 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

tree_sitter_usda-0.8.0-cp310-abi3-macosx_10_9_x86_64.whl (47.9 kB view details)

Uploaded CPython 3.10+macOS 10.9+ x86-64

File details

Details for the file tree_sitter_usda-0.8.0.tar.gz.

File metadata

  • Download URL: tree_sitter_usda-0.8.0.tar.gz
  • Upload date:
  • Size: 94.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.7.20

File hashes

Hashes for tree_sitter_usda-0.8.0.tar.gz
Algorithm Hash digest
SHA256 b5cc80f745a9731894aea45f7755c45e9b8408a311a1de11bd1cf3fdd25ef816
MD5 59200e3aa1412f9ec1c88f65d87bd2ee
BLAKE2b-256 beffc53529db84d66e40dc75dc40fcf652f703637a235f6d1b55a29e43beb190

See more details on using hashes here.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 973925fa982ade805a977dad3caf172a54077eb8d8caf761dde5bfdf0a07ee53
MD5 e2e56d0f80c0de81e740723f23c7c16d
BLAKE2b-256 805e5251721c610a992d4d04a5fdaa3ab97a0a575d58b98f038581acae7d676e

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 9b726ecbfc6be41efaddbd9ab6cf438844c83084a3c56dee27583cff35864170
MD5 502bdbd20f5c9bb87b1ae9b2667ad7b0
BLAKE2b-256 36f1f0a72ca59fb8dffc679c81a9eb5b72cf5f6d131e98cae2fe46b7d4ac37eb

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 77aab3ac76e30107d11818ea49cddc1a07e9a3bc3103d3ab3ab32c1182a4ca40
MD5 7a901a70b459f21448ad380366a6d823
BLAKE2b-256 89c5e13e67065349caef81b1eb9f83f0b9b4ab8f1a0971a493aa4a2454dd0f4e

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 703110a4bafa4fca7b20188339abad2b3550d854f380812bc50fd7ac4e7b7c20
MD5 37d2f1aa7d61588ade34f5283d12329e
BLAKE2b-256 8046230aac5a97c77aa4bf5ca12aaee89ab8ab9c4faf5d003aff0650f55a3252

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl
Algorithm Hash digest
SHA256 71b036f3094caf362f32b2a5f1a02d96866c79f3e24cca4cb776a2391b48c23a
MD5 cc4741dd0ff5fe409661a066300ec71d
BLAKE2b-256 ccf6eff3a1e71a37d76aa23363db7fcea7b0845ec242d57f87415dd1b4c580f7

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a439a0a2f12ef1736c7212e9268931d806e16996fe26419180642e217192b9f3
MD5 c3d1e4853e78bbec6764628cb19109c4
BLAKE2b-256 47d3ca18ec8ec02f089659e63253745d80b188e7b36ec7710e494bff78de50b3

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tree_sitter_usda-0.8.0-cp310-abi3-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for tree_sitter_usda-0.8.0-cp310-abi3-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 4b1f7ef537f203a31ec8450b6532a9c59c260ee996592b25b5b650668aa30a33
MD5 0be4efbe6019aa71f334ede588d23051
BLAKE2b-256 123f0382e96a7ed37b9cedb5080a0eee0abf094f4fd2514aec73ca779e6182f0

See more details on using hashes here.

Provenance

The following attestation bundles were made for tree_sitter_usda-0.8.0-cp310-abi3-macosx_10_9_x86_64.whl:

Publisher: publish.yml on ColinKennedy/tree-sitter-usd

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.8.0 This release

8 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page