Skip to main content

lang-parsing-substrate

A shared language-parsing substrate for static-analysis tools: tree-sitter grammar dispatch, language detection, and a growing set of language-agnostic analysis primitives (import/call graphs, control-flow graphs, structural fingerprinting, suppression comments) built on top of a unified LanguageInfo registry across 16 languages — compiled in at build time via Cargo feature flags.

What's in the substrate

Module Provides
registry Language detection by extension, the LanguageInfo table, SLOC comment-style metadata
query Iterative (non-recursive) tree-sitter traversal helpers: find_descendants, find_first_descendant, node_text, ancestor lookups
imports Per-file import/use-statement extraction, for building efferent-coupling (Ce) edges
calls Per-file call-graph edge extraction (callercallee), with external-call detection
cfg Control-flow graph / basic-block construction for a function body (c, cpp, rust)
c_standard Best-effort lower bound on the C standard (C99/C11/C23) a file's syntax requires
fingerprint Structural hashing of function-like subtrees, for duplicate/clone detection across a corpus
regions tools:off / tools:on ignored-region markers
suppressions tools:suppress TOOL:RULE single-line suppression comments
path_ignore Compiled glob ignore-pattern sets for path filtering

Everything below the registry is deliberately per-file: a module extracts what one parse tree contains, and leaves assembling a corpus-wide graph, dedup report, or coupling metric to the caller. This keeps the substrate's job narrow (one authoritative, correct answer per file) and lets each consumer choose its own storage model (in-memory, SQLite, whatever) without the substrate needing to know about it.

Language coverage varies by module — the registry knows about all 16 languages, but modules like cfg only model the languages they've been built out for. A module never fabricates a result for a language it doesn't support; it returns None (or an empty result) instead of guessing.

Feature flags

Not every consumer needs all 16 languages. Each language is an optional Cargo feature; the all-languages convenience feature (enabled by default) pulls in the full set.

Feature Language Grammar crate
lang-c C tree-sitter-c
lang-cpp C++ tree-sitter-cpp
lang-rust Rust tree-sitter-rust
lang-python Python tree-sitter-python
lang-javascript JavaScript tree-sitter-javascript
lang-typescript TypeScript tree-sitter-typescript
lang-go Go tree-sitter-go
lang-java Java tree-sitter-java
lang-csharp C# tree-sitter-c-sharp
lang-kotlin Kotlin tree-sitter-kotlin-ng
lang-swift Swift tree-sitter-swift
lang-php PHP tree-sitter-php
lang-ada Ada tree-sitter-ada
lang-fortran Fortran (free-form) tree-sitter-fortran
lang-scala Scala tree-sitter-scala
lang-lua Lua tree-sitter-lua
all-languages All of the above

A consumer that only cares about C/C++, for example, would declare:

lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }

Usage

# Cargo.toml — full language set (default)
lang-parsing-substrate = "0.5"

# Cargo.toml — C/C++ only
lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }
use lang_parsing_substrate::{language_for_file, languages, supported_languages_report};
use std::path::Path;

// Detect language for a file
if let Some(lang) = language_for_file(Path::new("main.c")) {
    let mut parser = tree_sitter::Parser::new();
    parser.set_language(&lang).unwrap();
    // parse...
}

// Enumerate compiled-in languages (reflects feature flags)
for info in languages() {
    println!("{}: {:?}", info.name, info.extensions);
}

// Human-readable summary (for --supported-languages flags)
print!("{}", supported_languages_report());

Grammar crates are re-exported so consumers reach them transitively:

// No direct tree-sitter-rust dependency needed in your Cargo.toml
use lang_parsing_substrate::tree_sitter_rust;

Analysis primitives

use lang_parsing_substrate::{call_edges, import_sources, build_function_cfg, structural_hash};

// Call-graph edges for every named function/macro in a parsed file
let edges = call_edges(tree.root_node(), source);

// Import/use-statement sources, for Ce/Ca coupling metrics
let imports = import_sources(&tree, source.as_bytes(), "rust");

// Control-flow graph for a single function body (c/cpp/rust)
if let Some(cfg) = build_function_cfg(func_node, source, "rust") {
    println!("{} basic blocks", cfg.block_count());
}

// Best-effort lower bound on the C standard a file requires
if let Some(standard) = detect_min_c_standard(&tree, source.as_bytes()) {
    println!("requires at least {standard:?}");
}

API

  • languages() -> &'static [LanguageInfo] — compiled-in language set
  • language_for_file(path: &Path) -> Option<Language> — grammar dispatch by extension
  • language_for_key(key: &str) -> Option<Language> — grammar dispatch by registry key
  • language_info_for_file(path: &Path) -> Option<&'static LanguageInfo>
  • is_source_extension / is_parseable_extension(ext: &OsStr) -> bool — recursive-discovery gates
  • supported_languages_report() -> String — human-readable language summary
  • LanguageInfo / SlocMode — registry metadata and comment-style enum (drives SLOC calculation)
  • find_descendants / find_first_descendant / find_ancestor / node_text and friends — traversal helpers (query)
  • import_sources / distinct_import_count — import extraction (imports)
  • call_edges / CallEdge / is_function_kind / get_function_name — call-graph extraction (calls)
  • build_function_cfg / FunctionCfg / BasicBlock / CfgEdge — control-flow graphs (cfg)
  • detect_min_c_standard / CStandard — C standard lower-bound detection (c_standard)
  • function_fingerprints / duplicate_groups / Fingerprint / CorpusFingerprint — structural hashing (fingerprint)
  • ignored_regions / IgnoredRegiontools:off/tools:on markers (regions)
  • suppressions / Suppressiontools:suppress comments (suppressions)
  • PathIgnore — compiled glob ignore sets (path_ignore)

Building

cargo build                                          # all languages (default)
cargo build --no-default-features --features lang-c,lang-cpp  # subset
cargo test

Requires no C compiler — tree-sitter grammar crates ship pre-generated C sources and compile via the cc crate.

Python bindings

The pyo3 Cargo feature (off by default) exposes the substrate's language-agnostic analysis primitives as a Python extension module, published to PyPI as prebuilt abi3 wheels (CPython 3.10+, one wheel per platform — see docs/releasing.md):

pip install lang-parsing-substrate

Building it yourself from source uses maturin:

pip install maturin
maturin build --release          # writes target/wheels/lang_parsing_substrate-*.whl
pip install target/wheels/lang_parsing_substrate-*.whl
import lang_parsing_substrate as lps

src = "fn helper(x: i32) -> i32 { x + 1 }\nfn main() { helper(41); }\n"
edges = lps.call_edges("rust", src)          # [CallEdge(caller='main', callee='helper', ...)]
cfg = lps.function_cfg("rust", src, "main")  # FunctionCfg | None
fps = lps.function_fingerprints("rust", src, min_nodes=1)

Python can't hand this crate a tree_sitter::Node/Tree directly — this crate's tree-sitter version has no ABI relationship to tree-sitter's own, separate Python bindings — so every bound function takes (language_key, source), parses internally, and returns owned data (CallEdge, FunctionCfg, Fingerprint, Suppression, IgnoredRegion, LanguageInfo, all plain attribute-holding classes). A consumer that also needs to walk the tree itself (e.g. for domain-specific semantics this crate doesn't model) still parses separately with a language-specific tree-sitter Python package; the bindings here only cover the substrate's own primitives.

invoke build-wheel builds the wheel locally for testing. The actual PyPI release happens in CI on a vX.Y.Z tag push, via Trusted Publishing (OIDC, no API token) — see docs/releasing.md.

License

MIT — see LICENSE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lang_parsing_substrate-0.5.1.tar.gz (88.9 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lang_parsing_substrate-0.5.1-cp310-abi3-win_amd64.whl (3.2 MB view details)

Uploaded CPython 3.10+Windows x86-64

lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_aarch64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

lang_parsing_substrate-0.5.1-cp310-abi3-macosx_11_0_arm64.whl (3.6 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

lang_parsing_substrate-0.5.1-cp310-abi3-macosx_10_12_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file lang_parsing_substrate-0.5.1.tar.gz.

File metadata

  • Download URL: lang_parsing_substrate-0.5.1.tar.gz
  • Upload date:
  • Size: 88.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for lang_parsing_substrate-0.5.1.tar.gz
Algorithm Hash digest
SHA256 35244fa5ae6d2d5596ea513aaa6368610e8f61c51e607dec565214f956e84631
MD5 70ec143add157179d3cff02a9beeb0d0
BLAKE2b-256 240703653b9aa3a250090d9b62c8b1abf9f8b8ec82036dd8936c58a2642498a3

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 17aa97ad6aba133604c27688069eab43f966b831f8e081604336891c6e6ce630
MD5 eb31898b1986c4590994b1a04047d54b
BLAKE2b-256 30835d73b3e362096888c6931e494b99ecd2d79a10927483403d77a6eea98d51

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 f2a6b9746b29dbe548bb2cb279c3e14ce25c859fd423273113d38b90532da62f
MD5 6bfe3517f6677adb5403bf9d5ab9c5ab
BLAKE2b-256 464cef5109b9b3a8bbdd571f5d41c63a5baed005ea11ca7d6a7df26a173652d9

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 b95b38492e2d6673963b6d91cbcb0fb661f84c55d93d870b7635276ba779d545
MD5 ee4ef141ed6ebf649f69d5a701aab1b7
BLAKE2b-256 b6182ef4d95c19f82b06c31b5559c1839c6567d22dd67b3398b1b38216a77b0a

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d0d3409bd842e31bcfb1ab6c215cff2c7c749a1553f41d34b6d43e098884a2b8
MD5 5221ce8691fe5b9a6b4332627b15ffb4
BLAKE2b-256 24bb7ce14a18cb2a65bb2aad38546723764d5a4eb2e9b3a7cadb10958a989e0a

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 dae5e801be124e0e38616f4286e641c3fd0e607c855e3eedbe3a5ce07d784433
MD5 dcfe23f6e25f6b7bbb4eac3ce9f17c2d
BLAKE2b-256 620a5b61cb55fb644808dd1c54cd1a3339edb9d19994a8d4e5df7935d0ed1cb3

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 7ebc408315e26e1f778ea7ec6973f95c7ed07e657045c9ba34cff1b57888e8ea
MD5 b580ddb088bfd5f4b753aa7495455889
BLAKE2b-256 0a9ff519a53009bf98d98d969dbedd177ea0f98eb8b6bb18393d33701ed46df1

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 4bf6771f001d1544d76bb0d3f20e3d64f237ca6dde96b71776a46acdaafcbbec
MD5 488b5cd9f6390ef500aeb14a6e86b13b
BLAKE2b-256 c652137135e61575c6801e79c10e5d41736478ee1d53f13a0145adb824a9fe94

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.9.0

8 files

0.8.0

8 files

0.7.1

8 files

0.7.0

8 files

0.6.2

8 files

0.6.1

8 files

0.6.0

8 files

0.5.2

8 files

This release

0.5.1 This release

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page