Skip to main content

lang-parsing-substrate

A shared language-parsing substrate for static-analysis tools: tree-sitter grammar dispatch, language detection, and a growing set of language-agnostic analysis primitives (import/call graphs, control-flow graphs, structural fingerprinting, suppression comments) built on top of a unified LanguageInfo registry across 16 languages — compiled in at build time via Cargo feature flags.

What's in the substrate

Module Provides
registry Language detection by extension, the LanguageInfo table, SLOC comment-style metadata
query Iterative (non-recursive) tree-sitter traversal helpers: find_descendants, find_first_descendant, node_text, ancestor lookups
imports Per-file import/use-statement extraction, for building efferent-coupling (Ce) edges
calls Per-file call-graph edge extraction (callercallee), with external-call detection
cfg Control-flow graph / basic-block construction for a function body (c, cpp, rust)
c_standard Best-effort lower bound on the C standard (C99/C11/C23) a file's syntax requires
fingerprint Structural hashing of function-like subtrees, for duplicate/clone detection across a corpus
regions tools:off / tools:on ignored-region markers
suppressions tools:suppress TOOL:RULE single-line suppression comments
dead_code Preprocessor dead-code regions for C/C++ (#if 0, __cplusplus-gated branches, locally-provable macro definedness)
dead_code_swift Dead-code regions for Swift's #if/#elseif/#else conditional compilation (compile-time-constant boolean conditions only — see module docs for why the C/C++ macro-definedness sub-problem doesn't apply to Swift)
dead_code_csharp Dead-code regions for C#'s #if/#elif/#else conditional compilation — AST-based like dead_code_swift, but ports both C/C++ sub-problems (constant conditions and locally-provable #define/#undef symbol definedness) since C# has real nested preprocessor nodes and real #define
path_ignore Compiled glob ignore-pattern sets for path filtering

Everything below the registry is deliberately per-file: a module extracts what one parse tree contains, and leaves assembling a corpus-wide graph, dedup report, or coupling metric to the caller. This keeps the substrate's job narrow (one authoritative, correct answer per file) and lets each consumer choose its own storage model (in-memory, SQLite, whatever) without the substrate needing to know about it.

Language coverage varies by module — the registry knows about all 16 languages, but modules like cfg only model the languages they've been built out for. A module never fabricates a result for a language it doesn't support; it returns None (or an empty result) instead of guessing.

Feature flags

Not every consumer needs all 16 languages. Each language is an optional Cargo feature; the all-languages convenience feature (enabled by default) pulls in the full set.

Feature Language Grammar crate
lang-c C tree-sitter-c
lang-cpp C++ tree-sitter-cpp
lang-rust Rust tree-sitter-rust
lang-python Python tree-sitter-python
lang-javascript JavaScript tree-sitter-javascript
lang-typescript TypeScript tree-sitter-typescript
lang-go Go tree-sitter-go
lang-java Java tree-sitter-java
lang-csharp C# tree-sitter-c-sharp
lang-kotlin Kotlin tree-sitter-kotlin-ng
lang-swift Swift tree-sitter-swift
lang-php PHP tree-sitter-php
lang-ada Ada tree-sitter-ada
lang-fortran Fortran (free-form) tree-sitter-fortran
lang-scala Scala tree-sitter-scala
lang-lua Lua tree-sitter-lua
all-languages All of the above

A consumer that only cares about C/C++, for example, would declare:

lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }

Usage

# Cargo.toml — full language set (default)
lang-parsing-substrate = "0.5"

# Cargo.toml — C/C++ only
lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }
use lang_parsing_substrate::{language_for_file, languages, supported_languages_report};
use std::path::Path;

// Detect language for a file
if let Some(lang) = language_for_file(Path::new("main.c")) {
    let mut parser = tree_sitter::Parser::new();
    parser.set_language(&lang).unwrap();
    // parse...
}

// Enumerate compiled-in languages (reflects feature flags)
for info in languages() {
    println!("{}: {:?}", info.name, info.extensions);
}

// Human-readable summary (for --supported-languages flags)
print!("{}", supported_languages_report());

Grammar crates are re-exported so consumers reach them transitively:

// No direct tree-sitter-rust dependency needed in your Cargo.toml
use lang_parsing_substrate::tree_sitter_rust;

Analysis primitives

use lang_parsing_substrate::{call_edges, import_sources, build_function_cfg, structural_hash};

// Call-graph edges for every named function/macro in a parsed file
let edges = call_edges(tree.root_node(), source);

// Import/use-statement sources, for Ce/Ca coupling metrics
let imports = import_sources(&tree, source.as_bytes(), "rust");

// Control-flow graph for a single function body (c/cpp/rust)
if let Some(cfg) = build_function_cfg(func_node, source, "rust") {
    println!("{} basic blocks", cfg.block_count());
}

// Best-effort lower bound on the C standard a file requires
if let Some(standard) = detect_min_c_standard(&tree, source.as_bytes()) {
    println!("requires at least {standard:?}");
}

API

  • languages() -> &'static [LanguageInfo] — compiled-in language set
  • language_for_file(path: &Path) -> Option<Language> — grammar dispatch by extension
  • language_for_key(key: &str) -> Option<Language> — grammar dispatch by registry key
  • language_info_for_file(path: &Path) -> Option<&'static LanguageInfo>
  • is_source_extension / is_parseable_extension(ext: &OsStr) -> bool — recursive-discovery gates
  • supported_languages_report() -> String — human-readable language summary
  • LanguageInfo / SlocMode — registry metadata and comment-style enum (drives SLOC calculation)
  • find_descendants / find_first_descendant / find_ancestor / node_text and friends — traversal helpers (query)
  • import_sources / distinct_import_count — import extraction (imports)
  • call_edges / CallEdge / is_function_kind / get_function_name — call-graph extraction (calls)
  • build_function_cfg / FunctionCfg / BasicBlock / CfgEdge — control-flow graphs (cfg)
  • detect_min_c_standard / CStandard — C standard lower-bound detection (c_standard)
  • function_fingerprints / duplicate_groups / Fingerprint / CorpusFingerprint — structural hashing (fingerprint)
  • ignored_regions / IgnoredRegiontools:off/tools:on markers (regions)
  • suppressions / Suppressiontools:suppress comments (suppressions)
  • dead_code_ranges / DeadCodeRegion / DeadCodeReason — preprocessor dead-code regions, C/C++ only (dead_code)
  • swift_dead_code_regions / SwiftDeadCodeRegion — Swift #if/#elseif/#else dead-code regions (dead_code_swift)
  • csharp_dead_code_regions / CSharpDeadCodeRegion — C# #if/#elif/#else dead-code regions (dead_code_csharp)
  • PathIgnore — compiled glob ignore sets (path_ignore)

Building

cargo build                                          # all languages (default)
cargo build --no-default-features --features lang-c,lang-cpp  # subset
cargo test

Requires no C compiler — tree-sitter grammar crates ship pre-generated C sources and compile via the cc crate.

Python bindings

The pyo3 Cargo feature (off by default) exposes the substrate's language-agnostic analysis primitives as a Python extension module, published to PyPI as prebuilt abi3 wheels (CPython 3.10+, one wheel per platform — see docs/releasing.md):

pip install lang-parsing-substrate

Building it yourself from source uses maturin:

pip install maturin
maturin build --release          # writes target/wheels/lang_parsing_substrate-*.whl
pip install target/wheels/lang_parsing_substrate-*.whl
import lang_parsing_substrate as lps

src = "fn helper(x: i32) -> i32 { x + 1 }\nfn main() { helper(41); }\n"
edges = lps.call_edges("rust", src)          # [CallEdge(caller='main', callee='helper', ...)]
cfg = lps.function_cfg("rust", src, "main")  # FunctionCfg | None
fps = lps.function_fingerprints("rust", src, min_nodes=1)

Python can't hand this crate a tree_sitter::Node/Tree directly — this crate's tree-sitter version has no ABI relationship to tree-sitter's own, separate Python bindings — so every bound function takes (language_key, source), parses internally, and returns owned data (CallEdge, FunctionCfg, Fingerprint, Suppression, IgnoredRegion, LanguageInfo, all plain attribute-holding classes). A consumer that also needs to walk the tree itself (e.g. for domain-specific semantics this crate doesn't model) still parses separately with a language-specific tree-sitter Python package; the bindings here only cover the substrate's own primitives.

invoke build-wheel builds the wheel locally for testing. The actual PyPI release happens in CI on a vX.Y.Z tag push, via Trusted Publishing (OIDC, no API token) — see docs/releasing.md.

License

MIT — see LICENSE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lang_parsing_substrate-0.9.0.tar.gz (134.8 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lang_parsing_substrate-0.9.0-cp310-abi3-win_amd64.whl (3.2 MB view details)

Uploaded CPython 3.10+Windows x86-64

lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_aarch64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

lang_parsing_substrate-0.9.0-cp310-abi3-macosx_11_0_arm64.whl (3.6 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

lang_parsing_substrate-0.9.0-cp310-abi3-macosx_10_12_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file lang_parsing_substrate-0.9.0.tar.gz.

File metadata

  • Download URL: lang_parsing_substrate-0.9.0.tar.gz
  • Upload date:
  • Size: 134.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for lang_parsing_substrate-0.9.0.tar.gz
Algorithm Hash digest
SHA256 d8c0f20dacdc11ed80737c12e3b6a4f71079ba9e8febe7408baf6dc2429d4b43
MD5 a1404e05449b79545fdc85f54c5d4132
BLAKE2b-256 8a050599caaba15f3bdbfeb64c345674e6ab5ecb6dba84016eddf7eae9bb8795

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 547c8665388a0cf9f4f65d9a40e173fa109af9a208dfc19bd15725fd2211b42f
MD5 40a14dc2f377103bd03c21fda93f5710
BLAKE2b-256 dfbf138d41b169df0ed3f689f5eaee47864e4bfae8cbe2eebe307938d974d452

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 92578d06b80ec0908b76d2d05b3fe2343bc546a719b7427e00ea31644ccafd7a
MD5 9b222554509482c6bedd38c8b5b4fceb
BLAKE2b-256 df8121bc5fa3d37dcca6aaa5779cc2286927823dca65cb7e3750c2244521be97

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 3b2cd39e957f96ead1bdbcd644bae518aaed10e2f3f6d8f7a89b9a666e690b6b
MD5 126de25b9acf5d3f2fe6a1e232835726
BLAKE2b-256 7014246200f229cb74d5eb43c592edffb3a9cde5c92216445ec9150213a78145

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 113a49943fcd203abb85162f83807ca5e06705073fc54f0245626f67a9e4c47f
MD5 5c6f37f721a2f8cc09625df62fb12235
BLAKE2b-256 9650c95f0c696be3c1662adfe0f523bb876b6e262871b51057bc623701560501

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 4eb91cf8f677b53726adb0f151485b829335ae530766ae5b9bd74a513b1d74e0
MD5 4244b6e6da0e4765467c966470c479ff
BLAKE2b-256 8d39b48f454ae6b2b60f02ad81de5e4661da8b1235578bc9829617243391da3a

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 24aa3ba1043e9b41b0a913396895c25a2ef742561232ae05e148ea1adaa682ea
MD5 e815264634cb4ed381107e78ad14c3c2
BLAKE2b-256 c53c58c633bbe9ad26ec3617a76b52108fd16dcb690471e8e409ced6c32a732f

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.9.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.9.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 b4d3352fc0a72e9d50d36a4bbe6558f5823319e277ae8f96b1c303054b04a9af
MD5 132d5ccae15913377a912f9cea29acb0
BLAKE2b-256 8229ceca1bf16848b541811eac8aa4dc3ad4147665a963bf9e856221ea3779f6

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.9.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.9.0 This release

8 files

0.8.0

8 files

0.7.1

8 files

0.7.0

8 files

0.6.2

8 files

0.6.1

8 files

0.6.0

8 files

0.5.2

8 files

0.5.1

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page