lang-parsing-substrate

A shared language-parsing substrate for static-analysis tools: tree-sitter grammar dispatch, language detection, and a growing set of language-agnostic analysis primitives (import/call graphs, control-flow graphs, structural fingerprinting, suppression comments) built on top of a unified LanguageInfo registry across 16 languages — compiled in at build time via Cargo feature flags.

What's in the substrate

Module Provides
registry Language detection by extension, the LanguageInfo table, SLOC comment-style metadata
query Iterative (non-recursive) tree-sitter traversal helpers: find_descendants, find_first_descendant, node_text, ancestor lookups
imports Per-file import/use-statement extraction, for building efferent-coupling (Ce) edges
calls Per-file call-graph edge extraction (caller → callee), with external-call detection
cfg Control-flow graph / basic-block construction for a function body (c, cpp, rust)
c_standard Best-effort lower bound on the C standard (C99/C11/C23) a file's syntax requires
fingerprint Structural hashing of function-like subtrees, for duplicate/clone detection across a corpus
regions tools:off / tools:on ignored-region markers
suppressions tools:suppress TOOL:RULE single-line suppression comments
dead_code Preprocessor dead-code regions for C/C++ (#if 0, __cplusplus-gated branches, locally-provable macro definedness)
dead_code_swift Dead-code regions for Swift's #if/#elseif/#else conditional compilation (compile-time-constant boolean conditions only — see module docs for why the C/C++ macro-definedness sub-problem doesn't apply to Swift)
dead_code_csharp Dead-code regions for C#'s #if/#elif/#else conditional compilation — AST-based like dead_code_swift, but ports both C/C++ sub-problems (constant conditions and locally-provable #define/#undef symbol definedness) since C# has real nested preprocessor nodes and real #define
path_ignore Compiled glob ignore-pattern sets for path filtering

Everything below the registry is deliberately per-file: a module extracts what one parse tree contains, and leaves assembling a corpus-wide graph, dedup report, or coupling metric to the caller. This keeps the substrate's job narrow (one authoritative, correct answer per file) and lets each consumer choose its own storage model (in-memory, SQLite, whatever) without the substrate needing to know about it.

Language coverage varies by module — the registry knows about all 16 languages, but modules like cfg only model the languages they've been built out for. A module never fabricates a result for a language it doesn't support; it returns None (or an empty result) instead of guessing.

Feature flags

Not every consumer needs all 16 languages. Each language is an optional Cargo feature; the all-languages convenience feature (enabled by default) pulls in the full set.

Feature Language Grammar crate
lang-c C tree-sitter-c
lang-cpp C++ tree-sitter-cpp
lang-rust Rust tree-sitter-rust
lang-python Python tree-sitter-python
lang-javascript JavaScript tree-sitter-javascript
lang-typescript TypeScript tree-sitter-typescript
lang-go Go tree-sitter-go
lang-java Java tree-sitter-java
lang-csharp C# tree-sitter-c-sharp
lang-kotlin Kotlin tree-sitter-kotlin-ng
lang-swift Swift tree-sitter-swift
lang-php PHP tree-sitter-php
lang-ada Ada tree-sitter-ada
lang-fortran Fortran (free-form) tree-sitter-fortran
lang-scala Scala tree-sitter-scala
lang-lua Lua tree-sitter-lua
all-languages All of the above —

A consumer that only cares about C/C++, for example, would declare:

lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }

Usage

# Cargo.toml — full language set (default)
lang-parsing-substrate = "0.5"

# Cargo.toml — C/C++ only
lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }

use lang_parsing_substrate::{language_for_file, languages, supported_languages_report};
use std::path::Path;

// Detect language for a file
if let Some(lang) = language_for_file(Path::new("main.c")) {
    let mut parser = tree_sitter::Parser::new();
    parser.set_language(&lang).unwrap();
    // parse...
}

// Enumerate compiled-in languages (reflects feature flags)
for info in languages() {
    println!("{}: {:?}", info.name, info.extensions);
}

// Human-readable summary (for --supported-languages flags)
print!("{}", supported_languages_report());

Grammar crates are re-exported so consumers reach them transitively:

// No direct tree-sitter-rust dependency needed in your Cargo.toml
use lang_parsing_substrate::tree_sitter_rust;

Analysis primitives

use lang_parsing_substrate::{call_edges, import_sources, build_function_cfg, structural_hash};

// Call-graph edges for every named function/macro in a parsed file
let edges = call_edges(tree.root_node(), source);

// Import/use-statement sources, for Ce/Ca coupling metrics
let imports = import_sources(&tree, source.as_bytes(), "rust");

// Control-flow graph for a single function body (c/cpp/rust)
if let Some(cfg) = build_function_cfg(func_node, source, "rust") {
    println!("{} basic blocks", cfg.block_count());
}

// Best-effort lower bound on the C standard a file requires
if let Some(standard) = detect_min_c_standard(&tree, source.as_bytes()) {
    println!("requires at least {standard:?}");
}

API

  • languages() -> &'static [LanguageInfo] — compiled-in language set
  • language_for_file(path: &Path) -> Option<Language> — grammar dispatch by extension
  • language_for_key(key: &str) -> Option<Language> — grammar dispatch by registry key
  • language_info_for_file(path: &Path) -> Option<&'static LanguageInfo>
  • is_source_extension / is_parseable_extension(ext: &OsStr) -> bool — recursive-discovery gates
  • supported_languages_report() -> String — human-readable language summary
  • LanguageInfo / SlocMode — registry metadata and comment-style enum (drives SLOC calculation)
  • find_descendants / find_first_descendant / find_ancestor / node_text and friends — traversal helpers (query)
  • import_sources / distinct_import_count — import extraction (imports)
  • call_edges / CallEdge / is_function_kind / get_function_name — call-graph extraction (calls)
  • build_function_cfg / FunctionCfg / BasicBlock / CfgEdge — control-flow graphs (cfg)
  • detect_min_c_standard / CStandard — C standard lower-bound detection (c_standard)
  • function_fingerprints / duplicate_groups / Fingerprint / CorpusFingerprint — structural hashing (fingerprint)
  • ignored_regions / IgnoredRegion — tools:off/tools:on markers (regions)
  • suppressions / Suppression — tools:suppress comments (suppressions)
  • dead_code_ranges / DeadCodeRegion / DeadCodeReason — preprocessor dead-code regions, C/C++ only (dead_code)
  • swift_dead_code_regions / SwiftDeadCodeRegion — Swift #if/#elseif/#else dead-code regions (dead_code_swift)
  • csharp_dead_code_regions / CSharpDeadCodeRegion — C# #if/#elif/#else dead-code regions (dead_code_csharp)
  • PathIgnore — compiled glob ignore sets (path_ignore)

Building

cargo build                                          # all languages (default)
cargo build --no-default-features --features lang-c,lang-cpp  # subset
cargo test

Requires no C compiler — tree-sitter grammar crates ship pre-generated C sources and compile via the cc crate.

Python bindings

The pyo3 Cargo feature (off by default) exposes the substrate's language-agnostic analysis primitives as a Python extension module, published to PyPI as prebuilt abi3 wheels (CPython 3.10+, one wheel per platform — see docs/releasing.md):

pip install lang-parsing-substrate

Building it yourself from source uses maturin:

pip install maturin
maturin build --release          # writes target/wheels/lang_parsing_substrate-*.whl
pip install target/wheels/lang_parsing_substrate-*.whl

import lang_parsing_substrate as lps

src = "fn helper(x: i32) -> i32 { x + 1 }\nfn main() { helper(41); }\n"
edges = lps.call_edges("rust", src)          # [CallEdge(caller='main', callee='helper', ...)]
cfg = lps.function_cfg("rust", src, "main")  # FunctionCfg | None
fps = lps.function_fingerprints("rust", src, min_nodes=1)

Python can't hand this crate a tree_sitter::Node/Tree directly — this crate's tree-sitter version has no ABI relationship to tree-sitter's own, separate Python bindings — so every bound function takes (language_key, source), parses internally, and returns owned data (CallEdge, FunctionCfg, Fingerprint, Suppression, IgnoredRegion, LanguageInfo, all plain attribute-holding classes). A consumer that also needs to walk the tree itself (e.g. for domain-specific semantics this crate doesn't model) still parses separately with a language-specific tree-sitter Python package; the bindings here only cover the substrate's own primitives.

invoke build-wheel builds the wheel locally for testing. The actual PyPI release happens in CI on a vX.Y.Z tag push, via Trusted Publishing (OIDC, no API token) — see docs/releasing.md.

License

MIT — see LICENSE.

Project links

Data verified by PyPI on Aug 26, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.9.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for szarta from gravatar.com szarta

Credits

Author: Brandon Arrendondo

GitHub Statistics

Data verified by PyPI on Aug 26, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.10

Tags

tree-sitter parsing language-detection static-analysis

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lang_parsing_substrate-0.6.0.tar.gz (108.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lang_parsing_substrate-0.6.0-cp310-abi3-win_amd64.whl (3.2 MB view details)

Uploaded CPython 3.10+Windows x86-64

lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_aarch64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

lang_parsing_substrate-0.6.0-cp310-abi3-macosx_11_0_arm64.whl (3.6 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

lang_parsing_substrate-0.6.0-cp310-abi3-macosx_10_12_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file lang_parsing_substrate-0.6.0.tar.gz.

File metadata

  • Download URL: lang_parsing_substrate-0.6.0.tar.gz
  • Upload date:
  • Size: 108.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for lang_parsing_substrate-0.6.0.tar.gz
Algorithm Hash digest
SHA256 14af613df593e1d605c3b83e5b59474f0d5b342d089638599b6ab90ca6c61da7
MD5 e106c5949fe5ffc8276559626e6948f4
BLAKE2b-256 dfa4a3f01aa7fbf6fdf8010cf39eb776e7f4353f9e4cec0b84360f8028e17b67

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 6cf1f372b73f6c78fad9168bdb10452c67f93435bfad4f006d5c56f7de72ff36
MD5 ee19d151adebfea2f0beec6a7b6d2325
BLAKE2b-256 3a442e124b796e6558f8218af55f1560132980af20a9733c29a177823728e49c

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 d9362f55521695f90e660536f758e9a61d1e98f6dad9bce20a527e574df43dc5
MD5 30d9b9e2795038352233bab9724f461b
BLAKE2b-256 80a10b1701e19b6c544f463d7bff4013a72fb32696871693aeb93b9b214ce591

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 d3bb8c8823c8bb8e9cd312369ee99b94bd36dfc91c3c4b92ea8f3a6fdf9331cb
MD5 a9cb8cc969a72b6bd7338149d6510602
BLAKE2b-256 5f12c357862c17851bd23eed21da1c552c85a1a608add5351b3e0f8ea54da1fc

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c07ed5fd54de33e4e5a23a90925f1538ea6b19aa17cdcb35368dc095bd29dc58
MD5 73818e5f013d5e8e5fc47f33b40b0333
BLAKE2b-256 69deb2c1cc55d6770a39ec28558515cc6a75706914795be53e6653b3eeb38993

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 0f6b7826ca57dc0ff98a9519537278933aae563f182d19f537ff8003ec29535f
MD5 b1d43569a9ce6f860ab1920de093444f
BLAKE2b-256 15123a656f6c4c01d53d02d55b782e42675826fff7ac02feeb2edd3bdae8b749

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6e2eb8271b8c66869bfca70f9a3d8c9bffdf6df2817eae7ebbf7f18c358470c6
MD5 e3700c7785a78916c7172faa4afe3cfe
BLAKE2b-256 766c0ad0660a6e96e29f3fa36717fd1502c74f6001d2db56d8c2244195c93961

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.6.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.6.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 f72acad76d3906b7aa85ec5e476ed10cc5471d66b3ad08b18ce45ce0816cd69f
MD5 b80456389573586cea7b2ee738648481
BLAKE2b-256 3cda979073150fccaa050d34798f43f6d5df0642abe5cdb1450a9cf17746c605

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.6.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.9.0

8 files

0.8.0

8 files

0.7.1

8 files

0.7.0

8 files

0.6.2

8 files

0.6.1

8 files

This release

0.6.0 This release

8 files

0.5.2

8 files

0.5.1

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page