Skip to main content

lang-parsing-substrate

A shared language-parsing substrate for static-analysis tools: tree-sitter grammar dispatch, language detection, and a growing set of language-agnostic analysis primitives (import/call graphs, control-flow graphs, structural fingerprinting, suppression comments) built on top of a unified LanguageInfo registry across 16 languages — compiled in at build time via Cargo feature flags.

What's in the substrate

Module Provides
registry Language detection by extension, the LanguageInfo table, SLOC comment-style metadata
query Iterative (non-recursive) tree-sitter traversal helpers: find_descendants, find_first_descendant, node_text, ancestor lookups
imports Per-file import/use-statement extraction, for building efferent-coupling (Ce) edges
calls Per-file call-graph edge extraction (callercallee), with external-call detection
cfg Control-flow graph / basic-block construction for a function body (c, cpp, rust)
c_standard Best-effort lower bound on the C standard (C99/C11/C23) a file's syntax requires
fingerprint Structural hashing of function-like subtrees, for duplicate/clone detection across a corpus
regions tools:off / tools:on ignored-region markers
suppressions tools:suppress TOOL:RULE single-line suppression comments
path_ignore Compiled glob ignore-pattern sets for path filtering

Everything below the registry is deliberately per-file: a module extracts what one parse tree contains, and leaves assembling a corpus-wide graph, dedup report, or coupling metric to the caller. This keeps the substrate's job narrow (one authoritative, correct answer per file) and lets each consumer choose its own storage model (in-memory, SQLite, whatever) without the substrate needing to know about it.

Language coverage varies by module — the registry knows about all 16 languages, but modules like cfg only model the languages they've been built out for. A module never fabricates a result for a language it doesn't support; it returns None (or an empty result) instead of guessing.

Feature flags

Not every consumer needs all 16 languages. Each language is an optional Cargo feature; the all-languages convenience feature (enabled by default) pulls in the full set.

Feature Language Grammar crate
lang-c C tree-sitter-c
lang-cpp C++ tree-sitter-cpp
lang-rust Rust tree-sitter-rust
lang-python Python tree-sitter-python
lang-javascript JavaScript tree-sitter-javascript
lang-typescript TypeScript tree-sitter-typescript
lang-go Go tree-sitter-go
lang-java Java tree-sitter-java
lang-csharp C# tree-sitter-c-sharp
lang-kotlin Kotlin tree-sitter-kotlin-ng
lang-swift Swift tree-sitter-swift
lang-php PHP tree-sitter-php
lang-ada Ada tree-sitter-ada
lang-fortran Fortran (free-form) tree-sitter-fortran
lang-scala Scala tree-sitter-scala
lang-lua Lua tree-sitter-lua
all-languages All of the above

A consumer that only cares about C/C++, for example, would declare:

lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }

Usage

# Cargo.toml — full language set (default)
lang-parsing-substrate = "0.5"

# Cargo.toml — C/C++ only
lang-parsing-substrate = { version = "0.5", default-features = false, features = ["lang-c", "lang-cpp"] }
use lang_parsing_substrate::{language_for_file, languages, supported_languages_report};
use std::path::Path;

// Detect language for a file
if let Some(lang) = language_for_file(Path::new("main.c")) {
    let mut parser = tree_sitter::Parser::new();
    parser.set_language(&lang).unwrap();
    // parse...
}

// Enumerate compiled-in languages (reflects feature flags)
for info in languages() {
    println!("{}: {:?}", info.name, info.extensions);
}

// Human-readable summary (for --supported-languages flags)
print!("{}", supported_languages_report());

Grammar crates are re-exported so consumers reach them transitively:

// No direct tree-sitter-rust dependency needed in your Cargo.toml
use lang_parsing_substrate::tree_sitter_rust;

Analysis primitives

use lang_parsing_substrate::{call_edges, import_sources, build_function_cfg, structural_hash};

// Call-graph edges for every named function/macro in a parsed file
let edges = call_edges(tree.root_node(), source);

// Import/use-statement sources, for Ce/Ca coupling metrics
let imports = import_sources(&tree, source.as_bytes(), "rust");

// Control-flow graph for a single function body (c/cpp/rust)
if let Some(cfg) = build_function_cfg(func_node, source, "rust") {
    println!("{} basic blocks", cfg.block_count());
}

// Best-effort lower bound on the C standard a file requires
if let Some(standard) = detect_min_c_standard(&tree, source.as_bytes()) {
    println!("requires at least {standard:?}");
}

API

  • languages() -> &'static [LanguageInfo] — compiled-in language set
  • language_for_file(path: &Path) -> Option<Language> — grammar dispatch by extension
  • language_for_key(key: &str) -> Option<Language> — grammar dispatch by registry key
  • language_info_for_file(path: &Path) -> Option<&'static LanguageInfo>
  • is_source_extension / is_parseable_extension(ext: &OsStr) -> bool — recursive-discovery gates
  • supported_languages_report() -> String — human-readable language summary
  • LanguageInfo / SlocMode — registry metadata and comment-style enum (drives SLOC calculation)
  • find_descendants / find_first_descendant / find_ancestor / node_text and friends — traversal helpers (query)
  • import_sources / distinct_import_count — import extraction (imports)
  • call_edges / CallEdge / is_function_kind / get_function_name — call-graph extraction (calls)
  • build_function_cfg / FunctionCfg / BasicBlock / CfgEdge — control-flow graphs (cfg)
  • detect_min_c_standard / CStandard — C standard lower-bound detection (c_standard)
  • function_fingerprints / duplicate_groups / Fingerprint / CorpusFingerprint — structural hashing (fingerprint)
  • ignored_regions / IgnoredRegiontools:off/tools:on markers (regions)
  • suppressions / Suppressiontools:suppress comments (suppressions)
  • PathIgnore — compiled glob ignore sets (path_ignore)

Building

cargo build                                          # all languages (default)
cargo build --no-default-features --features lang-c,lang-cpp  # subset
cargo test

Requires no C compiler — tree-sitter grammar crates ship pre-generated C sources and compile via the cc crate.

Python bindings

The pyo3 Cargo feature (off by default) exposes the substrate's language-agnostic analysis primitives as a Python extension module, published to PyPI as prebuilt abi3 wheels (CPython 3.10+, one wheel per platform — see docs/releasing.md):

pip install lang-parsing-substrate

Building it yourself from source uses maturin:

pip install maturin
maturin build --release          # writes target/wheels/lang_parsing_substrate-*.whl
pip install target/wheels/lang_parsing_substrate-*.whl
import lang_parsing_substrate as lps

src = "fn helper(x: i32) -> i32 { x + 1 }\nfn main() { helper(41); }\n"
edges = lps.call_edges("rust", src)          # [CallEdge(caller='main', callee='helper', ...)]
cfg = lps.function_cfg("rust", src, "main")  # FunctionCfg | None
fps = lps.function_fingerprints("rust", src, min_nodes=1)

Python can't hand this crate a tree_sitter::Node/Tree directly — this crate's tree-sitter version has no ABI relationship to tree-sitter's own, separate Python bindings — so every bound function takes (language_key, source), parses internally, and returns owned data (CallEdge, FunctionCfg, Fingerprint, Suppression, IgnoredRegion, LanguageInfo, all plain attribute-holding classes). A consumer that also needs to walk the tree itself (e.g. for domain-specific semantics this crate doesn't model) still parses separately with a language-specific tree-sitter Python package; the bindings here only cover the substrate's own primitives.

invoke build-wheel builds the wheel locally for testing. The actual PyPI release happens in CI on a vX.Y.Z tag push, via Trusted Publishing (OIDC, no API token) — see docs/releasing.md.

License

MIT — see LICENSE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lang_parsing_substrate-0.5.2.tar.gz (91.0 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lang_parsing_substrate-0.5.2-cp310-abi3-win_amd64.whl (3.2 MB view details)

Uploaded CPython 3.10+Windows x86-64

lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_x86_64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_aarch64.whl (3.6 MB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

lang_parsing_substrate-0.5.2-cp310-abi3-macosx_11_0_arm64.whl (3.6 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

lang_parsing_substrate-0.5.2-cp310-abi3-macosx_10_12_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file lang_parsing_substrate-0.5.2.tar.gz.

File metadata

  • Download URL: lang_parsing_substrate-0.5.2.tar.gz
  • Upload date:
  • Size: 91.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for lang_parsing_substrate-0.5.2.tar.gz
Algorithm Hash digest
SHA256 1a50cca2e4c01ac0ab4e2f9f086c1b0c2a9d8384f546bcf562d6cde4ac7e32e3
MD5 ebd09f9c84c687ed9c1be8375b594ad2
BLAKE2b-256 f47595953e9718f31880969ceed576427785708758870f248b753a19838b5cad

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2.tar.gz:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 78051e17415ded956f63dc749cabeffdc265094b42e9a8f5517a1f5c35501a63
MD5 43d3418739ab82a0b9978269b60e0d22
BLAKE2b-256 531e9b826ce90537622caf2e667127b5ac143a971fb11be191a2476bd49cf370

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-win_amd64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 49556faed8f004d3adf1667758347d825fcaa22138d7d99aec4aa491cfa241e7
MD5 ec9b1e90cbd69e8107d30b6bccc7ce93
BLAKE2b-256 10514a706c71a670343287a1bc7996bec6df81d91f6c5598fd592b789ca7a73f

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 92e33e493b6bf7e508cb45e0a1ff179f775c1c0f361d40e888c4b879313e4c3c
MD5 f8174b9b36ea70ee4268e0a314538660
BLAKE2b-256 aee083e84df089854b5305bf62a21a64650e1711ae910d7762e4187cb074b214

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-musllinux_1_2_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5190f45b76246e771bbb7e0d417f4ef338abaa6f4f81095478a3e162123850c8
MD5 9e9412ffeb6bedfc5c344ff80a626490
BLAKE2b-256 1942745a64ffc262aa8daf0ed10fd2a24b54849ac72d68b1594e0b9b66f5bf7a

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e14cfaff156645c66ac0eb4e7ae9ca3839d1932dd1705e5fe7648e02e00412c9
MD5 42046f34ce02812387c46fe05c984517
BLAKE2b-256 c65bc9e09fc433d6379a361da6fecdce397d1d23ac68f3c7f8f4a40e47aa56cc

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 339bb8069c97111431af91cae57ee30dde411e0f603c8c4b39aded29e89f22d1
MD5 2e01854411a7c214fc69dd1dfcd4a8c1
BLAKE2b-256 6d0bfb01f7ca46f2039649b896d651c78f3f52495d49a623dba9dece747a8462

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lang_parsing_substrate-0.5.2-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for lang_parsing_substrate-0.5.2-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 f916ab6a5b8c95d7acf7de63c7bb01aba56633f3a5c15d0f0e53d15ff027494b
MD5 2fe78cce87998088d38fb8cf00d95360
BLAKE2b-256 0b686b3b5e782dab58fd1f197f4ec1a0e1741f990461c13bc9f86918f776c1fc

See more details on using hashes here.

Provenance

The following attestation bundles were made for lang_parsing_substrate-0.5.2-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: wheels.yml on brandon-arrendondo/lang_parsing_substrate

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.9.0

8 files

0.8.0

8 files

0.7.1

8 files

0.7.0

8 files

0.6.2

8 files

0.6.1

8 files

0.6.0

8 files

This release

0.5.2 This release

8 files

0.5.1

8 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page