lambda-graphs: Multi-View Code Representation Tools

lambda-graphs aims to generate combined multi-code view graphs that can be used with various types of machine learning models (sequence models, graph neural networks, etc).

Overview

lambda-graphs is a CLI tool and Python library for generating code representations (AST, CFG, DFG) from source code. It supports C, C++, Java, and JavaScript, at both method-level and file-level granularity.

• AST (Abstract Syntax Tree)
• CFG (Control Flow Graph)
• DFG (Data Flow Graph)
• Combined graphs (any combination of the above)

lambda-graphs provides both a CLI and a Python API so you can generate graphs programmatically without shelling out. It is designed to be easily extendable to various programming languages. This is primarily because we use tree-sitter, a highly efficient incremental parser that supports over 40 languages.

Setup

1. Create a conda environment:

conda create -n lambda-graphs python=3.11

2. Activate the environment:

conda activate lambda-graphs

3. Install the package in development mode:

pip install -e .

4. Install GraphViz (Optional - for visualization):

GraphViz is only required if you want to generate DOT, PNG, or SVG output files.

Ubuntu/Debian:

sudo apt install graphviz

MacOS:

brew install graphviz

Windows: Download from graphviz.org

Generating Graphs

Using CLI

After setup, use the lambda-graphs command directly.

The attributes and options supported by the CLI are well documented and can be viewed by running:

lambda-graphs --help

Single File Analysis:

Generate a combined CFG and DFG graph for a C++ file:

lambda-graphs --lang "cpp" --code-file ./test.cpp --graphs "cfg,dfg"

Generate an AST for a C file with output in JSON format:

lambda-graphs --lang "c" --code-file ./example.c --graphs "ast" --output "json"

Folder Analysis (Multi-file Projects):

lambda-graphs can analyze entire projects by combining multiple source files from a folder:

lambda-graphs --lang "c" --code-folder ./project/src --graphs "cfg,dfg" --output "json"

This will:

1. Recursively scan the folder for all .c and .h files
2. Combine them into a single temporary file (preserving includes, declarations, definitions)
3. Generate the requested codeviews from the combined source
4. Output results to the output/ directory

You can customize the combined output file name:

lambda-graphs --lang "cpp" --code-folder ./mylib --combined-name "myproject" --graphs "ast,cfg"

Inline Code Analysis:

You can also analyze code snippets directly without a file:

lambda-graphs --lang "c" --code "int main() { int x = 5; return x; }" --graphs "ast,cfg"

Additional CLI Options:

Flag-Codeview Compatibility:

Examples:

# Generate all output formats (DOT, JSON, PNG)
lambda-graphs --lang "c" --code-file test.c --graphs "cfg" --output "all"

# Collapse duplicate variable nodes in DFG
lambda-graphs --lang "cpp" --code-file test.cpp --graphs "ast" --collapsed

# Add last definition tracking to DFG
lambda-graphs --lang "c" --code-file test.c --graphs "dfg" --last-def

# Exclude specific AST node types
lambda-graphs --lang "c" --code-file test.c --graphs "ast,cfg" --blacklisted "comment,string_literal"

Using the Python API

You can also use lambda-graphs as a library to get graph objects directly (no file I/O):

from lambda_graphs import generate

# -- Generate from a code string -------------------------------------------
result = generate(
    "cpp",
    code="int main() { int x = 5; return x; }",
    graphs=["ast", "cfg", "dfg"],
)

# Each graph is a networkx.MultiDiGraph
print(result.ast.nodes(data=True))
print(result.cfg.nodes(data=True))
print(result.dfg.nodes(data=True))
print(result.combined.nodes(data=True))

# Graph-level metadata (the "graph" key in JSON output)
print(result.combined.graph)  # {"language": "cpp", "views": ["ast", "cfg", "dfg"]}

# -- Generate from a file or folder ------------------------------------------
result = generate("cpp", code_file="./test.cpp", graphs=["cfg", "dfg"])
result = generate("c", code_folder="./project/src", graphs=["cfg", "dfg"])

# -- With extra options ------------------------------------------------------
result = generate(
    "cpp",
    code="...",
    graphs=["ast", "dfg"],
    collapsed=True,                              # collapse duplicate variable nodes
    last_def=True,                               # add last-def info to DFG edges
    blacklisted=["comment", "number_literal"],   # exclude AST node types
)

# -- Export to disk ----------------------------------------------------------
result.to_json("output.json")   # JSON
result.to_dot("output.dot")     # DOT
result.to_png("output.png")     # PNG image
result.to_svg("output.svg")     # SVG image

generate() returns a GraphsResult object with the following attributes:

generate() parameters:

See docs/json-output-format.md for details on the JSON output format.

Limitations

While lambda-graphs provides method-level and file-level support, it's important to note the following limitations and known issues:

General Limitations

• Syntax Errors in Code: To ensure accurate codeviews, the input code must be free of syntax errors. Code with syntax errors may not be correctly parsed and displayed in the generated codeviews. Note that the code does not need to be compilable, only syntactically valid.

C++ Specific Limitations

In addition to the general limitations, the tool has the following limitations specific to C++:

• Limited Template Metaprogramming Support: Complex template metaprogramming patterns may not be fully captured in the generated codeviews.

• Partial Preprocessor Directive Support: Preprocessor directives (e.g., #define, #ifdef) are parsed but not fully processed. Conditional compilation may not be accurately reflected in the codeviews.

• Limited Support for Advanced C++ Features: Some advanced C++ features such as:

  • Complex inheritance hierarchies
  • Multiple inheritance with virtual functions
  • Template specializations
  • SFINAE patterns
  • Concepts (C++20)

  may not be fully represented in the generated codeviews.

Acknowledgments

This tool builds upon the tree-sitter parsing framework and is inspired by research on source code representation learning for AI-driven software engineering tasks.