ASTAligner 0.8.0

ASTAligner is designed to align tokens from source code snippets to Abstract Syntax Tree (AST) nodes using Tree-sitter for AST generation and various HuggingFace tokenizers for language tokenization. The library supports a wide range of programming languages and Fast tokenizers, enabling precise mapping between source code elements and their AST representations.

AST-Alignment Tool

Aligns the tokens from a code snippet to their corresponding nodes in an AST representation.

Description

A Large Language Model (LLM) is a type of AI model designed to understand and generate human-like text based on vast amounts of data. Trained on diverse source code datasets, LLMs can automate Software Engineering tasks across various contexts, such as code translation, code summarization, test-case generation, and code completion. A critical component of LLMs is the tokenizer, which breaks down text into smaller units, typically words or subwords, that the model can process. The tokenizer's role is essential because it converts source code into a format the model can understand, ensuring efficient and accurate code processing and generation. In the context of Interpretability for AI, post-hoc techniques such as ASTScore, rely on alienation functions (phi) to match the tokens generated by an LLM’s tokenizer with their corresponding nodes in the AST representation of a snippet.

Goals

This project has two goals:

(1) Create a library for aligning the tokens from a code snippet to their corresponding nodes in the AST representation

(2) Create a tool to visualize the alignment of the tokens with their matching AST.

Additional Information

For more information regarding this project's background and dependencies, please refer to these readings:

(1) Evaluating and Explaining Large Language Models for Code Using Syntactic Structures

(2) Tree-Sitter Programming Language Parser

(3) Hugging Face Tokenizer

Installation

Use the package manager pip to install all backend dependencies needed for the AST-Alignment Tool. All required packages for the backend can be downloaded using requirements.txt, which can found in the base repository.

pip install -r /path/to/requirements.txt

Supported Features

• 11 supported languages
  • Python
  • C
  • C++
  • C#
  • Java
  • JavaScript
  • Ruby
  • HTML
  • GO
  • Kotlin
  • Rust
• 6 Tokenizers
  • Bert-Base-Uncased
  • CodeLlama
  • GPT2
  • DialoGPT
  • Roberta-Base
  • Qwen

Library Usage

ASTalign

Using the ASTalign method asks that the user provide:

• A snippet of code as a string, or a filepath to a text file containing code.
• The language of the code snippet as one of the following strings:
  • python
  • c
  • cpp
  • csharp
  • java
  • javascript
  • ruby
  • html
  • go
  • kotlin
  • rust
  • haskell
• A tokenizer specification as one of the following strings:
  • codellama
  • gpt2
  • bert-base-uncased
  • roberta-base
  • dialogpt
  • qwen
• (Optional) include_whitespace_and_special_tokens flag. Set to False by default, this flag allows the user to specify whether or not to show whitespaces and special characters in the tokens.

The method returns a dictionary of TSTree nodes to a list of tokens from the code snippet that overlap with those nodes. Example usage:

alignments[node] yields [tok1, tok2, ... , tokn]

Contributing

License

MIT