stump-cli 0.2.0

Stump: AST-aware code chunking (tree-sitter). Derived from ASTChunk; cite the cAST paper for research.

Stump

Stump is an AST-aware code chunking tool and library: it splits source using tree-sitter parse trees so chunks respect syntactic boundaries. The name is a nod to what is left when you “cut” a syntax tree into pieces.

This project is actively developed by DevAldrete on top of the ASTChunk research codebase. Original algorithms and the scientific credit belong to the ASTChunk authors; Stump adds its own packaging, CLI, and ongoing changes. See NOTICE and LICENSE for copyright and derivative-work attribution.

Cite the original research

The method is described in the cAST paper. Please cite it when you use Stump in academic work:

cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree
Yilin Zhang, Xinran Zhao, Zora Zhiruo Wang, Chenyang Yang, Jiayi Wei, Tongshuang Wu

Bibtex for citations:

@misc{zhang-etal-2025-astchunk,
      title={cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree}, 
      author={Yilin Zhang and Xinran Zhao and Zora Zhiruo Wang and Chenyang Yang and Jiayi Wei and Tongshuang Wu},
      year={2025},
      url={https://arxiv.org/abs/2506.15655}, 
}

Installation

From PyPI (after you publish the stump package):

pip install stump

From source:

git clone https://github.com/DevAldrete/stump.git
cd stump
pip install -e .

If your GitHub repository is still named differently (for example astchunk-cli), use that clone URL and directory name, and align Homepage / Repository in pyproject.toml with the real URL.

After install, the CLI is stump (stump --help). You can also run python main.py from the repository root when dependencies are available (the shim prepends src/ for imports).

Upstream library: the original project remains at github.com/yilinjz/astchunk and may publish under a different PyPI name; this line is independent.

Docker

Build and run the CLI in a container (mount your code to chunk a directory on the host):

docker build -t stump .
docker run --rm stump --help
docker run --rm -v "$PWD":/work -w /work stump chunk-repo . -o /work/chunks.json

Building wheels for PyPI

With the optional publish dependencies (pip install '.[publish]'), from the repo root:

python -m build
twine check dist/*
# twine upload dist/*

Stump depends on tree-sitter for parsing. The required language parsers are installed with the package:

# Core dependencies (automatically installed)
pip install numpy pyrsistent tree-sitter
pip install tree-sitter-python tree-sitter-java tree-sitter-c-sharp tree-sitter-typescript

Configuration Options

• max_chunk_size: Maximum non-whitespace characters per chunk
• language: Programming language for parsing
• metadata_template: Format for chunk metadata
• repo_level_metadata (optional): Repository-level metadata (e.g., repo name, file path)
• chunk_overlap (optional): Number of AST nodes to overlap between chunks
• chunk_expansion (optional): Whether to perform chunk expansion (i.e., add metadata headers to chunks)
• chunk_strategy (optional): size (default, greedy AST windows), definition (one top-level definition per chunk; may exceed max_chunk_size), or hybrid (definition boundaries with greedy splitting inside oversized definitions)

Symbol metadata (default and coderagbench-repoeval templates)

Chunks include symbols (simple names) and symbol_count. A name is listed when its tree-sitter definition overlaps the chunk’s line span (inclusive). Line numbers in metadata use tree-sitter’s 0-based row indices, matching start_line_no / end_line_no. Duplicate simple names from different scopes appear once in symbols.

CLI: pass --chunk-strategy size|definition|hybrid to chunk and chunk-repo.

Quick Start

from stump import ASTChunkBuilder

# Your source code
code = """
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

class Calculator:
    def add(self, a, b):
        return a + b
    
    def multiply(self, a, b):
        return a * b
"""

# Initialize the chunk builder
configs = {
    "max_chunk_size": 100,             # Maximum non-whitespace characters per chunk
    "language": "python",              # Supported: python, java, csharp, typescript
    "metadata_template": "default"     # Metadata format for output
}
chunk_builder = ASTChunkBuilder(**configs)

# Create chunks
chunks = chunk_builder.chunkify(code)

# Each chunk contains content and metadata
for i, chunk in enumerate(chunks):
    print(f"[Chunk {i+1}]")
    print(f"{chunk['content']}")
    print(f"Metadata: {chunk['metadata']}")
    print("-" * 50)

Advanced Usage

Customizing Chunk Parameters

# Add repo-level metadata
configs['repo_level_metadata'] = {
    "filepath": "src/calculator.py"
}

# Enable overlapping between chunks
configs['chunk_overlap'] = 1

# Add chunk expansion (metadata headers)
configs['chunk_expansion'] = True

# NOTE: max_chunk_size apply to the chunks before overlapping or chunk expansion.
# The final chunk size after overlapping or chunk expansion may exceed max_chunk_size.


# Extend current code for illustration
code += """
def divide(self, a, b):
    if b == 0:
        raise ValueError("Cannot divide by zero")
    return a / b

# This is a comment
# Another comment

def subtract(self, a, b):
    return a - b

def exponent(self, a, b):
    return a ** b
"""


# Create chunks
chunks = chunk_builder.chunkify(code, **configs)

for i, chunk in enumerate(chunks):
    print(f"[Chunk {i+1}]")
    print(f"{chunk['content']}")
    print(f"Metadata: {chunk['metadata']}")
    print("-" * 50)

Working with Files

# Process a single file
with open("example.py", "r") as f:
    code = f.read()

# Alternatively, you can also create single-use configs for the optional arguments for each chunkify() call
single_use_configs = {
    "repo_level_metadata": {
        "filepath": "example.py"
    },
    "chunk_expansion": True
}

chunks = chunk_builder.chunkify(code, **single_use_configs)

# Save chunks to separate files
for i, chunk in enumerate(chunks):
    with open(f"chunk_{i+1}.py", "w") as f:
        f.write(chunk['content'])

Processing Multiple Languages

# Python code
python_builder = ASTChunkBuilder(
    max_chunk_size=1500,
    language="python",
    metadata_template="default"
)

# Java code  
java_builder = ASTChunkBuilder(
    max_chunk_size=2000,
    language="java", 
    metadata_template="default"
)

# TypeScript code
ts_builder = ASTChunkBuilder(
    max_chunk_size=1800,
    language="typescript",
    metadata_template="default"
)

Supported Languages

Language	File Extensions	Status
Python	.py	✅ Full support
Java	.java	✅ Full support
C#	.cs	✅ Full support
TypeScript	.ts, .tsx	✅ Full support
Go	.go	✅ Full support
Rust	.rs	✅ Full support
JavaScript	.js, .jsx, .mjs, .cjs	✅ Full support

License

MIT License. Attribution for the original ASTChunk work and Stump changes is summarized in NOTICE; the full legal text is in LICENSE.

Version

Current version: 0.1.0