astchunk-extended 0.2.0

AST-based code chunking with dynamic multi-language support (fork of astchunk)

astchunk-extended

Drop-in replacement for astchunk with dynamic multi-language support.

What's different

• 15 languages instead of 4 (C, C++, Go, Rust, Ruby, Bash, Kotlin, Scala, Lua, PHP, Zig + original Python, Java, C#, TypeScript)
• Dynamic parser discovery — if a tree-sitter parser is installed, it works automatically
• No hardcoded imports — parsers loaded via importlib on demand
• Optional extras — install only the languages you need

Installation

# All languages
pip install astchunk-extended[all]

# Or pick what you need
pip install astchunk-extended[core]        # Python, Java, C#, TypeScript
pip install astchunk-extended[systems]     # C, C++, Go, Rust, Zig
pip install astchunk-extended[scripting]   # Ruby, Bash, Lua, PHP
pip install astchunk-extended[jvm]         # Java, Kotlin, Scala

With LEANN:

uv tool install leann-core --with leann --with "astchunk-extended[all]"
python -c "from astchunk.patch_leann import apply; apply()"

Usage

from astchunk import ASTChunkBuilder, get_available_languages

# See what's installed
print(get_available_languages())
# ['bash', 'c', 'cpp', 'go', 'java', 'kotlin', 'lua', 'php', 'python', 'rust', ...]

# Chunk C++ code
builder = ASTChunkBuilder(max_chunk_size=300, language="cpp", metadata_template="default")
chunks = builder.chunkify(code)

Custom languages

from astchunk import register_language

register_language("haskell", "tree_sitter_haskell")
builder = ASTChunkBuilder(max_chunk_size=300, language="haskell", metadata_template="default")

License

MIT (same as upstream astchunk)