ChunkHive

Hierarchical, semantic code chunking with embeddings for AI systems.

ChunkHive is a production-grade code chunking engine designed for modern AI workflows such as code embeddings, retrieval-augmented generation (RAG), agentic systems, and dataset synthesis.

It converts raw repositories into clean, structured, semantically accurate chunks with byte-level precision and preserved hierarchy.

---

🚀 Why ChunkHive?

Modern AI systems need more than naive text splitting.

chunkhive provides:

• AST-first semantic correctness
• Hierarchical structure awareness
• Byte-accurate spans
• Robust parsing across real-world repositories

---

🧠 Core Principle

AST is the Authority, Tree-sitter is Enrichment

• Primary source of truth: Language AST (semantic accuracy)
• Fallback & enrichment: Tree-sitter (structural robustness)
• Result: Maximum parsing success across diverse codebases

---

✨ Features

• Semantic AST-first chunking (no filename-based chunks)
• Preserves hierarchy: Module → Class → Method / Function
• Accurate parent–child relationships
• Byte-level precision (start_byte, end_byte)
• Clean symbol naming (ast.name)
• Import & decorator capture
• Robust handling of edge cases (empty files, __init__.py)
• Supports documentation + code chunking flows

✨ New Features

• 🌳 Hierarchical Chunking - Preserves code structure
• 🧠 Semantic Embeddings - Turn code into vectors (NEW in v0.3.1!)
• 🔍 Vector Search - Find code by meaning, not just names
• 🤖 4 Pre-trained Models - CodeBERT, UniXcoder, and more
• ⚡ Fast & Scalable - Process large codebases efficiently

Semantic Search (NEW!)

# Generate embeddings
chunkhive embed generate chunks.jsonl -o chunks_embedded.jsonl

# Generate with model choice
chunkhive embed generate chunks.jsonl --model unixcoder -o embedded_unixcoder.jsonl

# Create index
chunkhive embed index data\embeddings\chunks_embedded -o chunks_embedded.jsonl

# search query :
chunkhive embed search-index "your query" --index data\embeddings\crewai_index -k 2

# List available models
chunkhive embed models

# Search semantically ( for debugging,testing etc)
chunkhive embed search "authentication function" --chunks chunks_embedded.jsonl

What's New in v0.3.1

🎉 Semantic Embeddings!

• Generate vector embeddings for code chunks
• 4 pre-trained models (CodeBERT, UniXcoder, CodeSage, MiniLM)
• Fast similarity search with FAISS
• Natural language code search

---

🔄 Multi-Language Support

Currently Supported:

Python: Full AST parsing with decorators, imports, docstrings

Markdown/RST: Documentation chunking with code block detection

Configuration Files: JSON, YAML, TOML, INI, Dockerfiles

Text Files: README, LICENSE, requirements.txt, scripts

🔄 Coming Soon:

JavaScript/TypeScript

C++/Java/Go

🗂 Supported Chunk Types

module

class

method

function

documentation

configuration (JSON, YAML, TOML)

text

imports

🏢 Production Features

Deterministic IDs: Same code → same chunk ID across runs

Progress Indicators: Real-time processing feedback

Error Resilience: Graceful handling of malformed code

Statistics Generation: Detailed analytics and metrics

Batch Processing: Process multiple repositories from config file

Permission Handling: Intelligent output path resolution

📦 Installation

pip install chunkhive

Quick Start

Basic usage (creates in current directory)

chunkhive chunk local ./my_project

With output directory

chunkhive chunk local ./my_project -o ./output

With custom name and statistics

chunkhive chunk local ./my_project --name my_dataset --stats

chunkhive chunk repo https://github.com/user/repo --name my_dataset --stats

Clone and chunk any GitHub repository

chunkhive chunk repo https://github.com/user/repo

With filtering and limits

chunkhive chunk repo https://github.com/langchain-ai/langchain
--extensions .py,.md
--max-files 100
--name langchain_chunks

Single File Processing

chunkhive chunk file example.py

chunkhive chunk file example.py -o ./chunks.jsonl --stats

Repository Analysis

Analyze repository metadata

chunkhive analyze https://github.com/crewAIInc/crewAI

chunkhive analyze ./local/repo --output analysis.json

Show Examples

chunkhive examples

Check Version & Info

chunkhive version # Show current version

chunkhive info # Show system information

📦 Output Schema (Simplified)

{
  "chunk_id": "primary_a1b2c3d4",
  "file_path": "src/example.py",
  "chunk_type": "function",
  "language": "python",
  "code": "...",
  "ast": {
    "name": "my_function",
    "parent": "MyClass",
    "symbol_type": "function",
    "docstring": "Function documentation",
    "decorators": ["@decorator"],
    "imports": ["import module"]
  },
  "span": {
    "start_byte": 123,
    "end_byte": 456,
    "start_line": 10,
    "end_line": 25
  },
  "hierarchy": {
    "parent_id": "parent_chunk_id",
    "children_ids": ["child1", "child2"],
    "depth": 2,
    "is_primary": true
  },
  "metadata": {
    "byte_accuracy": "exact_bytes",
    "repo_info": {
      "agentic_detection": {"langchain": "usage"},
      "dependencies": {"python_packages": ["pandas", "numpy"]},
      "git": {"remote_url": "https://github.com/user/repo"},
      "structure": {"file_types": {".py": 50, ".md": 10}}
    },
    "repository_context": {
      "similar_files": ["src/other.py"],
      "total_similar_files": 5
    }
  }
}

🛠 Use Cases

Code embedding model training

RAG pipelines

Agentic AI systems

Code search & navigation

QA dataset generation

Static analysis & tooling

Enterprise codebase intelligence

AI training data generation

📜 License

Apache License 2.0 — free to use, modify, and distribute, including commercial use.