chunkhive 0.1.0

Hierarchical, semantic code chunking for AI systems

chunkhive

Semantic, hierarchical code chunking for AI systems

Chunkhive is a production-grade code chunking engine designed for modern AI workflows such as code embeddings, retrieval-augmented generation (RAG), agentic systems, and dataset synthesis.

It converts raw repositories into clean, structured, semantically accurate chunks with byte-level precision and preserved hierarchy.

---

🚀 Why chunkhive?

Modern AI systems need more than naive text splitting.

chunkhive provides:

• AST-first semantic correctness
• Hierarchical structure awareness
• Byte-accurate spans
• Robust parsing across real-world repositories

---

🧠 Core Principle

AST is the Authority, Tree-sitter is Enrichment

• Primary source of truth: Language AST (semantic accuracy)
• Fallback & enrichment: Tree-sitter (structural robustness)
• Result: Maximum parsing success across diverse codebases

---

✨ Features

• Semantic AST-first chunking (no filename-based chunks)
• Preserves hierarchy: Module → Class → Method / Function
• Accurate parent–child relationships
• Byte-level precision (start_byte, end_byte)
• Clean symbol naming (ast.name)
• Import & decorator capture
• Robust handling of edge cases (empty files, __init__.py)
• Supports documentation + code chunking flows

---

🗂 Supported Chunk Types

• module
• class
• method
• function
• documentation

---

📦 Installation

pip install chunkhive


---

## 📦 Output Schema (Simplified)

```json
{
  "chunk_id": "ts_123_456",
  "file_path": "src/example.py",
  "chunk_type": "function",
  "language": "python",
  "code": "...",
  "ast": {
    "name": "my_function",
    "parent": "MyClass"
  },
  "span": {
    "start_byte": 123,
    "end_byte": 456,
    "start_line": 10,
    "end_line": 25
  },
  "metadata": {
    "byte_accuracy": "exact_bytes"
  }
}

🛠 Use Cases

Code embedding model training

RAG pipelines

Agentic AI systems

Code search & navigation

QA dataset generation

Static analysis & tooling

📜 License

Apache License 2.0 — free to use, modify, and distribute, including commercial use.