HasteContext 0.1.0

Parser-backed code-context compression using Tree-sitter

HASTE

Parser-backed code-context compression for Python using Tree-sitter. It builds a structured index of functions/classes, ranks relevant functions for a free‑form query with lexical BM25 (optionally fused with semantic embeddings), expands along the call graph, then assembles a compact, LLM-ready payload. A minimal CLI is included for single-file workflows; the library API supports repository-level indexing and hybrid selection.

Import name is now haste.

Key features

• Hybrid retrieval: BM25 over rich function docs; optional semantic fusion
• Strict top‑k seed selection, BFS expansion over callers/callees
• Identifier TF‑IDF, PageRank on call graph, structure/complexity features
• CAST chunking: byte‑safe, newline‑aligned split/merge with token caps
• JSON payload with selected functions/classes and optional code blob

---

Installation

Using Poetry (recommended)

poetry install

Using pip

python -m venv .venv
. .venv/Scripts/Activate.ps1  # Windows PowerShell
pip install -e .

From PyPI (when published)

pip install haste

Python 3.11+ is required. Core runtime dependencies are declared in pyproject.toml (notably: tree-sitter, tree-sitter-language-pack, tiktoken, numpy, rank-bm25, openai).

Optional: set your OpenAI API key when using semantic reranking or embeddings-backed flows.

$env:OPENAI_API_KEY = "your_key_here"  # Windows PowerShell

---

Quickstart (programmatic)

Use the single-import public API facade for end-to-end flows:

from haste import select_from_file, build_payload_from_repo

# Single file, mirrors CLI output structure (nodes/classes/selected/code)
out = select_from_file(
    "path/to/file.py",
    query="find dataloader and training loop",
    top_k=6,
    bfs_depth=1,
)
print(out["nodes"][:2])
print(out["code"][:500])

# Repository-level payload (index the tree and select relevant code)
payload = build_payload_from_repo(
    "path/to/repo",
    include_code=True,
    top_k=50,
    depth=1,
    query="http handler metrics",
)

This reduces import boilerplate and keeps a stable, public surface.

---

CLI (single Python file)

The minimal CLI operates on a single .py file and prints JSON.

haste path\to\file.py --query "find dataloader and training loop" \
  --top-k 6 --prefilter 300 --bfs-depth 1 --max-add 12 \
  --hard-cap 1200 --soft-cap 1800 [--semantic] [--sem-model text-embedding-3-small]

Flags:

• --query (required): free‑form text
• --top-k: seed size (default 6)
• --prefilter: lexical candidate pool before rerank (default 300)
• --bfs-depth: expansion hops over same‑module call edges (default 1)
• --max-add: cap on nodes added by BFS (default 12)
• --semantic: enable OpenAI embeddings rerank (requires OPENAI_API_KEY)
• --sem-model: embeddings model (default text-embedding-3-small)
• --hard-cap, --soft-cap: CAST token caps used during chunk split/merge

Example output shape:

{
  "summary": {"total_functions": 12, "total_classes": 3},
  "nodes": [ {"type": "function", "name": "train", "qname": "module::train", "path": "...", "lineno": 10, "end_lineno": 120, "signature": "train(cfg)", "docstring": "...", "score": 0.71} ],
  "classes": [ {"type": "class", "name": "DataLoader", "qname": "module::DataLoader", "path": "..."} ],
  "selected": {"roots": ["module::train"], "functions": ["module::train", "module::step"], "classes": ["module::DataLoader"]},
  "code": "...stitched code under token caps..."
}

Also runnable from source without installing the script:

python -m haste.cli path\to\file.py --query "..."

---

Advanced usage (lower-level building blocks)

If you need full control, the lower-level modules remain available (indexing, metrics, selection, assembly). See haste.index, haste.metrics, and haste.selection for granular APIs.

---

How it works

1. Index with Tree‑sitter: collect functions/classes, call edges, decorators, docstrings, variables, and module API hints
2. Score: compute PageRank on the call graph; TF‑IDF over identifiers; cyclomatic complexity and structure richness
3. Retrieve: BM25 over rich function docs; optionally fuse semantic rankings via embeddings + RRF
4. Select: enforce strict top‑k seeds; expand via BFS over callers/callees; re‑rank by fused score
5. Compress: CAST split/merge spans with hard/soft token caps; stitch to a contiguous code blob

---

Requirements & notes

• Python 3.11+
• Tree‑sitter runtime and tree-sitter-language-pack for Python
• OpenAI API key only needed for --semantic or when using OpenAIEmbedder
• This README intentionally ignores pipeline.py, reports/, and ad‑hoc test scripts, which were used only for internal metrics.

---

Contributing

PRs welcome. Use Poetry for the dev environment (poetry install). Run linters/formatters as you normally would; keep public API changes minimal and documented.

License

MIT. See file headers (SPDX identifiers). Add a LICENSE file if distributing publicly.