codecompass-ai 0.1.0

AI-powered repository onboarding assistant

CodeCompass

AI-Powered Repository Onboarding Assistant

Setup

RAG

Retrieval Strategy Evaluation

Evaluated retrieval strategies on 19 test queries across 5 categories (feature search, concept search, API search, debug search, natural language).

Results

Strategy	Recall@5	Precision@5	MRR
HyDE	0.733	0.295	0.671
Query Expansion	0.718	0.295	0.737
Baseline (vector search)	0.644	0.263	0.754
Query Expansion + Context	0.428	0.189	0.399

HyDE (Hypothetical Document Embedding) performed best, improving recall by 13.9% over baseline. This approach generates hypothetical code matching the query, then searches for similar real code.

Context-Aware Expansion: Negative Result

The context-aware strategy (providing repo imports to the LLM) performed surprisingly poorly. I hypothesized the prompt was suboptimal, so I tested 4 variations:

Variant	Description	Recall@5
v4: Rule-based	No LLM, keyword matching	0.691
v2: Pick from list	LLM selects relevant imports	0.665
v3: Minimal prompt	Simple prompt	0.644
v1: Fewer imports	15 imports instead of 100	0.600
Original	100 imports, verbose prompt	0.428

Finding: Even the best context variant (rule-based, no LLM) underperformed simple query expansion. Adding repository context introduces noise rather than helping retrieval.

Takeaway: Simpler strategies (HyDE, query expansion) outperform complex context-aware approaches for code search.

Finetuning

I considered finetuning with xlam-60k from salesforce but after running an tool-calling evaluation script on base model with curated dataset, it already does a good job. So we only finetune for better code explanation.

Considred datasets like CodeSearchNet, code_x_glue_ct_code_to_text but found Magicoder-OSS-Instruct-75K to have the best detailed yet concise explanations suitable for our case. Since we expect user queries to be more abstract questions where code explanation is not the only response to the query, in fact it's only for a subset of most queries, I decided to use a two-pass architecture with hot-swap adapaters, using only the fine-tuned model for explanation portion so we don't see performance degradation for other tasks.

│ User Query │ │ ↓ │ │ [Base Model] → Tool selection (95% accurate, no training) │ │ ↓ │ │ [Tools] → Code chunks │ │ ↓ │ │ [Fine-tuned Model] → Explanations only ← TRAIN THIS ONLY │ │ ↓ │ │ [Base Model] → Final synthesized answer

And the user shouldn't be downloading two full qwen models so I decided to fine tune with LoRA.