codemine 0.1.0

Add your description here

Codemine

A Python project to embed Git repositories into a Pinecone vector database for semantic search.

Features

• Embeds code chunks from Git repositories.
• Supports incremental updates (remove outdated chunks).
• Uses Tree-sitter for language-aware code splitting.
• Search embedded code chunks using natural language queries.

Prerequisites

• Python >= 3.12
• uv (recommended) or pip

Setup

1. Clone the repository:

   git clone <repository_url>
   cd codemine
   

2. Install dependencies:

   Using uv:

   uv sync
   

   Using pip:

   pip install -e .
   

3. Configure Environment Variables:

   Create a .env file in the root directory with the following variables:

   PINECONE_API_KEY=your_pinecone_api_key
   GITHUB_TOKEN=your_github_token
   OPENAI_API_KEY=your_openai_api_key
   # Optional: Defaults to "https://openrouter.ai/api/v1"
   # OPENAI_BASE_URL=https://api.openai.com/v1
   

Usage

The project installs a codemine CLI command.

Embed a Repository

To embed a repository into the vector store:

codemine embed-repo \
  --repo-owner <owner> \
  --repo-name <repo> \
  [--create-index] \
  [--remove-outdated-chunks] \
  [--ignore-glob "**/tests/**"]

Options:

• --repo-owner: The owner of the GitHub repository (required).
• --repo-name: The name of the GitHub repository (required).
• --create-index: Create a new Pinecone index if it doesn't exist.
• --remove-outdated-chunks: Remove chunks that are no longer in the repository.
• --ignore-glob: Glob pattern to ignore files (can be used multiple times).

Example:

codemine embed-repo \
  --repo-owner jackharrington \
  --repo-name codebase-embedding \
  --create-index \
  --ignore-glob "**/__pycache__/**"

Search Chunks

To search for code chunks:

codemine search-chunks --query "How does the embedding client work?"

Options:

• --query: The search query (required).