📦 Repo Scrapper API & AI CLI

An automated, AI-powered GitHub repository search and scraping platform. Features a Grok AI-driven FastAPI backend deployed on Azure Kubernetes Service (AKS) via Terraform & GitHub Actions CI/CD, alongside an interactive PyPI CLI tool (repo-scrapper-cli).

🏛️ End-to-End System Architecture

┌────────────────────────────────────────────────────────────────────────┐
│                        PyPI Package Client                             │
│                    pip install repo-scrapper-cli                       │
└───────────────────────────────────┬────────────────────────────────────┘
                                     │
                  ┌──────────────────┴──────────────────┐
                  │                                      │
        [ Primary Hybrid Route ]                [ Offline Fallback Route ]
        FastAPI Backend (AKS)                   Direct GitHub REST API
                  │                                      │
    ┌─────────────┴─────────────┐            ┌───────────┴───────────┐
    │ Grok AI Dual-Key Rotator  │            │   Rule-Based Engine    │
    │ (Key 1 ➔ Key 2 Failover)  │            │   Regex Extractor      │
    └─────────────┬─────────────┘            └───────────┬───────────┘
                  │                                       │
                  └──────────────────┬────────────────────┘
                                     │
                                     ▼
                       ┌──────────────────────────┐
                       │   GitHub Search REST API │
                       └──────────────────────────┘

💻 1. For End Users: CLI Quickstart (PyPI)

If you just want to use the tool to find repositories, you don't need to download the source code — just install it via pip.

Install the CLI tool:

pip install repo-scrapper-cli

Run a search query:

repo-scrapper "top 3 python microservices with docker" --limit 3

Connect to a custom backend URL:

repo-scrapper "react templates" --url "http://your-aks-cluster-ip:8000"

🛠️ 2. For Developers: Local Backend Setup

If you want to run the AI-powered FastAPI backend locally, follow these steps.

Step 1: Clone the repository

git clone https://github.com/Haidar-Karamat/Repo_Scrapper.git
cd Repo_Scrapper

Step 2: Set up a virtual environment

# Windows
python -m venv venv
.\venv\Scripts\Activate.ps1

# Linux / Mac
python3 -m venv venv
source venv/bin/activate

Step 3: Install dependencies

cd server
pip install -r requirements.txt

Step 4: Configure environment variables

Create a .env file inside the server/ directory:

# Grok AI API Keys (Optional, fallback works without it)
GROK_API_KEY_1=your_first_key_here
GROK_API_KEY_2=your_second_key_here

# GitHub Token (Increases rate limit to 5000 req/hr)
GITHUB_TOKEN=ghp_your_github_token_here

Step 5: Run the FastAPI server

uvicorn app.main:app --reload --port 8000

API documentation will be available at: http://localhost:8000/docs

🐳 3. For DevOps: Docker Deployment

To run the backend isolated in a container:

# Build the Docker image
docker build -t repo-scrapper-backend ./server

# Run the container (make sure your .env file is ready)
docker run -d -p 8000:8000 --name repo-backend --env-file ./server/.env repo-scrapper-backend

☁️ 4. Cloud Deployment (Terraform & Azure AKS)

Step 1: Authenticate with Azure

az login

Step 2: Provision infrastructure with Terraform

cd terraform
terraform init
terraform plan
terraform apply -auto-approve

Step 3: Deploy to Kubernetes

az aks get-credentials --resource-group repo-scrapper-rg --name repo-scrapper-aks
kubectl apply -f k8s/

📄 License

This project is licensed under the MIT License.