Supabase Overleaf Template Crawler

This project crawls LaTeX templates from Overleaf and stores them in a Supabase instance. It can be set up to work with both local and cloud-based Supabase deployments.

Table of Contents

• Supabase Overleaf Template Crawler
  • Table of Contents
  • Setup
    • Supabase CLI Installation
      • macOS
      • Windows
      • Linux
    • Local Supabase Setup
    • Cloud Supabase Setup
  • Project Installation
  • Configuration
  • Usage
  • Project Structure
  • Database Setup
  • Contributing
  • License

Setup

Supabase CLI Installation

The Supabase CLI is a powerful tool that simplifies the process of setting up and managing your Supabase projects. Here's how to install it on different operating systems:

macOS

Using Homebrew:

brew install supabase/tap/supabase

Windows

Using Scoop:

scoop bucket add supabase https://github.com/supabase/scoop-bucket.git
scoop install supabase

Linux

Using Apt (Debian/Ubuntu):

sudo apt-get update
sudo apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.supabase.com/public/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/supabase-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/supabase-archive-keyring.gpg] https://dl.supabase.com/apt stable main" | sudo tee /etc/apt/sources.list.d/supabase.list
sudo apt-get update
sudo apt-get install supabase

For other Linux distributions, you can use the official install script:

curl -s https://packages.supabase.com/setup.sh | sudo bash

After installation, verify the CLI is installed correctly:

supabase --version

Local Supabase Setup

1. Initialize a new Supabase project:

   supabase init
   

2. Start the local Supabase instance:

   supabase start
   

3. This will provide you with local URLs and keys. Make note of these for configuration.

Cloud Supabase Setup

1. Log in to your Supabase account via CLI:

   supabase login
   

2. Initialize a new project:

   supabase init
   

3. Link your local project to a Supabase project:

   supabase link --project-ref your-project-ref
   

   Replace your-project-ref with your actual project reference.

4. Push your local schema to the cloud project:

   supabase db push
   

Project Installation

1. Clone this repository:

   git clone https://github.com/yourusername/supabase-overleaf-crawler.git
   cd supabase-overleaf-crawler
   

2. Install dependencies:

   pip install -r requirements.txt
   

Configuration

1. Copy the example configuration file:

   cp src/config/config.example.py src/config/config.py
   

2. Edit src/config/config.py with your Supabase details:

   • For local setup, use the URL and key provided by supabase start
   • For cloud setup, use the URL and key from your Supabase project dashboard

Usage

Run the crawler:

python src/main.py

Project Structure

• src/: Contains the source code
  • main.py: Main entry point
  • crawler_general.py: Crawls general template information
  • crawler_detail.py: Crawls detailed template information
  • db_handler.py: Handles database operations
  • config/: Contains configuration files
• data/: (Optional) For storing any local data
• requirements.txt: List of Python dependencies
• README.md: This file
• supabase/: Contains Supabase configuration files (created by supabase init)

Database Setup

The crawler expects certain tables to exist in your Supabase database. You can set these up by running the SQL commands in supabase/migrations/your-migration-file.sql.

For both local and cloud setups, the migration will be applied automatically when you run supabase db push.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.