pudim-hunter-driver-scraper 1.0.0

Playwright-based scraper implementation for The Pudim Hunter platform

Pudim Hunter Driver Scraper 🍮

A Python package that provides a Playwright-based scraper implementation for The Pudim Hunter platform. This package extends the pudim-hunter-driver interface to provide a common base for implementing job board scrapers.

Table of Contents

1. Features
2. Installation
   • PyPI Installation
   • Development Installation
3. Usage
   • Interface Overview
   • Example Implementation
   • Usage Examples
4. Development Setup
   • Virtual Environment
   • Prerequisites
   • Setup Instructions
5. Project Structure
6. Testing
7. Contributing
   • Getting Started
   • Pull Request Process
8. License

Features

• Playwright-based web scraping
• Headless browser automation
• Easy-to-extend base classes for job board implementations
• Built-in error handling and resource management
• Type hints and validation using Pydantic
• Advanced anti-detection scraping with PhantomPlaywrightScraper
  • WebDriver detection evasion
  • WebGL vendor spoofing
  • Chrome properties emulation
  • Plugin spoofing
  • Language preferences customization
  • And more stealth features

Installation

PyPI Installation

You can install the package directly from PyPI:

pip install pudim-hunter-driver-scraper

Or add to your requirements.txt:

pudim-hunter-driver-scraper>=0.0.1  # Replace with the version you need

Development Installation

For development:

git clone [e-mail]:luismr/pudim-hunter-driver-scraper.git
cd pudim-hunter-driver-scraper
pip install -e .

Usage

Interface Overview

This package provides the base scraper implementation for job search drivers. To create a scraper for a specific job board, you'll need to extend the ScraperJobDriver class and implement the required methods.

1. ScraperJobDriver (ABC) - The base scraper class that implements JobDriver:

   • Required Methods:
     • build_search_url(query: JobQuery) -> str - Build the search URL for the job board
     • get_selectors() -> Dict[str, str] - Get CSS selectors for job elements
     • extract_raw_job_data() -> Optional[Any] - Extract raw job data from the page
     • transform_job(data: Dict[str, Any]) -> Optional[Job] - Transform raw data into Job model
     • has_pagination() -> bool - Check if the job board has pagination
     • get_next_page_url(page_number: int) -> Optional[str] - Get URL for next page
     • has_pagination_items_per_page() -> bool - Check if pagination supports items per page
     • has_description_support_enabled() -> bool - Check if description support is enabled
     • get_description(job: Job) -> Optional[str] - Get job description
     • has_qualifications_support_enabled() -> bool - Check if qualifications support is enabled
     • get_qualifications(job: Job) -> Optional[List[str]] - Get job qualifications

2. PlaywrightScraper - The base scraper implementation:

   • Handles browser lifecycle
   • Provides navigation and data extraction methods
   • Context manager support with with statement

3. PhantomPlaywrightScraper - Enhanced scraper with anti-detection:

   • All features of base PlaywrightScraper
   • Advanced bot detection evasion
   • Stealth mode configurations

4. Exceptions:

   • Inherits all exceptions from pudim-hunter-driver
   • Adds scraper-specific error handling

Example Implementation

from typing import Dict, Any, Optional, List
from datetime import datetime
from pudim_hunter_driver.models import JobQuery, Job
from pudim_hunter_driver_scraper import ScraperJobDriver
from pudim_hunter_driver_scraper.driver import ScraperType

class MyPhantomJobDriver(ScraperJobDriver):
    def __init__(self):
        super().__init__(scraper_type=ScraperType.PHANTOM)
    
    def build_search_url(self, query: JobQuery) -> str:
        """Build the search URL for the job board."""
        return f"https://example.com/jobs?q={query.keywords}&l={query.location}"
    
    def get_selectors(self) -> Dict[str, str]:
        """Define CSS selectors for job elements."""
        return {
            "job_list": ".job-listing",
            "title": ".job-title",
            "company": ".company-name",
            "location": ".job-location"
        }
    
    def extract_raw_job_data(self) -> Optional[List[Dict[str, Any]]]:
        """Extract job data from the page."""
        jobs_data = []
        job_elements = self.scraper._page.query_selector_all(self.get_selectors()["job_list"])
        
        for job in job_elements:
            title = job.query_selector(self.get_selectors()["title"])
            company = job.query_selector(self.get_selectors()["company"])
            
            if title and company:
                jobs_data.append({
                    "title": title.inner_text(),
                    "company": company.inner_text()
                })
        
        return jobs_data
    
    def transform_job(self, data: Dict[str, Any]) -> Optional[Job]:
        """Transform raw job data into Job model."""
        if not data.get("title"):
            return None
            
        return Job(
            id=f"job-{hash(data['title'])}",
            title=data["title"],
            company=data["company"],
            location="",
            summary="",
            url="",
            remote=False,
            source="Example",
            posted_at=datetime.now()
        )
    
    def has_pagination(self) -> bool:
        """Check if the job board has pagination."""
        return True
    
    def get_next_page_url(self, page_number: int) -> Optional[str]:
        """Get URL for next page of results."""
        return f"https://example.com/jobs?page={page_number}"
    
    def has_pagination_items_per_page(self) -> bool:
        """Check if pagination supports items per page."""
        return True
    
    def has_description_support_enabled(self) -> bool:
        """Check if description support is enabled."""
        return True
    
    def get_description(self, job: Job) -> Optional[str]:
        """Get job description."""
        # Implement description extraction logic
        return None
    
    def has_qualifications_support_enabled(self) -> bool:
        """Check if qualifications support is enabled."""
        return True
    
    def get_qualifications(self, job: Job) -> Optional[List[str]]:
        """Get job qualifications."""
        # Implement qualifications extraction logic
        return None

Usage Examples

# Using the driver
driver = MyPhantomJobDriver()
query = JobQuery(
    keywords="software engineer",
    location="San Francisco",
    page=1,
    items_per_page=20
)

job_list = driver.fetch_jobs(query)
for job in job_list.jobs:
    print(f"{job.title} at {job.company}")

For more detailed examples, check the test files:

• tests/test_driver_phantom.py: Complete job driver implementation example
• tests/test_scraper_phantom.py: Anti-detection features testing
• tests/test_scraper_phantom_sites.py: Real-world site scraping examples

Development Setup

Virtual Environment

We strongly recommend using a virtual environment for development and testing.

Prerequisites

• Python 3.9 or higher
• pip (Python package installer)
• venv module (usually comes with Python 3)

Setup Instructions

1. Create and activate virtual environment:

python3.9 -m venv venv
source venv/bin/activate  # On Windows: .\venv\Scripts\activate

2. Install dependencies:

pip install -r requirements.txt
pip install -e .

3. Install Playwright browsers:

playwright install chromium

Project Structure

pudim-hunter-driver-scraper/
├── src/
│   └── pudim_hunter_driver_scraper/
│       ├── __init__.py          # Package initialization
│       ├── scraper.py           # PlaywrightScraper implementation
│       └── driver.py            # ScraperJobDriver implementation
├── tests/                       # Test directory
│   └── __init__.py
├── README.md                    # This file
├── requirements.txt             # Direct dependencies
├── setup.py                     # Package setup
└── pyproject.toml              # Project configuration

Testing

Run the tests:

pytest tests/

Key test files:

• test_driver_phantom.py: Tests for phantom job driver implementation
• test_scraper_phantom.py: Tests for anti-detection capabilities
• test_scraper_phantom_sites.py: Tests for real-world site scraping

Contributing

Getting Started

1. Fork and clone the repository:

git clone [e-mail]:luismr/pudim-hunter-driver-scraper.git
cd pudim-hunter-driver-scraper

2. Create your feature branch:

git checkout -b feature/amazing-feature

3. Set up development environment:

python3.9 -m venv venv
source venv/bin/activate
pip install -e .

Pull Request Process

1. Update documentation as needed
2. Add/update tests as needed
3. Ensure all tests pass
4. Submit PR for review

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Copyright (c) 2024-2025 Luis Machado Reis