Scrape Bing

A robust Python package for scraping search results from Bing with built-in rate limiting, retry mechanisms, and result cleaning features.

Features

• 🔍 Clean and structured search results
• 🔄 Automatic retry mechanism for failed requests
• ⏱️ Built-in rate limiting to prevent blocking
• 🧹 URL cleaning and validation
• 🔄 User agent rotation
• 💪 Type hints and proper error handling
• 📝 Comprehensive documentation

Installation

You can install the package using pip:

pip install scrape-bing

For development installation:

git clone https://github.com/affanshaikhsurab/scrape-bing.git
cd scrape_bing
pip install -e .

Quick Start

from scrape_bing import BingScraper

# Initialize the searcher
scraper= BingScraper(
    max_retries=3,
    delay_between_requests=1.0
)

# Perform a search
results = scraper.search("python programming", num_results=5)

# Process results
for result in results:
    print(f"\nTitle: {result.title}")
    print(f"URL: {result.url}")
    print(f"Description: {result.description}")

Advanced Usage

Custom Configuration

# Configure with custom parameters
scraper = BingScraper(
    max_retries=5,                # Maximum retry attempts
    delay_between_requests=2.0    # Delay between requests in seconds
)

Error Handling

from scrape_bing import BingScraper

scraper = BingScraper()

try:
    results = scraper.search("python programming")
  
except ValueError as e:
    print(f"Invalid input: {e}")
  
except ConnectionError as e:
    print(f"Network error: {e}")
  
except RuntimeError as e:
    print(f"Parsing error: {e}")

Search Result Structure

Each search result contains:

• title: The title of the search result
• url: The cleaned and validated URL
• description: The description snippet (if available)

# Access result attributes
for result in results:
    print(result.title)       # Title of the page
    print(result.url)         # Clean URL
    print(result.description) # Description (may be None)

API Reference

BingSearch Class

class BingScraper:
    def __init__(self, max_retries: int = 3, delay_between_requests: float = 1.0):
        """
        Initialize the BingSearch scraper.
  
        Args:
            max_retries: Maximum number of retry attempts for failed requests
            delay_between_requests: Minimum delay between requests in seconds
        """
        pass

    def search(self, query: str, num_results: int = 10) -> List[SearchResult]:
        """
        Perform a Bing search and return results.
  
        Args:
            query: Search query string
            num_results: Maximum number of results to return
  
        Returns:
            List of SearchResult objects
  
        Raises:
            ValueError: If query is empty
            ConnectionError: If network connection fails
            RuntimeError: If parsing fails
        """
        pass

SearchResult Class

@dataclass
class SearchResult:
    title: str                    # Title of the search result
    url: str                      # Cleaned URL
    description: Optional[str]     # Description (may be None)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Running Tests

# Install development dependencies
pip install -e ".[dev]"

# Run tests
python -m pytest tests/

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Beautiful Soup 4 for HTML parsing
• Requests library for HTTP requests
• Python typing for type hints

Support

If you encounter any issues or have questions, please file an issue on the GitHub repository.