mfdev-scraper 0.1.6

A lightweight Python SDK for web scraping, data cleaning, and CSV/Excel export.

MFDev Scraper SDK

A lightweight Python SDK for web scraping and data processing.
It provides robust HTTP request handling, data cleaning utilities, and convenient export helpers for CSV and Excel formats.

Features

• 🚀 HTTP Requests with Retry Logic
  Reliable GET/POST requests with configurable retries, custom error handling, and browser impersonation support (via curl_cffi).

• 🛡️ Custom Exceptions
  Clear and descriptive errors for configuration issues, HTTP status mismatches, max retries exceeded, invalid dictionaries, and file I/O errors.

• 🧹 Data Cleaning
  Utility to remove duplicates from lists of dictionaries, with strict or lenient validation modes.

• 📊 Data Export
  Export your data directly to CSV or Excel with built-in helpers.

---

Installation

You can install directly from source:

pip install -e .

Or once published to PyPI:

pip install mfdev-scraper

---

---

---

Usage - Example

Import

  from mfdev_scraper_sdk import MFDevScraper

  scraper = MFDevScraper()

Use request_mfdev method

  '''
  Perform a Request

  You can perform a request using the request_mfdev method.
  This method allows you to customize parameters such as:

  * status_code_accepted → The HTTP status code considered a successful response.

  * max_retries → The maximum number of retry attempts if the request fails.

  * timeout → The time (in seconds) to wait before the request times out.

  Note: If you do not provide these parameters, the method will automatically use the default values.

  '''

  # Perform a GET request
  response = client.request_mfdev(
      method="GET",
      url="https://httpbin.org/get",
      status_code_accepted=200,   # optional
      max_retries=3,              # optional
      timeout=10.0                # optional
  )

  print("Status Code:", response.status_code)
  print("Response Body:", response.text)

Use clean_duplicates_dict method

  '''
  Clean Duplicate Records

  The clean_duplicates_dict method removes duplicate records from a list of dictionaries.

  field → The key used to check for duplicates.

  If not provided, the default value is "id".

  strict → Defines how to handle records missing the specified field.

  Default: True → Raises an error if the field does not exist.

  If set to False, records without the field will simply be ignored.

  '''

  # Example data with duplicates - strict = False
  records = [
      {"id": 1, "name": "Alice"},
      {"id": 2, "name": "Bob"},
      {"id": 1, "name": "Alice"},   # Duplicate
      {"id": 3, "name": "Charlie"},
      {"name": "No ID"}             # Missing field
  ]

  # Clean duplicates by "id"
  cleaned = client.clean_duplicates_dict(records, field="id", strict=False)

  print("Cleaned Records:")
  for item in cleaned:
      print(item)

  # Output (with strict=False)
  '''
  {'id': 1, 'name': 'Alice'}
  {'id': 2, 'name': 'Bob'}
  {'id': 3, 'name': 'Charlie'}
  {'name': 'No ID'}

  '''

  ## Example data with duplicates - strict = true

  records = [
      {"id": 1, "name": "Alice"},
      {"name": "No ID"}   # Missing "id"
  ]

  try:
      # This will raise an error because "id" is missing in one record
      cleaned = client.clean_duplicates_dict(records, field="id", strict=True)
  except DataValidationError as e:
      print("Error:", e)

  # Output (with strict=True)
  '''
  Error: Missing required field 'id' in record

  '''

Use generate_csv method

  '''
  Export to CSV

  To export your data to a CSV file, simply provide the file name without the extension.
  Optionally, you can also specify a folder where the file will be saved.

  '''

  csv_path = scraper.generate_csv(cleaned, "output")
  print(f"CSV saved at: {csv_path}")

Use generate_excel method

  '''
  Export to Excel

  To export your data to a Excel file, simply provide the file name without the extension.
  Optionally, you can also specify a folder where the file will be saved.

  '''
  excel_path = scraper.generate_excel(cleaned, "output", "files")
  print(f"Excel saved at: {excel_path}")

---

---

---

Custom Errors

The SDK exposes several custom exceptions you can catch:

• ConfigurationError
• RequestError
• HTTPStatusError
• MaxRetriesExceeded
• InvalidResponseError
• DataValidationError
• MissingFieldError
• NotADictError
• FileIOError
• CSVWriteError
• ExcelWriteError

---

Development

Clone the repository and install dependencies:

git clone https://github.com/yourusername/mfdev-scraper.git
cd mfdev-scraper
pip install -e .
pip install pytest

Run tests:

pytest -q

---

License

This project is licensed under the MIT License.

---

Author

Miguel Fernandez
Homepage: https://mfdev.vercel.app