dsl-geral 1.0.0

A general purpose module with utility functions for data processing

DSL Geral

A general purpose Python module with utilities for data processing and web scraping.

Installation

pip install dsl_geral

Main Features

• Web scraping utilities with CAPTCHA handling (get, get3, get_contents, get_json, get_response)
• Text cleaning and normalization functions (clean, clean_text, clean_for_csv, deep_clean, remove_accents)
• Text extraction utilities (clext, ext, extract, trim)
• Selenium WebDriver integration (webdriver_get, webdriver_get_page, waitForLoad, xpath_* functions)

Usage Examples

Basic Web Scraping

from dsl_geral import get

html = get("https://example.com")
print(html)

Text Extraction

from dsl_geral import clext

text = "Some text [START]extract this[END] more text"
extracted = clext(text, "[START]", "[END]") 
print(extracted)  # "extract this"

Date Conversion

from dsl_geral import date

formatted = date("31/12/2023")
print(formatted)  # "2023-12-31"

Text Cleaning

from dsl_geral import clean, remove_accents

text = "  Some   messy  text with accents: áéíóú  "
cleaned = clean(text)  # "Some messy text with accents: áéíóú"
no_accents = remove_accents(cleaned)  # "Some messy text with accents: aeiou"

Full Function Documentation

Web Scraping

• get(url): Fetch HTML with CAPTCHA handling
• get3(url): Fetch raw response with CAPTCHA handling
• get_contents(url): Fetch response contents
• get_json(url): Fetch JSON data
• get_response(url): Fetch full response object
• webdriver_get(url): Fetch page using Selenium
• webdriver_get_page(url): Get page source using Selenium
• waitForLoad(xpath): Wait for element to load (Selenium)
• xpath_* functions: Interact with page elements (Selenium)

Text Processing

• clean(text): Normalize whitespace and clean text
• clean_text(text): Alternative cleaning function
• clean_for_csv(text): Clean text for CSV output
• deep_clean(text): Deep clean including HTML entities
• remove_accents(text): Remove accents and special chars
• clext(text, start, end): Clean and extract text between delimiters
• ext(text, start, end): Extract text between delimiters
• extract(text, start, end): Alternative extraction function
• trim(text, start, end): Trim text between markers

License

MIT - See LICENSE for details.