scrapely-client 1.0.1

Python client for Scrapely browser automation service - simple, intuitive web scraping and automation

Scrapely Client

A Python client library for the Scrapely browser automation service. Scrapely provides a simple, intuitive API for web scraping and browser automation through WebSocket connections.

Installation

pip install scrapely-client

Prerequisites

Before using the Scrapely client, you must have the Scrapely service running. This client is a lightweight wrapper that connects to your self-hosted Scrapely server.

Step 1: Setup the Scrapely Service

The Scrapely service provides the browser automation backend with anti-bot capabilities. You need to deploy it first:

🔗 Clone and setup the service from: https://github.com/YahyaRehman6/scrapely

Quick setup with Docker:

# Clone the Scrapely service repository
git clone https://github.com/YahyaRehman6/scrapely.git
cd scrapely

# Build and run with Docker
docker build -t scrapely .
docker run -d \
  --name scrapely-server \
  -p 5050:5050 \
  --restart unless-stopped \
  scrapely

# Verify it's running
docker logs scrapely-server

Step 2: Install the Client

Once your Scrapely service is running, install this client package:

pip install scrapely-client

Quick Start

Basic Usage with Context Manager (Recommended)

from scrapely import Scrapely

# Connect to Scrapely service
with Scrapely("ws://localhost:5050/connect") as browser:
    # Navigate to a website
    browser.goto("https://example.com")
    
    # Find and interact with elements
    search_box = browser.find("#search")
    search_box.type("Python web scraping")
    
    # Click a button
    browser.click("#search-button")
    
    # Extract data
    results = browser.find_all(".result-item")
    for result in results:
        title = result.get_text()
        link = result.get_attr("href")
        print(f"{title}: {link}")

Manual Connection Management

from scrapely import Scrapely

browser = Scrapely("ws://localhost:5050/connect")
browser.connect()

try:
    browser.goto("https://example.com")
    text = browser.get_text("h1")
    print(text)
finally:
    browser.close()

Core Features

Navigation

# Navigate to URL
browser.goto("https://example.com", timeout=15, wait_for_load=True)

# Wait for specific page
browser.wait_for_page("https://example.com/results")

# Get current URL
current = browser.current_url()
print(current)

Finding Elements

# Find single element
button = browser.find("#submit-btn")

# Find multiple elements
items = browser.find(".list-item", multiple=True)

# Check if element exists
if browser.exists("#optional-element"):
    print("Element found!")

# Find elements within a parent
container = browser.find("#main-container")
child = container.find(".child-element")

Element Interactions

# Click elements
browser.click("#button")
button = browser.find("#button")
button.click()

# Type into input fields
browser.type("[e-mail]", "#email")
browser.type("password123", "#password", clear=True)

# Scroll to element
browser.scroll_to("#footer")
element = browser.find("#section")
element.scroll_to()

Extracting Data

# Get text content
title = browser.get_text("h1")
element = browser.find(".article-title")
title = element.get_text()

# Get attributes
link = browser.get_attr("href", "a.download")
image_src = browser.find("img").get_attr("src")
data_id = browser.get_attr("data-id", ".item")

Advanced Features

JavaScript Execution

# Execute JavaScript in page context
result = browser.run_js("return document.title")

# Execute JavaScript on specific element
browser.run_js_on_element("this.style.border = '2px solid red'", "#highlight")

XPath Support

# Click element using XPath
browser.click_xpath("//button[contains(text(), 'Submit')]")

Dropdown Selection

# Select by value
browser.select_option("#country", value="us")

# Select by index
browser.select_option("#country", index=2)

Shadow DOM

# Access shadow DOM elements
shadow_root = browser.get_shadow_root("#shadow-host")
shadow_element = browser.find(element_id=shadow_root["text"])
text = shadow_element.get_text()

Drag and Drop

# Drag one element to another
browser.drag_and_drop_to(".draggable-item", ".drop-zone")

Batch Operations

Execute multiple operations efficiently in a single request:

operations = [
    {
        "method": "goto",
        "kwargs": {"url": "https://example.com"}
    },
    {
        "method": "type",
        "kwargs": {"selector": "#email", "text": "[e-mail]"}
    },
    {
        "method": "click",
        "kwargs": {"selector": "#submit"}
    },
    {
        "method": "get_text",
        "kwargs": {"selector": ".result"}
    }
]

results = browser.batch_operations(operations)
for result in results:
    print(f"Status: {result['status']}, Data: {result['data']}")

Complete Examples

E-commerce Scraping

with Scrapely() as browser:
    browser.goto("https://shop.example.com")
    
    # Search for products
    search = browser.find("#search-input")
    search.type("laptop")
    browser.click("#search-button")
    
    # Wait for results
    browser.wait_for_page("https://shop.example.com/search")
    
    # Extract product information
    products = browser.find(".product-card", multiple=True)
    
    for product in products:
        name = product.find(".product-name").get_text()
        price = product.find(".price").get_text()
        link = product.find("a").get_attr("href")
        
        print(f"Product: {name}")
        print(f"Price: {price}")
        print(f"URL: {link}")
        print("-" * 40)

Form Automation

with Scrapely() as browser:
    browser.goto("https://forms.example.com")
    
    # Fill out form
    browser.type("John Doe", "#name")
    browser.type("[e-mail]", "#email")
    browser.type("555-0123", "#phone")
    
    # Select dropdown
    browser.select_option("#country", value="us")
    
    # Check checkbox
    browser.click("#terms-checkbox")
    
    # Submit form
    browser.click("#submit-button")
    
    # Get confirmation message
    message = browser.get_text(".confirmation-message")
    print(message)

Dynamic Content Handling

with Scrapely() as browser:
    browser.goto("https://dynamic-site.example.com")
    
    # Scroll to load more content
    for _ in range(5):
        browser.scroll_to(".load-more-trigger")
        time.sleep(1)  # Wait for content to load
    
    # Extract all loaded items
    items = browser.find(".content-item", multiple=True)
    print(f"Loaded {len(items)} items")
    
    for item in items:
        title = item.find("h2").get_text()
        description = item.find(".description").get_text()
        print(f"{title}: {description}")

Configuration

Custom WebSocket URL

# Connect to custom service URL
browser = Scrapely("ws://192.168.1.100:8080/connect")

Connection Timeout

# Set custom connection timeout (in seconds)
browser = Scrapely("ws://localhost:5050/connect", timeout=60)

Error Handling

from scrapely import Scrapely, ConnectionError, OperationError

try:
    with Scrapely() as browser:
        browser.goto("https://example.com")
        browser.click("#non-existent-button")
        
except ConnectionError as e:
    print(f"Failed to connect: {e}")
    
except OperationError as e:
    print(f"Operation failed: {e}")

API Reference

Scrapely Class

Connection Methods

• connect() - Establish WebSocket connection
• close() - Close connection and cleanup

Navigation Methods

• goto(url, timeout, wait_for_load) - Navigate to URL
• wait_for_page(url, timeout) - Wait for page URL to match
• current_url() - Get current page URL

Element Finding Methods

• find(selector, element_id, timeout, multiple) - Find element(s)
• exists(selector, element_id) - Check if element exists

Interaction Methods

• click(selector, element_id, timeout) - Click element
• type(text, selector, element_id, timeout, clear) - Type text
• scroll_to(selector, element_id, timeout) - Scroll to element

Data Extraction Methods

• get_text(selector, element_id, timeout) - Get element text
• get_attr(attribute, selector, element_id, timeout) - Get attribute value

Advanced Methods

• run_js(script, *args) - Execute JavaScript
• run_js_on_element(script, selector, element_id, timeout) - Execute JS on element
• click_xpath(xpath) - Click using XPath
• select_option(selector, index, value, timeout) - Select dropdown option
• get_shadow_root(selector, element_id, timeout) - Access shadow DOM
• drag_and_drop_to(draggable, droppable, timeout) - Drag and drop

Batch Methods

• batch_operations(operations) - Execute multiple operations

Element Class

Element objects support chainable method calls:

• click(timeout) - Click this element
• type(text, timeout, clear) - Type into this element
• get_text(timeout) - Get text content
• get_attr(attribute, timeout) - Get attribute value
• scroll_to(timeout) - Scroll to this element
• run_js(script, timeout) - Execute JavaScript on this element
• exists(timeout) - Check if element still exists
• find(selector, timeout) - Find child element
• find_all(selector, timeout) - Find all child elements

Requirements

• Python 3.7+
• websocket-client
• Running Scrapely service instance

License

Apache License

Support

For issues, questions, or contributions, please visit the project repository.

Changelog

Version 1.0.0

• Initial release
• Core browser automation features
• Element caching and chaining
• Batch operations support
• Shadow DOM support
• Drag and drop functionality