scrapy-selenium-grid 0.0.1

Minimum required versions

• Python >= 3.8

• Scrapy >= 2.0

• aiohttp

SELENIUM_GRID_BROWSER_NAME

Type str, default chrome

The browser type to be used in Selenium Grid, e.g. chrome, edge, firefox, ie, safari.

SELENIUM_GRID_URL

Type str, default http://127.0.0.1:4444

The Selenium Grid hub url.

SELENIUM_GRID_IMPLICIT_WAIT_INSEC

Type int, default 0

Selenium has a built-in way to automatically wait for elements.

This is a global setting that applies to every element location call for the entire session. The default value is 0, which means that if the element is not found, it will immediately return an error. If an implicit wait is set, the driver will wait for the duration of the provided value before returning the error. Note that as soon as the element is located, the driver will return the element reference and the code will continue executing, so a larger implicit wait value won’t necessarily increase the duration of the session.

selenium_grid

Type bool, default False

If set to a value that evaluates to True the request will be processed by Selenium Grid.

return scrapy.Request("https://example.org", meta={"selenium_grid": True})

selenium_grid_driver

Type scrapy_selenium_grid.webdriver.WebDriver

This will be set with asynchronous Selenium Driver when you enabled selenium_grid in request meta.

import scrapy
from scrapy_selenium_grid.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

def start_requests(self):
    yield scrapy.Request(
        url="https://httpbin.org/get",
        meta={"selenium_grid": True},
    )

async def parse(self, response, **kwargs):
    driver = response.meta["selenium_grid_driver"]

    await ActionChains(driver).key_down(Keys.F12).key_up(Keys.F12).perform()

    inp_userid = await driver.find_element(By.CSS_SELECTOR, 'input[name="userid"]')
    assert await inp_userid.is_displayed() == True
    await inp_userid.send_keys("Username")

    print(await driver.get_log('browser'))

selenium_grid_browser

Type str, default None

Same values as SELENIUM_GRID_BROWSER_NAME but you set it per request.