A simple webscraper library
Project description
XSCRAPERS
The XSCRAPERS package provides an OOP interface to some simple webscraping techniques.
A base use case can be to load some pages to Beautifulsoup Elements. This package allows to load the URLs concurrently using multiple threads, which allows to safe an enormous amount of time.
import xscrapers.webscraper as ws
URLS = [
"https://www.google.com/",
"https://www.amazon.com/",
"https://www.youtube.com/",
]
PARSER = "html.parser"
web_scraper = ws.Webscraper(PARSER, verbose=True)
web_scraper.load(URLS)
web_scraper.parse()
Note that herein, the data scraped is stored in the data
attribute of the webscraper.
The URLs parsed are stored in the url
attribute.
Downloading the Firefox Geckodriver
Linux
See this link for a good explanation. In short, the steps are:
-
Download the geckodriver from the mozilla GitHub release page, note to change the
X
for the version you want to downloadwget https://github.com/mozilla/geckodriver/releases/download/vX.XX.X/geckodriver-vX.XX.X-linux64.tar.gz
-
Extract the file with
tar -xvzf geckodriver*
-
Make it executable
chmod +x geckodriver
-
In the last step, the driver can be added to the
PATH
environment variable, moved to theusr/local/bin
folder, or can be given as full path to theWebdriver
class asexe_path
argumentexport PATH=$PATH:/path-to-extracted-file/ sudo mv geckodriver /usr/local/bin/
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for xscrapers-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbe08f1426007a58b8678981fc890f36ed21c43443ff61c9765822f690d07d43 |
|
MD5 | 3d983d45ad37e2467b70d3f3e0c2cdb2 |
|
BLAKE2b-256 | bb591028f19bc039e212394d60448d9db4f3421a9dc2a4bd28c218e086d19f12 |