Skip to main content

Systemitized tool for scraping

Project description


Scrape_it is a tool for extracting valueble information from the website of interest. Save your time on reading and crawling through the website and leave it for Scrape_it!


Scrape_it is avalable on Pypi, you can install it using pip:

pip install scrape-it

Install the lastest version:

pip install git+

Scrape_it object

As a baseline Scrape_it relies on the model (dictionary) which could be customized although specific methods should be defined too.

Currently the object's base-line model is set up to scrape contact information, address, social media links, links for website's policies pages and if posiible condence its' texts.

To initialize the object specify the url as a string. For more precision provide some more details if known:

  • country: 'us' for United States of America, 'gb' for Great Britain/United Kingdom, 'au' for Australia
  • geo_key: API key for address verification, test is set up to work with this API
  • method: 'requests' for usual get request or 'webdriver' for request capable of rendering JavaScript code and dynamically changing webpages


Initialize Scrape_it object (find an example in

from scrape_it import Scrape_it

with open('/home/val/Downloads/geo-key.txt', 'r') as key:
    geo_key =

scrape_it = Scrape_it(url='', country='us', geo_key=geo_key, method='webdriver')


The output will like this:

Scraping ...
url :
country : us
category : None
company_name : All
contact_link : None
phones : {'+18009290927'}
address : 6561 W Post Rd
state : NV
county : Clark
city : Las Vegas
street : W Post Rd
housenumber : 6561
postalcode : 89118
district : Spring Valley
email : None
facebook :
instagram :
linkedin : None
pinterest : None
twitter :
youtube :
faq_link : None
privacy_link :
return_link : None
shipping_link : None
terms_link : None
warranty_link : None
faq_text : None
privacy_text : None
return_text : None
shipping_text : None
terms_text : None
warranty_text : None


The Scrape_it is by no means a perfect package and can be improved for sure. If you have any ideas, issues or would like to improve code or documentation please feel free to open issue or pull request. It is my honor to be at help if I can.


Q: The object returns the emplty dictionary. What do I do?

A: It could be the case the tools used did not find anything though it is certainly an exception rather than a rule. What you can try though: use 'webdriver' method to ensure JavaScript is rendered too, try specify the country, use proxy/VPN in case the website might block requests from your location

Q: Should I pass a root link or any would work?

A: Yes, for now at least. Scrape_it will scrape some information still but it relies on finding additional links to scrape the most information possible and I did not set the pipeline to process non-root links yet (I am working on it)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrape-it, version 0.3.8
Filename, size File type Python version Upload date Hashes
Filename, size scrape_it-0.3.8-py2.py3-none-any.whl (32.9 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size scrape_it-0.3.8.tar.gz (25.3 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page