Skip to main content

Using proxycrawl api to scrape similarweb data

Project description


similarweb_scraperis is a python library for scraping similarweb with proxycrawl api and it can bypass the distil projection so far. It also provides some functionality for transforming scraped data into pd dataframe.


Use the package manager pip to install foobar.

pip install similarweb-scraper

## Usage

from similarweb_scraper import scraper

### get the website html
web_scrape = scraper()
web_scrape.login(#api key from
web_scrape.webpage_scrape(#websit e.g:

### get the html code
soup = web_scrape.og_soup
### get the html code as json format
web_json = web_scrape.json_storage

### get data into json format
df = web_scrape.metrics_to_df(#str(metrics_type))
##metrics_type name :
# engagement',
# more function will be available soon


Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.



Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

similarweb_scraper-0.0.3.tar.gz (3.9 kB view hashes)

Uploaded source

Built Distribution

similarweb_scraper-0.0.3-py3-none-any.whl (5.5 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page