Skip to main content

A Python class that acts as wrapper for ProxyCrawl scraping and crawling API

Project description

ProxyCrawl API Python class

A lightweight, dependency free Python class that acts as wrapper for ProxyCrawl API.


Choose a way of installing:

  • Download the python class from Github.
  • Or use PyPi Python package manager. pip install proxycrawl

Then import the CrawlingAPI, ScraperAPI, etc as needed.

from proxycrawl import CrawlingAPI, ScraperAPI, LeadsAPI

Crawling API

First initialize the CrawlingAPI class.

api = CrawlingAPI({ 'token': 'YOUR_PROXYCRAWL_TOKEN' })

GET requests

Pass the url that you want to scrape plus any options from the ones available in the API documentation.

api.get(url, options = {})


response = api.get('')
if response['status_code'] == 200:

You can pass any options from ProxyCrawl API.


response = api.get('', {
    'user_agent': 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
    'format': 'json'
if response['status_code'] == 200:

POST requests

Pass the url that you want to scrape, the data that you want to send which can be either a json or a string, plus any options from the ones available in the API documentation., dictionary or string data, options = {})


response ='', { 'text': 'example search' })
if response['status_code'] == 200:

You can send the data as application/json instead of x-www-form-urlencoded by setting option post_content_type as json.

import json
response ='', json.dumps({ 'some_json': 'with some value' }), { 'post_content_type': 'json' })
if response['status_code'] == 200:

Javascript requests

If you need to scrape any website built with Javascript like React, Angular, Vue, etc. You just need to pass your javascript token and use the same calls. Note that only .get is available for javascript and not .post.

api = CrawlingAPI({ 'token': 'YOUR_JAVASCRIPT_TOKEN' })
response = api.get('')
if response['status_code'] == 200:

Same way you can pass javascript additional options.

response = api.get('', { 'page_wait': 5000 })
if response['status_code'] == 200:

Scraper API

The usage of the Scraper API is very similar, just change the class name to initialize.

scraper_api = ScraperAPI({ 'token': 'YOUR_NORMAL_TOKEN' })

response = scraper_api.get('')
if response['status_code'] == 200:
    print(response['json']['name']) # Will print the name of the Amazon product

Original status

You can always get the original status and proxycrawl status from the response. Read the ProxyCrawl documentation to learn more about those status.

response = api.get('')

If you have questions or need help using the library, please open an issue or contact us.

Leads API

To find email leads you can use the leads API, you can check the full API documentation if needed.

leads_api = LeadsAPI({ 'token': 'YOUR_NORMAL_TOKEN' })

response = leads_api.get_from_domain('')

if response['status_code'] == 200:

Custom timeout

If you need to use a custom timeout, you can pass it to the class instance creation like the following:

api = CrawlingAPI({ 'token': 'TOKEN', 'timeout': 120 })

Timeout is in seconds.

Copyright 2020 ProxyCrawl

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxycrawl-3.1.0.tar.gz (4.6 kB view hashes)

Uploaded source

Built Distributions

proxycrawl-3.1.0-py3-none-any.whl (10.3 kB view hashes)

Uploaded py3

proxycrawl-3.1.0-py2-none-any.whl (10.3 kB view hashes)

Uploaded py2

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page