Skip to main content

A Python class that acts as wrapper for ProxyCrawl scraping and crawling API

Project description

ProxyCrawl API Python class

A lightweight, dependency free Python class that acts as wrapper for ProxyCrawl API.

Installing

Choose a way of installing:

  • Download the python class from Github.
  • Or use PyPi Python package manager. pip install proxycrawl

Then import the ProxyCrawlAPI

Python2:

from proxycrawl import ProxyCrawlAPI

Python3:

from proxycrawl.proxycrawl_api import ProxyCrawlAPI

Class usage

First initialize the ProxyCrawlAPI class

api = ProxyCrawlAPI({ 'token': 'YOUR_PROXYCRAWL_TOKEN' })

GET requests

Pass the url that you want to scrape plus any options from the ones available in the API documentation.

api.get(url, options = {})

Example:

response = api.get('https://www.facebook.com/britneyspears')
if response['status_code'] == 200:
    print(response['body'])

You can pass any options from ProxyCrawl API.

Example:

response = api.get('https://www.reddit.com/r/pics/comments/5bx4bx/thanks_obama/', {
    'user_agent': 'Mozilla/5.0 (Windows NT 6.2; rv:20.0) Gecko/20121202 Firefox/30.0',
    'format': 'json'
})
if response['status_code'] == 200:
    print(response['body'])

POST requests

Pass the url that you want to scrape, the data that you want to send which can be either a json or a string, plus any options from the ones available in the API documentation.

api.post(url, dictionary or string data, options = {})

Example:

response = api.post('https://producthunt.com/search', { 'text': 'example search' })
if response['status_code'] == 200:
    print(response['body'])

You can send the data as application/json instead of x-www-form-urlencoded by setting option post_content_type as json.

import json
response = api.post('https://httpbin.org/post', json.dumps({ 'some_json': 'with some value' }), { 'post_content_type': 'json' })
if response['status_code'] == 200:
    print(response['body'])

Javascript requests

If you need to scrape any website built with Javascript like React, Angular, Vue, etc. You just need to pass your javascript token and use the same calls. Note that only .get is available for javascript and not .post.

api = ProxyCrawlAPI({ 'token': 'YOUR_JAVASCRIPT_TOKEN' })
response = api.get('https://www.nfl.com')
if response['status_code'] == 200:
    print(response['body'])

Same way you can pass javascript additional options.

response = api.get('https://www.freelancer.com', { 'page_wait': 5000 })
if response['status_code'] == 200:
    print(response['body'])

Original status

You can always get the original status and proxycrawl status from the response. Read the ProxyCrawl documentation to learn more about those status.

response = api.get('https://craiglist.com')
print(response['headers']['original_status'])
print(response['headers']['pc_status'])

If you have questions or need help using the library, please open an issue or contact us.


Copyright 2019 ProxyCrawl

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxycrawl-2.0.1.tar.gz (3.6 kB view details)

Uploaded Source

Built Distributions

proxycrawl-2.0.1-py3-none-any.whl (8.1 kB view details)

Uploaded Python 3

proxycrawl-2.0.1-py2-none-any.whl (8.1 kB view details)

Uploaded Python 2

File details

Details for the file proxycrawl-2.0.1.tar.gz.

File metadata

  • Download URL: proxycrawl-2.0.1.tar.gz
  • Upload date:
  • Size: 3.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.18.4 setuptools/40.7.3 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/2.7.16

File hashes

Hashes for proxycrawl-2.0.1.tar.gz
Algorithm Hash digest
SHA256 03c97e44f7a24f3c561b53053f3685a5ee58f348cb441105dacb19ba0591f4c6
MD5 f94936a661918109f7d32236ba139b91
BLAKE2b-256 bdf267918850f8bf968f53be76dfaadd313e1525edfe121cd2ac57958f4487c9

See more details on using hashes here.

File details

Details for the file proxycrawl-2.0.1-py3-none-any.whl.

File metadata

  • Download URL: proxycrawl-2.0.1-py3-none-any.whl
  • Upload date:
  • Size: 8.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.18.4 setuptools/40.7.3 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/2.7.16

File hashes

Hashes for proxycrawl-2.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 b154117cd9f94449b618f7d590b2357cb8841bcc1a9f14607a70d730acb91720
MD5 ee7eb7d5d1ce06f181a79faad9fad160
BLAKE2b-256 493ab10a4c43ce4bebc1ec1ab716bec69a9175f4440f215647ac3102149916b6

See more details on using hashes here.

File details

Details for the file proxycrawl-2.0.1-py2-none-any.whl.

File metadata

  • Download URL: proxycrawl-2.0.1-py2-none-any.whl
  • Upload date:
  • Size: 8.1 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.18.4 setuptools/40.7.3 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/2.7.16

File hashes

Hashes for proxycrawl-2.0.1-py2-none-any.whl
Algorithm Hash digest
SHA256 bbf4c819c8c3c40cefa59da2d1a97e79841c4f4856077f53ca6a6877f5577a3b
MD5 def3bc4dfcf068720b1e10f17a5f7c84
BLAKE2b-256 38132986d14d17a13008f08ab4fd4c18dad714606e07130c174868bd20f6c7c9

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page