pa-scraper 0.1.2

Python wrapper for Prompt API's Scraper API

Prompt API - Scraper API - Python Package

pa-scraper is a python wrapper for scraper api with few more extra cream and sugar.

Requirements

1. You need to signup for Prompt API
2. You need to subscribe scraper api, test drive is free!!!
3. You need to set PROMPTAPI_TOKEN environment variable after subscription.

then;

$ pip install pa-scraper

---

Example Usage

Examples can be found here.

from scraper import Scraper

url = 'https://pypi.org/classifiers/'
scraper = Scraper(url)
response = scraper.get()

if response.get('error', None):
    # response['error']  returns error message
    # response['status'] returns http status code
    # {'error': 'Not Found', 'status': 404}
    print(response)
else:
    result = response['result']

    print(result['headers'])   # returns response headers 
    print(result['data'])      # returns fetched html
    print(result['url'])       # returns fetched url
    print(response['status'])  # returns http status code

    save_result = scraper.save('/tmp/my-html.html')  # save to file
    if save_result.get('error', None):
        # we have save error
        pass
    else:
        print(save_result)    # contains saved file path and file size
        # {'file': '/tmp/my-html.html', 'size': 321322}

You can add url parameters for extra operations. Valid parameters are:

• auth_password: for HTTP Realm auth password
• auth_username: for HTTP Realm auth username
• cookie: URL Encoded cookie header.
• country: 2 character country code. If you wish to scrape from an IP address of a specific country.
• referer: HTTP referer header

from scraper import Scraper

    url = 'https://pypi.org/classifiers/'
    scraper = Scraper(url)

    fetch_params = dict(country='EE')
    response = scraper.get(params=fetch_params)

    if response.get('error', None):
        # response['error']  returns error message
        # response['status'] returns http status code
        # {'error': 'Not Found', 'status': 404}
        print(response)
    else:
        result = response['result']
        status = response['status']

        print(result['headers'])   # returns response headers 
        print(result['data'])      # returns fetched html
        print(result['url'])       # returns fetched url
        print(response['status'])  # returns http status code

        save_result = scraper.save('/tmp/my-html.html')  # save to file
        if save_result.get('error', None):
            # we have save error
            pass
        else:
            print(save_result)    # contains saved file path and file size
            # {'file': '/tmp/my-html.html', 'size': 321322}

---

TODO

• Add xpath extractor.

License

This project is licensed under MIT

---

Contributer(s)

• Prompt API - Creator, maintainer

---

Contribute

All PR’s are welcome!

1. fork (https://github.com/promptapi/scraper-py/fork)
2. Create your branch (git checkout -b my-feature)
3. commit yours (git commit -am 'Add awesome features...')
4. push your branch (git push origin my-feature)
5. Than create a new Pull Request!

This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

---