opportunity-scraper 0.1.6

Web scrapers for various innovation opportunity sites

opportunity_scraper

opportunity_scraper is a Python3 script for scraping R&D competition websites and dumping the results into the SuiteCRM V8 API.

Installation

Install with pip:

pip3 install opportunity_scraper

Dependencies

opportunity_scraper uses chromedriver, and will not work without it. Download the appropriate chromedriver version for your Chrome or Chromium version. Some distros (Fedora, at the very least) may have a packaged version that corresponds to the versions of Chrome available.

Supported resources

• UK Government Innovation Funding Service
• KTN Innovation Exchange (requires registration)

Before use

SuiteCRM OAuth2 credentials

Before using the scraper, you need to create some SuiteCRM OAuth2 client credentials for the script to use.

Click for the official SuiteCRM documentation. It's a little confusing on what to do, so for clarity:

• Navigate to the OAuth2 Clients and Tokens admin page (https://www.your-suitecrm-instance.com/suitecrm/index.php?module=OAuth2Clients)
• Create a new Client Credentials Client, by clicking "New Client Credentials Client", giving it a name, and entering a secret password in the input box labelled "Change secret". Despite the wording, do not leave the box blank when creating your credentials.
• After saving, you'll be presented with a Client ID.

The Client ID and Client Secret will be used in configuration.

Configuration

Configuration is via a YAML file.

The default location is ~/.config/opportunity_scraper/settings.yaml, but this can be changed with the --config CLI option at runtime.

The following can be used as a template for the config file. All values are required.

browser:
    # Location of chromedriver
    chromedriver: "/usr/bin/chromedriver"
    # Credentials for KTN Innovation Exchange
    ktn_username: "[e-mail]"
    ktn_password: "somesecretpassword"

oauth:
    # SuiteCRM OAuth2 credentials
    token_url: "https://example.com/suitecrm/Api/access_token"
    client_id: "some-uuid-token"
    client_secret: "anotherpassword"

suitecrm:
    # SuiteCRM API
    api_url: "https://example.com/suitecrm/Api"
    # Sales Account ID for .gov.uk competitions
    govuk_account_id: "another-uuid"
    # Sales Account ID for KTN competitions
    ktn_account_id: "another-uuid"
    # Default user to be assigned new opportunities
    assigned_user_id: "1"

Notes on configuration

browser

• chromedriver: path to the chromedriver executable. Depending on whether you install it from a distro package or the Chromedriver website, it may or may not end up in $PATH, so for simplicity we specify it here.
• ktn_username, ktn_password: KTN Innovation Exchange credentials, which can be created here.

oauth

• token_url: Full URL to the SuiteCRM V8 API OAuth2 endpoint - usually of the form https://your-suitecrm-instance.com/suitecrm/Api/access_token
• client_id, client_secret: These must be created in the SuiteCRM admin, as above.

suitecrm

• api_url: Full URL to the SuiteCRM V8 API - usually of the form https://your-suitecrm-instance.com/suitecrm/Api.
• govuk_account_id, ktn_account_id: The ID (in UUID form) of the SuiteCRM Sales Account that will be assigned to new opportunities scraped from the gov.uk and KTN sites. The UUID can be found in the URL when examining a particular Sales Account in SuiteCRM (https://your-suitecrm-instance.com/suitecrm/index.php?module=Accounts&action=index).

The same SuiteCRM account can be used for .gov.uk- and KTN-sourced opportunities if desired - just use the same ID for both values.

Running

usage: opportunity_scraper [-h] [-c CONFIG]

Scrape R&D competitions and push the results to the SuiteCRM API.

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Location of config file

License

GPLv3