Skip to main content

Python package to scrape NRC Event Reports.

Project description

Set of modules to scrape Event Reports from the website.




Download all event reports into yearly csv files at .\data\ :

python bulk_downloader --start_year 2004 --end_year 2020 --threads 16

Or go one by one:

    # import a Session that has correct properties to keep a connnection
    from nrc_scrape import session

    # generate single event notification report event
    url3 = ''

    h = EventNotificationReport.from_url(url3, session)

    # get event notification urls from 2019 to 2020
    er_urls = generate_nrc_event_report_urls(2019, 2020)

    from random import sample
    urls = sample(list(extract_nested_values(er_urls)), 10)

    sl = logging.getLogger('success_log')
    el = logging.getLogger('error_log')
    fl = logging.getLogger('fof_log')

    # get a subsample of the urls, and log attempts to request 
    enrs, errors, fofs = fetch_enrs(urls, session)

    # convert an event to dataframe
    df = enrs[0].events[0].to_dataframe()

    # event notification report to dataframe
    enr_df = enrs[1].to_dataframe()

    # event notification reports to dataframe
    enrs_df = pd.concat([enr.to_dataframe() for enr in enrs])

    #dump to csv

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for nrc-scrape, version 0.0.5
Filename, size File type Python version Upload date Hashes
Filename, size nrc_scrape-0.0.5-py3-none-any.whl (6.9 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size nrc-scrape-0.0.5.tar.gz (5.7 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page