Skip to main content

Python package to scrape NRC Event Reports.

Project description

Set of modules to scrape Event Reports from the NRC.gov website.

Tests

pytest 

Usage

Download all event reports into yearly csv files:

python bulk_downloader --start_year 2004 --end_year 2020
    # generate single event notification report event
    url3 = 'https://www.nrc.gov/reading-rm/doc-collections/event-status/event/2019/20190612en.html'

    h = EventNotificationReport.from_url(url3, headers)

    # get event notification urls from 2019 to 2020
    er_urls = generate_nrc_event_report_urls(2019, 2020)

    from random import sample
    urls = sample(list(extract_nested_values(er_urls)), 10)

    sl = logging.getLogger('success_log')
    el = logging.getLogger('error_log')
    fl = logging.getLogger('fof_log')

    # get a subsample of the urls, and log attempts to request 
    enrs, errors, fofs = fetch_enrs(urls)

    # convert an event to dataframe
    df = enrs[0].events[0].to_dataframe()

    # event notification report to dataframe
    enr_df = enrs[1].to_dataframe()

    # event notification reports to dataframe
    enrs_df = pd.concat([enr.to_dataframe() for enr in enrs])

    #dump to csv
    enrs_df.to_csv('enrs.csv')

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for nrc-scrape, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size nrc_scrape-0.0.4-py3-none-any.whl (10.9 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size nrc-scrape-0.0.4.tar.gz (9.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page