irradpy2 0.2.7

Multi-source solar radiation dataset downloader and forecasting toolkit.

irradpy2 Developer README

1. Project Overview

irradpy2 is a Python toolkit for downloading, standardizing, and forecasting solar radiation data from multiple public data sources.

This document is intended for developers, maintainers, and collaborators who need to understand the internal structure of the project, modify existing modules, or extend the toolkit with new data sources and forecasting workflows.

The project currently includes two major parts:

1. Data acquisition modules
2. Forecasting modules

---

2. Project Structure

A typical structure is as follows:

irradpy2/
├── __init__.py
├── necessary_main.py
├── midc.py
├── bsrn.py
├── sauran.py
├── solrad.py
├── surfrad.py
├── srml.py
├── forecast.py
└── README.md

The downloader modules are named after their corresponding data-source websites, while forecast.py is used for forecasting.

3. Core Modules

3.1 necessary_main.py

This file contains shared utility functions used by multiple modules.

Typical responsibilities include:

• date initialization and normalization
• URL construction
• HTTP downloading with retry
• directory creation
• timezone-safe localization
• UTC offset formatting
• timestamp clipping
• local time attachment

This file mainly provides shared utility functions for the downloader modules, including date handling, URL construction, directory creation, and timezone-related processing.

---

3.2 midc.py

This module downloads and standardizes radiation data from the NREL MIDC website.

Main responsibilities:

• split requests by month
• download monthly data
• parse MIDC responses
• reconstruct timestamps
• handle timezone localization and DST ambiguity
• keep selected site-specific variables
• export standardized CSV output

Public API:

download_midc(site, begin, end, save_path)

MIDC homepage：https://midcdmz.nrel.gov/apps/data_api_doc.pl?BMS

3.3 bsrn.py

This module downloads and parses BSRN shortwave radiation data.

Main responsibilities:

• connect to the BSRN FTP server
• download monthly source files
• extract compressed files
• parse radiation records
• generate standardized CSV output
• sort results by UTC timestamp
• clean temporary files

Public API:

download_bsrn(site, begin, end, username, password, save_path)

BSRN homepage： https://bsrn.awi.de/

3.4 sauran.py

This module downloads SAURAN data and converts it into a unified CSV format.

Main responsibilities:

• build API request URLs
• handle HTTPS and HTTP fallback
• parse response headers and data rows
• support multiple timestamp formats
• convert local timestamps to UTC
• generate standardized time fields
• export CSV output

Public API:

download_sauran(site, begin, end, save_path)

SAURAN homepage： https://sauran.ac.za/

3.5 solrad.py

This module downloads NOAA SOLRAD radiation data.

Main responsibilities:

• build URLs for realtime and archive data
• parse metadata lines
• parse fixed-format data rows
• apply quality-control filtering
• generate standardized time fields
• export CSV output

Public API:

download_solrad(site, begin, end, save_path)

SOLRAD homepage：https://gml.noaa.gov/grad/solrad/

3.6 surfrad.py

This module downloads NOAA SURFRAD radiation data.

Main responsibilities:

• build candidate URLs
• parse station data rows
• apply quality-control filtering
• deduplicate records by UTC timestamp
• export standardized CSV output

Public API:

download_surfrad(site, begin, end, save_path)

SURFRAD homepage:https://gml.noaa.gov/grad/surfrad/index.html

3.7 srml.py

This module downloads SRML data from the University of Oregon.

Main responsibilities:

• map user-facing site names to file prefixes
• try multiple SRML file types
• parse monthly ASCII files
• decode variable codes and quality flags
• reconstruct measurement timestamps
• localize timestamps and generate UTC time
• export standardized CSV output

Public API:

download_srml(site, begin, end, save_path)

SRML homepage:http://solardata.uoregon.edu/index.html

3.8 forecast.py

This module provides the forecasting pipeline for standardized radiation datasets.

Main responsibilities:

• automatically match common meteorological columns
• standardize input CSV files
• construct sequence datasets
• support RNN, LSTM, and Informer-style models
• train, validate, and test models
• save checkpoints and prediction results

Public API:

run_forecast(csv_path, save_path, model, seq_len=72, horizon=6, epochs=50, batch_size=64)

---

4. Standard Output Format

Most downloader modules produce CSV files with standardized time-related columns such as:

• UTC_Time
• Local_Time
• UTC_Offset

Depending on the source, common radiation and meteorological variables may include:

• GHI
• DNI
• DHI
• AirTemp
• RH
• Pressure
• WindSpeed
• WindDir

---

This file mainly provides shared utility functions for the downloader modules, including date handling, URL construction, directory creation, and timezone-related processing.

5. Extensibility

In this package, there are two main places where users can extend the implementation according to their own needs.

5.1 Extend site-specific variable mappings in the MIDC module

Users can modify the site-specific variable dictionary in the MIDC module to support additional stations or adjust the retained variables for existing stations.

COLUMNS_TO_KEEP = {
    'BMS': {
        'Global CMP22 (vent/cor) [W/m^2]': 'ghi',
        'Direct CHP1-1 [W/m^2]': 'dni_chp1',
        'Direct NIP [W/m^2]': 'dni_nip',
        'Diffuse CM22-1 (vent/cor) [W/m^2]': 'dhi',
        'Avg Wind Speed @ 6ft [m/s]': 'wind_speed',
        'Tower Dry Bulb Temp [deg C]': 'temp_air',
        'Station Pressure [mBar]': 'air_pressure',
        'Tower RH [%]': 'relative_humidity'
    },
    'UOSMRL': {
        'Global CMP22 [W/m^2]': 'ghi',
        'Direct CHP1 [W/m^2]': 'dni_chp1',
        'Diffuse [W/m^2]': 'dhi',
        'Direct NIP [W/m^2]': 'dni_nip',
        'Air Temperature [deg C]': 'temp_air',
        'Relative Humidity [%]': 'relative_humidity',
        'Avg Wind Speed @ 10m [m/s]': 'wind_speed'
    }
}

By extending this dictionary, users can control which variables are kept and how they are mapped for different MIDC sites.

5.2 Extend column matching rules in the forecasting module

Users can adapt the forecasting module to their own datasets by modifying the column-matching dictionary used for automatic feature detection.

COLUMN_KEYWORDS = {
    "Global": [
        "global", "ghi", "avg_global", "global_irradiance",
        "total_irradiance", "dw_psp", "dwpsp",
        "dw_solar", "dwsolar", "ghi_", "ghi2"
    ],
    "Direct": [
        "direct", "dni", "avg_direct",
        "beam_irradiance", "dnic", "dnic"
    ],
    "Diffuse": [
        "diffuse", "dhi", "diffuse_irradiance"
    ],
    "AirTemp": [
        "airtemp", "temp", "temperature", "tair",
        "drybulb", "airtc", "towerdrybulb"
    ],
    "RH": [
        "rh", "humidity", "relative_humidity"
    ],
    "Pressure": [
        "pressure", "press", "mbar", "bp_mbar"
    ]
}

By updating this dictionary, users can make the forecasting pipeline recognize the column names used in their own datasets.