AEMO Monthly Data Archive Tool

A CLI utility to find and obtain data made available through AEMO's MMS Monthly Data Archive.

Note

This package and its CLI have some overlap with NEMOSIS and NEMSEER. However:

• NEMOSIS does not provide access to the monthly data archive, and does not allow a user to download certain tables such as bid data tables (BIDPEROFFER).
• NEMSEER only handles PASA and pre-dispatch tables available in the monthly data archive, though it provides a lot of useful utilities for handling these tables

Installation

Recommended that you use pipx to install the CLI as it prevents dependency conflicts.

pipx install mms-monthly-cli

However, you can still install via pip if you are only using the Python package:

pip install mms-monthly-cli

Usage

You can use this package as a module (source code functions within mms_monthly.py) or as a CLI tool to:

1. Inspect what data is available
2. Download and unzip a table CSV for a particular period (i.e. a month within a year)

Python module

Simply import mms_monthly_cli as follows:

import mms_monthly_cli.mms_monthly as mms_monthly

This will expose the functions listed below (accessed using mms_monthly.<func_name>).

[!NOTE] The data_dir argument requires you to specify which folder within MMSDM_YYYY_MM you are interested in. Most users will be interested in tables within DATA (i.e. data_dir="DATA").

You can also access PREDISP_ALL_DATA (pre-dispatch data) and P5MIN_ALL_DATA (5-minute pre-dispatch data). Data tables in the PREDISP_ALL_DATA folders contain all forecasts for an interval of interest, whereas PREDISPATCH data tables in the DATA folder only contain the latest interval (refer to this part of the NEMSEER docs for more detail). It is unclear if the same is true for P5MIN_ALL_DATA; P5MIN data tables in DATA appear have the same data with some minor differences (e.g. one column name is different in the constraint files).

If you are accessing pre-dispatch data, consider using NEMSEER.

get_years_and_months() -> Dict[int, List[int]]

Years and months with data on NEMWeb MMSDM Historical Data Archive
Returns:
    Months mapped to each year. Data is available for each of these months.

get_available_tables(year: int, month: int, data_dir: str) -> List[str]

Tables that can be requested from MMSDM Historical Data Archive for a
   particular month and year.
Args:
    year: Year
    month: Month
    data_dir: Directory within monthly archives

Returns:
    List of tables associated with that forecast type for that period

get_table_names_and_sizes(year: int, month: int, data_dir: str) -> Dict

Returns table names and sizes from MMSDM Historical Data Archive page

For a year and month in the MMSDM Historical Data Archive, returns a list of
tuples each consisting of:
- A table name (obtained via captured regex group)
- The size of the associated zip file

Args:
    year: Year
    month: Month
    data_dir: Directory within monthly archives
Returns:
    Tuple of table names and file sizes

get_and_unzip_table_csv(year: int, month: int, data_dir: str, table: str, cache: pathlib.Path) -> None

Unzipped (single) csv file downloaded from `url` to `cache`

This function:

1. Downloads zip file in chunks to limit memory use and enable progress bar
2. Validates that the zip contains a single file that has the same name as the zip

Args:
    year: Year
    month: Month
    data_dir: Directory within monthly archives
    table: Table name
    cache: Path to save zip.
Returns:
    None. Extracts csv to `cache`

CLI tool

The CLI tool uses Typer.

_{^{GIF produced using asciinema and agg}}

Usage is documented within the tool:

mms-monthly-cli --help

...which returns:

Usage: mms-monthly-cli [OPTIONS] COMMAND [ARGS]...

 A CLI utility to find and obtain data made available through AEMO's MMS Monthly Data
 Archive: http://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/

╭─ Options ───────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                 │
│ --show-completion             Show completion for the current shell, to copy it or      │
│                               customize the installation.                               │
│ --help                        Show this message and exit.                               │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────╮
│ available-periods  Displays years and the months within them for which data is          │
│                    available                                                            │
│ available-tables   Displays available tables for a period (i.e. supplied month and      │
│                    year)                                                                │
│ get-table          Download and unzip monthly data zip file to get data table CSV in    │
│                    cache. To see available periods, use the `available_periods` command │
│                    To see available tables for a given period, use the                  │
│                    `available_tables` command                                           │
╰─────────────────────────────────────────────────────────────────────────────────────────╯

License

This tool and associated source code (reused from nemseer, which is licensed under GNU GPL-3.0-or-later) was created by Abhijith Prakash with contributions from Matthew Davis.

This tool and its source code is licensed under the terms of GNU GPL-3.0-or-later licences.