A package and CLI utility to check for available data and download tables from AEMO's Monthly Data Archive
Project description
AEMO Monthly Data Archive Tool
A CLI utility to find and obtain data made available through AEMO's MMS Monthly Data Archive.
Note
This package and its CLI have some overlap with NEMOSIS and NEMSEER. However:
- NEMOSIS does not provide access to the monthly data archive, and does not allow a user to download certain tables such as bid data tables (
BIDPEROFFER
).- NEMSEER only handles PASA and pre-dispatch tables available in the monthly data archive, though it provides a lot of useful utilities for handling these tables
Installation
Recommended that you use pipx
to install the CLI as it prevents dependency conflicts.
pipx install mms-monthly-cli
However, you can still install via pip
if you are only using the Python package:
pip install mms-monthly-cli
Usage
You can use this package as a module (source code functions within mms_monthly.py
) or as a CLI tool to:
- Inspect what data is available
- Download and unzip a table CSV for a particular period (i.e. a month within a year)
Python module
Simply import mms_monthly_cli
as follows:
import mms_monthly_cli.mms_monthly as mms_monthly
This will expose the following functions (accessed using mms_monthly.<func_name>
):
get_years_and_months() -> Dict[int, List[int]]
Years and months with data on NEMWeb MMSDM Historical Data Archive
Returns:
Months mapped to each year. Data is available for each of these months.
get_available_tables(year: int, month: int, data_dir: str) -> List[str]
Tables that can be requested from MMSDM Historical Data Archive for a
particular month and year.
Args:
year: Year
month: Month
data_dir: Directory within monthly archives
Returns:
List of tables associated with that forecast type for that period
get_table_names_and_sizes(year: int, month: int, data_dir: str) -> Dict
Returns table names and sizes from MMSDM Historical Data Archive page
For a year and month in the MMSDM Historical Data Archive, returns a list of
tuples each consisting of:
- A table name (obtained via captured regex group)
- The size of the associated zip file
Args:
year: Year
month: Month
data_dir: Directory within monthly archives
Returns:
Tuple of table names and file sizes
get_and_unzip_table_csv(year: int, month: int, data_dir: str, table: str, cache: pathlib.Path) -> None
Unzipped (single) csv file downloaded from `url` to `cache`
This function:
1. Downloads zip file in chunks to limit memory use and enable progress bar
2. Validates that the zip contains a single file that has the same name as the zip
Args:
year: Year
month: Month
data_dir: Directory within monthly archives
table: Table name
cache: Path to save zip.
Returns:
None. Extracts csv to `cache`
CLI tool
The CLI tool uses Typer.
GIF produced using asciinema and agg
Usage is documented within the tool:
mms-monthly-cli --help
...which returns:
Usage: mms-monthly-cli [OPTIONS] COMMAND [ARGS]...
A CLI utility to find and obtain data made available through AEMO's MMS Monthly Data
Archive: http://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/
╭─ Options ───────────────────────────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to copy it or │
│ customize the installation. │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────╮
│ available-periods Displays years and the months within them for which data is │
│ available │
│ available-tables Displays available tables for a period (i.e. supplied month and │
│ year) │
│ get-table Download and unzip monthly data zip file to get data table CSV in │
│ cache. To see available periods, use the `available_periods` command │
│ To see available tables for a given period, use the │
│ `available_tables` command │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
License
This tool and associated source code (reused from nemseer
, which is licensed under GNU GPL-3.0-or-later) was created by Abhijith Prakash.
This tool and its source code is licensed under the terms of GNU GPL-3.0-or-later licences.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for mms_monthly_cli-1.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd8a32f7561d63ca638d3f33702248593853da298814597ca2da21fc7591b7d2 |
|
MD5 | 97a1a8b1293cc875ee028a2ede0e6ac6 |
|
BLAKE2b-256 | 65ca0af2114b68cd7c2e831e16e4d119f530d1e1568549153518e4c2f981571a |