Skip to main content

A dashboard for visualising bidding data from the Australian National Electricity Market

Project description

Introduction

nem-bidding-dashboard is a web app and python package for collating, processing and visualising data relevant to understanding participant behaviour in the Australian National Electricity Market wholesale spot market.

The web app is intended to make reviewing the bidding behaviour of market participants as easy as possible. Aggregate behaviour can be visualised at a whole of market, regional, or technology level. Alternatively, the non-aggregated bids of dispatch units, and stations can be visualised.

We have additionally published the code required to run the web app as a python package, so that it can be used to help visualise and analyse bidding behaviour in alternative or more sophisticated ways than allowed by the web app.

The development of nem-bidding-dashboard was funded by the Digital Grid Futures Institute

Web app

nem-bidding-dashboard is hosted as a web app here: https://nembiddingdashboard.org

Python package / API

The python api can be used to:

  • run the web app interface locally
  • download publicly available bidding and operational data from the Australian Energy Market Operator
  • process and aggregate bidding and operational data into the format used in the web app
  • build and populate a PostgresSQL database for efficiently querying and aggregating bidding and operational data

Installation

pip install nem_bidding_dashboard

Quick examples

Below are some quick examples that provide a taste of the api capabilities, see the full set of examples and api documentation for a complete guide.

Get raw data

To get the raw data used by nem-bidding-dashboard before preprocessing use functions in the fetch_data module, e.g. get_volume_bids.

from nem_bidding_dashboard import fetch_data

volume_bids = fetch_data.volume_bids(
    start_time='2022/01/01 00:00:00',
    end_time='2022/01/01 00:05:00',
    raw_data_cache='D:/nemosis_data_cache')

print(volume_bids.head(5))
#        SETTLEMENTDATE     DUID  ... PASAAVAILABILITY   INTERVAL_DATETIME
# 309360     2021-12-31  ADPBA1G  ...              6.0 2022-01-01 00:05:00
# 309361     2021-12-31  ADPBA1G  ...              NaN 2022-01-01 00:05:00
# 309362     2021-12-31  ADPBA1G  ...              NaN 2022-01-01 00:05:00
# 309363     2021-12-31  ADPBA1G  ...              NaN 2022-01-01 00:05:00
# 309364     2021-12-31  ADPBA1G  ...              NaN 2022-01-01 00:05:00
#
# [5 rows x 18 columns]

Get processed data

To get data in the format stored by nem-bidding-dashboard in the PostgresSQL database use functions in the module fetch_and_preprocess, e.g. bid_data.

from nem_bidding_dashboard import fetch_and_preprocess

bids = fetch_and_preprocess.bid_data(
    start_time='2022/01/01 00:00:00',
    end_time='2022/01/01 00:05:00',
    raw_data_cache='D:/nemosis_data_cache')

print(bids.head(5))
#        INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDVOLUMEADJUSTED  BIDPRICE  ONHOUR
# 0    2022-01-01 00:05:00  ADPBA1G        8          6                0.0    998.00   False
# 462  2022-01-01 00:05:00   REECE1        2         45               45.0    -55.03   False
# 463  2022-01-01 00:05:00   REECE1        4         74               74.0     -0.85   False
# 464  2022-01-01 00:05:00   REECE2        2         35               35.0    -54.77   False
# 465  2022-01-01 00:05:00   REECE2        4         84               84.0     -0.86   False

Setup a PostgresSQL database

Create tables for storing processed data and functions, then populate the database with historical data.

from nem_bidding_dashboard import postgres_helpers, populate_postgres_db

con_string = postgres_helpers.build_connection_string(
    hostname='localhost',
    dbname='bidding_dashboard_db',
    username='bidding_dashboard_maintainer',
    password='1234abcd',
    port=5433)

raw_data_cache = "D:/nemosis_cache"
start = "2022/01/01 00:00:00"
end = "2022/02/01 00:00:00"

populate_postgres_db.duid_info(con_string, raw_data_cache)
populate_postgres_db.bid_data(con_string, raw_data_cache, start, end)
populate_postgres_db.region_data(con_string, raw_data_cache, start, end)
populate_postgres_db.unit_dispatch(con_string, raw_data_cache, start, end)

Query and aggregate bidding data from PostgresSQL database

Filter bids by time and region, and then aggregate into price bands. Other functions in the module query_postgres_db provide querying and aggregation and for each table in the db.

from nem_bidding_dashboard import postgres_helpers, query_postgres_db

con_string = postgres_helpers.build_connection_string(
    hostname='localhost',
    dbname='bidding_dashboard_db',
    username='bidding_dashboard_maintainer',
    password='1234abcd',
    port=5433)

agg_bids = query_postgres_db.aggregate_bids(connection_string=con_string,
                                            start_time="2022/01/01 00:00:00",
                                            end_time="2022/01/01 01:00:00",
                                            regions=['QLD', 'NSW', 'SA'],
                                            dispatch_type='Generator',
                                            tech_types=[],
                                            resolution='hourly',
                                            adjusted='adjusted')

print(agg_bids)
#       INTERVAL_DATETIME        BIN_NAME   BIDVOLUME
# 0   2022-01-01 01:00:00   [-1000, -100)  9673.93400
# 1   2022-01-01 01:00:00       [-100, 0)   366.70236
# 2   2022-01-01 01:00:00         [0, 50)  1527.00000
# 3   2022-01-01 01:00:00       [50, 100)  1290.00000
# 4   2022-01-01 01:00:00      [100, 200)   908.00000
# 5   2022-01-01 01:00:00      [200, 300)  1217.00000
# 6   2022-01-01 01:00:00      [300, 500)   943.00000
# 7   2022-01-01 01:00:00     [500, 1000)   240.00000
# 8   2022-01-01 01:00:00    [1000, 5000)   210.00000
# 9   2022-01-01 01:00:00   [5000, 10000)   125.00000
# 10  2022-01-01 01:00:00  [10000, 15500)  6766.00000

Contributing

Interested in contributing? Check out the contributing guidelines.

Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.

License and Disclaimer

nem-bidding-dashboard was created by Nicholas Gorman and Patrick Chambers. It is licensed under the terms of the BSD-3-Clause license. Please, also read the disclaimer accompanying the licence

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nem_bidding_dashboard-1.0.8.tar.gz (57.9 kB view details)

Uploaded Source

Built Distribution

nem_bidding_dashboard-1.0.8-py3-none-any.whl (69.2 kB view details)

Uploaded Python 3

File details

Details for the file nem_bidding_dashboard-1.0.8.tar.gz.

File metadata

  • Download URL: nem_bidding_dashboard-1.0.8.tar.gz
  • Upload date:
  • Size: 57.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.2.2 CPython/3.8.6 Windows/10

File hashes

Hashes for nem_bidding_dashboard-1.0.8.tar.gz
Algorithm Hash digest
SHA256 8b89aec36141afe05950b312dcf9ea0c0f5419203773f467c586b5a449dcbb44
MD5 37a61c2b09ff4d6e132e75becc7b2b9e
BLAKE2b-256 0a15f46d46f424cdb6072aaaf06bd57ec04037610b93092c3da24bfed7db74eb

See more details on using hashes here.

File details

Details for the file nem_bidding_dashboard-1.0.8-py3-none-any.whl.

File metadata

File hashes

Hashes for nem_bidding_dashboard-1.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 cf0102d88e5a21630df12b4b6dd000b1fb3d11a7a24f6c2d1ecdb670d26f50b8
MD5 b27a86cee646a503181d9c1351ede226
BLAKE2b-256 246fe462d6cb8d4cd5ce6deb038c3f23f72b5e650cf5e02aa0225b0d88a92233

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page