OpenNEM Australian Energy Data
Project description
OpenNEM Energy Market Data Access
The OpenNEM project aims to make the wealth of public National Electricity Market (NEM) data more accessible to a wider audience.
This toolkit enables downloading, mirroring and accessing energy data from various networks
Project homepage at https://opennem.org.au
Available on Docker at https://hub.docker.com/r/opennem/opennem
Currently supports:
- Australian NEM: https://www.nemweb.com.au/
- West Australia Energy Market: http://data.wa.aemo.com.au/
Requirements
- Python 3.7+ (see
.python-version
withpyenv
) - Docker and
docker-compose
if you want to run the local dev stack
Quickstart
With poetry:
$ poetry install
$ source .venv/bin/activate
$ ./init.sh
With pip + venv:
$ pip -m venv .venv
$ pip install -r requirements.txt
$ source .venv/bin/activate
$ ./init.sh
Install
You can install this project with python pip
:
$ pip install opennem
Or alternatively with docker:
$ docker pull opennem/opennem
Bundled with sqlite support. Other database drivers are optional and not installed by default. Install a supported database driver:
Postgres:
$ pip install psycopg2
Install Extras
The package contains extra modules that can be installed:
$ poetry install -E postgres
The list of extras are:
postgres
- Postgres database driversserver
- API server
Usage
List the crawlers
$ scrapy list
Crawl
$ scrapy crawl au.nem.current.dispatch_scada
Development
This project uses the new pyproject.toml
project and build specification file. To make use of it use the poetry
tool which can be installed on Windows, MacOS and Linux:
https://python-poetry.org/docs/
Installation instructions for Poetry are at:
https://python-poetry.org/docs/#installation
By default poetry will install virtual environments in your home metadata directory. A good alternative is to install the venv
locally for each project with the following setting:
$ poetry config virtualenvs.in-project true
This will create the virtual environment within the project folder in a folder called .venv
. This folder is ignored by git by default.
Setting up a virtual environment and installing requiements using Poetry:
$ poetry install
To activate the virtual environment either run:
$ poetry shell
Or you can just activate the standard venv
$ source .venv/bin/activate
Settings are read from environment variables. Environment variables can be read from a .env
file in the root of the folder. Setup the environment by copying the .env.sample
file to .env
. The defaults in the sample file map to the settings in docker-compose.yml
There is a docker-compose
file that will bring a local database:
$ docker-compose up -d
Bring up the database migrations using alembic:
$ alembic upgrade head
Run scrapy in the root folder for options:
$ scrapy
The opennem
cli interface provides other options and settings:
$ opennem -h
Settings for Visual Studio Code are stored in .vscode
. Code is kept formatted and linted using pylint
, black
and isort
with settings defined in pyproject.toml
Testing
Tests are in tests/
Run tests with:
$ pytest
Run background test watcher with
$ ptw
Build Release
The script build-release.sh
will tag a new release, build the docker image, tag the git version, push to GitHub and push the latest
release to PyPi
Architecture overview
This project uses Scrapy to obtain data from supported energy markets and SQLAlchemy to store data, and Alembic for database migrations. Database storage has been tested with sqlite, postgres and mysql.
Overview of scrapy architecture:
Code Navigation
- Spider definitions in
opennem/spiders
- Processing pipelines for crawls in
opennem/pipelines
- Database models for supported energy markets are stored in
opennem/db/models
Deploy Crawlers
You can deploy the crawlers to the scrapyd server with:
$ scrapyd-deploy
If you don't have that command and it isn't available install it with:
$ pip install scrapyd-client
Which installs the scrapyd-client tools. Project settings are read from scrapy.cfg
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for opennem-3.3.0a3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05600c9e251fff212b3d16b24ddf715bfc58e3b1061e414c7c1a6de699d0698b |
|
MD5 | 48f7c05d6df4a58f1b4da19c7f009f01 |
|
BLAKE2b-256 | c823d930c3dbf929d12a5741f773e0edca43792ef2ebbba78e7c2f11b550392e |