Live forecast for the OCF nowcasting project
Project description
Nowcasting Forecast
Making live forecasts for the nowcasting project. The aim is to produce Grid Supply Point (GSP) Solar generation Forecasts. This is done by loading live PV results, Numerical Weather Predictions (NWPs) and Satellite images, and running this data through various different ML models.
You can run the application locally by:
python nowcasting_forecast/app.py --db-url='sqlite:///test.db'
Directories and files
The following shows the main files
+-- nowcasting_forecast
| +-- config
| +-- mvp_v0.yaml
| +-- database
| +-- connection.py
| +-- models.py
| +-- national.py
| +-- read.py
| +-- save.py
| +-- models
| +-- nwp_solar_simple.py
| +-- app.py
| +-- batch.py
| +-- dataloader.py
+-- scripts
+-- tests
☀️ nowcasting_forecast
app.py
is the main entry point for running the forecast module. It contains the following arguments:
- --db-url: the database url that the forecasts will be saved too
- --fake: Create fake forecast, mainly used for testing
The app has three main steps
- Make batched data
- Run forecasting model to make forecasts
- Save forecasts to the database
batch.py
is used to load the data and create batches of data. This uses mainly used ManagerLive
dataloader.py
is used to load the batched data in an efficient way. This is current a stripped down version of this dataloader.
The config
directory contains configurations that are used to load different data sources.
The configuration is used by batch.py
to convert large amounts of data into batched data, which is then ready for ML models.
The database
directory contains database models and functions to interact with the database. See below for more information
The model
directory contains ml models to make forecasts. See below for more information
Models
v0
This model is a simple model that uses NWP solar irradiance data to predict solar generation. ⚠️ This model takes the average solar irradiance and divides by 10, which will not produce a very accurate forecast. The main purpose of this model is to test the data pipeline.
The model performs the following steps:
- Creates a data loader
- Run simple model by looping over batches for all GSP
- Make national forecast
v1
TODO
Data model
The data model has been made using sqlalchemy
with a mirrored model in pydantic
.
⚠️ Database tables are currently made automatically, but in the future there should be a migration process
Future: The data model could be moved, to be a more modular solution.
models.py
All models are in nowcasting_forecast.database.models.py
.
The diagram below shows how the different tables are connected.
connection.py
nowcasting_forecast.database.connection.py
contains a connection class which can be used to make a sqlalchemy session.
from nowcasting_forecast.database.connection import DatabaseConnection
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
# do something with the database
pass
👓 read.py
nowcasting_forecast.database.read.py
contains functions to read the database.
The idea is that these are easy to use functions that query the database in an efficient and easy way.
- get_latest_forecast: Get the latest
Forecast
for a specific GSP. - get_all_gsp_ids_latest_forecast: Get the latest
Forecast
for all GSPs. - get_forecast_values: Gets the latest
ForecastValue
for a specific GSP - get_latest_national_forecast: Returns the latest national forecast
- get_location: Gets a
Location
object
from nowcasting_forecast.database.connection import DatabaseConnection
from nowcasting_forecast.database.read import get_latest_forecast
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
f = get_latest_forecast(session=session, gsp_id=1)
💾 save.py
nowcasting_forecast.database.save.py
has one functions to save a list of Forecast
to the database
🇬🇧 national.py
nowcasting_forecast.database.fake.py
has a useful function for adding up forecasts for all GSPs into a national Forecast.
fake.py
nowcasting_forecast.database.fake.py
Functions used to make fake model data.
🩺 Testing
Tests are run by using the following command
docker-compose -f test-docker-compose.yml run forecast
These sets up postgres
in a docker container and runs the tests in another docker container.
This slightly more complicated testing framework is needed (compared to running pytest
)
as some queries can not be fully tested on a sqlite
database
🛠️ infrastructure
.github/workflows
contains a number of CI actions
- linters.yaml: Runs linting checks on the code
- release.yaml: Make and pushes docker files on a new code release
- test-docker.yaml': Runs tests on every push
The docker file is in the folder infrastructure/docker/
The version is bumped automatically for any push to main
.
Environmental Variables
- DB_URL: The database url which the forecasts will be saved too
- NWP_PATH: Override NWP data path. THis is useful when running this locally.
- FAKE: Option to make fake/dummy forecasts
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nowcasting_forecast-0.0.51.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | aaa3dd5f04b2b27d6376866f06f9915177d589d92a301878a61c6dd472f9e1a3 |
|
MD5 | 9375b375120771d8d427304779d2230d |
|
BLAKE2b-256 | cf04793275e9a534cd807507b6a2356a3cb3e2e58d548cfaef0f1354166d15fc |
Hashes for nowcasting_forecast-0.0.51-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b3e9529011b45c9d79fd6bb1d8f674be2da154275f49e6cdd1f76b19b7327f3 |
|
MD5 | 6c3181f8cec9233f206809ab472ad751 |
|
BLAKE2b-256 | a057086a2d8d2bdff9eb7b527f3b1281aa80e063375eeb090515315c6a7bad0e |