Data Model for the OCF nowcasting project
Project description
nowcasting_datamodel
Datamodel for the nowcasting project
The data model has been made using sqlalchemy
with a mirrored model in pydantic
.
⚠️ Database tables are currently made automatically, but in the future there should be a migration process
Future: The data model could be moved, to be a more modular solution.
nowcasting_datamodel
models.py
All models are in nowcasting_datamodel.models.py
.
The diagram below shows how the different tables are connected.
connection.py
nowcasting_datamodel.connection.py
contains a connection class which can be used to make a sqlalchemy session.
from nowcasting_datamodel.connection import DatabaseConnection
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
# do something with the database
pass
👓 read.py
nowcasting_datamodel.read.py
contains functions to read the database.
The idea is that these are easy to use functions that query the database in an efficient and easy way.
- get_latest_forecast: Get the latest
Forecast
for a specific GSP. - get_all_gsp_ids_latest_forecast: Get the latest
Forecast
for all GSPs. - get_forecast_values: Gets the latest
ForecastValue
for a specific GSP - get_latest_national_forecast: Returns the latest national forecast
- get_location: Gets a
Location
object
from nowcasting_datamodel.connection import DatabaseConnection
from nowcasting_datamodel.read import get_latest_forecast
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
f = get_latest_forecast(session=session, gsp_id=1)
💾 save.py
nowcasting_datamodel.save.py
has one functions to save a list of Forecast
to the database
🇬🇧 national.py
nowcasting_datamodel.fake.py
has a useful function for adding up forecasts for all GSPs into a national Forecast.
fake.py
nowcasting_datamodel.fake.py
Functions used to make fake model data.
🩺 Testing
Tests are run by using the following command
docker stop $(docker ps -a -q)
docker-compose -f test-docker-compose.yml build
docker-compose -f test-docker-compose.yml run tests
These sets up postgres
in a docker container and runs the tests in another docker container.
This slightly more complicated testing framework is needed (compared to running pytest
)
as some queries can not be fully tested on a sqlite
database
Mac M1 users
An upstream builds issue of libgp may cause the following error:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above
As suggested in this thread, a temporary fix is to set the env variable DOCKER_DEFAULT_PLATFORM=linux/amd64
prior to building the test images - although this reportedly comes with performance penalties.
🛠️ infrastructure
.github/workflows
contains a number of CI actions
- linters.yaml: Runs linting checks on the code
- release.yaml: Make and pushes docker files on a new code release
- test-docker.yaml': Runs tests on every push
The docker file is in the folder infrastructure/docker/
The version is bumped automatically for any push to main
.
Environmental Variables
- DB_URL: The database url which the forecasts will be saved too
Contributors ✨
Thanks goes to these wonderful people (emoji key):
Brandon Ly 💻 |
Chris Lucas 💻 |
James Fulton 💻 |
Rosheen Naeem 💻 |
Henri Dewilde 💻 |
This project follows the all-contributors specification. Contributions of any kind welcome!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nowcasting_datamodel-1.5.53.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8377db45ea08ce94354a99f8f7b9fdd11b7734133596a3edc75cc14f026d3a0f |
|
MD5 | 2c7671754957bc13ef2f679360d92563 |
|
BLAKE2b-256 | 4e725ab80a064c7258344aa03936dbb0e789a78d8e7deddce4786a2af209aec9 |
Hashes for nowcasting_datamodel-1.5.53-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 485ba18e2817178e43fac6e9e55c27b9dd5807eb782cdc42da3ed119d573cfec |
|
MD5 | b6f23a406281151226c3368569b5e051 |
|
BLAKE2b-256 | 3fbde006fd8dcd13f06cf266c5ef66bec1f8c239c4ee289f9852b0c7c623afdf |