Data Model for the OCF nowcasting project
Project description
nowcasting_datamodel
Datamodel for the nowcasting project
The data model has been made using sqlalchemy
with a mirrored model in pydantic
.
⚠️ Database tables are currently made automatically, but in the future there should be a migration process
Future: The data model could be moved, to be a more modular solution.
nowcasting_datamodel
models.py
All models are in nowcasting_datamodel.models.py
.
The diagram below shows how the different tables are connected.
connection.py
nowcasting_datamodel.connection.py
contains a connection class which can be used to make a sqlalchemy session.
from nowcasting_datamodel.connection import DatabaseConnection
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
# do something with the database
pass
👓 read.py
nowcasting_datamodel.read.py
contains functions to read the database.
The idea is that these are easy to use functions that query the database in an efficient and easy way.
- get_latest_forecast: Get the latest
Forecast
for a specific GSP. - get_all_gsp_ids_latest_forecast: Get the latest
Forecast
for all GSPs. - get_forecast_values: Gets the latest
ForecastValue
for a specific GSP - get_latest_national_forecast: Returns the latest national forecast
- get_location: Gets a
Location
object
from nowcasting_datamodel.connection import DatabaseConnection
from nowcasting_datamodel.read import get_latest_forecast
# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')
# make sessions
with db_connection.get_session() as session:
f = get_latest_forecast(session=session, gsp_id=1)
💾 save.py
nowcasting_datamodel.save.py
has one functions to save a list of Forecast
to the database
🇬🇧 national.py
nowcasting_datamodel.fake.py
has a useful function for adding up forecasts for all GSPs into a national Forecast.
fake.py
nowcasting_datamodel.fake.py
Functions used to make fake model data.
🩺 Testing
Tests are run by using the following command
docker-compose -f test-docker-compose.yml run datamodel
These sets up postgres
in a docker container and runs the tests in another docker container.
This slightly more complicated testing framework is needed (compared to running pytest
)
as some queries can not be fully tested on a sqlite
database
🛠️ infrastructure
.github/workflows
contains a number of CI actions
- linters.yaml: Runs linting checks on the code
- release.yaml: Make and pushes docker files on a new code release
- test-docker.yaml': Runs tests on every push
The docker file is in the folder infrastructure/docker/
The version is bumped automatically for any push to main
.
Environmental Variables
- DB_URL: The database url which the forecasts will be saved too
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nowcasting_datamodel-0.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3fd7ed3897fa033a0e1d47946c255c44f137bb910a7ae12cf391e2f74add5278 |
|
MD5 | 7adf5b9e8376c360fd0f354a1bd688ec |
|
BLAKE2b-256 | 00bd4589f5fedc342e62337fd961ec3185e50f40ec3a68da4ac22fc6d0c97036 |
Hashes for nowcasting_datamodel-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ed148e9247743638e5fa0ecd9d30b97b5c37c5b548523c7cd744d233fb415df |
|
MD5 | bf25fde772c4ee10feff823e1167af39 |
|
BLAKE2b-256 | 2db976eb9492f3b39a5317a5408ee759dbb7f48b9b0676c792c63ecb86e0f862 |