Skip to main content

Data Model for the OCF nowcasting project

Project description

nowcasting_datamodel

All Contributors

Datamodel for the nowcasting project

The data model has been made using sqlalchemy with a mirrored model in pydantic.

⚠️ Database tables are currently made automatically, but in the future there should be a migration process

Future: The data model could be moved, to be a more modular solution.

nowcasting_datamodel

models.py

All models are in nowcasting_datamodel.models.py. The diagram below shows how the different tables are connected.

Models Models

connection.py

nowcasting_datamodel.connection.py contains a connection class which can be used to make a sqlalchemy session.

from nowcasting_datamodel.connection import DatabaseConnection

# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')

# make sessions
with db_connection.get_session() as session:

    # do something with the database
    pass

👓 read.py

nowcasting_datamodel.read.py contains functions to read the database. The idea is that these are easy to use functions that query the database in an efficient and easy way.

  • get_latest_forecast: Get the latest Forecast for a specific GSP.
  • get_all_gsp_ids_latest_forecast: Get the latest Forecast for all GSPs.
  • get_forecast_values: Gets the latest ForecastValue for a specific GSP
  • get_latest_national_forecast: Returns the latest national forecast
  • get_location: Gets a Location object
from nowcasting_datamodel.connection import DatabaseConnection
from nowcasting_datamodel.read import get_latest_forecast

# make connection object
db_connection = DatabaseConnection(url='sqlite:///test.db')

# make sessions
with db_connection.get_session() as session:
    f = get_latest_forecast(session=session, gsp_id=1)

💾 save.py

nowcasting_datamodel.save.py has one functions to save a list of Forecast to the database

🇬🇧 national.py

nowcasting_datamodel.fake.py has a useful function for adding up forecasts for all GSPs into a national Forecast.

fake.py

nowcasting_datamodel.fake.py

Functions used to make fake model data.

🩺 Testing

Tests are run by using the following command

docker stop $(docker ps -a -q)
docker-compose -f test-docker-compose.yml build
docker-compose -f test-docker-compose.yml run tests

These sets up postgres in a docker container and runs the tests in another docker container. This slightly more complicated testing framework is needed (compared to running pytest) as some queries can not be fully tested on a sqlite database

Mac M1 users

An upstream builds issue of libgp may cause the following error:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SCRAM authentication requires libpq version 10 or above

As suggested in this thread, a temporary fix is to set the env variable DOCKER_DEFAULT_PLATFORM=linux/amd64 prior to building the test images - although this reportedly comes with performance penalties.

🛠️ infrastructure

.github/workflows contains a number of CI actions

  1. linters.yaml: Runs linting checks on the code
  2. release.yaml: Make and pushes docker files on a new code release
  3. test-docker.yaml': Runs tests on every push

The docker file is in the folder infrastructure/docker/

The version is bumped automatically for any push to main.

Environmental Variables

  • DB_URL: The database url which the forecasts will be saved too

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Brandon Ly
Brandon Ly

💻
Chris Lucas
Chris Lucas

💻
James Fulton
James Fulton

💻
Rosheen Naeem
Rosheen Naeem

💻
Henri Dewilde
Henri Dewilde

💻

This project follows the all-contributors specification. Contributions of any kind welcome!

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nowcasting_datamodel-1.5.53.tar.gz (216.2 kB view details)

Uploaded Source

Built Distribution

nowcasting_datamodel-1.5.53-py3-none-any.whl (45.8 kB view details)

Uploaded Python 3

File details

Details for the file nowcasting_datamodel-1.5.53.tar.gz.

File metadata

  • Download URL: nowcasting_datamodel-1.5.53.tar.gz
  • Upload date:
  • Size: 216.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for nowcasting_datamodel-1.5.53.tar.gz
Algorithm Hash digest
SHA256 8377db45ea08ce94354a99f8f7b9fdd11b7734133596a3edc75cc14f026d3a0f
MD5 2c7671754957bc13ef2f679360d92563
BLAKE2b-256 4e725ab80a064c7258344aa03936dbb0e789a78d8e7deddce4786a2af209aec9

See more details on using hashes here.

File details

Details for the file nowcasting_datamodel-1.5.53-py3-none-any.whl.

File metadata

File hashes

Hashes for nowcasting_datamodel-1.5.53-py3-none-any.whl
Algorithm Hash digest
SHA256 485ba18e2817178e43fac6e9e55c27b9dd5807eb782cdc42da3ed119d573cfec
MD5 b6f23a406281151226c3368569b5e051
BLAKE2b-256 3fbde006fd8dcd13f06cf266c5ef66bec1f8c239c4ee289f9852b0c7c623afdf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page