Skip to main content

An I/O HDF5-based utility to assist in the distribution, execution and integration of data coming from different sources (e.g. results of mathematical optimization models).

Project description

dirB

A generic tool working over HDF5 files to share experiments data, both input and output, in a standardized way.

JSON dictionaries are the default objects used within the HDF5 files in order to collect input parameters and resulting data.

Features

The main features available in dirB are:

  • Create new cases with input parameters and metadata (dirB.base.creaNuevoCaso(inputParamsDict, metadataDict))
  • Load already existing cases from HDF5 files (dirB.base.cargaCaso(hdf5FileName))
  • Merge several files containing solutions (dirB.merge.mergeListaDeSoluciones(baseCase, listOfFilesWithSolutions, outputFile))

Features to be implemented:

  • Retrieval of solutions fulfilling a certain criteria, in order to process all them together

Examples

To create a new case from scratch:

from dirB.base import zsan_DirB
from datetime import date, datetime
import time
import shutil

diccionarioCasoBase = { 'x': 12,
                        'y': 7}
diccionarioAtributosCasoBase = {
                        'Autor': 'Steve Vai',
                        'Dia':  str(date.today()),
                        'Escenario': 'Acido',
                        'Numero Caso': 23
                        }

casoBase = zsan_DirB()
casoBase.creaNuevoCaso(diccionarioCasoBase, diccionarioAtributosCasoBase, nombreDeFichero = "casoBase.hdf5")

For new partners looking to run this case within their own environments, the first step is to copy the file. Then, run the model with the considered solvers and save them into regular Python dictionaries, which should be added to the new file:

shutil.copy("casoBase.hdf5", "casoConSolucion_01.hdf5")
time.sleep(2)

# Assume these are the 2 resulting dictionaries
diccionarioSolucionCaso_1 = {'ex1': 'xxx',
                             'ex2': 'yyy',
                             'ex3': 'zzz'}
diccionarioAtributosSolucionCaso_1 = {
                            'Autor': 'Michel Gondry',
                            'Dia':  str(datetime.now()),
                            'Escenario': 'Amargo',
                            }

solucion_1 = zsan_DirB()
solucion_1.cargaCaso("casoConSolucion_01.hdf5")
solucion_1.guardaNuevaSolucion(diccionarioSolucionCaso_1, diccionarioAtributosSolucionCaso_1)

For every new experiment, the same process should be followed.

Finally, when all the runs have been finished, they should be added up into the same file. As an example, consider we've got a total of 3 runs and that we want to merge them all into our base case file. An additional parameter should be provided to specify the resulting file name, as the base case one will never be overwritten by this method.

from dirB.merge import mergeListaDeSoluciones

casoBase = 'casoBase.hdf5'
listaDeCasos = ['casoConSolucion_01.hdf5', 'casoConSolucion_02.hdf5', 'casoConSolucion_03.hdf5']
id_salida = "salida.hdf5"

mergedList = mergeListaDeSoluciones(casoBase, listaDeCasos, id_salida)

Complete example notebooks have been added to this repository.

Documentation

Here is the official documentation. For any modification you'd like to do it, it can be generated locally. In order to do this, we recommend to create a brand new environment for sphinx, in which you will install the proper dependencies by means of pip install docs/{requirements_file}:

  • docs/requirements.in: contains the base dependencies
  • docs/requirements.txt: contains the base dependencies plus the whole bunch of subdependencies recursively gotten for complete reproducibility

Then, you should run the make command for the HTML creation:

cd docs
make html

With the following command we'll get a clean documentation env to regenerate the whole documentation in case we need to:

make clean html
make html

Acknowledgements

© Copyright 2023, Germán Navarro $^\dagger$, Santiago Fernández Prieto $^{\ddagger,1}$, Ricardo Enríquez Miranda $^\ddagger$, Javier Hernanz Zájara $^\ddagger$

$^\dagger$ Barcelona Supercomputing Center
$^\ddagger$ Repsol Quantum Advisory Team

$^1$ Repsol-BSC Research Center

Developed within the framework of the project CUCO. Financed by the CDTI and with the support of the Spanish Ministry of Science and Innovation under the Recovery, Transformation and Resilience Plan (RTRP).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dirB-0.2.3.tar.gz (20.0 kB view details)

Uploaded Source

Built Distribution

dirB-0.2.3-py3-none-any.whl (17.5 kB view details)

Uploaded Python 3

File details

Details for the file dirB-0.2.3.tar.gz.

File metadata

  • Download URL: dirB-0.2.3.tar.gz
  • Upload date:
  • Size: 20.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.9.18

File hashes

Hashes for dirB-0.2.3.tar.gz
Algorithm Hash digest
SHA256 ed28f5f69e799891e7a83da7dadff7f31849b7b7f8de683d8d466968c6d6ba1b
MD5 97d252b293fbd3c67a2f2379e3d6724b
BLAKE2b-256 9922ab03bd9990b433845cfeef0cdb5ed8dee1369ef6b1f55b5052529ac577a8

See more details on using hashes here.

File details

Details for the file dirB-0.2.3-py3-none-any.whl.

File metadata

  • Download URL: dirB-0.2.3-py3-none-any.whl
  • Upload date:
  • Size: 17.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.9.18

File hashes

Hashes for dirB-0.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 7a8a96ac0105b81ff5585faceab4e87ca9f745ec4e3a9a2daf770362bd134d26
MD5 219fcb7c939722f7b1657527f1f7ef7d
BLAKE2b-256 671fc6822158a70e0e1f56308360d5c2477e6292cfc90b18b3c251a9608eb64b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page