Skip to main content

Python package to translate between gdx (GAMS data) and pandas.

Project description

gdx-pandas: Python package to translate between gdx (GAMS data) and pandas.

USE

There are two main ways to use gdxpds. The first use case is the one that was
initially supported: direct conversion between GDX files on disk and pandas
DataFrames or a csv version thereof. The Version 1.0.0 rewrite intoduces a
second style of use, that is, interfacing with GDX files and symbols via the
`gdxpds.gdx.GdxFile` and `gdxpds.gdx.GdxSymbol` classes.

USE -- Direct Conversion

The two primary points of reference for the direct conversion utilities are GDX
files on disk and python dicts of {symbol_name: pandas.DataFrame}, where
each pandas.DataFrame contains data for a single set, parameter, equation, or
variable. For sets and parameters, the last column of the DataFrame is assumed to
contain the value of the element, which for sets should be `True`, and for
parameters should be a `float` (or one of the `gdxpds.gdx.NUMPY_SPECIAL_VALUES`).
Equations and variables have additional 'value' columns, in particular a level,
a marginal value, a lower bound, an upper bound, and a scale, as enumerated in
`gdxpds.gdx.GamsValueType`. These values are all assumed to be found in the last
five columns of the DataFrame, also see `gdxpds.gdx.GAMS_VALUE_COLS_MAP`.

The basic interface to convert from GDX to DataFrames is:

import gdxpds

gdx_file = 'C:\path_to_my_gdx\data.gdx'
dataframes = gdxpds.to_dataframes(gdx_file)
for symbol_name, df in dataframes.items():
print("Doing work with {}.".format(symbol_name))

And vice-versa:

import gdxpds

# assume we have a DataFrame df with last column 'value'
data_ready_for_GAMS = { 'symbol_name': df }

gdx_file = 'C:\path_to_my_output_gdx\data_to_send_to_gams.gdx'
gdx = gdxpds.to_gdx(data_ready_for_GAMS, gdx_file)

Note that providing a gdx_file is optional, and the returned gdx is an object of
type `gdxpds.gdx.GdxFile`.

The package also includes command line utilities for converting between GDX and
CSV: gdx_to_csv.py and csv_to_gdx.py.

USE -- Backend Classes

The basic functionalities described above can also be achieved with direct use
of the backend classes now available in `gdxpds.gdx`. To duplicate the GDX read
functionality shown above one would write:

import gdxpds

gdx_file = 'C:\path_to_my_gdx\data.gdx'
with gdxpds.gdx.GdxFile(lazy_load=False) as f:
f.read(gdx_file)
for symbol in f:
symbol_name = symbol.name
df = symbol.dataframe
print("Doing work with {}:\n{}".format(symbol_name,df.head()))

The backend especially gives more control over creating new data in GDX format.
For example:

import gdxpds

out_file = 'my_new_gdx_data.gdx'
with gdxpds.gdx.GdxFile() as gdx:
# Create a new set with one dimension
gdx.append(gdxpds.gdx.GdxSymbol('my_set',gdxpds.gdx.GamsDataType.Set,dims=['u']))
data = pds.DataFrame([['u' + str(i)] for i in range(1,11)])
data['Value'] = True
gdx[-1].dataframe = data
# Create a new parameter with one dimension
gdx.append(gdxpds.gdx.GdxSymbol('my_parameter',gdxpds.gdx.GamsDataType.Parameter,dims=['u']))
data = pds.DataFrame([['u' + str(i), i*100] for i in range(1,11)],
columns=(gdx[-1].dims + gdx[-1].value_col_names))
gdx[-1].dataframe = data
gdx.write(out_file)


DEPENDENCIES

- Python 2.6 or higher 2.X; Python 3.4 or higher 3.X
- pandas (In general you will want the SciPy stack. Anaconda comes with it, or see [my notes for Windows](http://elainethale.wordpress.com/programming-notes/python-environment-set-up/).)
- For Python versions < 3.4, enum34. Also **uninstall the enum package** if it is installed.
- psutil (optional--for monitoring memory use)
- pytest (optional--for running tests)
- GAMS Python bindings
- See GAMS/win64/XX.X/apifiles/readme.txt on Windows,
GAMS/gamsXX.X_osx_x64_64_sfx/apifiles/readme.txt on Mac, or
/opt/gams/gamsXX.X_linux_x64_64_sfx/apifiles/readme.txt on Linux
- Run the following for the correct version of the Python bindings

python setup.py install

or

python setup.py build --build-base=/path/to/somwhere/you/have/write/access install

with the latter being for the case when you can install packages into
Python but don't have GAMS directory write access.

- .../apifiles/Python/api/setup.py works for Python 2.7
- For other versions of Python, especially 3.X, use
.../apifiles/Python/api_XX/setup.py. For Python 3.X in particular you will
need GAMS version >= 24.5.1 (Python 3.4, Windows and Linux),
24.7.4 (Python 3.4, Mac OS X), or >= 24.8.4 (Python 3.6)


TESTING

After installation, you can test the package using pytest:

pytest --pyargs gdxpds

If the tests fail due to permission IOErrors, apply `chmod g+x` and `chmod a+x`
to the `gdx-pandas/gdxpds/test` folder.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gdxpds, version 1.1.0
Filename, size File type Python version Upload date Hashes
Filename, size gdxpds-1.1.0.tar.gz (641.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page