Cloud-native, scalable, and user-friendly multi dimensional energy data!
Project description
"MDIO" is a library to work with large multidimensional energy datasets. The primary motivation behind MDIO is to represent multidimensional time series data in a format that makes it easier to use in resource assessment, machine learning, and data processing workflows.
Features
Shared Features
- Abstractions for common energy data types (see below).
- Cloud native chunked storage based on Zarr and fsspec.
- Lossy and lossless data compression using Blosc and ZFP.
- Distributed reads and writes using Dask.
- Powerful command-line-interface (CLI) based on Click
Domain Specific Features
- Oil & Gas Data
- Import and export 2D - 5D seismic data types stored in SEG-Y.
- Import seismic interpretation, horizon, data. FUTURE
- Optimized chunking logic for various seismic types. FUTURE
- Wind Resource Assessment
- Numerical weather prediction models with arbitrary metadata. FUTURE
- Optimized chunking logic for time-series analysis and mappin. FUTURE
- Xarray interface. FUTURE
The features marked as FUTURE will be open-sourced at a later date.
Installation
You can install MDIO via pip from PyPI:
pip install multidimio
You can also install some optional dependencies (extras) like this:
pip install multidimio[distributed]
pip install multidimio[cloud]
pip install multidimio[lossy]
distributed
installs Dask for parallel, distributed processing.
cloud
installs fsspec backed I/O libraries for AWS' S3,
Google's GCS, and Azure ABS.
lossy
will install the ZFPY library for lossy chunk compression.
Usage
Please see the Command-line Reference for details.
For Python API please see the API Reference for details.
Requirements
Minimal
Chunked storage and parallelization: zarr
, dask
, numba
, and psutil
.
SEG-Y Parsing: segyio
CLI and Progress Bars: click
, click-params
, and tqdm
.
Optional
Distributed computing [distributed]
: distributed
and bokeh
.
Cloud Object Store I/O [cloud]
: s3fs
, gcsfs
, and adlfs
.
Lossy Compression [lossy]
: zfpy
Contributing
Contributions are very welcome. To learn more, see the Contributor Guide.
License
Distributed under the terms of the Apache 2.0 license, MDIO is free and open source software.
Issues
If you encounter any problems, please file an issue along with a detailed description.
Credits
This project was established at TGS. Original authors and current maintainers are Altay Sansal and Sri Kainkaryam; with the support of many more great colleagues.
This project template is based on @cjolowicz's Hypermodern Python Cookiecutter template.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for multidimio-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b10e5fb45cceeedc63ebe146dcbf026f3e6127266b310982ee7bb57ddfc4136a |
|
MD5 | 257d38546d329bddf709db2ff20f980d |
|
BLAKE2b-256 | 3a05d607da95c786f9c717a2da695d00b7d7184e1bf2411a7f55c730d3458e55 |