Skip to main content

Collect, manage, and visualize geospace data.

Project description

GeospaceLab (geospacelab)

License Python DOI

To collect, manage, and visualize geospace data in an easy and fast way. The documentation can be found on readthedocs.io.

Features

  • Class-based data manager, including
    • DataHub: the core module to manage data from multiple sources,
    • DatasetModel: the base class to download, load, and process data from a data source,
    • VariableModel: the base class to store the value, error, and other attributes for a variable.
  • Extendable
    • Provide a standard procedure from downloading, loading, and post-processing the data.
    • Easy to extend for a data source which has not been included in the package.
    • Flexible to add functions for post-processing.
  • Visualization
    • Time series plots with
      • automatically adjustable time ticks and tick labels.
      • dynamical panels (easily adding or removing panels).
      • useful marking tools (vertical line crossing panels, shadings, top bars, etc, see Example 2 in Usage)
    • Map projection
      • Polar views with
        • coastlines in either GEO or AACGM (APEX) coordinate system.
        • mapping in either fixed lon/mlon mode or in fixed LST/MLT mode.
      • Support 1-D or 2-D plots with
        • satellite tracks (time ticks and labels)
        • nadir colored 1-D plots
        • gridded surface plots
  • Space coordinate system transformation
  • Toolboxes for data analysis
    • Basic toolboxes for numpy array, datetime, logging, python dict, list, and class.
    • Coordinate system transformation.

Built-in data sources:

Data Source Variables File Format Downloadable Express Status
CDAWeb/OMNI Solar wind and IMF cdf True OMNIDashboard stable
Madrigal/EISCAT Ionospheric Ne, Te, Ti, ... EISCAT-hdf5, Madrigal-hdf5 True EISCATDashboard stable
Madrigal/GNSS/TECMAP Ionospheric GPS TEC map hdf5 True - beta
Madrigal/DMSP/s1 DMSP SSM, SSIES, etc hdf5 True DMSPTSDashboard beta
Madrigal/DMSP/s4 DMSP SSIES hdf5 True DMSPTSDashboard beta
Madrigal/DMSP/e DMSP SSJ hdf5 True DMSPTSDashboard beta
JHUAPL/DMSP/SSUSI DMSP SSUSI netcdf True DMSPSSUSIDashboard beta
JHUAPL/AMPERE/fitted AMPERE FAC netcdf False AMPEREDashboard stable
SuperDARN/POTMAP SuperDARN potential map ascii False - stable
WDC/Dst Dst index IAGA2002-ASCII True - stable
WDC/ASYSYM ASY/SYM indices IAGA2002-ASCII True OMNIDashboard stable
WDC/AE AE indices IAGA2002-ASCII True OMNIDashboard stable
GFZ/Kp Kp/Ap indices ASCII True - stable
GFZ/Hpo Hp30 or Hp60 indices ASCII True - stable
GFZ/SNF107 SN, F107 ASCII True - stable
ESA/SWARM/EFI_LP_1B SWARM Ne, Te, etc. netcdf True - stable
ESA/SWARM/AOB_FAC_2F SWARM FAC, auroral oval boundary netcdf True - beta
UTA/GITM/2DALL GITM 2D output binary, IDL-sav False - beta
UTA/GITM/3DALL GITM 3D output binary, IDL-sav False - beta

Installation

1. The python distribution "Anaconda" is recommended:

The package was tested with the anaconda distribution and with PYTHON>=3.7 under Ubuntu 20.04 and MacOS Big Sur.

With Anaconda, it may be easier to install some required dependencies listed below, e.g., cartopy, using the conda command. It's also recommended installing the package and dependencies in a virtual environment with anaconda.

After installing the anaconda distribution, a virtual environment can be created by the code below in the terminal:

conda create --name [YOUR_ENV_NAME] python=3.8 spyder

The package "spyder" is a widely-used python IDE. Other IDEs, like "VS Code" or "Pycharm" also work.

After creating the virtual environement, you need to activate the virtual environment:

conda activate [YOUR_ENV_NAME]

and then to install the package as shown below or to start the IDE "spyder".

More detailed information to set the anaconda environment can be found here,

2. Installation

Quick install from the pre-built release (recommended):

pip install geospacelab

Install from Github (not recommended):

pip install git+https://github.com/JouleCai/geospacelab@master

2. Dependencies

The package dependencies need to be installed before or after the installation of the package. Several dependencies will be installed automatically with the package installation, including toml, requests, bueatifulsoup4, numpy, scipy, matplotlib, h5py, netcdf4, cdflib, madrigalweb, and aacgmv2.

Other dependencies will be needed if you see a ImportError or ModuleNotFoundError displayed in the python console. Some frequently used modules and their installation methods are listed below:

  • cartopy: Map projection for geospatial data.
    • conda install -c conda-forge cartopy
  • apexpy *: Apex and Quasi-Dipole geomagnetic coordinate system.
    • pip install apexpy
  • geopack: The geopack and Tsyganenko models in Python.
    • pip install geopack

(*): The gcc or gfortran compilers are required before installing the package.

  • MacOS: brew install gcc
  • Linux: sudo apt install gcc gfortran

Please install the packages above, if needed.

Note: The package is currently pre-released. The installation methods may be changed in the future.

4. First-time startup and basic configuration

Some basic configurations will be made with the first-time import of the package. Following the messages prompted in the python console, the first configuration is to set the root directory for storing the data.

When the modules to access the online Madrigal database is imported, it will ask for the inputs of user's full name, email, and affiliation.

The user's configuration can be found from the toml file below:

[your_home_directory]/.geospacelab/config.toml

The user can set or change the preferences in the configuration file. For example, to change the root directory for storing the data, modify or add the lines in "config.toml":

[datahub]
data_root_dir = "YOUR_ROOT_DIR"

To set the Madrigal cookies, change the lines:

[datahub.madrigal]
user_fullname = "YOUR_NAME"
user_email = "YOU_EMAIL"
user_affiliation = "YOUR_AFFILIATION"

5. Upgrade

If the package is installed from the pre-built release. Update the package via:

pip install geospacelab --upgrade

6. Uninstallation

Uninstall the package via:

pip uninstall geospacelab

If you don't need the user's configuration, delete the file at [your_home_directory]/.geospacelab/config.toml

Usage

Example 1: Dock a sourced dataset and get variables:

The core of the data manager is the class Datahub. A Datahub instance will be used for docking a buit-in sourced dataset, or adding a temporary or user-defined dataset.

The "dataset" is a Dataset instance, which is used for loading and downloading the data.

Here is an example to load the EISCAT data from the online service. The module will download EISCAT data automatically from the EISCAT schedule page with the preset loading mode "AUTO" and file type "eiscat-hdf5". In addition, the package can load data by assigning the data file paths.

In "example1.py":

import datetime

from geospacelab.datahub import DataHub

# settings
dt_fr = datetime.datetime.strptime('20210309' + '0000', '%Y%m%d%H%M')   # datetime from
dt_to = datetime.datetime.strptime('20210309' + '2359', '%Y%m%d%H%M')   # datetime to
database_name = 'madrigal'      # built-in sourced database name 
facility_name = 'eiscat'        # facility name

site = 'UHF'                # facility attributes required, check from the eiscat schedule page
antenna = 'UHF'
modulation = 'ant'

# create a datahub instance
dh = DataHub(dt_fr, dt_to)
# dock a dataset
ds_1 = dh.dock(datasource_contents=[database_name, facility_name],
                      site=site, antenna=antenna, modulation=modulation, data_file_type='eiscat-hdf5')
# load data
ds_1.load_data()
# assign a variable from its own dataset to the datahub
n_e = dh.assign_variable('n_e')
T_i = dh.assign_variable('T_i')

# get the variables which have been assigned in the datahub
n_e = dh.get_variable('n_e')
T_i = dh.get_variable('T_i')
# if the variable is not assigned in the datahub, but exists in the its own dataset:
comp_O_p = dh.get_variable('comp_O_p', dataset=ds_1)     # O+ ratio
# above line is equivalent to
comp_O_p = dh.datasets[1]['comp_O_p']

# The variables, e.g., n_e and T_i, are the class Variable's instances, 
# which stores the variable values, errors, and many other attributes, e.g., name, label, unit, depends, ....
# To get the value of the variable, use variable_isntance.value, e.g.,
print(n_e.value)        # return the variable's value, type: numpy.ndarray, axis 0 is always along the time, check n_e.depends.items{}
print(n_e.error)

Example 2: EISCAT quicklook plot

The EISCAT quicklook plot shows the GUISDAP analysed results in the same format as the online EISCAT quicklook plot. The figure layout and quality are improved. In addition, several marking tools like vertical lines, shadings, top bars can be added in the plot. See the example script and figure below:

In "example2.py"

import datetime
import geospacelab.express.eiscat_dashboard as eiscat

dt_fr = datetime.datetime.strptime('20201209' + '1800', '%Y%m%d%H%M')
dt_to = datetime.datetime.strptime('20201210' + '0600', '%Y%m%d%H%M')

site = 'UHF'
antenna = 'UHF'
modulation = '60'
load_mode = 'AUTO'
dashboard = eiscat.EISCATDashboard(
    dt_fr, dt_to, site=site, antenna=antenna, modulation=modulation, load_mode='AUTO'
)
dashboard.quicklook()

# dashboard.save_figure() # comment this if you need to run the following codes
# dashboard.show()   # comment this if you need to run the following codes.

"""
As the viewer is an instance of the class EISCATViewer, which is a heritage of the class Datahub.
The variables can be retrieved in the same ways as shown in Example 1. 
"""
n_e = dashboard.assign_variable('n_e')
print(n_e.value)
print(n_e.error)

"""
Several marking tools (vertical lines, shadings, and top bars) can be added as the overlays 
on the top of the quicklook plot.
"""
# add vertical line
dt_fr_2 = datetime.datetime.strptime('20201209' + '2030', "%Y%m%d%H%M")
dt_to_2 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
dashboard.add_vertical_line(dt_fr_2, bottom_extend=0, top_extend=0.02, label='Line 1', label_position='top')
# add shading
dashboard.add_shading(dt_fr_2, dt_to_2, bottom_extend=0, top_extend=0.02, label='Shading 1', label_position='top')
# add top bar
dt_fr_3 = datetime.datetime.strptime('20201210' + '0130', "%Y%m%d%H%M")
dt_to_3 = datetime.datetime.strptime('20201210' + '0430', "%Y%m%d%H%M")
dashboard.add_top_bar(dt_fr_3, dt_to_3, bottom=0., top=0.02, label='Top bar 1')

# save figure
dashboard.save_figure()
# show on screen
dashboard.show()

Output:

alt text

Example 3: OMNI data and geomagnetic indices (WDC + GFZ):

In "example3.py"

import datetime
import geospacelab.express.omni_dashboard as omni

dt_fr = datetime.datetime.strptime('20160314' + '0600', '%Y%m%d%H%M')
dt_to = datetime.datetime.strptime('20160320' + '0600', '%Y%m%d%H%M')

omni_type = 'OMNI2'
omni_res = '1min'
load_mode = 'AUTO'
dashboard = omni.OMNIDashboard(
    dt_fr, dt_to, omni_type=omni_type, omni_res=omni_res, load_mode=load_mode
)
dashboard.quicklook()

# data can be retrieved in the same way as in Example 1:
dashboard.list_assigned_variables()
B_x_gsm = dashboard.get_variable('B_x_GSM', dataset_index=1)
# save figure
dashboard.save_figure()
# show on screen
dashboard.show()

Output:

alt text

Example 4: Mapping geospatial data in the polar map.

import datetime
import matplotlib.pyplot as plt

import geospacelab.visualization.mpl.geomap.geodashboards as geomap

dt_fr = datetime.datetime(2015, 9, 8, 8)
dt_to = datetime.datetime(2015, 9, 8, 23, 59)
time1 = datetime.datetime(2015, 9, 8, 20, 21)
pole = 'N'
sat_id = 'f18'
band = 'LBHS'

dashboard = geomap.GeoDashboard(dt_fr=dt_fr, dt_to=dt_to, figure_config={'figsize': (5, 5)})

# If the orbit_id is specified, only one file will be downloaded. This option saves the downloading time.
# dashboard.dock(datasource_contents=['jhuapl', 'dmsp', 'ssusi', 'edraur'], pole='N', sat_id='f17', orbit_id='46863')
# If not specified, the data during the whole day will be downloaded.
dashboard.dock(datasource_contents=['jhuapl', 'dmsp', 'ssusi', 'edraur'], pole=pole, sat_id=sat_id, orbit_id=None)
dashboard.set_layout(1, 1)

lbhs = dashboard.assign_variable('GRID_AUR_' + band, dataset_index=1)
dts = dashboard.assign_variable('DATETIME', dataset_index=1).value.flatten()
mlat = dashboard.assign_variable('GRID_MLAT', dataset_index=1).value
mlon = dashboard.assign_variable('GRID_MLON', dataset_index=1).value
mlt = dashboard.assign_variable(('GRID_MLT'), dataset_index=1).value

ind_t = dashboard.datasets[1].get_time_ind(ut=time1)

panel1 = dashboard.add_polar_map(row_ind=0, col_ind=0, style='mlt-fixed', cs='AACGM', mlt_c=0., pole=pole, ut=time1, boundary_lat=65., mirror_south=True)
# panel1 = dashboard.add_polar_map(row_ind=0, col_ind=0, style='mlt-fixed', cs='AACGM', mlt_c=0., pole=pole, ut=time1, mirror_south=True)
# panel1 = dashboard.add_polar_map(row_ind=0, col_ind=0, style='lst-fixed', cs='GEO', lst_c=3., pole='N', ut=time1)
# panel1 = dashboard.add_polar_map(row_ind=0, col_ind=0, style='lst-fixed', cs='GEO', lst_c=0, pole='S', ut=time1, mirror_south=True)
# panel1 = dashboard.add_polar_map(row_ind=0, col_ind=0, style='lon-fixed', cs='GEO', lon_c=0., pole='S', ut=time1,
#                           boundary_lat=0, mirror_south=False)
# panel1 = dashboard.add_polar_map(row_ind=0, col_ind=0, style='lon-fixed', cs='GEO', lon_c=0., pole='N', ut=time1,
#                          boundary_lat=0, mirror_south=False)
panel1.add_coastlines()

lbhs_ = lbhs.value[ind_t, :, :]
pcolormesh_config = lbhs.visual.plot_config.pcolormesh
pcolormesh_config.update(c_scale='log')
pcolormesh_config.update(c_lim=[100, 1500])
import geospacelab.visualization.mpl.colormaps as cm
cmap = cm.cmap_gist_ncar_modified()
cmap = 'viridis'
pcolormesh_config.update(cmap=cmap)
ipc = panel1.add_pcolor(lbhs_, coords={'lat': mlat[ind_t, ::], 'lon': mlon[ind_t, ::], 'mlt': mlt[ind_t, ::], 'height': 250.}, cs='AACGM', **pcolormesh_config)
panel1.add_colorbar(ipc, c_label=band + " (R)", c_scale=pcolormesh_config['c_scale'], left=1.1, bottom=0.1,
                    width=0.05, height=0.7)

panel1.add_gridlines(lat_res=5, lon_label_separator=5)

polestr = 'North' if pole == 'N' else 'South'
panel1.add_title(title='DMSP/SSUSI, ' + band + ', ' + sat_id.upper() + ', ' + polestr + ', ' + time1.strftime('%Y-%m-%d %H%M UT'))
plt.savefig('DMSP_SSUSI_' + time1.strftime('%Y%m%d-%H%M') + '_' + band + '_' + sat_id.upper() + '_' + pole, dpi=300)
plt.show()

Output:

alt text

This is an example showing the HiLDA aurora in the dayside polar cap region (see also DMSP observations of the HiLDA aurora (Cai et al., JGR, 2021)).

Other examples for the time-series plots and map projections can be found here

Citation and Acknowledgements

Please acknowledge or cite GeospaceLab, if the library contributes to a project that leads to a scientific publication.

Notes

  • The current version is a pre-released version. Many features will be added soon.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for geospacelab, version 0.3.1
Filename, size File type Python version Upload date Hashes
Filename, size geospacelab-0.3.1-py3-none-any.whl (248.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size geospacelab-0.3.1.tar.gz (134.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page