Python tools for Cassini UVIS data

These details have not been verified by PyPI

Project links

Homepage

Development Status
- 4 - Beta
Intended Audience
- Developers
License
- OSI Approved :: Apache Software License
Natural Language
- English
Programming Language
- Python :: 3.7
- Python :: 3.8

Project description

pyuvis

This package provides tools to work with Cassini UVIS data.

How to install

It is recommended to manage your Python environments with conda or its new and faster sibling mamba.

After installation of mamba/conda, create a new Python environment with the name py38 (or whatever name you prefer) with:

conda create -n py38 python=3.8 -c conda-forge

{% include note.html content='If you installed mambaforge in the previous step, the conda-forge channel is the default source for conda packages and you wouldn’t need to add it here.' %}If you need a refresher or intro into managing your Python environment with conda, have a look at my tutorials here (long version) and here (short version).

Once you have created the new environment, activate it:

conda activate py38

(or your <env> name)

and install the latest requirements for the package:

mamba install -c conda-forge hvplot xarray pandas

followed by a pip install for this package (a pyuvis conda package will be created later):

pip install pyuvis

The package depends on planetarypy to organize data discovery, download, and local data management and the pip install should pull in planetarypy with the minimum version of 0.15.

How to use

When launched for the first time, the planetarypy package will ask for a general storage path for all data managed by planetarypy. You should give it a folder path where you have sufficient space to store the planetary data you want to work with.

The different available modules were born out of special use cases for analysis of UVIS data, but for the general user the most important classes are UVPDS and the UVISObs.

UVPDS

The UVPDS class is designed to deal with exactly one PDS product for either EUV or FUV data:

from pyuvis import UVPDS

pid = "FUV2005_172_03_35"
uv = UVPDS(pid)

If the data file does not exist on your local storage yet, planetarypy will go ahead and download it for you and provide the local path to UVPDS for reading it.

You don't have to do anything to get the data and you don't have to decide anything about where to store the data. :) If you are interested in seeing how the planetarypy.uvis module achieves this, have a look at https://michaelaye.github.io/nbplanetary/cassini_uvis

While the data qube itself is accessible as a numpy.array via the data attribute:

uv.data[100]

array([[0, 0, 0, ..., 0, 2, 1],
       [0, 1, 0, ..., 0, 1, 0],
       [0, 0, 0, ..., 0, 1, 2],
       ...,
       [0, 0, 1, ..., 0, 0, 0],
       [0, 1, 0, ..., 0, 1, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=uint16)

... it is recommended to use the xarray attribute for a more advanced data container with labeled coordinate axes

("spectral", "spatial", "samples") :

uv.xarray

:root { --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1)); --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54)); --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38)); --xr-border-color: var(--jp-border-color2, #e0e0e0); --xr-disabled-color: var(--jp-layout-color3, #bdbdbd); --xr-background-color: var(--jp-layout-color0, white); --xr-background-color-row-even: var(--jp-layout-color1, white); --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee); }

html[theme=dark], body.vscode-dark { --xr-font-color0: rgba(255, 255, 255, 1); --xr-font-color2: rgba(255, 255, 255, 0.54); --xr-font-color3: rgba(255, 255, 255, 0.38); --xr-border-color: #1F1F1F; --xr-disabled-color: #515151; --xr-background-color: #111111; --xr-background-color-row-even: #111111; --xr-background-color-row-odd: #313131; }

.xr-wrap { display: block; min-width: 300px; max-width: 700px; }

.xr-text-repr-fallback { /* fallback to plain text repr when CSS is not injected (untrusted notebook) */ display: none; }

.xr-header { padding-top: 6px; padding-bottom: 6px; margin-bottom: 4px; border-bottom: solid 1px var(--xr-border-color); }

.xr-header > div, .xr-header > ul { display: inline; margin-top: 0; margin-bottom: 0; }

.xr-obj-type, .xr-array-name { margin-left: 2px; margin-right: 10px; }

.xr-obj-type { color: var(--xr-font-color2); }

.xr-sections { padding-left: 0 !important; display: grid; grid-template-columns: 150px auto auto 1fr 20px 20px; }

.xr-section-item { display: contents; }

.xr-section-item input { display: none; }

.xr-section-item input + label { color: var(--xr-disabled-color); }

.xr-section-item input:enabled + label { cursor: pointer; color: var(--xr-font-color2); }

.xr-section-item input:enabled + label:hover { color: var(--xr-font-color0); }

.xr-section-summary { grid-column: 1; color: var(--xr-font-color2); font-weight: 500; }

.xr-section-summary > span { display: inline-block; padding-left: 0.5em; }

.xr-section-summary-in:disabled + label { color: var(--xr-font-color2); }

.xr-section-summary-in + label:before { display: inline-block; content: '►'; font-size: 11px; width: 15px; text-align: center; }

.xr-section-summary-in:disabled + label:before { color: var(--xr-disabled-color); }

.xr-section-summary-in:checked + label:before { content: '▼'; }

.xr-section-summary-in:checked + label > span { display: none; }

.xr-section-summary, .xr-section-inline-details { padding-top: 4px; padding-bottom: 4px; }

.xr-section-inline-details { grid-column: 2 / -1; }

.xr-section-details { display: none; grid-column: 1 / -1; margin-bottom: 5px; }

.xr-section-summary-in:checked ~ .xr-section-details { display: contents; }

.xr-array-wrap { grid-column: 1 / -1; display: grid; grid-template-columns: 20px auto; }

.xr-array-wrap > label { grid-column: 1; vertical-align: top; }

.xr-preview { color: var(--xr-font-color3); }

.xr-array-preview, .xr-array-data { padding: 0 5px !important; grid-column: 2; }

.xr-array-data, .xr-array-in:checked ~ .xr-array-preview { display: none; }

.xr-array-in:checked ~ .xr-array-data, .xr-array-preview { display: inline-block; }

.xr-dim-list { display: inline-block !important; list-style: none; padding: 0 !important; margin: 0; }

.xr-dim-list li { display: inline-block; padding: 0; margin: 0; }

.xr-dim-list:before { content: '('; }

.xr-dim-list:after { content: ')'; }

.xr-dim-list li:not(:last-child):after { content: ','; padding-right: 5px; }

.xr-has-index { font-weight: bold; }

.xr-var-list, .xr-var-item { display: contents; }

.xr-var-item > div, .xr-var-item label, .xr-var-item > .xr-var-name span { background-color: var(--xr-background-color-row-even); margin-bottom: 0; }

.xr-var-item > .xr-var-name:hover span { padding-right: 5px; }

.xr-var-list > li:nth-child(odd) > div, .xr-var-list > li:nth-child(odd) > label, .xr-var-list > li:nth-child(odd) > .xr-var-name span { background-color: var(--xr-background-color-row-odd); }

.xr-var-name { grid-column: 1; }

.xr-var-dims { grid-column: 2; }

.xr-var-dtype { grid-column: 3; text-align: right; color: var(--xr-font-color2); }

.xr-var-preview { grid-column: 4; }

.xr-var-name, .xr-var-dims, .xr-var-dtype, .xr-preview, .xr-attrs dt { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; padding-right: 10px; }

.xr-var-name:hover, .xr-var-dims:hover, .xr-var-dtype:hover, .xr-attrs dt:hover { overflow: visible; width: auto; z-index: 1; }

.xr-var-attrs, .xr-var-data { display: none; background-color: var(--xr-background-color) !important; padding-bottom: 5px !important; }

.xr-var-attrs-in:checked ~ .xr-var-attrs, .xr-var-data-in:checked ~ .xr-var-data { display: block; }

.xr-var-data > table { float: right; }

.xr-var-name span, .xr-var-data, .xr-attrs { padding-left: 25px !important; }

.xr-attrs, .xr-var-attrs, .xr-var-data { grid-column: 1 / -1; }

dl.xr-attrs { padding: 0; margin: 0; display: grid; grid-template-columns: 125px auto; }

.xr-attrs dt, .xr-attrs dd { padding: 0; margin: 0; float: left; padding-right: 10px; width: auto; }

.xr-attrs dt { font-weight: normal; grid-column: 1; }

.xr-attrs dt:hover span { display: inline-block; background: var(--xr-background-color); padding-right: 10px; }

.xr-attrs dd { grid-column: 2; white-space: pre-wrap; word-break: break-all; }

.xr-icon-database, .xr-icon-file-text2 { display: inline-block; vertical-align: middle; width: 1em; height: 1.5em !important; stroke-width: 0; stroke: currentColor; fill: currentColor; } </style>

<xarray.DataArray 'FUV2005_172_03_35' (spectral: 1024, spatial: 60, samples: 164)> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Coordinates:

spectral (spectral) float64 111.5 111.6 111.7 111.8 ... 191.1 191.2 191.3
spatial (spatial) int64 2 3 4 5 6 7 8 9 10 ... 53 54 55 56 57 58 59 60 61
samples (samples) int64 1 2 3 4 5 6 7 8 ... 158 159 160 161 162 163 164 Attributes: units: Counts long_name: FUV raw data n_bands: 1024 integration_duration: Quantity(value=240.0, units='SECOND')

xarray.DataArray

'FUV2005_172_03_35'

spectral: 1024
spatial: 60
samples: 164

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 1, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
...
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint16)

Coordinates: (3)

spectral
(spectral)
float64
111.5 111.6 111.7 ... 191.2 191.3
units :
nm
long_name :
Wavelength
```
array([111.535, 111.613, 111.69 , ..., 191.131, 191.209, 191.287])
```

spatial

(spatial)

int64

2 3 4 5 6 7 8 ... 56 57 58 59 60 61

long_name :: Spatial lines

array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
56, 57, 58, 59, 60, 61])

samples

(samples)

int64

1 2 3 4 5 6 ... 160 161 162 163 164

array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,
15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,  28,
29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,  42,
43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,  56,
57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,  70,
71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,  84,
85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126,
127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154,
155, 156, 157, 158, 159, 160, 161, 162, 163, 164])

Attributes: (4)
units :
Counts
long_name :
FUV raw data
n_bands :
1024
integration_duration :
Quantity(value=240.0, units='SECOND')

One can see that the spatial range was already reduced to the valid range of 2..61 according to the label data.

Note that the sample count starts at 1, to be fitting to the UVIS manual.

I usually story the xarray.DataArray into a da or arr variable for further analysis and plotting:

arr = uv.xarray

A built-in example plot method shows you how to use hvplot to generate interactive plots:

uv.plot()

The widget to the right allows you to scroll through the samples dimension.

Hint:You can look at the source code of any method using the double question mark:

uv.plot??

[0;31mSignature:[0m
[0muv[0m[0;34m.[0m[0mplot[0m[0;34m([0m[0;34m[0m
[0;34m[0m    [0mprecise[0m[0;34m:[0m [0mbool[0m [0;34m=[0m [0;32mFalse[0m[0;34m,[0m[0;34m[0m
[0;34m[0m    [0mpercentiles[0m[0;34m:[0m [0mtuple[0m [0;34m=[0m [0;34m([0m[0;36m0.5[0m[0;34m,[0m [0;36m99.5[0m[0;34m)[0m[0;34m,[0m[0;34m[0m
[0;34m[0m    [0mclim[0m[0;34m:[0m [0mtuple[0m [0;34m=[0m [0;32mNone[0m[0;34m,[0m[0;34m[0m
[0;34m[0m    [0mcmap[0m[0;34m:[0m [0mstr[0m [0;34m=[0m [0;34m'viridis'[0m[0;34m,[0m[0;34m[0m
[0;34m[0m    [0mcalibrated[0m[0;34m=[0m[0;32mFalse[0m[0;34m,[0m[0;34m[0m
[0;34m[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;31mSource:[0m   
    [0;32mdef[0m [0mplot[0m[0;34m([0m[0;34m[0m
[0;34m[0m        [0mself[0m[0;34m,[0m[0;34m[0m
[0;34m[0m        [0mprecise[0m[0;34m:[0m [0mbool[0m [0;34m=[0m [0;32mFalse[0m[0;34m,[0m  [0;31m# switch to choose more precise quadmesh plot[0m[0;34m[0m
[0;34m[0m        [0mpercentiles[0m[0;34m:[0m [0mtuple[0m [0;34m=[0m [0;34m([0m[0;34m[0m
[0;34m[0m            [0;36m0.5[0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0;36m99.5[0m[0;34m,[0m[0;34m[0m
[0;34m[0m        [0;34m)[0m[0;34m,[0m  [0;31m# percentiles to be applied as minimum/maximum stretch[0m[0;34m[0m
[0;34m[0m        [0mclim[0m[0;34m:[0m [0mtuple[0m [0;34m=[0m [0;32mNone[0m[0;34m,[0m  [0;31m# Set the visual stretch manually instead of via percentiles[0m[0;34m[0m
[0;34m[0m        [0mcmap[0m[0;34m:[0m [0mstr[0m [0;34m=[0m [0;34m"viridis"[0m[0;34m,[0m  [0;31m# default colormap. Other nice ones are 'plasma' or 'magma'[0m[0;34m[0m
[0;34m[0m        [0mcalibrated[0m [0;34m=[0m [0;32mFalse[0m[0;34m,[0m  [0;31m# switch to control if to plot raw or calibrated data[0m[0;34m[0m
[0;34m[0m    [0;34m)[0m[0;34m:[0m[0;34m[0m
[0;34m[0m        [0;34m"""Create default hvplot for the data.[0m
[0;34m[0m
[0;34m        Due to non-equidistant wavelengths, one should use the quadmesh plot,[0m
[0;34m        but that is less performant than a constant raster and creates an annoying[0m
[0;34m        aliasing structure when zoomed out (however correct, though).[0m
[0;34m[0m
[0;34m        I am investigating if that aliasing can be avoided, it might come from gridlines.[0m
[0;34m        So I leave it to the user to switch to the raster plot using the `precise` switch."""[0m[0;34m[0m
[0;34m[0m        [0mdata[0m [0;34m=[0m [0mself[0m[0;34m.[0m[0mxarray[0m [0;32mif[0m [0;32mnot[0m [0mcalibrated[0m [0;32melse[0m [0mself[0m[0;34m.[0m[0mcalibrated[0m[0;34m[0m
[0;34m[0m        [0;31m# define good stretch by using percentiles:[0m[0;34m[0m
[0;34m[0m        [0mstretch[0m [0;34m=[0m [0;34m([0m[0;34m[0m
[0;34m[0m            [0mtuple[0m[0;34m([0m[0mnp[0m[0;34m.[0m[0mpercentile[0m[0;34m([0m[0mdata[0m[0;34m,[0m [0mpercentiles[0m[0;34m)[0m[0;34m)[0m [0;32mif[0m [0mclim[0m [0;32mis[0m [0;32mNone[0m [0;32melse[0m [0mclim[0m[0;34m[0m
[0;34m[0m        [0;34m)[0m[0;34m[0m
[0;34m[0m        [0;31m# use `frameswise=False` to allow zooming in survice scrolling over samples[0m[0;34m[0m
[0;34m[0m        [0mkwargs[0m [0;34m=[0m [0mdict[0m[0;34m([0m[0;34m[0m
[0;34m[0m            [0mx[0m[0;34m=[0m[0;34m"spectral"[0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0my[0m[0;34m=[0m[0;34m"spatial"[0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0mframewise[0m[0;34m=[0m[0;32mFalse[0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0mcmap[0m[0;34m=[0m[0mcmap[0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0mclim[0m[0;34m=[0m[0mstretch[0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0mclabel[0m[0;34m=[0m[0mdata[0m[0;34m.[0m[0mattrs[0m[0;34m[[0m[0;34m'units'[0m[0;34m][0m[0;34m,[0m[0;34m[0m
[0;34m[0m            [0mtitle[0m[0;34m=[0m[0mdata[0m[0;34m.[0m[0mattrs[0m[0;34m[[0m[0;34m'long_name'[0m[0;34m][0m[0;34m,[0m[0;34m[0m
[0;34m[0m        [0;34m)[0m[0;34m[0m
[0;34m[0m        [0;32mif[0m [0mprecise[0m[0;34m:[0m[0;34m[0m
[0;34m[0m            [0mkwargs[0m[0;34m[[0m[0;34m"kind"[0m[0;34m][0m [0;34m=[0m [0;34m"quadmesh"[0m[0;34m[0m
[0;34m[0m        [0;32mreturn[0m [0mdata[0m[0;34m.[0m[0mhvplot[0m[0;34m([0m[0;34m**[0m[0mkwargs[0m[0;34m)[0m[0;34m.[0m[0mopts[0m[0;34m([0m[0mgridstyle[0m[0;34m=[0m[0;34m{[0m[0;34m"grid_line_color"[0m[0;34m:[0m [0;34m"white"[0m[0;34m}[0m[0;34m)[0m[0;34m.[0m[0mopts[0m[0;34m([0m[0maxiswise[0m[0;34m=[0m[0;32mTrue[0m[0;34m)[0m[0;34m[0m[0;34m[0m[0m
[0;31mFile:[0m      ~/Dropbox/src/pyuvis/pyuvis/io.py
[0;31mType:[0m      method

Plot style

I chose to use the less precise (due to our non-equidistant wavelengths) raster plot by default, instead of the more precise quadmesh, because the latter creates an annoying aliasing effect,possibly due to gridlines, I'm am investigating.

If you need to be really precise on the wavelength position, use the precise=True keyword:

uv.plot(precise=True)

Control the visual limits using the percentile stretch or the clim parameter directly:

uv.plot(percentiles=(2, 98))

uv.plot(clim=(None, 10))

Control the colormap using cmap:

uv.plot(precise=False, cmap='magma')

All the options added to the UVPDS.plot method can be applied to the xarray directly, if you need the direct control.

For example, you can select a specific sample using the xarray.sel method and plot only that:

arr.sel(samples=100).hvplot(x='spectral', y='spatial', cmap='viridis')

One can also apply a video scrubber to the samples instead:

np.percentile(arr, (1, 99))  # determine good color stretch for inner 98% of the data

array([ 0., 55.])

arr.hvplot(
    x="spectral",
    y="spatial",
    widget_type="scrubber",
    widget_location="bottom",
    clim=(0, 55),
    cmap="viridis",
)

Calibrated data

Calibration masks are being downloaded automatically by the planetarypy.uvis module.

It is applied following the qube description in the calibration data label:

uv.caliblabel.QUBE.DESCRIPTION

'This UVIS Qube contains a (1,024x64) matrix of 4 byte IEEE reals. This matrix is made up of calibration values used to convert raw detector counts into rayleighs. Use these instructions to generate calibrated data: 1. multiply each sample of the raw data by the matrix in this object. 2. multiply each product by the CORE_MULTIPLIER value defined in this object The resulting object contains calibrated data in units of kiloRayleighs. CORE_NULL values correspond to pixels which do not contain valid data and which should not be used in any application of calibrated data. CORE_NULL values can be eliminated by interpolation which is a process described in SOFTWARE/CALIB/UVIS_INVALID_PIXEL_DISCUSSION.DOC. The data in this Qube was produced using the algorithm found in: GET_FUV_2010_LAB_CALIBRATION.PRO and data found in: FUV_1999_LAB_CAL.DAT'

uv.calibrated