Python RINEX 2/3 NAV/OBS reader with speed and simplicity.
Project description
GeoRinex
RINEX 3 and RINEX 2 reader in Python -- reads NAV and OBS GPS RINEX data into xarray.Dataset for easy use in analysis and plotting. This gives remarkable speed vs. legacy iterative methods, and allows for HPC / out-of-core operations on massive amounts of GNSS data. GeoRinex works in Python ≥ 3.6.
Pure compiled language RINEX processors such as within Fortran NAPEOS give perhaps 500x faster performance than this Python program. However, the initial goal of this Python program was to be for one-time offline conversion of ASCII (and compressed ASCII) RINEX to HDF5/NetCDF4, where ease of cross-platform install and correctness are primary goals.
Inputs
- RINEX 3 or RINEX 2
- NAV
- OBS
- Plain ASCII or seamlessly read compressed ASCII in:
.gz
GZIP.Z
LZW.zip
- Hatanaka compressed RINEX (plain
.crx
or.crx.gz
etc.)
Output
- File: NetCDF4 (subset of HDF5), with
zlib
compression. This yields orders of magnitude speedup in reading/converting RINEX data and allows filtering/processing of gigantic files too large to fit into RAM. - In-memory: Xarray.Dataset. This allows all the database-like indexing power of Pandas to be unleashed.
Install
Latest stable release:
pip install georinex
Current development version:
git clone https://github.com/scivision/georinex
cd georinex
python -m pip install -e .
Optional Hatanaka
If you need to use .crx
Hatanaka compressed RINEX, compile the crx2rnx
code by:
make install -C rnxcmp
Usage
The simplest command-line use is through the top-level ReadRinex
script.
- Read RINEX3 or RINEX 2 Obs or Nav file:
ReadRinex myrinex.XXx
- you can read multiple files bounded by
--tlim
likeReadRinex ~/data/ --tlim 2017-01-02-T21 2017-01-03T01
- you can read multiple files bounded by
- Read NetCDF converted RINEX data:
ReadRinex myrinex.nc
It's suggested to save the GNSS data to NetCDF4 (a subset of HDF5) with the -o
option,
as NetCDF4 is also human-readable, yet say 1000x faster to load than RINEX.
You can also of course use the package as a python imported module as in the following examples. Each example assumes you have first done:
import georinex as gr
read RINEX
This convenience function reads any possible RINEX 2/3 OBS/NAV or .nc file:
obs = gr.load('tests/demo.10o')
read times in OBS, NAV file(s)
Print start, stop times and measurement interval in a RINEX OBS or NAV file:
TimeRinex ~/my.rnx
Print start, stop times and measurement interval for all files in a directory:
TimeRinex ~/data *.rnx
Get xarray.DataArray
of times in RINEX file:
times = gr.gettimes('~/my.rnx')
read Obs
If you desire to specifically read a RINEX 2 or 3 OBS file:
obs = gr.load('tests/demo_MO.rnx')
This returns an xarray.Dataset of data within the .XXo observation file.
NaN is used as a filler value, so the commands typically end with .dropna(dim='time',how='all') to eliminate the non-observable data vs time. As per pg. 15-20 of RINEX 3.03 specification, only certain fields are valid for particular satellite systems. Not every receiver receives every type of GNSS system. Most Android devices in the Americas receive at least GPS and GLONASS.
Time limits
For very large files, time bounds can be set -- load only data between those time bounds with the
--tlim start stop
option, where start
and stop
are formatted like 2017-02-23T12:00
Use Signal and Loss of Lock indicators
By default, the SSI and LLI (loss of lock indicators) are not loaded to speed up the program and save memory.
If you need them, the -useindicators
option loads SSI and LLI.
get OBS header
To get a dict()
of the RINEX file header:
hdr = gr.rinexheader('myfile.rnx')
Index OBS data
assume the OBS data from a file is loaded in variable obs
.
Select satellite(s) (here, G13
) by
obs.sel(sv='G13').dropna(dim='time',how='all')
Pick any parameter (say, L1
) across all satellites and time (or index via .sel()
by time and/or satellite too) by:
obs['L1'].dropna(dim='time',how='all')
Indexing only a particular satellite system (here, Galileo) using Boolean indexing.
import georinex as gr
obs = gr.load('myfile.o', use='E')
would load only Galileo data by the parameter E.
ReadRinex
allow this to be specified as the -use command line parameter.
If however you want to do this after loading all the data anyway, you can make a Boolean indexer
Eind = obs.sv.to_index().str.startswith('E') # returns a simple Numpy Boolean 1-D array
Edata = obs.isel(sv=Eind) # any combination of other indices at same time or before/after also possible
Plot OBS data
Plot for all satellites L1C:
from matplotlib.pyplot import figure, show
ax = figure().gca()
ax.plot(obs.time, obs['L1C'])
show()
Suppose L1C pseudorange plot is desired for G13
:
obs['L1C'].sel(sv='G13').dropna(dim='time',how='all').plot()
read Nav
If you desire to specifically read a RINEX 2 or 3 NAV file:
nav = gr.load('tests/demo_MN.rnx')
This returns an xarray.Dataset
of the data within the RINEX 3 or RINEX 2 Navigation file.
Indexed by time x quantity
Index NAV data
assume the NAV data from a file is loaded in variable nav
.
Select satellite(s) (here, G13
) by
nav.sel(sv='G13')
Pick any parameter (say, M0
) across all satellites and time (or index by that first) by:
nav['M0']
Analysis
A significant reason for using xarray
as the base class of GeoRinex is that big data operations are fast, easy and efficient.
It's suggested to load the original RINEX files with the -use
or use=
option to greatly speed loading and conserve memory.
A copy of the processed data can be saved to NetCDF4 for fast reloading and out-of-core processing by:
obs.to_netcdf('process.nc', group='OBS')
georinex.__init.py__
shows examples of using compression and other options if desired.
Join data from multiple files
Please see documentation for xarray.concat
and xarray.merge
for more details.
Assuming you loaded OBS data from one file into obs1
and data from another file into obs2
, and the data needs to be concatenated in time:
obs = xarray.concat((obs1, obs2), dim='time')
The xarray.concat
operation may fail if there are different SV observation types in the files.
you can try the more general:
obs = xarray.merge((obs1, obs2))
Converting to Pandas DataFrames
Although Pandas DataFrames are 2-D, using say df = nav.to_dataframe()
will result in a reshaped 2-D DataFrame.
Satellites can be selected like df.loc['G12'].dropna(0, 'all')
using the usual
Pandas Multiindexing methods.
Benchmarks
Done on 5 year old Haswell laptop:
time ./ReadRinex.py tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz -u E
real 48.6 s
time ./ReadRinex.py tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz -u E -m C1C
real 17.6 s
Profiling
using
conda install line_profiler
and ipython
:
%load_ext line_profiler
%lprun -f gr.obs3._epoch gr.load('tests/CEDA00USA_R_20182100000_23H_15S_MO.rnx.gz', use='E', meas='C1C')
shows that np.genfromtxt()
is consuming about 30% of processing time, and xarray.concat
and xarray.Datasetnested inside
concat` takes over 60% of time.
Notes
RINEX 3.03 specification
- GPS satellite position is given for each time in the NAV file as Keplerian parameters, which can be converted to ECEF.
- https://downloads.rene-schwarz.com/download/M001-Keplerian_Orbit_Elements_to_Cartesian_State_Vectors.pdf
- http://www.gage.es/gFD
RINEX OBS reader algorithm
- read overall OBS header (so we know what to expect in the rest of the OBS file)
- fill the xarray.Dataset with the data by reading in blocks -- another key difference from other programs out there, instead of reading character by character, I ingest a whole time step of text at once, helping keep the processing closer to CPU cache making it much faster.
Data
For capable Android devices, you can log RINEX 3 using the built-in GPS receiver.
Here is a lot of RINEX 3 data to work with:
Likewise here's a bunch of RINEX 2 data:
Hatanaka compressed RINEX .crx
The compressed Hatanaka .crx
or .crx.gz
files are supported seamlessly via crx2rnx
as noted in the Install section.
There are distinct from the supported .rnx
, .gz
, or .zip
RINEX files.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.