Geostatistical methods for spatial modeling
Project description
GeostatsPy Package
The GeostatsPy Package brings GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functions to Python. GSLIB is extremely robust and practical set of code for building spatial modeling workflows. I need it in Python to support my students in my Data Analytics, Geostatistics and Machine Learning courses.
I find my students benefit from handson opportunities, in fact it is hard to imagine teaching these topics without providing the opportunity to handle the numerical methods and build workflows. I tried to have the use the original FORTRAN executables last year and even with support and worked out examples, it was an uphill battle.
In addition, all my students and I are now working in Python for our research. Having geostatistical methods in Python directly impact the research of my group. Finally, I like to code. I have over 25 years of experience in FORTRAN, C++ and Visual Basic programing. This includes frontend (Qt interfaces in C++) and backend development with small and at times very large engineering and geoscience projects.
This package includes 2 parts:

geostatspy.geostats includes GSLIB functions rewritten in Python. This currently includes all the variogram, distribution transformations, and spatial estimation and simulation (SGSIM soon) methods. I will continue adding functions to support modeling operations for practical subsurface model cosntruction.

geostatspy.GSLIB includes reimplimentation of the GSLIB visualizations and low tech wrappers of the numerical methods (note: the lowtech wrapper require access to GSLIB executables).
The Authors
This package is being developed at the University of Texas in the Texas Center for Geostatistics.

Michael J. Pyrcz, Ph.D., P.Eng.  associate professor with the University of Texas at Austin. Primary author of the package.

Anton Kupenko  bug fixes, added docstrings, code refractory for PEP8, removed duplicated functions and variables. Thank you Anton!

Wendi Liu  Ph.D. student working with Michael Pyrcz at the University of Texas at Austin. GSLIB compiles in Mac OSX, 3D variogram calculation lowtech wrapper.

Alex E. Gigliotti  undergraduate student working with Michael Pyrcz at the University of Texas at Austin. Established unit testing.
Package Inventory
Here's a list and some details on each of the functions available.
geostatspy.GSLIB Functions
Utilities to support moving between Python DataFrames and ndarrays, and Data Tables, Gridded Data and Models in GeoEAS file format (standard to GSLIB):
 ndarray2GSLIB  utility to convert 1D or 2D numpy ndarray to a GSLIB GeoEAS file for use with GSLIB methods
 GSLIB2ndarray  utility to convert GSLIB GeoEAS files to a 1D or 2D numpy ndarray for use with Python methods
 Dataframe2GSLIB(data_file,df)  utility to convert pandas DataFrame to a GSLIB GeoEAS file for use with GSLIB methods
 GSLIB2Dataframe  utility to convert GSLIB GeoEAS files to a pandas DataFrame for use with Python methods
 DataFrame2ndarray  take spatial data from a DataFrame and make a sparse 2D ndarray (NaN where no data in cell)
Visualization functions with the same parameterization as GSLIB using matplotlib:
 pixelplt  reimplemention in Python of GSLIB pixelplt with matplotlib methods
 pixelplt_st  reimplemention in Python of GSLIB pixelplt with matplotlib methods with support for sub plots
 pixelplt_log_st  reimplemention in Python of GSLIB pixelplt with matplotlib methods with support for sub plots and log color bar
 locpix  pixel plot and location map, reimplementation in Python of a GSLIB MOD with MatPlotLib methods
 locpix_st  pixel plot and location map, reimplementation in Python of a GSLIB MOD with MatPlotLib methods with support for sub plots
 locpix_log_st  pixel plot and location map, reimplementation in Python of a GSLIB MOD with MatPlotLib methods with support for sub plots and log color bar
 hist  histograms reimplemented in Python of GSLIB hist with MatPlotLib methods
 hist_st  histograms reimplemented in Python of GSLIB hist with MatPlotLib methods with support for sub plots
Data transformations
 affine  affine distribution transformation to correct feature mean and standard deviation
 nscore  normal score transform, wrapper for nscore from GSLIB (GSLIB's nscore.exe must be in working directory)
 declus  cellbased declustering, 2D wrapper for declus from GSLIB (GSLIB's declus.exe must be in working directory)
Spatial Continuity
 make_variogram  make a dictionary of variogram parameters to for application with spatial estimation and simulation
 gamv  irregularly sampled variogram, 2D wrapper for gam from GSLIB (.exe must be in working directory)
 varmap  regular spaced data, 2D wrapper for varmap from GSLIB (.exe must be in working directory)
 varmapv  irregular spaced data, 2D wrapper for varmap from GSLIB (.exe must be in working directory)
 vmodel  variogram model, 2D wrapper for vmodel from GSLIB (.exe must be in working directory)
Spatial Modeling
 kb2d  kriging estimation, 2D wrapper for kb2d from GSLIB (GSLIB's kb2d.exe must be in working directory)
 sgsim_uncond  sequential Gaussian simulation, 2D unconditional wrapper for sgsim from GSLIB (GSLIB's sgsim.exe must be in working directory)
 sgsim  sequential Gaussian simulation, 2D wrapper for sgsim from GSLIB (GSLIB's sgsim.exe must be in working directory)
 cosgsim_uncond  sequential Gaussian simulation, 2D unconditional wrapper for sgsim from GSLIB (GSLIB's sgsim.exe must be in working directory)
Spatial Model Resampling
 sample  sample 2D model with provided X and Y and append to DataFrame
 gkern  make a Gaussian kernel for convolution, moving window averaging (from Teddy Hartano, Stack Overflow)
 regular_sample  extract regular spaced samples from a 2D spatial model
 random_sample  extract random samples from a 2D spatial model
 DataFrame2ndarray  convent spatial point data in a DataFrame to a sparse ndarray grid
geostatspy.geostats Functions
Numerical methods in GSLIB (Deutsch and Journel, 1998) translated to Python:
 correct_trend  correct the order relations of an indicatorbased trend model
 backtr  GSLIB's backtr function to transform a distribution
 declus  GSLIB's DECLUS program reimplimented for cellbased declustering in 2D
 gam  GSLIB's GAM program reimplimented for variogram calculation with regular data in 2D
 gamv  GSLIB's GAMV program reimplimented for variogram calculation with iregular data in 2D
 varmapv  GSLIB's VARMAP program reimplimented for irregularly spaced spatial data in 2D
 vmodel  GSLIB's VMODEL program reimplimented for visualization of nested variogram models in 2D
 nscore  GSLIB's NSCORE program reimplimented for normal score distribution transformation
 kb2d  GSLIB's KB2D program reimplimented for 2D spaitial estimation
 ik2d  GSLIB's IK3D program reimplimented for 2D indicatorbased kriging estimation
More functionality will be added soon.
Package Dependencies
The functions rely on the following packages:
 numpy  for ndarrays
 pandas  for DataFrames
 numpy.linalg  for linear algebra
 numba  for numerical speed up
 scipy  for fast nearest neighbor search
 matplotlib.pyplot  for plotting
These packages should be available with any modern Python distribution (e.g. https://www.anaconda.com/download/).
If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python m pip install [packagename]'. More assistance is available with the respective package docs.
Explanation of GeostatsPy
GeostatsPy includes functions that run 2D workflows in GSLIB from Python (i.e. low tech wrappers), Python translations and reimplementations of GSLIB methods, along with utilities to move between GSLIB's GeoEAS data sets and Pandas DataFrames, and grids and 2D NumPy ndarrays respectively and other useful operations such as resampling from regular datasets and rescaling distributions.
The reimplementations as of now include NSCORE, GAM, GAMV, VMODEL, DECLUS, KB2D, IK2D and SGSIM etc. and most of the visualizations using the standard GSLIB parametric inputs and matplotlib back end. The low tech wrappers simply write the GSLIB parameters, run the GSLIB executables and then read in the GSLIB output. This allows for construction of Python workflows with the very robust GSLIB programs.
Why make this package?
I wanted a set of functions for working with the very robust and numerically efficient GSLIB: Geostatistical Library (Deutsch and Journel, 1998) in Python. While there are other current solutions in Python. I found that these solutions are either proprietary (not open source), not maintained or missing vital functionality; therefore, I have not been able to use these other solutions to teach modeling workflows to students with little or no programming experience. Imagine getting 55 undergraduate students to role back to a previous version on Python because a single dependency of an available package is not available in a current Python version. Image a student about to submit an assignment, and the code won't run immediately before submission because of an update to a dependency. I need methods for my students that just work, are reliable and do not require students to complete a more complicated environment setup.
Deutsch and Journel (1998) gave the community GSLIB, an extremely robust and flexible set of tools to build spatial modeling workflows. I have spent almost 20 years working with GSLIB along with a wide variety of subsurface modeling software. The powerful flexibility of GSLIB may be lost in methods that attempt to 'can' the inputs and parameters into complicated and hard to work with objects or attempt to combine the programs into a single program. I love open source for teaching the theory, because students must see under the hood! The concept of basic building blocks and simple, common inputs is essential to GSLIB. I tried to preserve this by putting together functions with the same conventions as GSLIB, the result is a set of functions that (1) are practical for my students to use and (2) will move the GSLIB veterans into Python workflow construction. Honestly, I did nothing original, but that was my intention.
I'm a very busy new professor, I'll keep adding more functionality as I have time.
More on GSLIB
The GSLIB source is available from GSLIB.com. If you would like to get the executables, ready to use without any need to compile them, go to GSLIB.com for Windows and Linux. I failed to find any Mac OS X executables so my Ph.D. student Wendi Liu compiled them for us (thank you Wendi!) and we have posted them here https://github.com/GeostatsGuy/GSLIB_MacOS. If folks on Windows are encountering missing DLL's, I could post static builds. Wendi provided instructions to help Mac users with missing DLL issues at that same location above.
Making Images
The graphical / visualization methods have 2 variants. '_ST' specifies that function is suitable for building subplots (stacked / combined images) while those without '_ST' are for standalone images (an image is returned with the function call and an image file is saved in the working directory). The resolution and image file type are specified at the top of the GeostatPy.GSLIB functions.
Assistance Welcome
Found an issue (I'm sure there are issues!)? Got a new idea? Want to work on this? Let me know, submit revisions, I'm happy to collaborate.
Package Examples
There are many example workflow examples available on my GitHub account at https://github.com/GeostatsGuy/, specifically the GeostatsPy https://github.com/GeostatsGuy/GeostatsPy and PythonNumericalDemos https://github.com/GeostatsGuy/PythonNumericalDemos repositories. Most of these examples simply placed the code for the required functions directly in the Jupyter notebook. These were made before this Package was made as I was developing all the functions individually. To use these examples just make these modifications:
 install geostatspy with the command pip install geostatspy. I used the terminal Anaconda Navigator under the environments tab to make sure the package was accessible to Jupyter Notebooks.
 add import geostatspy.GSLIB as GSLIB and import geostatspy.geostats as geostats to the top of the workflow
 add GSLIB. or geostats. as a 'prefex to the GeostatsPy functions based on which set they belong to.
Over the next month I will update all workflows to use the geostatspy package instead of pasting code into the workflows.
Here's a simple exaple of declustering with the geostatspy package. It looks long because we include making a synthetic dataset, dropping samples to impose a sampling bias, declustering and all the visualization and diagnostics.
import geostatspy.GSLIB as GSLIB # GSLIB utilities, viz and wrapped functions import geostatspy.geostats as geostats # GSLIB converted to Python import matplotlib.pyplot as plt # plotting import scipy.stats # summary stats of ndarrays # Make a 2d simulation nx = 100; ny = 100; cell_size = 10 # grid number of cells and cell size xmin = 0.0; ymin = 0.0; # grid origin xmax = xmin + nx * cell_size; ymax = ymin + ny * cell_size# calculate the extent of model seed = 74073 # random number seed for stochastic simulation range_max = 1800; range_min = 500; azimuth = 65 # Porosity variogram ranges and azimuth vario = GSLIB.make_variogram(0.0,nst=1,it1=1,cc1=1.0,azi1=65,hmaj1=1800,hmin1=500) # assume variogram model mean = 10.0; stdev = 2.0 # Porosity mean and standard deviation vmin = 4; vmax = 16; cmap = plt.cm.plasma # color min and max and using the plasma color map # calculate a stochastic realization with standard normal distribution sim = GSLIB.sgsim_uncond(1,nx,ny,cell_size,seed,vario,"simulation") # 2d unconditional simulation sim = GSLIB.affine(sim,mean,stdev) # correct the distribution to a target mean and standard deviation # extract samples from the 2D realization sampling_ncell = 10 # sample every 10th node from the model samples = GSLIB.regular_sample(sim,xmin,xmax,ymin,ymax,sampling_ncell,10,10,nx,ny,'Realization') # remove samples to create a sample bias (preferentially removed low values to bias high) samples_cluster = samples.drop([80,79,78,73,72,71,70,65,64,63,61,57,56,54,53,47,45,42]) # this removes specific rows (samples) samples_cluster = samples_cluster.reset_index(drop=True) # we reset and remove the index (it is not sequential anymore) GSLIB.locpix(sim,xmin,xmax,ymin,ymax,cell_size,vmin,vmax,samples_cluster,'X','Y','Realization','Porosity Realization and Regular Samples','X(m)','Y(m)','Porosity (%)',cmap,"Por_Samples") # apply the declus program convert to Python wts,cell_sizes,averages = geostats.declus(samples_cluster,'X','Y','Realization',iminmax=1,noff=5,ncell=100,cmin=1,cmax=2000) samples_cluster['wts'] = wts # add the weights to the sample data samples_cluster.head() # plot the results and diagnostics for the declustering plt.subplot(321) GSLIB.locmap_st(samples_cluster,'X','Y','wts',xmin,xmax,ymin,ymax,0.0,2.0,'Declustering Weights','X (m)','Y (m)','Weights',cmap) plt.subplot(322) GSLIB.hist_st(samples_cluster['wts'],0.0,2.0,log=False,cumul=False,bins=20,weights=None,xlabel="Weights",title="Declustering Weights") plt.ylim(0.0,20) plt.subplot(323) GSLIB.hist_st(samples_cluster['Realization'],0.0,20.0,log=False,cumul=False,bins=20,weights=None,xlabel="Porosity",title="Naive Porosity") plt.ylim(0.0,20) plt.subplot(324) GSLIB.hist_st(samples_cluster['Realization'],0.0,20.0,log=False,cumul=False,bins=20,weights=samples_cluster['wts'],xlabel="Porosity",title="Naive Porosity") plt.ylim(0.0,20) # Plot the declustered mean vs. cell size to check the cell size selection plt.subplot(325) plt.scatter(cell_sizes,averages, c = "black", marker='o', alpha = 0.2, edgecolors = "none") plt.xlabel('Cell Size (m)') plt.ylabel('Porosity Average (%)') plt.title('Porosity Average vs. Cell Size') plt.ylim(8,12) plt.xlim(0,2000) print(scipy.stats.describe(wts)) plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=3.5, wspace=0.2, hspace=0.2) plt.show()
I also have various lecture notes, demonstrations and example workflows on GitHub (see link below), and some tutorials on my YouTube channel (see link below). I will continue to add more. I'm just getting started,
Michael
Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
More Resources Available at: Twitter  GitHub  Website  GoogleScholar  Book  YouTube  LinkedIn
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size  File type  Python version  Upload date  Hashes 

Filename, size geostatspy0.0.9py3noneany.whl (33.2 kB)  File type Wheel  Python version py3  Upload date  Hashes View 
Filename, size geostatspy0.0.9.tar.gz (32.2 kB)  File type Source  Python version None  Upload date  Hashes View 
Hashes for geostatspy0.0.9py3noneany.whl
Algorithm  Hash digest  

SHA256  bb35edbf411aee21f00e28a269283d1653d2d9a382c91d6cd874b2ddaa186c22 

MD5  e63a55f31f74e508515af0452ee3d970 

BLAKE2256  e814c38e047859e4c4f6dbadffcaae6d6a6b9ce4a97d19cead82f3e510af89bd 