## GeostatsPy: Univariate Spatial Trend Modeling for Subsurface Data Analytics in Python

### Michael Pyrcz, Associate Professor, University of Texas at Austin

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Univariate Spatial Trends Modeling for Subsurface Data Analytics in Python

Here's a simple workflow with basic univariate spatial trend modeling for subsurface modeling workflows. This should help you get started with building subsurface models that include deterministic and stochastic components.

#### Trend Modeling

Trend modeling is the modeling of local features, based on data and interpretation, that are deemed certain (known).  The trend is substracted from the data, leaving a residual that is modeled stochastically with uncertainty (treated as unknown).

* geostatistical spatial estimation methods will make an assumption concerning stationarity
    * in the presence of significant nonstationarity we can not rely on spatial estimates based on data + spatial continuity model
* if we observe a trend, we should model the trend.
    * then model the residuals stochastically

Steps:

1. model trend consistent with data and intepretation at all locations within the area of itnerest, integrate all available information and expertise.

\begin{equation}
m(\bf{u}_\beta), \, \forall \, \beta \in \, AOI
\end{equation}

2. substract trend from data at the $n$ data locations to formulate a residual at the data locations.

\begin{equation}
y(\bf{u}_{\alpha}) = z(\bf{u}_{\alpha}) - m(\bf{u}_{\alpha}), \, \forall \, \alpha = 1, \ldots, n
\end{equation}

3. characterize the statistical behavoir of the residual $y(\bf{u}_{\alpha})$ integrating any information sources and interpretations.  For example the global cumulative distribution function and a measure of spatial continuity shown here.

\begin{equation}
F_y(y) \quad \gamma_y(\bf{h})
\end{equation}

4. model the residual at all locations with $L$ multiple realizations.

\begin{equation}
Y^\ell(\bf{u}_\beta),  \, \forall \, \beta \, \in \, AOI; \, \ell = 1, \ldots, L
\end{equation}

5. add the trend back in to the stochastic residual realizations to calculate the multiple realizations, $L$, of the property of interest based on the composite model of known deterministic trend, $m(\bf{u}_\alpha)$ and unknown stochastic residual, $y(\bf{u}_\alpha)$

\begin{equation}
Z^\ell(\bf{u}_\beta) = Y^\ell(\bf{u}_\beta) + m(\bf{u}_\beta),  \, \forall \, \beta \in \, AOI; \, \ell = 1, \ldots, L
\end{equation}

6. check the model, including quantification of the proportion of variance treated as known (trend) and unknown (residual).

\begin{equation}
\sigma^2_{Z} = \sigma^2_{Y} + \sigma^2_{m} + 2 \cdot C_{Y,m}
\end{equation}

given $C_{Y,m} \to 0$:

\begin{equation}
\sigma^2_{Z} = \sigma^2_{Y} + \sigma^2_{m}
\end{equation}

I can now describe the proportion of variance allocated to known and unknown components as follows:

\begin{equation}
Prop_{Known} = \frac{\sigma^2_{m}}{\sigma^2_{Y} + \sigma^2_{m}} \quad Prop_{Unknown} = \frac{\sigma^2_{Y}}{\sigma^2_{Y} + \sigma^2_{m}}
\end{equation}

I provide some practical, data-driven methods for trend model, but I should indicate that:

1. trend modeling is very important in reservoir modeling as it has large impact on local model accuracy and on the undertainty model
2. trend modeling is used in almost every subsurface model, unless the data is dense enough to impose local trends
3. trend modeling includes a high degree of expert judgement combined with the integration of various information sources

We limit ourselves to simple data-driven methods, but acknowledge much more is needed. In fact, trend modeling requires a high degree of knowledge concerning local geoscience and engineering data and knowledge.

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trig etc.
import scipy.signal as signal             # kernel for moving window calculation
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")             # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv')     # load our data table (wrong name!)
```

It worked, we loaded our file into our DataFrame called 'df'. But how do you really know that it worked? Visualizing the DataFrame would be useful and we already leard about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
print(df.iloc[0:5,:])                   # display first 4 samples in the table as a preview
df.head(n=13)                           # we could also use this command for a table preview
```

         X    Y  Facies  Porosity       Perm
    0  100  900       1  0.115359   5.736104
    1  100  800       1  0.136425  17.211462
    2  100  600       1  0.135810  43.724752
    3  100  500       0  0.094414   1.609942
    4  100  100       0  0.113049  10.886001

      X
      Y
      Facies
      Porosity
      Perm

      0
      100
      900
      1
      0.115359
      5.736104

      1
      100
      800
      1
      0.136425
      17.211462

      2
      100
      600
      1
      0.135810
      43.724752

      3
      100
      500
      0
      0.094414
      1.609942

      4
      100
      100
      0
      0.113049
      10.886001

      5
      200
      800
      1
      0.154648
      106.491795

      6
      200
      700
      1
      0.153113
      140.976324

      7
      200
      500
      1
      0.126167
      12.548074

      8
      200
      400
      0
      0.094750
      1.208561

      9
      200
      100
      1
      0.150961
      44.687430

      10
      300
      800
      1
      0.199227
      1079.709291

      11
      300
      700
      1
      0.154220
      179.491695

      12
      300
      500
      1
      0.137502
      38.164911

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 2 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      289.0
      475.813149
      254.277530
      0.000000
      300.000000
      430.000000
      670.000000
      990.000000

      Y
      289.0
      529.692042
      300.895374
      9.000000
      269.000000
      549.000000
      819.000000
      999.000000

      Facies
      289.0
      0.813149
      0.390468
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      289.0
      0.134744
      0.037745
      0.058548
      0.106318
      0.126167
      0.154220
      0.228790

      Perm
      289.0
      207.832368
      559.359350
      0.075819
      3.634086
      14.908970
      71.454424
      5308.842566

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;           # range of porosity values
nx = 100; ny = 100; csize = 10.0
cmap = plt.cm.plasma                    # color map
```

Let's try out locmap. This is a reimplementation of GSLIB's locmap program that uses matplotlib. I hope you find it simpler than matplotlib, if you want to get more advanced and build custom plots lock at the source. If you improve it, send me the new code. Any help is appreciated. To see the parameters, just type the command name:

```python
GSLIB.locmap
```

    <function geostatspy.GSLIB.locmap(df, xcol, ycol, vcol, xmin, xmax, ymin, ymax, vmin, vmax, title, xlabel, ylabel, vlabel, cmap, fig_name)>

Now we can populate the plotting parameters and visualize the porosity data.

```python
plt.subplot(111)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's get some declustering weights. For more information see the demonstration on declustering.

```python
wts, cell_sizes, dmeans = geostats.declus(df,'X','Y','Porosity',iminmax = 1, noff= 10, ncell=100,cmin=10,cmax=2000)
df['Wts'] = wts                            # add weights to the sample data DataFrame
df.head()                                  # preview to check the sample data DataFrame
```

    There are 289 data with:
       mean of      0.13474387540138408
       min and max  0.058547873 and 0.228790002
       standard dev 0.03767982164385207

      X
      Y
      Facies
      Porosity
      Perm
      Wts

      0
      100
      900
      1
      0.115359
      5.736104
      3.064286

      1
      100
      800
      1
      0.136425
      17.211462
      1.076608

      2
      100
      600
      1
      0.135810
      43.724752
      0.997239

      3
      100
      500
      0
      0.094414
      1.609942
      1.165119

      4
      100
      100
      0
      0.113049
      10.886001
      1.224164

#### Trend by Convolution / Local Window Average

Let's first attempt a convolution-based trend model, this is a moving window average of the local data.

We have a convenience function that takes data with X and Y locations in a DataFrame and makes a sparse 2D array.  All cells without a data value are assigned to NumPy's NaN (null values, missing value). Let's see the inputs for this command.

```python
GSLIB.DataFrame2ndarray
```

    <function geostatspy.GSLIB.DataFrame2ndarray(df, xcol, ycol, vcol, xmin, xmax, ymin, ymax, step, nx, ny)>

Let's make an sparse array with the appropriate parameters. The reason we are doing this is that convolution programs in general work with ndarrays and not from DataFrames.

```python
por_grid = GSLIB.DataFrame2ndarray(df,'X','Y','Porosity',xmin, xmax, ymin, ymax, csize, nx, ny)
```

We have a ndarray (por_grid) with the data assigned to grid cells. Now we need a kernel. The kernel represents the weights within the moving window. If we use constant 1.0 in the moving window will get discontinuities in our trend model. A Gaussian kernel (weights highest in the middle of the window and decreasing to 0,0 at the edge) is useful to get a smooth trend. We can use the SciPy package's signal functions. Of course we will have to import that package and then we can make our kernel. Here's an example below. There shouldnt be any surprises.

```python
gkern1d = signal.gaussian(53,5).reshape(53, 1)
gkern2d = np.outer(gkern1d, gkern1d)
print('We have made a kernel of size, number of grid cells (ny, nx) ' + str(gkern2d.shape))

plt.subplot(111)
GSLIB.pixelplt_st(gkern2d,xmin=-265,xmax=265,ymin=-265,ymax=265,step=10,vmin=0,vmax=1,title='Kernel',xlabel='X(m)',ylabel='Y(m)',vlabel='weight',cmap=cmap)
plt.subplots_adjust(left=0.0, bottom=0.0, right=0.6, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

Now we need to convolve our sparse data assigned to a ndarray with our Gaussian kernel. There are many functions available for convolution. But we have a problem as we want to apply our Gaussian kernel to a sparse ndarray full of missing values. It turns out this is a common issue for our friends in Astronomy and so their Astropy package has a convolution method that will work well. I figured out the following (so you don't have to!).

```python
import astropy.convolution.convolve as convolve
porosity_trend = convolve(por_grid,gkern2d,boundary='extend',nan_treatment='interpolate',normalize_kernel=True)
```

No errors? It worked? Let's look at the results. We can plot and compare the original porosity data and the resulting trend to check for consistency.

```python
plt.subplot(131)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(132)
GSLIB.pixelplt_st(porosity_trend,xmin,xmax,ymin,ymax,csize,pormin,pormax,'Porosity Trend','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(133)
GSLIB.locpix_st(porosity_trend,xmin,xmax,ymin,ymax,csize,pormin,pormax,df,'X','Y','Porosity','Porosity Data and Trend','X(m)','Y(m)','Porosity (fraciton)',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

```

#### Other Methods for Trend Calculation

There are a variety of other methods for trend calculation. I will just mention them here.

1. hand-drawn, expert interpretation - many 3D modeling packages allow for experts to draw trends and allow for fast interpolation to build an exhaustive trend model.
2. kriging - kriging provides best linear unbiased estimates between data given a spatial continuity model (more on this when we cover spatial estimation).  One note of caution is that kriging is exact; therefore it will over fit unless it is use with averaged data values (e.g. over the vertical) or with a block kriging option (kriging at a volume support larger than the data).
3. regression - fit a function as a function of X, Y coordinates. This could be extended to more complicated prediction models from machine learning.

#### Trend Diagnotistics

Let's go back to the convolution trend and check it (to demonstrate the method of trend checking). Note, I haven't tried to perfect the result. I'm just demonstrating the method.

In addition to the previous visualization, let's look at the distributions and summary statistics of the original declustered porosity data and the trend.

```python
plt.subplot(121)
GSLIB.hist_st(df['Porosity'],pormin,pormax,False,False,20,df['Wts'],'Porosity (fraction)','Declustered Porosity')

plt.subplot(122)
GSLIB.hist_st(porosity_trend.flatten(),pormin,pormax,False,False,20,None,'Porosity Trend (fraction)','Porosity Trend')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.5, wspace=0.2, hspace=0.2)
plt.show()
```

We can also look at the summary statistics. Here's a function that calculates the weighted standard deviation (and the average). We can use this with the data and declustering weights and figure out the allocation of variance between the trend and the residual.

```python
# Weighted average and standard deviation
def weighted_avg_and_std(values, weights):                  # from Eric O Lebigot, stack overflow
    average = np.average(values, weights=weights)
    variance = np.average((values-average)**2, weights=weights)
    return (average, math.sqrt(variance))

wavg_por,wstd_por = weighted_avg_and_std(df['Porosity'],df['Wts'])

wavg_por_trend = np.average(porosity_trend)
wstd_por_trend = np.std(porosity_trend)

print('Declustered Porosity Data: Average ' + str(round(wavg_por,4)) + ', Var ' + str(round(wstd_por**2,5)))
print('Porosity Trend: Average            ' + str(round(wavg_por_trend,4)) + ', Var ' + str(round(wstd_por_trend**2,5)))
print('Proportion Trend / Known:          ' + str(round(wstd_por_trend**2/(wstd_por**2),3)))
print('Proportion Residual / Unknown:     ' + str(round((wstd_por**2 - wstd_por_trend**2)/(wstd_por**2),3)))
```

    Declustered Porosity Data: Average 0.1212, Var 0.00102
    Porosity Trend: Average            0.1233, Var 0.00064
    Proportion Trend / Known:          0.631
    Proportion Residual / Unknown:     0.369

Interesting, we have 63% of the variance being treated as known, modeled by trend, and 37% of the variance being treated as unknown, modeled by residual.

#### Adding Trend to DataFrame

Let's add the porosity trend to our DataFrame. We have a sample program in GeostatsPy that takes a 2D ndarray and extracts the values at the data locations and adds them as a new column.  Then we can do a little math to calculate and add the porosity residual also and visualize this all together as a final check.

```python
df = GSLIB.sample(porosity_trend,xmin,xmax,ymin,ymax,nx,ny,csize,"Por_Trend",df,'X','Y')
df['Por_Res'] = df['Porosity'] - df['Por_Trend'] # calculate the residual and add to DataFrame
```

Let's check out the DataFrame and confirm that we have everything now. We will need trend and residual in our DataFrame to support all subsequent modeling steps.

```python
df.head()
```

      X
      Y
      Facies
      Porosity
      Perm
      Wts
      Por_Trend
      Por_Res

      0
      100
      900
      1
      0.115359
      5.736104
      3.064286
      0.117365
      -0.002006

      1
      100
      800
      1
      0.136425
      17.211462
      1.076608
      0.123938
      0.012487

      2
      100
      600
      1
      0.135810
      43.724752
      0.997239
      0.128435
      0.007375

      3
      100
      500
      0
      0.094414
      1.609942
      1.165119
      0.112399
      -0.017985

      4
      100
      100
      0
      0.113049
      10.886001
      1.224164
      0.102791
      0.010258

That looks good. A quick check, confirm that the Porosity column is equal to the Por_Trend + the Por_Res columns.  As a final check let's visualize the original porosity data, porosity trends at the data locations and the porosity residuals.

```python
plt.subplot(131)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(132)
GSLIB.locmap_st(df,'X','Y','Por_Trend',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity Trend','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(133)
GSLIB.locmap_st(df,'X','Y','Por_Res',xmin,xmax,ymin,ymax,-0.01,0.01,'Well Data - Porosity Residual','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Does it look correct? There is a strong degree of consistency between the porosity data and trend and the porosity residual no longer has a trend, it has been detrended.

#### Comments

This was a basic demonstration of trend modeling. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)