<p align="center">

</p>

## Machine Learning based uncertainty models

## Subsurface Property Prediction with Neural Networks

### Eduardo Maldonado Cruz, Graduate Research Assistant, The University of Texas at Austin

#### [Twitter](https://twitter.com/emaldonadocruz) | [GitHub](https://github.com/emaldonadocruz) | [LinkedIn](https://www.linkedin.com/in/eduardo-maldonado-8386a898/)

### Michael Pyrcz, Associate Professor, The University of Texas at Austin

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

## Executive summary

Deep learning models focus on **prediction accuracy and minimizing prediction error**. Yet, for spatial models, **accurate predictions along with accurate and precise uncertainty models are critical.**

Due to limited data veracity, data sparsity, and feature heterogeneity in the spatial setting, uncertainty is significant; therefore, **predicting a single estimate must be replaced with a prediction of the uncertainty distribution**.

**We propose a novel and easy to use objective function to summarize uncertainty model performance**.
The proposed function tunes deep learning models for optimum uncertainty accuracy and precision.

## Objective

Tune deep learning models for maximum **goodness**, **accuracy** and **precision**.

## Introduction

Deep learning models focus on prediction accuracy and minimizing prediction error. Yet, for spatial models, accurate predictions along with accurate and precise uncertainty models are critical. Prediction of the entire uncertainty model is accomplished robustly with Bayesian Neural Networks but with high computational cost(MacKay, 1991; Hinton and Van Camp 1993). Leading to the development of stochastic regularization techniques, one of the most popular is dropout (Hinton et al., 2012; Srivastava et al., 2014, Gal, 2016, Gal and Ghahramani., 2016, Kendall and Gal, 2017, Osband et al., 2016) that can be interpreted as a Bayesian approximation with greater computationally efficiency.

Dropout is a technique that addresses overfitting in neural networks and provides an approach to improve prediction accuracy by ensemble learning. Dropout refers to randomly removing or reducing (also known as regularizing) network weights, removing or weakening the connections between the nodes in the neural network to introduce prediction model stochasticity during model training (Dahl, 2013; Srivastava, 2014; Hara, 2016).

Ensemble predictions benefit from reduced model variance and improved prediction model generalization. The proposed method uses the machine learning dropout hyperparameter to calculate accurate and precise uncertainty models of response features. This objective function is based on the goodness metric proposed by Deutsch (1996).

The goodness of the uncertainty model (Maldonado and Pyrcz, 2021)

\begin{equation}
\mathrm{f}\left(x\right)=\frac{1}{2}\left[\frac{1}{n}\sum_{i=1}^{n}\left|y\left(\mathbf{u}_{\mathrm{i}}\right)-\hat{y}\left(\mathbf{u}_{\mathrm{i}}\right)\right|\right]+\frac{1}{2}\left[\int_{0}^{1}\left[3a\left(p\right)-2\right]\left[\overline{\xi\left(p\right)}-p\right]dp\right]
\label{eq:Goodness} \tag{1}
\end{equation}

where the first term is the typical mean absolute error (MAE) in testing between $𝑦_𝑖 (𝐮_𝐢 )$ training response feature values and $𝑦_𝑖 (𝐮_𝐢 )$ estimates over n withheld testing data, and the second term is a measure of goodness.

Where $\xi(𝐮_𝑖;𝑝)$ is the indicator function at each point $𝐮_𝑖, 𝑖=1,…,𝑛,$ evaluated over a range of symmetric 𝑝-probability intervals:

\begin{equation}
\xi\left(\mathbf{u}_{\mathrm{i}};p\right)=\begin{cases}
1, & F_{y}\left(\mathbf{u}_{\mathrm{i}};y\left(\mathbf{u}_{\mathrm{i}}\right)\right)\in\left(p_{low},p_{upp}\right]\\
0, & \mathrm{Otherwise}
\end{cases}
\label{eq:IndicatorFunc} \tag{2}
\end{equation}

Next, we use $\ref{eq:Goodness}$ as the objective function to tune a very simple uncertainty model.

## Library importing

First we must import some libraries to train a deep learning model.

```python
# Set verbosity
import logging
logging.getLogger("tensorflow").setLevel(logging.ERROR)

# Libraries for data analysis and array manipulation
import pandas as pd
import numpy as np

# Libraries for plotting
import matplotlib.pyplot as plt
from matplotlib import colors

# Import tensorflow
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from keras.layers.core import Lambda
from tensorflow.keras import backend as K

#Librabry with useful analysis statistics
from scipy import stats
from scipy import integrate
import random

# Preprocessing and train-test split
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# library for hyperparameter search
import optuna

#To import, save and load jobs
import joblib
```

```python
tf.random.set_seed(0)
np.random.RandomState(38)
np.random.seed(38)
```

```python
!pip install UTuning
```

    Requirement already satisfied: UTuning in c:\users\em42363\anaconda3\envs\new\lib\site-packages (0.1.3)

```python
#import sys
#sys.path.insert(0, r'C:\Users\em42363\OneDrive\PhD\UTuning')

from UTuning import scorer, plots
```

```python
## Plots
import matplotlib.pyplot as plt
import numpy as np
import random

import matplotlib.tri as tri
from scipy.ndimage import gaussian_filter

#import matplotlib
#matplotlib.rcParams['text.usetex'] = True

def error_line(Prediction, Truth, Sigma, Frac=1):
    '''Simple function to draw an error line plot. It takes three arrays of the same length

    Parameters
    ----------

    Prediction : float array
        The predicted value array (Prediction)
    Truth : float array
        The truth value array (Truth)
    Sigma : float array
        The standard deviation array (Sigma)
    Frac : float
        Frac is the fraction of points to display randomly

    Returns
    -------
    Scatter plot
    '''
    yline = [min(Prediction.min(), Truth.min()),
             max(Prediction.max(), Truth.max())]
    xline = [min(Prediction.min(), Truth.min()),
             max(Prediction.max(), Truth.max())]

    fig, ax = plt.subplots(figsize=(10, 6))
    # To display randomly less points [Remove , Keep] in fraction
    mask = np.random.choice([False, True], len(Prediction), p=[(1-Frac), (Frac)])
    ax.errorbar(Prediction[mask], Truth[mask], xerr=Sigma[mask],
                fmt='k.',
                ecolor='k')
    ax.plot(xline, yline, '-k')
    ax.set_xlabel('Predicted value, $\hat{y}$')
    ax.set_ylabel('True value, $y$ ')
    plt.show()

def error_accuracy_plot(percentile,IF_array,Prediction_array,Truth,Sigma):
    '''Simple function to draw an error line plot and its corresponding accuracy plot.

    Parameters
    ----------

    Prediction : float array
        The predicted value array (Prediction)
    Truth : float array
        The truth value array (Truth)
    Sigma : float array
        The standard deviation array (Sigma)
    Frac : float
        Frac is the fraction of points to display randomly

    Returns
    -------
    Scatter plot
    '''
    avgIndFunc = np.mean(IF_array, axis=0)

    L = 10
    mean = np.empty((L, len(percentile)))

    for p_interv in range(len(percentile)):
        for l in np.arange(0, L):
            samples = random.choices(IF_array[:, p_interv],
                                     k=IF_array.shape[0])
            mean[l, p_interv] = np.mean(samples)

    fig,(ax1,ax2)=plt.subplots(1,2,figsize=(12,4))

    if len(Prediction_array.shape)>1:
        xline = [0,max(np.mean(Prediction_array,axis=1).max(),Truth.max())+max(np.mean(Prediction_array,axis=1).max(),Truth.max())*0.1]#
        yline = [0,xline[1]]#

        ax1.errorbar(np.mean(Prediction_array,axis=1), Truth, xerr=Sigma,
                     fmt='k.',
                     ecolor='k')
    else:
        xline = [0,max(Prediction_array.max(),Truth.max())+max(Prediction_array.max(),Truth.max())*0.1]#
        yline = [0,xline[1]]#

        ax1.errorbar(Prediction_array, Truth, xerr=Sigma,
                     fmt='k.',
                     ecolor='k')
    ax1.plot(xline, yline, '-k')
    ax1.set_xlabel('Predicted value, $\hat{y}$')
    ax1.set_ylabel('True value, $y$ ')

    ax2.plot(percentile, avgIndFunc,'-ok',markersize=5)
    ax2.plot(percentile,np.round(avgIndFunc+np.std(mean, axis=0), 3),'--k')
    ax2.plot(percentile,np.round(avgIndFunc-np.std(mean, axis=0), 3),'--k')
    ax2.plot([0, 1],[0, 1],'-k')
    ax2.set_ylabel(r"$\overline{\xi (p)}$")
    ax2.set_xlabel('Probability interval $p$')
    ax2.set_ylim(0,1)
    ax2.set_xlim(0,1)

    ax2.plot(percentile, avgIndFunc,'-ok',markersize=5)

def surface(x, y, z, levels, labels):

    fig, (ax1) = plt.subplots(nrows=1,figsize=(12,6))

    npoints=x.shape[0]
    smooth=1

    # Create grid values first.
    xi = np.linspace(x.min(), x.max(), npoints)
    yi = np.linspace(y.min(), y.max(), npoints)

    # Linearly interpolate the data (x, y) on a grid defined by (xi, yi).
    triang = tri.Triangulation(x, y)
    interpolator = tri.LinearTriInterpolator(triang, z)
    Xi, Yi = np.meshgrid(xi, yi)
    zi = interpolator(Xi, Yi)

    zi = gaussian_filter(zi, smooth)

    levels = levels

    ax1.contour(xi, yi, zi, levels=levels, linewidths=0.1, colors='k')

    cntr1 = ax1.contourf(xi, yi, zi, levels=levels, cmap="inferno",alpha=0.95)

    cbar = plt.colorbar(cntr1, ax=ax1)
    cbar.set_label(labels['z'], rotation=270,labelpad=30)

    ax1.set(xlim=(x.min(), x.max()),ylim=(y.min(), y.max()))

    ax1.scatter(x,y,s=7,color='white')
    ax1.set_xlabel(labels['x'])
    ax1.set_ylabel(labels['y'])

    plt.show()

```

```python
## Scorer
import numpy as np
import scipy.stats as stats
import scipy.integrate as integrate

class scorer:
    '''
    Scorer class, a class to represent the accuracy, precision and goodness
    score from an uncertainty model.

    Attributes
    ----------
    Accuracy : float
    Precision : float
    Goodness : float
    Overall uncertainty : float
    Indicator Function : float array
    '''
    def __init__(self, Prediction, Truth, Sigma):

        n_quantiles = 11
        self.perc = np.linspace(0.0, 1.01, n_quantiles)
        # self.A_array=np.zeros(Prediction.shape[0])
        # self.P_array=np.zeros(Prediction.shape[0])
        # self.G_array=np.zeros(Prediction.shape[0])
        #self.U_array=np.zeros(Prediction.shape[0])

        self.IF_array=np.zeros((Prediction.shape[0],n_quantiles))

        if len(Prediction.shape)>1:
            for i in range(Prediction.shape[0]):
                IF = APG_calc(Truth[i], Prediction[i,:], Sigma[i],n_quantiles)
                #self.U_array[i]=U
                # self.A_array[i]=A
                # self.P_array[i]=P
                # self.G_array[i]=G
                self.IF_array[i,:] = IF
        else:
            Pred_array = np.zeros((Sigma.shape[0],100))
            for i in range(Prediction.shape[0]):
                Pred_array[i,:] = np.random.normal(loc=Prediction[i],scale=Sigma[i],size=100)
                IF = APG_calc(Truth[i], Pred_array[i,:], Sigma[i],n_quantiles)
                #self.U_array[i]=U
                # self.A_array[i]=A
                # self.P_array[i]=P
                # self.G_array[i]=G
                self.IF_array[i,:] = IF

        self.avgIndFunc = np.mean(self.IF_array, axis=0)

        self.a = np.zeros(len(self.avgIndFunc))
        for i in range(len(self.avgIndFunc)):
            if self.avgIndFunc[i] > self.perc[i] or self.avgIndFunc[i] == self.perc[i]:
                self.a[i] = 1
            else:
                self.a[i] = 0

    def Accuracy(self):
        Accuracy = integrate.simps(self.a, self.perc)
        return Accuracy
        #return print('Accuracy = {0:2.2f}'.format(np.mean(self.A_array)))

    def Precision(self):
        Prec = self.a*(self.avgIndFunc-self.perc)
        Precision = 1-2*integrate.simps(Prec, self.perc)
        return Precision
        #return print('Precision = {0:2.2f}'.format(np.mean(self.P_array)))

    def Goodness(self):
        Sum = (3*self.a-2)*(self.avgIndFunc-self.perc)
        Goodness = 1-integrate.simps(Sum, self.perc)
        return Goodness
        #return print('Goodness = {0:2.2f}'.format(np.mean(self.G_array)))

    def Overall_uncertainty(self,Sigma):
        return Sigma.mean()
        #return print('Overall uncertainty = {0:2.2f}'.format(np.mean(self.G_array)))

    def IndicatorFunction(self):
        return self.IF_array

def APG_calc(Truth, Pred, Sigma,n_quantiles):

    mask = np.random.choice([False, True],
                            len(Pred),
                            p=[0, 1]) # To display randomly less points [Remove , Keep] in fraction

    Pred=Pred[mask]

    perc = np.linspace(0.0, 1.01, n_quantiles)

    F = np.zeros(Pred.shape[0])
    Indicator_func = np.zeros((Pred.shape[0], perc.shape[0]))

    # range of symmetric p-probability intervals
    plow = (1 - perc) / 2
    pupp = (1 + perc) / 2

    for i in range(len(Pred)):
        F[i] = stats.norm.cdf(Truth,
                          loc=Pred[i],
                          scale=Sigma)
        for proba_low, proba_upp in zip(plow, pupp):
            for k in range(len(plow)):
                if plow[k] < F[i] <= pupp[k]:
                    Indicator_func[i, k] = 1
                else:
                    Indicator_func[i, k] = 0

    avgIndFunc = np.mean(Indicator_func, axis=0)

    return avgIndFunc

```

```python
physical_devices = tf.config.list_physical_devices('GPU')
if len(physical_devices)>0:
    tf.config.experimental.set_memory_growth(physical_devices[0], True)
```

## Functions

First lets define some useful functions that will be later used for plotting and case evaluation

```python
# Custom functions
def get_dataframe(mc_predictions, y_s, X_s):
    '''
    This function takes the sampled distribution along with the testing data and obtains
    the mean, variance and standard deviation, from this function we get the dataframe
    to construct the accuracy plot.
    '''

    Mean = np.zeros((mc_predictions.shape[1]))
    Var = np.zeros((mc_predictions.shape[1]))
    STD = np.zeros((mc_predictions.shape[1]))

    for i in range(mc_predictions.shape[1]):
        Mean[i] = np.average(mc_predictions[:, i])
        Var[i] = np.var(mc_predictions[:, i])
        STD[i] = np.std(mc_predictions[:, i])

    mask = np.random.choice([False, True], len(Mean), p=[0.5, 0.5])

    df = pd.DataFrame()

    df['Pred'] = Mean[mask]
    df['Truth'] = y_s[mask]
    df['Var'] = Var[mask]
    df['Sigma'] = np.sqrt(Var[mask])

    return df

def get_GAP(MeanVal, Sigma, Truth, n_quantiles):
    '''
    This function takes the dataframe and calculates the indicator function and the average
    we then use this information to estimate the accuracy, goodness and precision of the model.
    '''
    perc = np.linspace(0.0, 1.00, n_quantiles)
    F = np.zeros(MeanVal.shape[0])
    Indicator_func = np.zeros((MeanVal.shape[0], perc.shape[0]))

    # range of symmetric p-probability intervals
    plow = (1 - perc) / 2
    pupp = (1 + perc) / 2

    for i in range(len(MeanVal)):
        F[i] = stats.norm.cdf(Truth[i],
                              loc=MeanVal[i],
                              scale=Sigma[i])
        for proba_low, proba_upp in zip(plow, pupp):
            for k in range(len(plow)):
                if plow[k] < F[i] <= pupp[k]:
                    Indicator_func[i, k] = 1
                else:
                    Indicator_func[i, k] = 0

    avgIndFunc = np.mean(Indicator_func, axis=0)

    a = np.zeros(len(avgIndFunc))
    for i in range(len(avgIndFunc)):
        if avgIndFunc[i] > perc[i] or avgIndFunc[i] == perc[i]:
            a[i] = 1
        else:
            a[i] = 0
    print(f'Overall uncertainty = {Sigma.mean():.6f}')
    U = Sigma.mean()

    ##% Goodness, Precision and Accuracy

    Accuracy = integrate.simps(a, perc)

    Prec = a*(avgIndFunc-perc)
    Precision = 1-2*integrate.simps(Prec, perc)

    Sum = (3*a-2)*(avgIndFunc-perc)
    Goodness = 1-integrate.simps(Sum, perc)

    return Goodness, Precision, Accuracy, U, Indicator_func,perc

def error_line(Mean,Truth,STD,r):
    '''
    Simple function to draw an error line plot. It takes the mean predicted value,
    the truth value and the standard deviation.
    '''

    xline = [0,1]#
    yline = [0,1]#
    plt.figure(figsize=(10, 6))
    mask = np.random.choice([False, True], len(Mean), p=[0.5, 0.5])
    plt.errorbar(Mean[mask], Truth[mask], xerr=STD[mask],
                 fmt='k.', ecolor='k',label="Pearson's r = {0:2.2f}".format(r))
    plt.legend()
    plt.plot(xline, yline, '-k')

    plt.xlabel('Predicted value, $\hat{y}$ [bl]')
    plt.ylabel('True value, $y$ [bl]')

    plt.show()

def accuracy_plot(Indicator_func,perc):
    '''
    This function takes the indicator function and percentiles to draw the accuracy plot.
    '''

    L = 100
    mean = np.empty((L, len(perc)))
    std = np.empty_like(mean)
    avgIndFunc = np.mean(Indicator_func, axis=0)
    for p_interv in range(len(perc)):
        for l in np.arange(0, L):
            samples = random.choices(Indicator_func[:, p_interv],
                                     k=Indicator_func.shape[0])
            mean[l, p_interv] = np.mean(samples)

    plt.figure(figsize=(10, 6))
    plt.plot(perc, avgIndFunc,'-ok',markersize=5)
    plt.plot(perc,np.round(avgIndFunc+np.std(mean, axis=0), 3),'--k')
    plt.plot(perc,np.round(avgIndFunc-np.std(mean, axis=0), 3),'--k')
    plt.plot([0, 1],[0, 1],'-k')
    plt.ylabel(r"$\overline{\xi (p)}$")
    plt.xlabel('Probability interval $p$')
    plt.ylim(0,1)
    plt.xlim(0,1)
    plt.show()

def histogram(mc_predictions):
    '''
    From the Monte Carlo predictions we draw a random point and construct the
    histogram of predictions from the model
    '''

    Avg = []
    rand=np.random.randint(0,mc_predictions.shape[1])
    for i in range(mc_predictions.shape[0]):
        Avg.append(np.average(mc_predictions[i,rand]))

    Std = np.std(Avg)

    # Histograms
    n_bins = 20
    fig, axs = plt.subplots(1, 1, figsize=(10, 6))
    N, bins, patches = axs.hist(Avg,
                                bins=n_bins,
                                label='$\sigma$ = %2.5f' % Std)
    #axs.set_title('Root Mean squared error in barrels for each cell');
    fracs = N / N.max()
    norm = colors.Normalize(fracs.min(), fracs.max())
    for thisfrac, thispatch in zip(fracs, patches):
        color = plt.cm.binary(norm(thisfrac))
        thispatch.set_facecolor(color)
    plt.legend()
    plt.ylabel('Number of cases')
    plt.xlabel('Normalized production value')

def histogram_fixed(Prediction,rand):
    '''
    From the Monte Carlo predictions we draw a random point and construct the
    histogram of predictions from the model
    '''

    Pred_array = np.zeros(100)
    Pred_array = np.random.normal(loc=np.mean(Prediction[:,rand]),scale=np.std(Prediction[:,rand]),size=500)

    # Histograms
    n_bins = 15
    fig, axs = plt.subplots(1, 1, figsize=(10, 6))
    N, bins, patches = axs.hist(Pred_array,
                                bins='auto',
                                color = 'black',
                                label='$\sigma$ = %2.5f' % np.std(Prediction[:,rand]),
                                rwidth = 0.85,
                                alpha =0.85)
    plt.legend()
    plt.ylabel('Number of cases')
    plt.xlabel('Normalized production value')

def plot_loss(history):
    '''
    Simple function to plot the loss for the training and testing data
    '''
    plt.figure(figsize=(12,6))
    plt.plot(history.history['loss'], label='Training loss')
    plt.plot(history.history['val_loss'], label='Testing loss')
    plt.xlabel('Epoch')
    plt.ylabel('Loss ')
    plt.legend()
    plt.grid(True)

def objective(trial):
    '''
    We use this function to define the search space for the hyperparameter search.
    This function creates, train and evaluates the model.
    '''

    dropout = trial.suggest_discrete_uniform('Dropout', 0.05, 0.95, 0.01)
    lr = float(trial.suggest_discrete_uniform('learn_rate', 0.001, .03, 0.001))

    model = create_model(dropout)

    opt = keras.optimizers.Adam(learning_rate=lr)

    model.compile(loss='mae',
                  optimizer=opt,
                  metrics='mse')

    model.fit(
        X_train,
        y_train,
        shuffle=True,
        batch_size=16,
        epochs=100,
        verbose=False,
    )
    score = model.evaluate(X_test, y_test, verbose=0)

    mc_predictions = evaluate_cases(model, X_test, 8, y_test,100)
    #df = get_dataframe(mc_predictions, y_test, X_test)
    #df.to_csv('ModelPredictions.csv')

    Sigma = np.std(mc_predictions , axis = 0)
    Pred_array = mc_predictions
    Pred = np.mean(mc_predictions , axis = 0)
    A, P, G = model_eval(y_test,mc_predictions,Sigma,np.mean(mc_predictions,axis=0))

    Score = 0.5*score[0]+0.5*(1-G)

    return Score

def APG_calc(Truth, Pred, Sigma,n_quantiles):

    mask = np.random.choice([False, True],
                            len(Pred),
                            p=[0.0, 1]) # To display randomly less points [Remove , Keep] in fraction

    Pred=Pred[mask]
    perc = np.linspace(0.0, 1.00, n_quantiles)

    F = np.zeros(Pred.shape[0])
    Indicator_func = np.zeros((Pred.shape[0], perc.shape[0]))

    # range of symmetric p-probability intervals
    plow = (1 - perc) / 2
    pupp = (1 + perc) / 2

    for i in range(len(Pred)):
        F[i] = stats.norm.cdf(Truth,
                          loc=Pred[i],
                          scale=Sigma)
        for proba_low, proba_upp in zip(plow, pupp):
            for k in range(len(plow)):
                if plow[k] < F[i] <= pupp[k]:
                    Indicator_func[i, k] = 1
                else:
                    Indicator_func[i, k] = 0

    avgIndFunc = np.mean(Indicator_func, axis=0)

    return avgIndFunc

def model_eval(y_test,Pred_array,Sigma,Pred):

    n_quantiles=11
    perc = np.linspace(0.0, 1.00, n_quantiles)

    Truth = y_test

    IF_array=np.zeros((Pred_array.shape[0],n_quantiles))

    for i in range(Pred_array.shape[0]):
        IF=APG_calc(Truth[i], Pred_array[:,i], Sigma[i],n_quantiles)
        IF_array[i,:] = IF

    avgIndFunc = np.mean(IF_array, axis=0)

    a = np.zeros(len(avgIndFunc))
    for i in range(len(avgIndFunc)):
        if avgIndFunc[i] > perc[i] or avgIndFunc[i] == perc[i]:
            a[i] = 1
        else:
            a[i] = 0

    Accuracy = integrate.simps(a, perc)

    Prec = a*(avgIndFunc-perc)

    Precision = 1-2*integrate.simps(Prec, perc)

    Sum = (3*a-2)*(avgIndFunc-perc)

    Goodness = 1-integrate.simps(Sum, perc)

    print('Accuracy = {0:1.2f}'.format(Accuracy))
    print('Precision = {0:1.2f}'.format(Precision))
    print('Goodness = {0:1.2f}'.format(Goodness))

    L = 100
    mean = np.empty((L, len(perc)))

    for p_interv in range(len(perc)):
        for l in np.arange(0, L):
            samples = random.choices(IF_array[:, p_interv],
                                     k=IF_array.shape[0])
            mean[l, p_interv] = np.mean(samples)

    mask = np.random.choice([False, True],
                            len(Pred),
                            p=[0.95, 0.05]) # To display randomly less points [Remove , Keep] in fraction

    fig,(ax1,ax2)=plt.subplots(1,2,figsize=(12,4))
    xline = [0,max(Pred.max(),Truth.max())+max(Pred.max(),Truth.max())*0.1]#
    yline = [0,xline[1]]#
    ax1.errorbar(Pred[mask], Truth[mask], xerr=Sigma[mask],
                 fmt='k.',
                 ecolor='k')
    ax1.plot(xline, yline, '-k')
    ax1.set_xlabel('Predicted value, $\hat{y}$')
    ax1.set_ylabel('True value, $y$ ')
    ax2.plot(perc, avgIndFunc,'-ok',markersize=5)
    ax2.plot(perc,np.round(avgIndFunc+np.std(mean, axis=0), 3),'--k')
    ax2.plot(perc,np.round(avgIndFunc-np.std(mean, axis=0), 3),'--k')
    ax2.plot([0, 1],[0, 1],'-k')
    ax2.set_ylabel(r"$\overline{\xi (p)}$")
    ax2.set_xlabel('Probability interval $p$')
    ax2.set_ylim(0,1)
    ax2.set_xlim(0,1)
    ax2.plot(perc, avgIndFunc,'-ok',markersize=5)
    plt.show()

    return Accuracy,Precision,Goodness

def evaluate_cases(model, X_s, batch_size,y_s,n_samples):
    '''
    This function takes the model, the testing data, a batch size and an integer representing
    the number of samples that the model will produce.
    '''
    #Take n_samples to draw a distribution
    mc_predictions = np.zeros((n_samples, y_s.shape[0]))
    for i in range(n_samples):
        y_p = model.predict(X_s, verbose=1, batch_size=batch_size)
        mc_predictions[i] = (y_p[:,0])

    return mc_predictions

```

## Dataset exploration

For this notebook example we have a problem that consists on predicting **Production** from **Porosity**, **Permeability**, **Brittleness** and **TOC**. We have selected this problem because we are primarily interested in capturing the uncertainty related to predictions of Production based on existing data.

This problem can be expanded to any prediction problem.

First lets import the dataset:

```python
df=pd.read_csv('https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/unconv_MV.csv')
```

Some information about our dataset, we have 5 columns including Porosity, Permeability, Acoustic impedance, Density and Facies.

```python
df.describe().transpose()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
      <th>min</th>
      <th>25%</th>
      <th>50%</th>
      <th>75%</th>
      <th>max</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>WellIndex</th>
      <td>1000.0</td>
      <td>500.500000</td>
      <td>288.819436</td>
      <td>1.000000</td>
      <td>250.75000</td>
      <td>500.50000</td>
      <td>750.250000</td>
      <td>1000.00000</td>
    </tr>
    <tr>
      <th>Por</th>
      <td>1000.0</td>
      <td>14.950460</td>
      <td>3.029634</td>
      <td>5.400000</td>
      <td>12.85750</td>
      <td>14.98500</td>
      <td>17.080000</td>
      <td>24.65000</td>
    </tr>
    <tr>
      <th>LogPerm</th>
      <td>1000.0</td>
      <td>1.398880</td>
      <td>0.405966</td>
      <td>0.120000</td>
      <td>1.13000</td>
      <td>1.39000</td>
      <td>1.680000</td>
      <td>2.58000</td>
    </tr>
    <tr>
      <th>AI</th>
      <td>1000.0</td>
      <td>2.982610</td>
      <td>0.577629</td>
      <td>0.960000</td>
      <td>2.57750</td>
      <td>3.01000</td>
      <td>3.360000</td>
      <td>4.70000</td>
    </tr>
    <tr>
      <th>Brittle</th>
      <td>1000.0</td>
      <td>49.719480</td>
      <td>15.077006</td>
      <td>-10.500000</td>
      <td>39.72250</td>
      <td>49.68000</td>
      <td>59.170000</td>
      <td>93.47000</td>
    </tr>
    <tr>
      <th>TOC</th>
      <td>1000.0</td>
      <td>1.003810</td>
      <td>0.504978</td>
      <td>-0.260000</td>
      <td>0.64000</td>
      <td>0.99500</td>
      <td>1.360000</td>
      <td>2.71000</td>
    </tr>
    <tr>
      <th>VR</th>
      <td>1000.0</td>
      <td>1.991170</td>
      <td>0.308194</td>
      <td>0.900000</td>
      <td>1.81000</td>
      <td>2.00000</td>
      <td>2.172500</td>
      <td>2.90000</td>
    </tr>
    <tr>
      <th>Production</th>
      <td>1000.0</td>
      <td>2247.295809</td>
      <td>1464.256312</td>
      <td>2.713535</td>
      <td>1191.36956</td>
      <td>1976.48782</td>
      <td>3023.594214</td>
      <td>12568.64413</td>
    </tr>
  </tbody>
</table>
</div>

Now we separate or data between predictor features and response features.
We use Porosity, Permeability, Brittleness and Total Oil Content values to predict production

```python
y = df['Production'].values
x = df[['Por', 'LogPerm', 'Brittle', 'TOC']].values
```

## Data normalization and split train-test

To speed up things, we can choose to randomly remove points of the dataset using the following code:

```python
mask = np.random.choice([False, True], len(x), p=[0.5, .5])  #Remove, Keep
```

```python
mask = np.random.choice([False, True], len(x), p=[0, 1])  #Remove, Keep
x=x[mask,:]
y=y[mask]
```

Then we perform a min max normalization over the reponse features

```python
ymin=y.min()
ymax=y.max()
y = (y-ymin)/(ymax-ymin)
```

And apply standarization over the predictor features

```python
scaler = StandardScaler()
x = scaler.fit_transform(x)
```

Next, we split our data in training-testing.

```python
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.33,random_state=20)
```

## Neural network model, deterministic

Next we define the neural network model under the class create_model. This is a classic deterministic model. We generate the exact same outcomes under a given set of initial conditions.

```python
class create_model(keras.Model):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.input_layer = keras.layers.Flatten(input_shape=[x.shape[1]])
        self.hidden1 = keras.layers.Dense(24, activation='relu')
        self.hidden2 = keras.layers.Dense(24, activation='relu')
        self.hidden3 = keras.layers.Dense(24, activation='relu')
        self.output_layer = keras.layers.Dense(1)

    def call(self, input, training=True):
        input_layer = self.input_layer(input)
        hidden1 = self.hidden1(input_layer)
        hidden2 = self.hidden2(hidden1)
        output_layer = self.output_layer(hidden2)
        return output_layer
```

The following information is used to save or load the model weights.

```python
path = ""
name = "Production_predict"
weights_path = path + name + '.h5'
```

### Training

Before training we must instantiate the model, lets use Mean Absolute Error as loss function to guide our model for better results.

```python
model = create_model()

opt = keras.optimizers.Adam(learning_rate=0.01)

model.compile(loss='mae',
              optimizer=opt,
              metrics='mse')
```

And fit our model

```python
history=model.fit(
    X_train,
    y_train,
    shuffle=True,
    batch_size=16,
    epochs=300,
    verbose=True,
    validation_data=(X_test,y_test)
)
```

    Epoch 1/300
    42/42 [==============================] - 0s 4ms/step - loss: 0.0814 - mse: 0.0144 - val_loss: 0.0416 - val_mse: 0.0034
    Epoch 2/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0321 - mse: 0.0020 - val_loss: 0.0348 - val_mse: 0.0023
    Epoch 3/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0275 - mse: 0.0015 - val_loss: 0.0276 - val_mse: 0.0017
    Epoch 4/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0192 - mse: 7.2834e-04 - val_loss: 0.0239 - val_mse: 0.0015
    Epoch 5/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0217 - mse: 9.6677e-04 - val_loss: 0.0188 - val_mse: 0.0011
    Epoch 6/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0197 - mse: 8.8205e-04 - val_loss: 0.0237 - val_mse: 0.0013
    Epoch 7/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0216 - mse: 9.7121e-04 - val_loss: 0.0174 - val_mse: 0.0010
    Epoch 8/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0169 - mse: 7.0821e-04 - val_loss: 0.0208 - val_mse: 0.0011
    Epoch 9/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0136 - mse: 4.2520e-04 - val_loss: 0.0151 - val_mse: 7.7642e-04
    Epoch 10/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0130 - mse: 3.7688e-04 - val_loss: 0.0139 - val_mse: 7.4347e-04
    Epoch 11/300
    42/42 [==============================] - 0s 2ms/step - loss: 0.0154 - mse: 5.1280e-04 - val_loss: 0.0196 - val_mse: 0.0010
    Epoch 12/300
    42/42 [==============================] - 0s 2ms/step - loss: 0.0147 - mse: 4.3246e-04 - val_loss: 0.0134 - val_mse: 6.1520e-04
    Epoch 13/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0146 - mse: 5.4891e-04 - val_loss: 0.0167 - val_mse: 9.1230e-04
    Epoch 14/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0148 - mse: 5.1257e-04 - val_loss: 0.0157 - val_mse: 8.3598e-04
    Epoch 15/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0109 - mse: 2.9622e-04 - val_loss: 0.0126 - val_mse: 6.3203e-04
    Epoch 16/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0154 - mse: 5.1265e-04 - val_loss: 0.0114 - val_mse: 5.4699e-04
    Epoch 17/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0100 - mse: 2.4569e-04 - val_loss: 0.0106 - val_mse: 5.0593e-04
    Epoch 18/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0113 - mse: 2.6259e-04 - val_loss: 0.0188 - val_mse: 6.8315e-04
    Epoch 19/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0105 - mse: 2.4215e-04 - val_loss: 0.0111 - val_mse: 4.5334e-04
    Epoch 20/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0100 - mse: 2.1796e-04 - val_loss: 0.0102 - val_mse: 4.5106e-04
    Epoch 21/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0100 - mse: 2.3250e-04 - val_loss: 0.0143 - val_mse: 6.4058e-04
    Epoch 22/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0111 - mse: 2.4717e-04 - val_loss: 0.0105 - val_mse: 4.8525e-04
    Epoch 23/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0116 - mse: 2.7866e-04 - val_loss: 0.0140 - val_mse: 6.2280e-04
    Epoch 24/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0104 - mse: 2.2747e-04 - val_loss: 0.0106 - val_mse: 4.1180e-04
    Epoch 25/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.9595e-04 - val_loss: 0.0081 - val_mse: 3.2163e-04
    Epoch 26/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0111 - mse: 2.3746e-04 - val_loss: 0.0088 - val_mse: 3.7616e-04
    Epoch 27/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0077 - mse: 1.2487e-04 - val_loss: 0.0099 - val_mse: 4.1804e-04
    Epoch 28/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0110 - mse: 2.7573e-04 - val_loss: 0.0101 - val_mse: 3.9446e-04
    Epoch 29/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.3862e-04 - val_loss: 0.0106 - val_mse: 3.4492e-04
    Epoch 30/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.2542e-04 - val_loss: 0.0089 - val_mse: 4.0501e-04
    Epoch 31/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.9394e-04 - val_loss: 0.0118 - val_mse: 3.9765e-04
    Epoch 32/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0096 - mse: 1.8953e-04 - val_loss: 0.0115 - val_mse: 4.5696e-04
    Epoch 33/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0107 - mse: 2.4836e-04 - val_loss: 0.0122 - val_mse: 4.6006e-04
    Epoch 34/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.6931e-04 - val_loss: 0.0073 - val_mse: 2.2432e-04
    Epoch 35/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.1949e-04 - val_loss: 0.0079 - val_mse: 2.1966e-04
    Epoch 36/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0074 - mse: 1.4117e-04 - val_loss: 0.0142 - val_mse: 5.1516e-04
    Epoch 37/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0087 - mse: 1.7177e-04 - val_loss: 0.0085 - val_mse: 2.3251e-04
    Epoch 38/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0090 - mse: 1.6235e-04 - val_loss: 0.0110 - val_mse: 3.9395e-04
    Epoch 39/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0110 - mse: 2.2397e-04 - val_loss: 0.0186 - val_mse: 6.4351e-04
    Epoch 40/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0110 - mse: 2.3720e-04 - val_loss: 0.0180 - val_mse: 6.3284e-04
    Epoch 41/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0107 - mse: 2.6746e-04 - val_loss: 0.0099 - val_mse: 2.7985e-04
    Epoch 42/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.6761e-04 - val_loss: 0.0085 - val_mse: 2.5701e-04
    Epoch 43/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0080 - mse: 1.3475e-04 - val_loss: 0.0072 - val_mse: 1.3918e-04
    Epoch 44/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.3504e-04 - val_loss: 0.0078 - val_mse: 1.8957e-04
    Epoch 45/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0074 - mse: 1.2305e-04 - val_loss: 0.0089 - val_mse: 2.9211e-04
    Epoch 46/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.5870e-04 - val_loss: 0.0083 - val_mse: 1.7516e-04
    Epoch 47/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0072 - mse: 1.2776e-04 - val_loss: 0.0069 - val_mse: 3.1085e-04
    Epoch 48/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0070 - mse: 1.1037e-04 - val_loss: 0.0090 - val_mse: 1.9641e-04
    Epoch 49/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.2342e-04 - val_loss: 0.0087 - val_mse: 2.9335e-04
    Epoch 50/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.2199e-04 - val_loss: 0.0071 - val_mse: 1.6971e-04
    Epoch 51/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0083 - mse: 1.4028e-04 - val_loss: 0.0111 - val_mse: 3.7563e-04
    Epoch 52/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.7357e-04 - val_loss: 0.0091 - val_mse: 2.7233e-04
    Epoch 53/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.2993e-04 - val_loss: 0.0093 - val_mse: 2.9423e-04
    Epoch 54/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0078 - mse: 1.3151e-04 - val_loss: 0.0072 - val_mse: 2.2456e-04
    Epoch 55/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0090 - mse: 1.6556e-04 - val_loss: 0.0071 - val_mse: 1.9486e-04
    Epoch 56/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0072 - mse: 1.2739e-04 - val_loss: 0.0113 - val_mse: 3.4445e-04
    Epoch 57/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0070 - mse: 1.0624e-04 - val_loss: 0.0068 - val_mse: 1.9313e-04
    Epoch 58/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0075 - mse: 1.1394e-04 - val_loss: 0.0073 - val_mse: 1.7004e-04
    Epoch 59/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0095 - mse: 1.6963e-04 - val_loss: 0.0075 - val_mse: 2.1849e-04
    Epoch 60/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0086 - mse: 1.5963e-04 - val_loss: 0.0094 - val_mse: 2.2123e-04
    Epoch 61/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 8.4366e-05 - val_loss: 0.0077 - val_mse: 2.2877e-04
    Epoch 62/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0077 - mse: 1.3357e-04 - val_loss: 0.0066 - val_mse: 1.8032e-04
    Epoch 63/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0066 - mse: 9.2224e-05 - val_loss: 0.0073 - val_mse: 2.0599e-04
    Epoch 64/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 8.2988e-05 - val_loss: 0.0082 - val_mse: 1.9170e-04
    Epoch 65/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0077 - mse: 1.2470e-04 - val_loss: 0.0117 - val_mse: 3.3148e-04
    Epoch 66/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0077 - mse: 1.2306e-04 - val_loss: 0.0099 - val_mse: 2.6347e-04
    Epoch 67/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0074 - mse: 1.2708e-04 - val_loss: 0.0066 - val_mse: 1.6140e-04
    Epoch 68/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 7.8551e-05 - val_loss: 0.0069 - val_mse: 1.6546e-04
    Epoch 69/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 9.2213e-05 - val_loss: 0.0056 - val_mse: 1.2738e-04
    Epoch 70/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.9408e-05 - val_loss: 0.0066 - val_mse: 1.7455e-04
    Epoch 71/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.1460e-04 - val_loss: 0.0080 - val_mse: 2.1001e-04
    Epoch 72/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0077 - mse: 1.3617e-04 - val_loss: 0.0071 - val_mse: 1.7298e-04
    Epoch 73/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0075 - mse: 1.1852e-04 - val_loss: 0.0095 - val_mse: 2.2529e-04
    Epoch 74/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0079 - mse: 1.1709e-04 - val_loss: 0.0068 - val_mse: 2.0934e-04
    Epoch 75/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0075 - mse: 1.2801e-04 - val_loss: 0.0067 - val_mse: 1.9504e-04
    Epoch 76/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0089 - mse: 1.5742e-04 - val_loss: 0.0080 - val_mse: 2.4277e-04
    Epoch 77/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0081 - mse: 1.3807e-04 - val_loss: 0.0064 - val_mse: 2.4792e-04
    Epoch 78/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 8.3497e-05 - val_loss: 0.0080 - val_mse: 1.6929e-04
    Epoch 79/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.7929e-05 - val_loss: 0.0094 - val_mse: 3.2079e-04
    Epoch 80/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 1.1046e-04 - val_loss: 0.0084 - val_mse: 2.7403e-04
    Epoch 81/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 8.1145e-05 - val_loss: 0.0073 - val_mse: 1.7889e-04
    Epoch 82/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 7.3867e-05 - val_loss: 0.0110 - val_mse: 2.8726e-04
    Epoch 83/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 1.0777e-04 - val_loss: 0.0079 - val_mse: 1.2290e-04
    Epoch 84/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 9.9660e-05 - val_loss: 0.0080 - val_mse: 1.7097e-04
    Epoch 85/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0078 - mse: 1.2465e-04 - val_loss: 0.0123 - val_mse: 2.4871e-04
    Epoch 86/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0078 - mse: 1.3280e-04 - val_loss: 0.0110 - val_mse: 4.5761e-04
    Epoch 87/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.2276e-04 - val_loss: 0.0070 - val_mse: 1.8337e-04
    Epoch 88/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 5.5426e-05 - val_loss: 0.0097 - val_mse: 2.6099e-04
    Epoch 89/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0078 - mse: 1.2732e-04 - val_loss: 0.0054 - val_mse: 1.0984e-04
    Epoch 90/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.7590e-05 - val_loss: 0.0053 - val_mse: 8.7908e-05
    Epoch 91/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 7.9880e-05 - val_loss: 0.0066 - val_mse: 1.5811e-04
    Epoch 92/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 8.1073e-05 - val_loss: 0.0062 - val_mse: 1.0965e-04
    Epoch 93/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 7.7131e-05 - val_loss: 0.0082 - val_mse: 2.6179e-04
    Epoch 94/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 6.9024e-05 - val_loss: 0.0071 - val_mse: 1.4102e-04
    Epoch 95/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0069 - mse: 1.0865e-04 - val_loss: 0.0062 - val_mse: 1.8163e-04
    Epoch 96/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0069 - mse: 9.1299e-05 - val_loss: 0.0061 - val_mse: 1.3345e-04
    Epoch 97/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 7.9084e-05 - val_loss: 0.0062 - val_mse: 1.7969e-04
    Epoch 98/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0072 - mse: 1.1458e-04 - val_loss: 0.0082 - val_mse: 1.8245e-04
    Epoch 99/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0068 - mse: 8.9391e-05 - val_loss: 0.0073 - val_mse: 1.4253e-04
    Epoch 100/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 7.8243e-05 - val_loss: 0.0066 - val_mse: 1.3368e-04
    Epoch 101/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 1.0142e-04 - val_loss: 0.0081 - val_mse: 1.7462e-04
    Epoch 102/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 7.1083e-05 - val_loss: 0.0090 - val_mse: 1.9999e-04
    Epoch 103/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 7.7310e-05 - val_loss: 0.0055 - val_mse: 1.2330e-04
    Epoch 104/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 6.9998e-05 - val_loss: 0.0061 - val_mse: 1.5232e-04
    Epoch 105/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0077 - mse: 1.3061e-04 - val_loss: 0.0112 - val_mse: 2.4474e-04
    Epoch 106/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 6.8639e-05 - val_loss: 0.0061 - val_mse: 1.4186e-04
    Epoch 107/300
    42/42 [==============================] - 0s 2ms/step - loss: 0.0064 - mse: 9.0115e-05 - val_loss: 0.0079 - val_mse: 1.3575e-04
    Epoch 108/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0074 - mse: 1.1147e-04 - val_loss: 0.0067 - val_mse: 1.2212e-04
    Epoch 109/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 7.5483e-05 - val_loss: 0.0065 - val_mse: 1.4294e-04
    Epoch 110/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 7.6710e-05 - val_loss: 0.0047 - val_mse: 9.3679e-05
    Epoch 111/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0069 - mse: 9.7101e-05 - val_loss: 0.0090 - val_mse: 2.9455e-04
    Epoch 112/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.4727e-05 - val_loss: 0.0058 - val_mse: 1.4088e-04
    Epoch 113/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 9.0757e-05 - val_loss: 0.0081 - val_mse: 2.5385e-04
    Epoch 114/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.0066e-04 - val_loss: 0.0098 - val_mse: 3.0367e-04
    Epoch 115/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 9.9026e-05 - val_loss: 0.0067 - val_mse: 1.8737e-04
    Epoch 116/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 7.5559e-05 - val_loss: 0.0088 - val_mse: 1.4902e-04
    Epoch 117/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0083 - mse: 1.1594e-04 - val_loss: 0.0080 - val_mse: 1.5817e-04
    Epoch 118/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 9.1852e-05 - val_loss: 0.0080 - val_mse: 2.3247e-04
    Epoch 119/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.2927e-04 - val_loss: 0.0061 - val_mse: 1.2387e-04
    Epoch 120/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 6.8821e-05 - val_loss: 0.0080 - val_mse: 1.8310e-04
    Epoch 121/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 9.2615e-05 - val_loss: 0.0068 - val_mse: 1.6435e-04
    Epoch 122/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.2821e-05 - val_loss: 0.0070 - val_mse: 1.1260e-04
    Epoch 123/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0070 - mse: 9.1530e-05 - val_loss: 0.0072 - val_mse: 2.3776e-04
    Epoch 124/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 8.6660e-05 - val_loss: 0.0061 - val_mse: 1.7995e-04
    Epoch 125/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 6.0655e-05 - val_loss: 0.0081 - val_mse: 3.0223e-04
    Epoch 126/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 6.6649e-05 - val_loss: 0.0067 - val_mse: 1.8752e-04
    Epoch 127/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 6.8041e-05 - val_loss: 0.0067 - val_mse: 1.9133e-04
    Epoch 128/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0068 - mse: 9.9772e-05 - val_loss: 0.0087 - val_mse: 1.8831e-04
    Epoch 129/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0070 - mse: 9.8302e-05 - val_loss: 0.0069 - val_mse: 1.7968e-04
    Epoch 130/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.9842e-05 - val_loss: 0.0071 - val_mse: 1.0258e-04
    Epoch 131/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 7.1312e-05 - val_loss: 0.0075 - val_mse: 1.4695e-04
    Epoch 132/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 8.5862e-05 - val_loss: 0.0100 - val_mse: 2.6444e-04
    Epoch 133/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0068 - mse: 9.0690e-05 - val_loss: 0.0118 - val_mse: 4.1715e-04
    Epoch 134/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 1.0103e-04 - val_loss: 0.0056 - val_mse: 1.4445e-04
    Epoch 135/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 4.8866e-05 - val_loss: 0.0052 - val_mse: 1.4140e-04
    Epoch 136/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 6.8648e-05 - val_loss: 0.0065 - val_mse: 2.3243e-04
    Epoch 137/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.2946e-04 - val_loss: 0.0070 - val_mse: 1.5256e-04
    Epoch 138/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 7.3030e-05 - val_loss: 0.0061 - val_mse: 1.5503e-04
    Epoch 139/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.2716e-05 - val_loss: 0.0081 - val_mse: 1.8602e-04
    Epoch 140/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 8.4998e-05 - val_loss: 0.0082 - val_mse: 1.5034e-04
    Epoch 141/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 6.2398e-05 - val_loss: 0.0057 - val_mse: 1.1957e-04
    Epoch 142/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 6.6173e-05 - val_loss: 0.0056 - val_mse: 1.7083e-04
    Epoch 143/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 6.0733e-05 - val_loss: 0.0057 - val_mse: 1.0661e-04
    Epoch 144/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 8.0522e-05 - val_loss: 0.0064 - val_mse: 1.1473e-04
    Epoch 145/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.6711e-05 - val_loss: 0.0079 - val_mse: 1.1944e-04
    Epoch 146/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 7.4152e-05 - val_loss: 0.0098 - val_mse: 2.1069e-04
    Epoch 147/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.6855e-05 - val_loss: 0.0060 - val_mse: 1.3591e-04
    Epoch 148/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 6.3899e-05 - val_loss: 0.0070 - val_mse: 1.6339e-04
    Epoch 149/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 7.5358e-05 - val_loss: 0.0052 - val_mse: 1.1788e-04
    Epoch 150/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0044 - mse: 4.2842e-05 - val_loss: 0.0048 - val_mse: 1.0681e-04
    Epoch 151/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0072 - mse: 1.2054e-04 - val_loss: 0.0095 - val_mse: 3.0137e-04
    Epoch 152/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0066 - mse: 8.1178e-05 - val_loss: 0.0073 - val_mse: 2.0710e-04
    Epoch 153/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.5720e-05 - val_loss: 0.0048 - val_mse: 1.0204e-04
    Epoch 154/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0069 - mse: 9.0737e-05 - val_loss: 0.0091 - val_mse: 2.3716e-04
    Epoch 155/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.4583e-05 - val_loss: 0.0068 - val_mse: 1.6047e-04
    Epoch 156/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 6.0576e-05 - val_loss: 0.0059 - val_mse: 1.2939e-04
    Epoch 157/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 6.3907e-05 - val_loss: 0.0081 - val_mse: 1.7977e-04
    Epoch 158/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 7.3962e-05 - val_loss: 0.0106 - val_mse: 3.1322e-04
    Epoch 159/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0082 - mse: 1.5051e-04 - val_loss: 0.0045 - val_mse: 5.7628e-05
    Epoch 160/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 6.6603e-05 - val_loss: 0.0060 - val_mse: 9.9346e-05
    Epoch 161/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 5.5048e-05 - val_loss: 0.0064 - val_mse: 1.1103e-04
    Epoch 162/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.4024e-05 - val_loss: 0.0074 - val_mse: 1.3934e-04
    Epoch 163/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 5.5069e-05 - val_loss: 0.0053 - val_mse: 1.3712e-04
    Epoch 164/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0047 - mse: 4.6915e-05 - val_loss: 0.0079 - val_mse: 1.4469e-04
    Epoch 165/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0054 - mse: 6.3470e-05 - val_loss: 0.0082 - val_mse: 2.7247e-04
    Epoch 166/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.0336e-04 - val_loss: 0.0094 - val_mse: 2.3560e-04
    Epoch 167/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 6.8783e-05 - val_loss: 0.0053 - val_mse: 8.4373e-05
    Epoch 168/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 5.9468e-05 - val_loss: 0.0068 - val_mse: 1.5222e-04
    Epoch 169/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0073 - mse: 1.1458e-04 - val_loss: 0.0053 - val_mse: 1.1981e-04
    Epoch 170/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.7259e-05 - val_loss: 0.0049 - val_mse: 1.0878e-04
    Epoch 171/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 8.1650e-05 - val_loss: 0.0064 - val_mse: 1.3175e-04
    Epoch 172/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 6.3932e-05 - val_loss: 0.0047 - val_mse: 9.2474e-05
    Epoch 173/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 9.6580e-05 - val_loss: 0.0057 - val_mse: 1.4144e-04
    Epoch 174/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 4.9481e-05 - val_loss: 0.0043 - val_mse: 8.3490e-05
    Epoch 175/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0054 - mse: 5.8062e-05 - val_loss: 0.0057 - val_mse: 1.3072e-04
    Epoch 176/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 6.8270e-05 - val_loss: 0.0063 - val_mse: 1.4447e-04
    Epoch 177/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 8.5488e-05 - val_loss: 0.0068 - val_mse: 1.1309e-04
    Epoch 178/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 1.0531e-04 - val_loss: 0.0084 - val_mse: 1.8008e-04
    Epoch 179/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 6.2985e-05 - val_loss: 0.0045 - val_mse: 5.8746e-05
    Epoch 180/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0054 - mse: 5.6227e-05 - val_loss: 0.0061 - val_mse: 1.0869e-04
    Epoch 181/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 7.0476e-05 - val_loss: 0.0086 - val_mse: 1.6577e-04
    Epoch 182/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 8.1801e-05 - val_loss: 0.0048 - val_mse: 6.9696e-05
    Epoch 183/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.8887e-05 - val_loss: 0.0063 - val_mse: 7.5618e-05
    Epoch 184/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 7.3379e-05 - val_loss: 0.0058 - val_mse: 1.3934e-04
    Epoch 185/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0068 - mse: 9.1260e-05 - val_loss: 0.0075 - val_mse: 2.0659e-04
    Epoch 186/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 9.0080e-05 - val_loss: 0.0047 - val_mse: 6.5638e-05
    Epoch 187/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 6.1882e-05 - val_loss: 0.0062 - val_mse: 7.2243e-05
    Epoch 188/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 4.7554e-05 - val_loss: 0.0059 - val_mse: 1.1862e-04
    Epoch 189/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 8.3313e-05 - val_loss: 0.0068 - val_mse: 1.3685e-04
    Epoch 190/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0075 - mse: 1.0537e-04 - val_loss: 0.0051 - val_mse: 7.0838e-05
    Epoch 191/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.7702e-05 - val_loss: 0.0078 - val_mse: 1.7585e-04
    Epoch 192/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 6.4796e-05 - val_loss: 0.0060 - val_mse: 8.8326e-05
    Epoch 193/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 4.7219e-05 - val_loss: 0.0062 - val_mse: 1.5796e-04
    Epoch 194/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 5.1677e-05 - val_loss: 0.0067 - val_mse: 1.3277e-04
    Epoch 195/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 6.9632e-05 - val_loss: 0.0079 - val_mse: 1.5329e-04
    Epoch 196/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 9.0809e-05 - val_loss: 0.0057 - val_mse: 7.1136e-05
    Epoch 197/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 5.8524e-05 - val_loss: 0.0059 - val_mse: 1.4199e-04
    Epoch 198/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 6.9521e-05 - val_loss: 0.0061 - val_mse: 7.4527e-05
    Epoch 199/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 7.2456e-05 - val_loss: 0.0052 - val_mse: 1.5655e-04
    Epoch 200/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0047 - mse: 4.3549e-05 - val_loss: 0.0047 - val_mse: 5.5717e-05
    Epoch 201/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0047 - mse: 4.4822e-05 - val_loss: 0.0042 - val_mse: 7.8345e-05
    Epoch 202/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0048 - mse: 4.8816e-05 - val_loss: 0.0046 - val_mse: 9.4846e-05
    Epoch 203/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0070 - mse: 1.0794e-04 - val_loss: 0.0058 - val_mse: 1.6083e-04
    Epoch 204/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.6473e-05 - val_loss: 0.0052 - val_mse: 9.5970e-05
    Epoch 205/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0054 - mse: 6.1393e-05 - val_loss: 0.0037 - val_mse: 5.3126e-05
    Epoch 206/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 5.2160e-05 - val_loss: 0.0075 - val_mse: 1.1487e-04
    Epoch 207/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 6.2329e-05 - val_loss: 0.0045 - val_mse: 8.6454e-05
    Epoch 208/300
    42/42 [==============================] - 0s 2ms/step - loss: 0.0056 - mse: 7.1358e-05 - val_loss: 0.0079 - val_mse: 2.2958e-04
    Epoch 209/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 7.1106e-05 - val_loss: 0.0057 - val_mse: 9.5187e-05
    Epoch 210/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0046 - mse: 3.5775e-05 - val_loss: 0.0039 - val_mse: 5.2507e-05
    Epoch 211/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 5.4912e-05 - val_loss: 0.0052 - val_mse: 8.2427e-05
    Epoch 212/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.3008e-05 - val_loss: 0.0069 - val_mse: 1.4625e-04
    Epoch 213/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 7.0521e-05 - val_loss: 0.0060 - val_mse: 1.0048e-04
    Epoch 214/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0047 - mse: 4.2954e-05 - val_loss: 0.0073 - val_mse: 1.1561e-04
    Epoch 215/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 6.2314e-05 - val_loss: 0.0058 - val_mse: 1.0344e-04
    Epoch 216/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 4.5261e-05 - val_loss: 0.0069 - val_mse: 1.4167e-04
    Epoch 217/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 4.4381e-05 - val_loss: 0.0078 - val_mse: 1.3661e-04
    Epoch 218/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0069 - mse: 9.3258e-05 - val_loss: 0.0069 - val_mse: 1.1803e-04
    Epoch 219/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 6.7221e-05 - val_loss: 0.0051 - val_mse: 9.8572e-05
    Epoch 220/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0048 - mse: 4.8778e-05 - val_loss: 0.0054 - val_mse: 1.0263e-04
    Epoch 221/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 5.3534e-05 - val_loss: 0.0048 - val_mse: 9.5925e-05
    Epoch 222/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 6.7529e-05 - val_loss: 0.0065 - val_mse: 1.1518e-04
    Epoch 223/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0076 - mse: 1.1581e-04 - val_loss: 0.0062 - val_mse: 9.1742e-05
    Epoch 224/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 6.0290e-05 - val_loss: 0.0052 - val_mse: 8.1851e-05
    Epoch 225/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 5.2247e-05 - val_loss: 0.0045 - val_mse: 8.3312e-05
    Epoch 226/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0056 - mse: 6.6073e-05 - val_loss: 0.0062 - val_mse: 1.0039e-04
    Epoch 227/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.6409e-05 - val_loss: 0.0067 - val_mse: 1.5190e-04
    Epoch 228/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 7.0581e-05 - val_loss: 0.0090 - val_mse: 2.0530e-04
    Epoch 229/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0070 - mse: 9.1087e-05 - val_loss: 0.0052 - val_mse: 1.2105e-04
    Epoch 230/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.5316e-05 - val_loss: 0.0057 - val_mse: 7.6297e-05
    Epoch 231/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 5.1771e-05 - val_loss: 0.0071 - val_mse: 1.3378e-04
    Epoch 232/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 7.0156e-05 - val_loss: 0.0047 - val_mse: 1.2579e-04
    Epoch 233/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0046 - mse: 4.3290e-05 - val_loss: 0.0070 - val_mse: 1.1702e-04
    Epoch 234/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 7.3415e-05 - val_loss: 0.0057 - val_mse: 1.2236e-04
    Epoch 235/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0066 - mse: 8.6901e-05 - val_loss: 0.0046 - val_mse: 1.0557e-04
    Epoch 236/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0047 - mse: 4.1018e-05 - val_loss: 0.0042 - val_mse: 7.6049e-05
    Epoch 237/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 4.9177e-05 - val_loss: 0.0064 - val_mse: 1.4833e-04
    Epoch 238/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.7200e-05 - val_loss: 0.0067 - val_mse: 2.1210e-04
    Epoch 239/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 8.0688e-05 - val_loss: 0.0067 - val_mse: 1.3086e-04
    Epoch 240/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 6.1370e-05 - val_loss: 0.0051 - val_mse: 9.5057e-05
    Epoch 241/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0068 - mse: 8.2176e-05 - val_loss: 0.0077 - val_mse: 1.6205e-04
    Epoch 242/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 5.2551e-05 - val_loss: 0.0051 - val_mse: 6.8989e-05
    Epoch 243/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0049 - mse: 4.6727e-05 - val_loss: 0.0075 - val_mse: 1.2030e-04
    Epoch 244/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 8.1997e-05 - val_loss: 0.0085 - val_mse: 1.3121e-04
    Epoch 245/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 6.4805e-05 - val_loss: 0.0092 - val_mse: 1.8028e-04
    Epoch 246/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.2186e-05 - val_loss: 0.0064 - val_mse: 1.7425e-04
    Epoch 247/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0048 - mse: 4.3518e-05 - val_loss: 0.0054 - val_mse: 1.2773e-04
    Epoch 248/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 6.6771e-05 - val_loss: 0.0054 - val_mse: 1.5097e-04
    Epoch 249/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0066 - mse: 9.5822e-05 - val_loss: 0.0046 - val_mse: 7.3126e-05
    Epoch 250/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0044 - mse: 4.1224e-05 - val_loss: 0.0062 - val_mse: 1.3743e-04
    Epoch 251/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0048 - mse: 4.6620e-05 - val_loss: 0.0044 - val_mse: 1.0271e-04
    Epoch 252/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 7.5948e-05 - val_loss: 0.0068 - val_mse: 8.7230e-05
    Epoch 253/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0054 - mse: 5.4778e-05 - val_loss: 0.0044 - val_mse: 6.5804e-05
    Epoch 254/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0055 - mse: 6.3418e-05 - val_loss: 0.0055 - val_mse: 1.0392e-04
    Epoch 255/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.6216e-05 - val_loss: 0.0054 - val_mse: 6.0154e-05
    Epoch 256/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.4180e-05 - val_loss: 0.0043 - val_mse: 5.8443e-05
    Epoch 257/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 7.1766e-05 - val_loss: 0.0078 - val_mse: 1.2650e-04
    Epoch 258/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 7.9813e-05 - val_loss: 0.0049 - val_mse: 1.2098e-04
    Epoch 259/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 6.5999e-05 - val_loss: 0.0048 - val_mse: 1.1527e-04
    Epoch 260/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0074 - mse: 8.7428e-05 - val_loss: 0.0072 - val_mse: 1.8533e-04
    Epoch 261/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0079 - mse: 1.2692e-04 - val_loss: 0.0076 - val_mse: 1.0674e-04
    Epoch 262/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.8040e-05 - val_loss: 0.0052 - val_mse: 1.2378e-04
    Epoch 263/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.0862e-05 - val_loss: 0.0069 - val_mse: 1.3954e-04
    Epoch 264/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.8548e-05 - val_loss: 0.0063 - val_mse: 1.0337e-04
    Epoch 265/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 7.5242e-05 - val_loss: 0.0057 - val_mse: 1.8311e-04
    Epoch 266/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0053 - mse: 5.4029e-05 - val_loss: 0.0049 - val_mse: 5.9990e-05
    Epoch 267/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0060 - mse: 7.1162e-05 - val_loss: 0.0046 - val_mse: 9.5780e-05
    Epoch 268/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0069 - mse: 7.8567e-05 - val_loss: 0.0075 - val_mse: 2.4731e-04
    Epoch 269/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 8.2296e-05 - val_loss: 0.0087 - val_mse: 1.5353e-04
    Epoch 270/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0068 - mse: 1.0643e-04 - val_loss: 0.0069 - val_mse: 1.2236e-04
    Epoch 271/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.7793e-05 - val_loss: 0.0048 - val_mse: 1.2510e-04
    Epoch 272/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0047 - mse: 4.1083e-05 - val_loss: 0.0063 - val_mse: 1.4366e-04
    Epoch 273/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.6742e-05 - val_loss: 0.0049 - val_mse: 1.6983e-04
    Epoch 274/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0058 - mse: 6.0277e-05 - val_loss: 0.0073 - val_mse: 1.5486e-04
    Epoch 275/300
    42/42 [==============================] - 0s 2ms/step - loss: 0.0056 - mse: 6.4204e-05 - val_loss: 0.0080 - val_mse: 3.2585e-04
    Epoch 276/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 1.0421e-04 - val_loss: 0.0054 - val_mse: 1.3319e-04
    Epoch 277/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0046 - mse: 4.1245e-05 - val_loss: 0.0057 - val_mse: 9.8379e-05
    Epoch 278/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0054 - mse: 5.6719e-05 - val_loss: 0.0064 - val_mse: 1.4399e-04
    Epoch 279/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 8.8348e-05 - val_loss: 0.0067 - val_mse: 1.2456e-04
    Epoch 280/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 8.4842e-05 - val_loss: 0.0060 - val_mse: 1.5448e-04
    Epoch 281/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 4.8014e-05 - val_loss: 0.0047 - val_mse: 9.8158e-05
    Epoch 282/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.0804e-05 - val_loss: 0.0064 - val_mse: 1.3491e-04
    Epoch 283/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0063 - mse: 7.7454e-05 - val_loss: 0.0051 - val_mse: 5.3983e-05
    Epoch 284/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 4.9816e-05 - val_loss: 0.0096 - val_mse: 1.6865e-04
    Epoch 285/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0066 - mse: 8.5170e-05 - val_loss: 0.0060 - val_mse: 8.4507e-05
    Epoch 286/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0059 - mse: 6.3074e-05 - val_loss: 0.0066 - val_mse: 1.1274e-04
    Epoch 287/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0064 - mse: 7.5881e-05 - val_loss: 0.0056 - val_mse: 1.1819e-04
    Epoch 288/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0051 - mse: 4.8760e-05 - val_loss: 0.0050 - val_mse: 8.6427e-05
    Epoch 289/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0071 - mse: 8.5625e-05 - val_loss: 0.0071 - val_mse: 1.2134e-04
    Epoch 290/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0067 - mse: 8.8004e-05 - val_loss: 0.0077 - val_mse: 1.1360e-04
    Epoch 291/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0061 - mse: 7.4987e-05 - val_loss: 0.0064 - val_mse: 2.5261e-04
    Epoch 292/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0057 - mse: 6.6741e-05 - val_loss: 0.0113 - val_mse: 2.2855e-04
    Epoch 293/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0093 - mse: 1.6509e-04 - val_loss: 0.0056 - val_mse: 1.0089e-04
    Epoch 294/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0052 - mse: 4.9412e-05 - val_loss: 0.0054 - val_mse: 6.5134e-05
    Epoch 295/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 5.0847e-05 - val_loss: 0.0082 - val_mse: 1.7439e-04
    Epoch 296/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.6717e-05 - val_loss: 0.0052 - val_mse: 8.8916e-05
    Epoch 297/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0050 - mse: 4.7274e-05 - val_loss: 0.0060 - val_mse: 1.0869e-04
    Epoch 298/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0065 - mse: 8.1333e-05 - val_loss: 0.0073 - val_mse: 1.6940e-04
    Epoch 299/300
    42/42 [==============================] - 0s 1ms/step - loss: 0.0062 - mse: 7.4596e-05 - val_loss: 0.0044 - val_mse: 6.9456e-05
    Epoch 300/300
    42/42 [==============================] - 0s 2ms/step - loss: 0.0054 - mse: 5.4123e-05 - val_loss: 0.0044 - val_mse: 4.4640e-05

We plot the loss function to diagnostic the model and the dataset, from the plot we can identify

- A separation between the test and train loss function might indicate overfitting
- A large gap between the curves might indicate an unrepresentative train dataset

```python
plot_loss(history)
```

Next we predict and compare our predictions with the observed occurrences. Now this model is very accurate and for every combination of predictor features we will have the exact same outcome.

```python
y_test_pred = model.predict(X_test)
plt.figure(figsize=(12,6))
plt.plot(X_test[:,0],y_test,'o',label='Test')
plt.plot(X_test[:,0],y_test_pred,'o',label='Predicted test')
plt.xlabel('Porosity')
plt.ylabel('Production')
plt.legend()
plt.grid(True)
```

The next thing we do is we test the goodness of the model. Lets evaluate the model 10 times to estimate the Goodness, Precision and Acurracy of the model.

We could only do this once, because of the deterministic nature of the model, all 10 evaluations we will give the same result

```python
mc_predictions = evaluate_cases(model, X_test, 8, y_test,10)
df = get_dataframe(mc_predictions, y_test, X_test)
```

    42/42 [==============================] - 0s 559us/step
    42/42 [==============================] - 0s 512us/step
    42/42 [==============================] - 0s 487us/step
    42/42 [==============================] - 0s 534us/step
    42/42 [==============================] - 0s 511us/step
    42/42 [==============================] - 0s 511us/step
    42/42 [==============================] - 0s 535us/step
    42/42 [==============================] - 0s 536us/step
    42/42 [==============================] - 0s 632us/step
    42/42 [==============================] - 0s 585us/step

How does the cross plot for predicted versus true value look like?

What about the Pearson's product moment correlation coefficient?

```python
r= stats.pearsonr(df.Pred,df.Truth)
error_line(df.Pred,df.Truth,df.Sigma,r[0])
```

What is the Goodness, Precision and Accuracy of the model?

```python
Sigma = np.sqrt(np.var(mc_predictions.T, axis=1))

score = scorer(mc_predictions.T, y_test, Sigma)

IF_array = score.IndicatorFunction()
n_quantiles = 11
perc = np.linspace(0.0, 1.00, n_quantiles)

```

    C:\Users\em42363\Anaconda3\envs\new\lib\site-packages\scipy\stats\_distn_infrastructure.py:1987: RuntimeWarning: divide by zero encountered in double_scalars
      x = np.asarray((x - loc)/scale, dtype=dtyp)

Very precise! But inaccurate

```python
print('Accuracy = {0:2.2f}'.format(score.Accuracy()))
print('Precision = {0:2.2f}'.format(score.Precision()))
print('Goodness = {0:2.2f}'.format(score.Goodness()))
```

    Accuracy = 0.03
    Precision = 1.00
    Goodness = -0.02

The precision is 1 and the goodness is 0!! The model is inaccurate and precise

How does the histogram of predictions for a random case look like? All 10 occurrences have the same result

```python
histogram(mc_predictions)
```

We can use the accuracy plot to diagnose our model. You can find more details on the use of the accuracy plots in the provided article.

How does the accuracy plot look like for a deterministic model?

```python
error_accuracy_plot(perc, IF_array, mc_predictions.T, y_test, Sigma)
```

Our model is **inaccurate and imprecise.**

## Neural network model, probabilistic

For this second model we use **dropout** to introduce randomness into our model

### Neural network model

Similarly as before, we define the model and include dropout layers in our deep learning model.

```python
def PermaDropout(rate):
    return Lambda(lambda x: K.dropout(x, level=rate)) # Fraction of the input units to drop

class create_model(keras.Model):
    def __init__(self,dropout, **kwargs):
        super().__init__(**kwargs)
        self.input_layer = keras.layers.Flatten(input_shape=[x.shape[1]])
        self.hidden1 = keras.layers.Dense(32, activation='relu')
        self.hidden2 = keras.layers.Dense(32, activation='relu')
        self.hidden3 = keras.layers.Dense(32, activation='relu')
        self.output_layer = keras.layers.Dense(1)
        self.dropout_layer = PermaDropout(rate=dropout)

    def call(self, input, training=True):
        input_layer = self.input_layer(input)
        input_layer = self.dropout_layer(input_layer)
        hidden1 = self.hidden1(input_layer)
        hidden1 = self.dropout_layer(hidden1, training=training)
        hidden2 = self.hidden2(hidden1)
        hidden2 = self.dropout_layer(hidden2, training=training)
        #hidden3 = self.hidden3(hidden2)
        #hidden3 = self.dropout_layer(hidden3, training=training)
        output_layer = self.output_layer(hidden2)
        return output_layer
```

### Training

We instantiate the model and define a dropout value that was previously optimized using the uncertainty model goodness metric

```python
#model = create_model(0.0525)
model = create_model(0.06)
opt = keras.optimizers.Adam(learning_rate=0.016)

model.compile(loss='mae',
              optimizer=opt,
              metrics='mse')
```

```python
history=model.fit(
    X_train,
    y_train,
    batch_size=16,
    epochs=100,
    verbose=True,
    validation_data=(X_test,y_test)
)
```

    Epoch 1/100
    42/42 [==============================] - 1s 3ms/step - loss: 0.0813 - mse: 0.0138 - val_loss: 0.0475 - val_mse: 0.0058
    Epoch 2/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0385 - mse: 0.0033 - val_loss: 0.0410 - val_mse: 0.0047
    Epoch 3/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0331 - mse: 0.0026 - val_loss: 0.0254 - val_mse: 0.0019
    Epoch 4/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0273 - mse: 0.0019 - val_loss: 0.0276 - val_mse: 0.0024
    Epoch 5/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0274 - mse: 0.0018 - val_loss: 0.0298 - val_mse: 0.0028
    Epoch 6/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0300 - mse: 0.0029 - val_loss: 0.0279 - val_mse: 0.0025
    Epoch 7/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0276 - mse: 0.0024 - val_loss: 0.0284 - val_mse: 0.0022
    Epoch 8/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0302 - mse: 0.0020 - val_loss: 0.0306 - val_mse: 0.0019
    Epoch 9/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0322 - mse: 0.0025 - val_loss: 0.0351 - val_mse: 0.0024
    Epoch 10/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0369 - mse: 0.0030 - val_loss: 0.0431 - val_mse: 0.0033
    Epoch 11/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0352 - mse: 0.0028 - val_loss: 0.0291 - val_mse: 0.0025
    Epoch 12/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0293 - mse: 0.0021 - val_loss: 0.0367 - val_mse: 0.0031
    Epoch 13/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0268 - mse: 0.0017 - val_loss: 0.0285 - val_mse: 0.0023
    Epoch 14/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0283 - mse: 0.0023 - val_loss: 0.0224 - val_mse: 0.0020
    Epoch 15/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0273 - mse: 0.0019 - val_loss: 0.0237 - val_mse: 0.0015
    Epoch 16/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0270 - mse: 0.0019 - val_loss: 0.0230 - val_mse: 0.0019
    Epoch 17/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0280 - mse: 0.0023 - val_loss: 0.0302 - val_mse: 0.0027
    Epoch 18/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0270 - mse: 0.0019 - val_loss: 0.0313 - val_mse: 0.0028
    Epoch 19/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0295 - mse: 0.0023 - val_loss: 0.0279 - val_mse: 0.0026
    Epoch 20/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0269 - mse: 0.0017 - val_loss: 0.0275 - val_mse: 0.0023
    Epoch 21/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0284 - mse: 0.0024 - val_loss: 0.0259 - val_mse: 0.0019
    Epoch 22/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0281 - mse: 0.0022 - val_loss: 0.0274 - val_mse: 0.0023
    Epoch 23/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0280 - mse: 0.0021 - val_loss: 0.0248 - val_mse: 0.0015
    Epoch 24/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0260 - mse: 0.0017 - val_loss: 0.0299 - val_mse: 0.0022
    Epoch 25/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0296 - mse: 0.0025 - val_loss: 0.0291 - val_mse: 0.0030
    Epoch 26/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0261 - mse: 0.0016 - val_loss: 0.0290 - val_mse: 0.0024
    Epoch 27/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0262 - mse: 0.0021 - val_loss: 0.0257 - val_mse: 0.0023
    Epoch 28/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0270 - mse: 0.0019 - val_loss: 0.0248 - val_mse: 0.0020
    Epoch 29/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0261 - mse: 0.0019 - val_loss: 0.0237 - val_mse: 0.0018
    Epoch 30/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0287 - mse: 0.0021 - val_loss: 0.0333 - val_mse: 0.0035
    Epoch 31/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0296 - mse: 0.0023 - val_loss: 0.0265 - val_mse: 0.0024
    Epoch 32/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0302 - mse: 0.0025 - val_loss: 0.0341 - val_mse: 0.0023
    Epoch 33/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0271 - mse: 0.0016 - val_loss: 0.0312 - val_mse: 0.0035
    Epoch 34/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0265 - mse: 0.0021 - val_loss: 0.0308 - val_mse: 0.0027
    Epoch 35/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0311 - mse: 0.0024 - val_loss: 0.0288 - val_mse: 0.0028
    Epoch 36/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0295 - mse: 0.0027 - val_loss: 0.0279 - val_mse: 0.0018
    Epoch 37/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0285 - mse: 0.0020 - val_loss: 0.0219 - val_mse: 0.0019
    Epoch 38/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0274 - mse: 0.0023 - val_loss: 0.0302 - val_mse: 0.0022
    Epoch 39/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0258 - mse: 0.0016 - val_loss: 0.0295 - val_mse: 0.0021
    Epoch 40/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0269 - mse: 0.0022 - val_loss: 0.0269 - val_mse: 0.0020
    Epoch 41/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0296 - mse: 0.0022 - val_loss: 0.0306 - val_mse: 0.0023
    Epoch 42/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0281 - mse: 0.0023 - val_loss: 0.0306 - val_mse: 0.0031
    Epoch 43/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0278 - mse: 0.0024 - val_loss: 0.0254 - val_mse: 0.0025
    Epoch 44/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0231 - mse: 0.0015 - val_loss: 0.0263 - val_mse: 0.0034
    Epoch 45/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0290 - mse: 0.0024 - val_loss: 0.0283 - val_mse: 0.0031
    Epoch 46/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0277 - mse: 0.0018 - val_loss: 0.0251 - val_mse: 0.0013
    Epoch 47/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0246 - mse: 0.0017 - val_loss: 0.0298 - val_mse: 0.0020
    Epoch 48/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0254 - mse: 0.0021 - val_loss: 0.0258 - val_mse: 0.0018
    Epoch 49/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0285 - mse: 0.0022 - val_loss: 0.0265 - val_mse: 0.0016
    Epoch 50/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0286 - mse: 0.0028 - val_loss: 0.0261 - val_mse: 0.0018
    Epoch 51/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0241 - mse: 0.0017 - val_loss: 0.0260 - val_mse: 0.0019
    Epoch 52/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0242 - mse: 0.0016 - val_loss: 0.0251 - val_mse: 0.0018
    Epoch 53/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0233 - mse: 0.0014 - val_loss: 0.0259 - val_mse: 0.0028
    Epoch 54/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0236 - mse: 0.0014 - val_loss: 0.0262 - val_mse: 0.0025
    Epoch 55/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0256 - mse: 0.0019 - val_loss: 0.0274 - val_mse: 0.0021
    Epoch 56/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0274 - mse: 0.0021 - val_loss: 0.0292 - val_mse: 0.0029
    Epoch 57/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0267 - mse: 0.0020 - val_loss: 0.0211 - val_mse: 0.0017
    Epoch 58/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0260 - mse: 0.0018 - val_loss: 0.0277 - val_mse: 0.0027
    Epoch 59/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0282 - mse: 0.0021 - val_loss: 0.0249 - val_mse: 0.0016
    Epoch 60/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0264 - mse: 0.0016 - val_loss: 0.0241 - val_mse: 0.0015
    Epoch 61/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0276 - mse: 0.0023 - val_loss: 0.0238 - val_mse: 0.0021
    Epoch 62/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0257 - mse: 0.0017 - val_loss: 0.0306 - val_mse: 0.0029
    Epoch 63/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0248 - mse: 0.0017 - val_loss: 0.0248 - val_mse: 0.0019
    Epoch 64/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0241 - mse: 0.0022 - val_loss: 0.0248 - val_mse: 0.0016
    Epoch 65/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0270 - mse: 0.0021 - val_loss: 0.0287 - val_mse: 0.0025
    Epoch 66/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0248 - mse: 0.0019 - val_loss: 0.0257 - val_mse: 0.0019
    Epoch 67/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0252 - mse: 0.0018 - val_loss: 0.0221 - val_mse: 0.0017
    Epoch 68/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0245 - mse: 0.0018 - val_loss: 0.0251 - val_mse: 0.0018
    Epoch 69/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0307 - mse: 0.0029 - val_loss: 0.0275 - val_mse: 0.0021
    Epoch 70/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0240 - mse: 0.0023 - val_loss: 0.0289 - val_mse: 0.0017
    Epoch 71/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0243 - mse: 0.0015 - val_loss: 0.0236 - val_mse: 0.0015
    Epoch 72/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0280 - mse: 0.0022 - val_loss: 0.0291 - val_mse: 0.0024
    Epoch 73/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0285 - mse: 0.0018 - val_loss: 0.0258 - val_mse: 0.0019
    Epoch 74/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0230 - mse: 0.0017 - val_loss: 0.0235 - val_mse: 0.0018
    Epoch 75/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0248 - mse: 0.0020 - val_loss: 0.0249 - val_mse: 0.0016
    Epoch 76/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0262 - mse: 0.0028 - val_loss: 0.0318 - val_mse: 0.0039
    Epoch 77/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0254 - mse: 0.0018 - val_loss: 0.0221 - val_mse: 0.0017
    Epoch 78/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0241 - mse: 0.0016 - val_loss: 0.0255 - val_mse: 0.0019
    Epoch 79/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0255 - mse: 0.0019 - val_loss: 0.0301 - val_mse: 0.0022
    Epoch 80/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0250 - mse: 0.0019 - val_loss: 0.0299 - val_mse: 0.0029
    Epoch 81/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0261 - mse: 0.0025 - val_loss: 0.0259 - val_mse: 0.0023
    Epoch 82/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0211 - mse: 0.0011 - val_loss: 0.0257 - val_mse: 0.0025
    Epoch 83/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0277 - mse: 0.0022 - val_loss: 0.0295 - val_mse: 0.0025
    Epoch 84/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0272 - mse: 0.0021 - val_loss: 0.0282 - val_mse: 0.0026
    Epoch 85/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0235 - mse: 0.0016 - val_loss: 0.0234 - val_mse: 0.0018
    Epoch 86/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0262 - mse: 0.0019 - val_loss: 0.0262 - val_mse: 0.0027
    Epoch 87/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0249 - mse: 0.0020 - val_loss: 0.0270 - val_mse: 0.0032
    Epoch 88/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0272 - mse: 0.0019 - val_loss: 0.0304 - val_mse: 0.0023
    Epoch 89/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0297 - mse: 0.0024 - val_loss: 0.0321 - val_mse: 0.0032
    Epoch 90/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0236 - mse: 0.0016 - val_loss: 0.0273 - val_mse: 0.0032
    Epoch 91/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0238 - mse: 0.0016 - val_loss: 0.0253 - val_mse: 0.0020
    Epoch 92/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0238 - mse: 0.0017 - val_loss: 0.0283 - val_mse: 0.0038
    Epoch 93/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0250 - mse: 0.0017 - val_loss: 0.0272 - val_mse: 0.0032
    Epoch 94/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0216 - mse: 0.0015 - val_loss: 0.0245 - val_mse: 0.0025
    Epoch 95/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0264 - mse: 0.0019 - val_loss: 0.0282 - val_mse: 0.0021
    Epoch 96/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0291 - mse: 0.0027 - val_loss: 0.0236 - val_mse: 0.0022
    Epoch 97/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0223 - mse: 0.0014 - val_loss: 0.0259 - val_mse: 0.0022
    Epoch 98/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0255 - mse: 0.0016 - val_loss: 0.0335 - val_mse: 0.0035
    Epoch 99/100
    42/42 [==============================] - 0s 1ms/step - loss: 0.0232 - mse: 0.0013 - val_loss: 0.0243 - val_mse: 0.0019
    Epoch 100/100
    42/42 [==============================] - 0s 2ms/step - loss: 0.0255 - mse: 0.0023 - val_loss: 0.0304 - val_mse: 0.0026

Next we plot the loss for the training and testing dataset. We observe no overfitting in our model. We had to increase the number of epochs because of the regularization parameter

```python
plot_loss(history)
```

```python
y_test_pred = model.predict(X_test)

plt.figure(figsize=(12,6))
plt.plot(X_test[:,0],y_test,'o',label='Test')
plt.plot(X_test[:,0],y_test_pred,'o',label='Predicted test')

plt.xlabel('Porosity')
plt.ylabel('Production')
plt.legend()
plt.grid(True)
```

### Goodness of the uncertainty model

Let's evaluate the goodness of the model.

This is no longer a deterministic approach and we will evaluate the model only 200 times, next we will print the Goodness, Precision and Acurracy of the model.

```python
mc_predictions = evaluate_cases(model, X_test, 8, y_test,200)
```

    42/42 [==============================] - 0s 576us/step
    42/42 [==============================] - 0s 551us/step
    42/42 [==============================] - 0s 512us/step
    42/42 [==============================] - 0s 525us/step
    42/42 [==============================] - 0s 572us/step
    42/42 [==============================] - 0s 539us/step
    42/42 [==============================] - 0s 537us/step
    42/42 [==============================] - 0s 579us/step
    42/42 [==============================] - 0s 540us/step
    42/42 [==============================] - 0s 517us/step
    42/42 [==============================] - 0s 517us/step
    42/42 [==============================] - 0s 563us/step
    42/42 [==============================] - 0s 543us/step
    42/42 [==============================] - 0s 553us/step
    42/42 [==============================] - 0s 518us/step
    42/42 [==============================] - 0s 504us/step
    42/42 [==============================] - 0s 530us/step
    42/42 [==============================] - 0s 553us/step
    42/42 [==============================] - 0s 565us/step
    42/42 [==============================] - 0s 526us/step
    42/42 [==============================] - 0s 543us/step
    42/42 [==============================] - 0s 537us/step
    42/42 [==============================] - 0s 502us/step
    42/42 [==============================] - 0s 529us/step
    42/42 [==============================] - 0s 532us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 495us/step
    42/42 [==============================] - 0s 520us/step
    42/42 [==============================] - 0s 531us/step
    42/42 [==============================] - 0s 525us/step
    42/42 [==============================] - 0s 522us/step
    42/42 [==============================] - 0s 503us/step
    42/42 [==============================] - 0s 538us/step
    42/42 [==============================] - 0s 504us/step
    42/42 [==============================] - 0s 514us/step
    42/42 [==============================] - 0s 541us/step
    42/42 [==============================] - 0s 500us/step
    42/42 [==============================] - 0s 506us/step
    42/42 [==============================] - 0s 523us/step
    42/42 [==============================] - 0s 483us/step
    42/42 [==============================] - 0s 516us/step
    42/42 [==============================] - 0s 507us/step
    42/42 [==============================] - 0s 484us/step
    42/42 [==============================] - 0s 534us/step
    42/42 [==============================] - 0s 510us/step
    42/42 [==============================] - 0s 554us/step
    42/42 [==============================] - 0s 492us/step
    42/42 [==============================] - 0s 504us/step
    42/42 [==============================] - 0s 503us/step
    42/42 [==============================] - 0s 508us/step
    42/42 [==============================] - 0s 523us/step
    42/42 [==============================] - 0s 545us/step
    42/42 [==============================] - 0s 535us/step
    42/42 [==============================] - 0s 533us/step
    42/42 [==============================] - 0s 502us/step
    42/42 [==============================] - 0s 483us/step
    42/42 [==============================] - 0s 533us/step
    42/42 [==============================] - 0s 510us/step
    42/42 [==============================] - 0s 537us/step
    42/42 [==============================] - 0s 539us/step
    42/42 [==============================] - 0s 514us/step
    42/42 [==============================] - 0s 535us/step
    42/42 [==============================] - 0s 535us/step
    42/42 [==============================] - 0s 521us/step
    42/42 [==============================] - 0s 537us/step
    42/42 [==============================] - 0s 499us/step
    42/42 [==============================] - 0s 509us/step
    42/42 [==============================] - 0s 489us/step
    42/42 [==============================] - 0s 512us/step
    42/42 [==============================] - 0s 500us/step
    42/42 [==============================] - 0s 536us/step
    42/42 [==============================] - 0s 583us/step
    42/42 [==============================] - 0s 585us/step
    42/42 [==============================] - 0s 556us/step
    42/42 [==============================] - 0s 512us/step
    42/42 [==============================] - 0s 485us/step
    42/42 [==============================] - 0s 536us/step
    42/42 [==============================] - 0s 549us/step
    42/42 [==============================] - 0s 545us/step
    42/42 [==============================] - 0s 522us/step
    42/42 [==============================] - 0s 511us/step
    42/42 [==============================] - 0s 508us/step
    42/42 [==============================] - 0s 497us/step
    42/42 [==============================] - 0s 542us/step
    42/42 [==============================] - 0s 548us/step
    42/42 [==============================] - 0s 522us/step
    42/42 [==============================] - 0s 521us/step
    42/42 [==============================] - 0s 512us/step
    42/42 [==============================] - 0s 533us/step
    42/42 [==============================] - 0s 493us/step
    42/42 [==============================] - 0s 493us/step
    42/42 [==============================] - 0s 522us/step
    42/42 [==============================] - 0s 518us/step
    42/42 [==============================] - 0s 520us/step
    42/42 [==============================] - 0s 527us/step
    42/42 [==============================] - 0s 499us/step
    42/42 [==============================] - 0s 543us/step
    42/42 [==============================] - 0s 527us/step
    42/42 [==============================] - 0s 562us/step
    42/42 [==============================] - 0s 532us/step
    42/42 [==============================] - 0s 528us/step
    42/42 [==============================] - 0s 528us/step
    42/42 [==============================] - 0s 508us/step
    42/42 [==============================] - 0s 601us/step
    42/42 [==============================] - 0s 517us/step
    42/42 [==============================] - 0s 525us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 538us/step
    42/42 [==============================] - 0s 553us/step
    42/42 [==============================] - 0s 494us/step
    42/42 [==============================] - 0s 497us/step
    42/42 [==============================] - 0s 563us/step
    42/42 [==============================] - 0s 514us/step
    42/42 [==============================] - 0s 496us/step
    42/42 [==============================] - 0s 510us/step
    42/42 [==============================] - 0s 507us/step
    42/42 [==============================] - 0s 494us/step
    42/42 [==============================] - 0s 544us/step
    42/42 [==============================] - 0s 489us/step
    42/42 [==============================] - 0s 527us/step
    42/42 [==============================] - 0s 561us/step
    42/42 [==============================] - 0s 504us/step
    42/42 [==============================] - 0s 503us/step
    42/42 [==============================] - 0s 513us/step
    42/42 [==============================] - 0s 487us/step
    42/42 [==============================] - 0s 533us/step
    42/42 [==============================] - 0s 504us/step
    42/42 [==============================] - 0s 511us/step
    42/42 [==============================] - 0s 553us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 501us/step
    42/42 [==============================] - 0s 511us/step
    42/42 [==============================] - 0s 508us/step
    42/42 [==============================] - 0s 559us/step
    42/42 [==============================] - 0s 554us/step
    42/42 [==============================] - 0s 516us/step
    42/42 [==============================] - 0s 566us/step
    42/42 [==============================] - 0s 501us/step
    42/42 [==============================] - 0s 619us/step
    42/42 [==============================] - 0s 657us/step
    42/42 [==============================] - 0s 754us/step
    42/42 [==============================] - 0s 657us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 570us/step
    42/42 [==============================] - 0s 542us/step
    42/42 [==============================] - 0s 463us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 539us/step
    42/42 [==============================] - 0s 543us/step
    42/42 [==============================] - 0s 497us/step
    42/42 [==============================] - 0s 525us/step
    42/42 [==============================] - 0s 512us/step
    42/42 [==============================] - 0s 524us/step
    42/42 [==============================] - 0s 536us/step
    42/42 [==============================] - 0s 499us/step
    42/42 [==============================] - 0s 545us/step
    42/42 [==============================] - 0s 520us/step
    42/42 [==============================] - 0s 556us/step
    42/42 [==============================] - 0s 555us/step
    42/42 [==============================] - 0s 544us/step
    42/42 [==============================] - 0s 548us/step
    42/42 [==============================] - 0s 486us/step
    42/42 [==============================] - 0s 485us/step
    42/42 [==============================] - 0s 499us/step
    42/42 [==============================] - 0s 523us/step
    42/42 [==============================] - 0s 485us/step
    42/42 [==============================] - 0s 528us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 469us/step
    42/42 [==============================] - 0s 475us/step
    42/42 [==============================] - 0s 473us/step
    42/42 [==============================] - 0s 574us/step
    42/42 [==============================] - 0s 486us/step
    42/42 [==============================] - 0s 536us/step
    42/42 [==============================] - 0s 499us/step
    42/42 [==============================] - 0s 514us/step
    42/42 [==============================] - 0s 514us/step
    42/42 [==============================] - 0s 642us/step
    42/42 [==============================] - 0s 566us/step
    42/42 [==============================] - 0s 532us/step
    42/42 [==============================] - 0s 496us/step
    42/42 [==============================] - 0s 513us/step
    42/42 [==============================] - 0s 514us/step
    42/42 [==============================] - 0s 534us/step
    42/42 [==============================] - 0s 561us/step
    42/42 [==============================] - 0s 518us/step
    42/42 [==============================] - 0s 584us/step
    42/42 [==============================] - 0s 698us/step
    42/42 [==============================] - 0s 559us/step
    42/42 [==============================] - 0s 585us/step
    42/42 [==============================] - 0s 584us/step
    42/42 [==============================] - 0s 560us/step
    42/42 [==============================] - 0s 584us/step
    42/42 [==============================] - 0s 618us/step
    42/42 [==============================] - 0s 535us/step
    42/42 [==============================] - 0s 511us/step
    42/42 [==============================] - 0s 515us/step
    42/42 [==============================] - 0s 584us/step
    42/42 [==============================] - 0s 778us/step
    42/42 [==============================] - 0s 608us/step

```python
df = get_dataframe(mc_predictions, y_test, X_test)
```

How does the cross validation plot look like?
What about the Pearson's product moment correlation coefficient?

```python
r= stats.pearsonr(df.Pred,df.Truth)
error_line(df.Pred,df.Truth,df.Sigma,r[0])
```

What is the Goodness, Precision and Accuracy of the model?

```python
Sigma = np.sqrt(np.var(mc_predictions.T, axis=1))

score = scorer(mc_predictions.T, y_test, Sigma)

IF_array = score.IndicatorFunction()
n_quantiles = 11
perc = np.linspace(0.0, 1.00, n_quantiles)

```

```python
print('Accuracy = {0:2.2f}'.format(score.Accuracy()))
print('Precision = {0:2.2f}'.format(score.Precision()))
print('Goodness = {0:2.2f}'.format(score.Goodness()))
```

    Accuracy = 0.03
    Precision = 1.00
    Goodness = 0.63

For a random point we can plot the histogram of predictions and estimate the standard deviation of the mean value

```python
histogram_fixed(mc_predictions,5);
```

How does the accuracy plot look like for the defined model?

```python
error_accuracy_plot(perc, IF_array, mc_predictions.T, y_test, Sigma)
```

## Exploration of the hyperparameter space

The model we instantiated in the last section was selected using the following algorithm:

**Input**: m hyperparameter combinations to evaluate
**Output**: optimal dropout with maximum uncertainty model goodness for a given learning rate
1. Use any design of experiments (DOE) to sample m dropout and learning rate values of the search space.
2. Use the objective function defined in equation 8
3. **while** i<m **do**

    1. Perform model training with specified dropout rate, d; for a given learning rate, lr;
    2. for l=1,…,L ensemble models calculate prediction realizations over the testing data to calculate the nonparametric cumulative distribution function $F_y (u_i)$;
	3. Calculate the average indicator function $\overline{\xi (p)}$ for each discretized symmetric probability interval to calculate the accuracy plot;
	4. Evaluate scores, precision, accuracy, and goodness of the uncertainty model with the proposed objective function;
	5. Evaluate overall uncertainty to split ties for models with similar scores from the proposed objective function
4. **return** optimum dropout with uncertainty model accuracy, precision and the least overall uncertainty.

We use Optuna for hyperparameter tuning and the Uncertainty Model Goodness to search for the optimum value of hyperparameters.

**NOTE**: the following piece of code might be too long to run. So feel free to skip the following lines and just load the results of this grid search  in the next section.

```python
d_space = np.linspace(0.05, 0.95, 30)
l_space = np.linspace(0.001, .03 , 20)
search_space = {"Dropout": d_space, "learn_rate": l_space}

study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space))
```

Save our job

```python
study.optimize(objective,
               n_trials=d_space.shape[0] * l_space.shape[0],
               show_progress_bar=True)
```

```python
StudyName='Search-Space'
joblib.dump(study, StudyName+'.pkl')
```

### Summary of the training

Next we present the summary of the training

```python
# StudyName='Search-Space'

# study = joblib.load(StudyName+'.pkl')
# df = study.trials_dataframe().drop(['state','datetime_start','datetime_complete'], axis=1)

# df.to_csv('Search-space.csv')

df=pd.read_csv('Search-space.csv')
```

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 600 entries, 0 to 599
    Data columns (total 8 columns):
     #   Column                     Non-Null Count  Dtype
    ---  ------                     --------------  -----
     0   Unnamed: 0                 600 non-null    int64
     1   number                     600 non-null    int64
     2   value                      600 non-null    float64
     3   duration                   600 non-null    object
     4   params_Dropout             600 non-null    float64
     5   params_learn_rate          600 non-null    float64
     6   system_attrs_grid_id       600 non-null    int64
     7   system_attrs_search_space  600 non-null    object
    dtypes: float64(3), int64(3), object(2)
    memory usage: 37.6+ KB

This is how the search space evaluated with the uncertainty model goodness metric looks like:

```python
import scipy.ndimage
import matplotlib.tri as tri
from scipy.ndimage.filters import gaussian_filter

z=df['value'].values
x=df['params_Dropout'].values
y=df['params_learn_rate'].values

fig, (ax1) = plt.subplots(nrows=1,figsize=(12,6))

npoints=300
smooth=2

# Create grid values first.
xi = np.linspace(x.min(), x.max(), npoints)
yi = np.linspace(y.min(), y.max(), npoints)

# Linearly interpolate the data (x, y) on a grid defined by (xi, yi).
triang = tri.Triangulation(x, y)
interpolator = tri.LinearTriInterpolator(triang, z)
Xi, Yi = np.meshgrid(xi, yi)
zi = interpolator(Xi, Yi)

zi = gaussian_filter(zi, smooth)

levels=10

ax1.contour(xi, yi, zi, levels=levels, linewidths=0.1, colors='k')
cntr1 = ax1.contourf(xi, yi, zi, levels=levels, cmap="inferno",alpha=0.9)

cbar = plt.colorbar(cntr1, ax=ax1)

cbar.set_label('Uncertainty model Goodness', rotation=270,labelpad=30)

ax1.set(xlim=(df.params_Dropout.min(), df.params_Dropout.max()),ylim=(df.params_learn_rate.min(), df.params_learn_rate.max()))
ax1.scatter(x,y,s=3,color='darkgray')
ax1.set_ylabel('Learn rate')
ax1.set_xlabel('Dropout')
plt.show()

```

## Conclusions and observations

* The work introduces a novel workflow to tune uncertainty models by using the uncertainty model goodness metric.

* This inclusion allows the calculation of robust uncertainty models from machine learning-based ensembles of estimates and robust forecasting.

* This approach is critical to any prediction problem with significant uncertainty.

### Eduardo Maldonado Cruz, Graduate Research Assistant, The University of Texas at Austin

#### [Twitter](https://twitter.com/emaldonadocruz) | [GitHub](https://github.com/emaldonadocruz) | [LinkedIn](https://www.linkedin.com/in/eduardo-maldonado-8386a898/)

### Michael Pyrcz, Associate Professor, The University of Texas at Austin

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)