```python
import sys
sys.path.append('..') # Add src to path
import os
os.environ["HDF5_USE_FILE_LOCKING"]='FALSE'
import datetime
import h5py
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from matplotlib.animation import FuncAnimation
from IPython.display import Image

import tensorflow as tf

from src.display import get_cmap
from src.utils import make_log_dir

# comment these out if you don't have cartopy
import cartopy.feature as cfeature
from src.display.cartopy import make_ccrs,make_animation

from make_dataset import NowcastGenerator,get_nowcast_train_generator,get_nowcast_test_generator

from unet_benchmark import create_model
from unet_benchmark import nowcast_mae, nowcast_mse
from IPython.core.debugger import set_trace
```

# Radar Nowcast Challenge

This notebook will describe the SEVIR radar nowcasting challenge.  Nowcasts (https://en.wikipedia.org/wiki/Nowcasting_(meteorology)) are short-term forecast of weather variables typically measured by weather radar or satellite.   Nowcasts are different from traditional weather forecasts (like those you see on the news, or on your phones weather app) in that they are based (mostly) on statistical extrapolations of recent data, rather than full physics-based numerical weather prediction (NWP) models.   The advantages of using statistical extrapolation techniques is that they generally run much faster than NWP models (seconds compared to hours), and because of this, nowcasts are able to  leverage the most recently observed data and use it for forecasting.  Because of this extremely low latency, nowcast out-perform traditional weather models in terms of accuracy and precision for short look-aheads (typically 1-2 hours at most).

Below is an example that a nowcast generated using the baseline model created in this notebook.  The first half of the movie is observed radar VIL.  The second half of the movie is the forecast generated by a model trained in this notebook.   The challenge is to create the best possible nowcast algorithm given the data in SEVIR.

### Contents:
* [Python environment](#env)
* [Problem Statement](#problem)
* [Training & testing dataset](#datasets)
    * [Visualizing training samples](#vistraining)
* [Model Training](#training)
* [Visualize Result](#visualize)
* [Forecast Scoring](#scoring)
* [Applying to full-sized images](#fillimgs)

## Python Environment

Running this notebook requires a python environment that includes the modules described here.  You can setup the environment with anaconda via

```
conda create --name sevir_challenges
conda activate sevir_challenges
conda install --file requirements.txt
```

## Problem Statement

The goal of this challenge is to predict 12 future frames of radar images in 5 minute steps (`vil` in this case) given the previous 13 images of `vil` (also in 5 minute steps) in addition to other modalities in SEVIR.

## Training and testing datasets

### MIT Supercloud

If you have access to MIT supercloud, the data for this challenge can be accessed directly:

### AWS Download

Sample training and testing datasets are provided on the SEVIR S3 bucket.  These can be obtained by running the following cell.

**This downloads and decompresses files that will take up roughly 40 GB so make sure you have sufficient disk space**

```python
data_path="/scratch/li.baol/SEVIR"
```

```python
# Target locations of sample training & testing data
DEST_TRAIN_FILE= os.path.join(data_path,'data/processed/nowcast_training_000.h5')
DEST_TRAIN_META=os.path.join(data_path, 'data/processed/nowcast_training_000_META.csv')
DEST_TEST_FILE= os.path.join(data_path, 'data/processed/nowcast_testing_000.h5')
DEST_TEST_META= os.path.join(data_path, 'data/processed/nowcast_testing_000_META.csv')
```

```python
# THIS DOWNLOADS APPROXIMATELY 40 GB DATASETS (AFTER DECOMPRESSION)
import boto3
from botocore.handlers import disable_signing
import tarfile
resource = boto3.resource('s3')
resource.meta.client.meta.events.register('choose-signer.s3.*', disable_signing)
bucket=resource.Bucket('sevir')

print('Dowloading sample training data')
if not os.path.exists(DEST_TRAIN_FILE):
    bucket.download_file('data/processed/nowcast_training_000.h5.tar.gz',DEST_TRAIN_FILE+'.tar.gz')
    bucket.download_file('data/processed/nowcast_training_000_META.csv',DEST_TRAIN_META)
    with tarfile.open(DEST_TRAIN_FILE+'.tar.gz') as tfile:
        tfile.extract('data/processed/nowcast_training_000.h5',data_path)
else:
    print('Train file %s already exists' % DEST_TRAIN_FILE)
print('Dowloading sample testing data')
if not os.path.exists(DEST_TEST_FILE):
    bucket.download_file('data/processed/nowcast_testing_000.h5.tar.gz',DEST_TEST_FILE+'.tar.gz')
    bucket.download_file('data/processed/nowcast_testing_000_META.csv',DEST_TEST_META)
    with tarfile.open(DEST_TEST_FILE+'.tar.gz') as tfile:
        tfile.extract('data/processed/nowcast_testing_000.h5',data_path)
else:
    print('Test file %s already exists' % DEST_TEST_FILE)
```

    Dowloading sample training data
    Train file /scratch/li.baol/SEVIR/data/processed/nowcast_training_000.h5 already exists
    Dowloading sample testing data
    Test file /scratch/li.baol/SEVIR/data/processed/nowcast_testing_000.h5 already exists

See the contents of the training data file using `h5ls`:

```python
#!h5ls /scratch/li.baol/SEVIR/data/processed/nowcast_training_000.h5
```

This file contains four modalities you can use as input to a nowcast model (keys start with `IN_` -- `vil`,`ir_ir107`,`ir069`, or `lght`).  The challenge is to use these inputs to predict the output vil `OUT_vil`.

If you have downloaded the full SEVIR dataset, you can extract even larger datasets of this format, or even modify the dataset for your puprose.  We assume SEVIR is located in the directory `$SEVIR_ROOT`.  To make your own dataset, run the script `make_dataset.py`:

```
python make_dataset.py --input_types vil --output_types vil --sevir_data $SEVIR_ROOT/data/ --sevir_catalog $SEVIR_ROOT/CATALOG.csv --output_location ../data/processed/
```

The rest of this notebook assumes you are using the sample training and test files located in the directory `../data/processed/` relative to this notebook.  Adjust as needed.

### Exploring the training dataset

First load some samples from the training data and assocaited metadata for georeferencing and plotting.  If the full training set is too large to fit into memory, adjust `N_TRAIN` and `N_TEST` below.  This will use `TRAIN_VAL_FRAC * N_TRAIN` training samples for a validation set.

NOTE:  For this baseline we will only use `IN_vil` frames as inputs.  However the other modalities may be useful for nowcasting.

```python
# Control how many samples are read.   Set to -1 to read all 5000 samples.
N_TRAIN=-1
TRAIN_VAL_FRAC=0.8
#set_trace()
N_TEST=-1
```

```python
# Loading data takes a few minutes
with h5py.File(DEST_TRAIN_FILE,'r') as hf:
    Nr = N_TRAIN if N_TRAIN>=0 else hf['IN_vil'].shape[0]
    X_train = hf['IN_vil'][:Nr]
    Y_train = hf['OUT_vil'][:Nr]
    training_meta = pd.read_csv(DEST_TRAIN_META).iloc[:Nr]
    X_train,X_val=np.split(X_train,[int(TRAIN_VAL_FRAC*Nr)])
    Y_train,Y_val=np.split(Y_train,[int(TRAIN_VAL_FRAC*Nr)])
    training_meta,val_meta=np.split(training_meta,[int(TRAIN_VAL_FRAC*Nr)])
#set_trace()
with h5py.File(DEST_TEST_FILE,'r') as hf:
    Nr = N_TEST if N_TEST>=0 else hf['IN_vil'].shape[0]
    X_test = hf['IN_vil'][:Nr]
    Y_test = hf['OUT_vil'][:Nr]
    testing_meta=pd.read_csv(DEST_TEST_META).iloc[:Nr]
```

Get a sample batch of image sequences.  Note the shape`[N,L,L,T]` where `N` is the batch size, `L` is the size of the image patch, and `T` is the number of time frames in the video (1 time step = 5 minutes).  The VIL images in SEVIR are scaled to the range [0-255], and can be decoded back to orginal units following the [SEVIR Tutorial](https://nbviewer.jupyter.org/github/MIT-AI-Accelerator/eie-sevir/blob/master/examples/SEVIR_Tutorial.ipynb).

```python
batch_size,batch_num=8,0
bs,be=batch_size*batch_num,batch_size*(batch_num+1)
X,Y,meta = X_train[bs:be],Y_train[bs:be],training_meta.iloc[bs:be]
print('Input X:',X.shape)
print('Output Y:',Y.shape)
print('META:',meta.shape)
```

    Input X: (8, 384, 384, 13)
    Output Y: (8, 384, 384, 12)
    META: (8, 14)

We'll start by visualizing some of the data.  This includes some colormaps for visualization.  The following plots a few frames with the custom colormap provided for the `vil` type:

```python
batch_index=1
cmap,norm,vmin,vmax=get_cmap('vil')
fig,axs=plt.subplots(1,4,figsize=(12,4))
for i in range(4):
    axs[i].imshow(X[batch_index,:,:,4*i],origin='lower',cmap=cmap,norm=norm,vmin=vmin,vmax=vmax) # every 4th frame
    axs[i].set_axis_off()
```

The variable `meta` contains additional information about the event pictured:

```python
meta.iloc[batch_index]
```

    Unnamed: 0                                                    19989
    id                                                          S819072
    time_utc                                        2019-04-09 17:10:00
    episode_id                                                 136472.0
    event_id                                                   819072.0
    event_type                                        Thunderstorm Wind
    minute_offsets    -60:-55:-50:-45:-40:-35:-30:-25:-20:-15:-10:-5...
    llcrnrlat                                                 41.434316
    llcrnrlon                                                -77.367946
    urcrnrlat                                                 43.830965
    urcrnrlon                                                -71.510706
    proj              +proj=laea +lat_0=38 +lon_0=-98 +units=m +a=63...
    height_m                                                   384000.0
    width_m                                                    384000.0
    Name: 1, dtype: object

Since this event has an associated `event_id`, more information can be obtained from NCEI storm event database associated to the year of the event (2019 in this case).  Look for the files named `StormEvents_details` for the particular years of interest here:

[https://www1.ncdc.noaa.gov/pub/data/swdi/stormevents/csvfiles](https://www1.ncdc.noaa.gov/pub/data/swdi/stormevents/csvfiles)

If you download relevant files, additional information for the event can be obtained by searching these files for the `episode_id` or `event_id` found above.

The metadata can be used to georeference the patches shown above.  If cartopy is installed in your environment, map features can be added to these plots:

```python
# REQUIRES CARTOPY (and an internet connection if this is your first time using cartopy)
proj,img_extent = make_ccrs(meta.iloc[batch_index])
xll,xur=img_extent[0],img_extent[1]
yll,yur=img_extent[2],img_extent[3]
fig,axs=plt.subplots(1,4,figsize=(12,4), subplot_kw={'projection':proj})
for i in range(4):
    axs[i].set_xlim((xll,xur))
    axs[i].set_ylim((yll,yur))
    axs[i].imshow(X[batch_index,:,:,4*i],interpolation='nearest',
                  origin='lower',transform=proj,extent=[xll,xur,yll,yur],
                  cmap=cmap,norm=norm,vmin=vmin,vmax=vmax) # every 4th frame
    axs[i].add_feature(cfeature.STATES)

```

The function `make_animation` creates a short movie loop:

```python
%%capture off
anim = make_animation(X[batch_index],meta.iloc[batch_index],title='Inputs')
anim.save('imgs/input_animation.gif', writer='imagemagick', fps=6)
anim = make_animation(Y[batch_index],meta.iloc[batch_index],title='Target')
anim.save('imgs/output_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/input_animation.gif')
```

    <IPython.core.display.Image object>

```python
Image('imgs/output_animation.gif')
```

    <IPython.core.display.Image object>

## Create baseline model

In this section we will initialize a baseline model and train it.

### Initialize U-Net model

The baseline model used in this notebook is the U-Net model based on the model described [here](https://proceedings.neurips.cc//paper/2020/hash/fa78a16157fed00d7a80515818432169-Abstract.html).

This model has the following structure, which takes a sequence of VIL images, and outputs another sequence that covers the next hour of time:

Code for the model is defined in `unet_benchmark.py`.

The following cell defines training hyperparamters we'll use to start.  These can be adjusted as needed.

```python
# Add more as needed
params={
    'start_neurons'   :16,      # Controls size of hidden layers in CNN, higher = more complexity
    'activation'      :'relu',  # Activation used throughout the U-Net,  see https://www.tensorflow.org/api_docs/python/tf/keras/activations
    'loss'            :'mae',   # Either 'mae' or 'mse', or others as https://www.tensorflow.org/api_docs/python/tf/keras/losses
    'loss_weights'    :0.021,    # Scale for loss.  Recommend squaring this if using MSE
    'opt'             :tf.keras.optimizers.Adam,  # optimizer, see https://www.tensorflow.org/api_docs/python/tf/keras/optimizers
    'learning_rate'   :0.001,   # Learning rate for optimizer
    'num_epochs'      :10,       # Number of epochs to train for
    'batch_size'      :8        # Size of batches during training
}
```

```python
unet = create_model(start_neurons=params['start_neurons'],activation=params['activation'])
unet.summary()
```

For a loss function, we'll start with mean absolute error scaled by an estimate of the standard deviation computed over the training data ($\sigma^{-1} \approx 0.021$)
\begin{equation}
L(Y,\hat{Y})=\frac{1}{\sigma}\sum_{i} ||Y-\hat{Y}||_1
\end{equation}

```python
opt=params['opt'](learning_rate=params['learning_rate'])
unet.compile(optimizer=opt, loss=params['loss'],loss_weights=[params['loss_weights']])
```

### Train Model

We'll set up two callbacks.  The first saves checkpoints whenever `val_loss` is minimized.  The other sets up tensorboard.

**NOTE:  Model checkpoints will be saved under the `experiments` directory in a newly created time-stamped directory.  A link named `experiments/latest` is also created that points to the most recent experiment.**

```python
# Training 10 epochs takes around 10-20 minutes on GPU
num_epochs=params['num_epochs']
batch_size=params['batch_size']
exprmt_dir=make_log_dir('experiments')

callbacks=[
    tf.keras.callbacks.ModelCheckpoint(exprmt_dir+'/nowcast-unet-{epoch:04d}-{val_loss:04f}.hdf5',
                    monitor='val_loss',save_best_only=True),
    tf.keras.callbacks.TensorBoard(log_dir=exprmt_dir+'/tboardlogs')
]

history = unet.fit(x=X_train, y=Y_train,
                  batch_size=batch_size,
                  epochs=num_epochs,
                   callbacks=callbacks,
                  validation_data=(X_val, Y_val))
```

Plot model training performance for the first 10 epochs

```python
plt.plot(history.history['loss'],label='Train loss')
plt.plot(history.history['val_loss'],label='Val loss')
plt.legend()
```

```python
# save model for later use
unet.save('unet_10epochs.h5')
```

```python
# Reload previously saved model
# Make sure to use `custom_objects` for custom loss functions

# NOTE: The best performing model (on val set) across all epochs was saved under experiments/latest,
# so you can also load that instead

unet = tf.keras.models.load_model('unet_10epochs.h5',custom_objects={})
```

## Visualize model on test samples

Start by visualizing the resulting model on test samples

```python
batch_size,batch_num=8,4
bs,be=batch_size*batch_num,batch_size*(batch_num+1)
x_test,y_test,meta = X_test[bs:be],Y_test[bs:be],testing_meta.iloc[bs:be]
```

```python
%%capture off
bidx=5
anim = make_animation(x_test[bidx],meta.iloc[bidx],title='Inputs')
anim.save('imgs/input_test_animation.gif', writer='imagemagick', fps=6)
anim = make_animation(y_test[bidx],meta.iloc[bidx],title='Target')
anim.save('imgs/output_test_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/input_test_animation.gif')
```

```python
Image('imgs/output_test_animation.gif')
```

Now compute the prediction

```python
%%capture off
y_pred = unet.predict(x_test)
anim = make_animation(y_pred[bidx],meta.iloc[bidx],title='Prediction')
anim.save('imgs/pred_test_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/pred_test_animation.gif')
```

What do you think?  You can adjust `batch_num` and `bidx` to view other cases from the test set.

The good:
* Seems to be moving the weather in the correct direction, and retaining some of the structure

The bad:
* A lot of the detail gets washed out, and for longer leads, the more intense sections of the storm wash away.

## Forecast Scoring

Let's apply some more quantitative verifications to establish baseline performance.

To define metrics used, let $T_i(t)$ and $F_i(t)$ represent target and forecast images, respectively, for test sample $i$ at lead times $t=5,10,...,60$ minutes.  The first metric to consider is mean-absolute error (MAE):

\begin{equation}
MAE(t) = \frac{1}{L^2 N_{test}}\sum_{i=1}^{N_{test}} ||T_i(t)-F_i(t)||_1
\end{equation}

MAE is nice because it is simple, however it often fails to capture forecast skill in predicting fine detail in storm, such as severe storm cores.  Therefore, we'll also apply more standard forecast verification metrics (see [https://www.cawcr.gov.au/projects/verification/](https://www.cawcr.gov.au/projects/verification/)).

Given a threshold $\tau$, we binarize both the target and forecast images, and label each pixel as a "hit" ($H$) if both target and forecast are $\geq \tau$, a "miss" ($M$) if $T\geq\tau$ and $F<\tau$, a "false alarm" ($FA$) if $T<\tau$ and $F\geq \tau$, and a correct rejection if both $T$ and $F$ are less than $\tau$.

These pixel scores are rolled up into Critical Success Index (CSI) (which is similar to Intersection over Union, which is another common name for this metric) and is computed as

\begin{equation}
CSI(t;\tau) = \frac{\#H}{\#H + \#M + \#FA}
\end{equation}

A CSI of 1 represents a perfect pixel-to-pixel forecast.  For scoring VIL nowcasts, we'll average this over three thresholds that are used in practice [16,74,133] representing low, medium, and high strom intensity.  We'll refer to this average as "mCSI", computed for each lead time $t$.

\begin{equation}
mCSI(t) = \frac{1}{3}(CSI(t;16) + CSI(t;74) + CSI(t;133))
\end{equation}

A "final" test scores can be computed by averaging $MAE$ and $mCSI$ over all lead times in 0-60 minutes.

This repo contains some code to compute CSI for the test set.

```python
# Run unet over the test set
y_pred = unet.predict(X_test,batch_size=4)
```

```python
# Compute metrics over the test set separately for each lead (takes a few minutes)
from tqdm import tqdm
from src.metrics import critical_success_index
CSI = lambda yt,yp,tau: critical_success_index(yt.astype(np.float32),
                                               yp.astype(np.float32),np.array([tau],dtype=np.float32)).numpy()

THRESHOLDS=np.array([16,74,133],dtype=np.float32)
nT=Y_test.shape[3]
mae_t=[]
csi_t = []
for t in tqdm(range(nT)):
    mae_t.append(np.mean(tf.keras.losses.MAE(Y_test[:,:,:,t:t+1],y_pred[:,:,:,t:t+1])))
    csi_t.append(np.mean([CSI(Y_test[:,:,:,t:t+1], y_pred[:,:,:,t:t+1],tau) for tau in THRESHOLDS]))

```

We'll also include another standard baseline model in nowcasting, the *persistence model*.  This model represents the "do nothing" forecast where the last image in the input sequence is just repeated for every future frame.

```python
# Also compute scores of persistence model
nT=Y_test.shape[3]
pers_mae_t=[]
pers_csi_t = []
for t in tqdm(range(nT)):
    pers_mae_t.append(np.mean(tf.keras.losses.MAE(Y_test[:,:,:,t:t+1],X_test[:,:,:,-1:].astype(np.float32))))
    pers_csi_t.append(np.mean([CSI(Y_test[:,:,:,t:t+1], X_test[:,:,:,-1:],tau) for tau in THRESHOLDS]))
```

```python
t=np.arange(5,65,5)
fig,ax=plt.subplots(1,2,figsize=(15,5))
ax[0].plot(t,mae_t,label='U-Net Baseline')
ax[0].plot(t,pers_mae_t,label='Persistence')
ax[0].set_xlabel('Lead time (minutes)')
ax[0].set_ylabel('Nowcast MAE (lower is better)')
ax[0].set_title('MAE')
ax[0].legend()

ax[1].plot(t,csi_t,label='U-Net Baseline')
ax[1].plot(t,pers_csi_t,label='Persistence')
ax[1].set_xlabel('Lead time (minutes)')
ax[1].set_ylabel('Nowcast mCSI (higher is better)')
ax[1].set_title('mCSI')
ax[1].legend()
plt.show()

print('Average MAE of U-Net on test set: ',np.mean(mae_t))
print('Average mCSI of U-Net on test set: ',np.mean(csi_t))

```

The baseline model beats persistence (phew!) but it can still be better!

##### Can you create a model that improves upon the above test scores?

## (Optional) Applying to US-sized images

The model trained above is fully convolutional and can be apllied to full-sized radar mosaics.  Sample US radar mosaics of VIL can be obtained from the MRMS system [https://mrms.ncep.noaa.gov/data/2D/VIL/](https://mrms.ncep.noaa.gov/data/2D/VIL/).  The past 24 hours of files are avaialble online for download.  The model *may* also work for other radar fields, e.g. reflectivity, so long as they are first normalized to the range [0-255] first.

MRMS data is not a part of this repo so you'll need to download and prepared this data yourself.  Before feeding this data to the CNN, some preparation is required first.  First, the MRMS files are available in `grib2` format and need to be converted to a numpy array.  Second, the units need to be converted to the [0-255] scale.  Code for doing these steps can be found in [this gist](https://gist.github.com/markveillette/048d7e6b6be35d26bac37378bd66ae01), which provides a script for doing this preprocessing.

```python
# MRMS updates every 2 minutes.  We will approximate 5 minute updates by switching between 4 and 6 minute
# steps.

# UPDATE WITH YOUR OWN LOCATION AND FILES
mrms_root='../data/MRMS'
files=[
    'MRMS_VIL_00.50_20201125-160039.grib2.npy','MRMS_VIL_00.50_20201125-160436.grib2.npy',
    'MRMS_VIL_00.50_20201125-161039.grib2.npy','MRMS_VIL_00.50_20201125-161442.grib2.npy',
    'MRMS_VIL_00.50_20201125-162039.grib2.npy','MRMS_VIL_00.50_20201125-162434.grib2.npy',
    'MRMS_VIL_00.50_20201125-163037.grib2.npy','MRMS_VIL_00.50_20201125-163436.grib2.npy',
    'MRMS_VIL_00.50_20201125-164041.grib2.npy','MRMS_VIL_00.50_20201125-164438.grib2.npy',
    'MRMS_VIL_00.50_20201125-165039.grib2.npy','MRMS_VIL_00.50_20201125-165436.grib2.npy',
    'MRMS_VIL_00.50_20201125-170042.grib2.npy']
vil=[]
for f in files:
    vil.append(np.load(mrms_root+'/'+f)[np.newaxis,:,:,np.newaxis])
vil=np.concatenate(vil,axis=3)
vil = np.pad(vil, ((0,0),(2,2),(4,4),(0,0))) # needed to make sure downsample layers can be done cleanly
m=vil[0:1,2:-2,4:-4,0]==255
vil[vil==255]=0
```

```python
# Load weights of your own saved model, but make sure input_shape is None so it works on full sized images
unet = create_model(start_neurons=params['start_neurons'],
                    activation=params['activation'],input_shape=(None,None,13))
unet.load_weights('experiments/unet_baseline/nowcast-unet-0039-0.133934.hdf5')
```

```python
# If this overwhelms your GPU, you might need to compute forecast in smaller subsections
forecast = unet.predict(vil)
```

```python
%%capture off
import cartopy.crs as crs
from cartopy.crs import Globe
import cartopy.feature as cfeature
from matplotlib import animation, rc
def make_mrms_animation(frames,img_type='vil',fig=None,
                   interval=100,title=None,**kwargs):
    """
    Makes animation of MRMS data
    """
    ellps='WGS84'
    globe=Globe(ellipse=ellps)
    proj=crs.PlateCarree(globe=globe)
    img_extent=(-14471586.0, -6679194.0322265625, 2226397.75, 6122594.1611328125) # MRMS extent
    xll,xur=img_extent[0],img_extent[1]
    yll,yur=img_extent[2],img_extent[3]
    if fig is None:
        fig=plt.gcf()
    ax=fig.add_subplot(1,1,1,projection=proj)
    ax.set_xlim((xll,xur))
    ax.set_ylim((yll,yur))
    cmap,norm,vmin,vmax=get_cmap(img_type)
    ax.add_feature(cfeature.STATES)
    ax.add_feature(cfeature.LAKES, alpha=0.5)
    ax.add_feature(cfeature.RIVERS, alpha=0.5)
    ax.add_feature(cfeature.COASTLINE)
    ax.add_feature(cfeature.BORDERS )

    im=ax.imshow(frames[:,:,0], interpolation='nearest',
          origin='upper', extent=[xll,xur,yll,yur],
          transform=proj,cmap=cmap,norm=norm,vmin=vmin,vmax=vmax);

    def init():
        return (im,)
    def animate(i):
        im.set_data(frames[:,:,i]);
        ftype='Analysis' if i<13 else 'Forecast'
        ax.set_title(ftype)
        return (im,)
    return animation.FuncAnimation(fig, animate, init_func=init,
                                   frames=range(frames.shape[2]),
                                   interval=interval, blit=True);

frames = np.concatenate((vil[:,2:-2,4:-4,:],forecast[:,2:-2,4:-4,:]),axis=3)
for i in range(frames.shape[3]):
    frames[:,:,:,i][m]=np.nan
plt.figure(figsize=(15,7))
anim = make_mrms_animation(frames[0])
anim.save('imgs/mrms_animation.gif', writer='imagemagick', fps=6)
```

```python
Image('imgs/mrms_animation.gif')
```


#  Pawnee Fire Analysis

> * 🔬 Data Science
* 🖥 Requires RasterAnalytics Portal Configuration
* 🖥 Requires GeoEnrichment Portal Configuration
* 🖥 Requires GeoAnalytics Portal Configuration

The Pawnee Fire was a large wildfire that burned in Lake County, California. The fire started on June 23, 2018 and burned a total of 15,185 acres (61 km2) before it was fully contained on July 8, 2018.

Table of Contents

Pawnee Fire Analysis

Remote Sensing using Sentinel-2 data

Data Preparation

Visual Assessment

Quantitative Assessment

Impact Assessment

Conclusion

## Remote Sensing using Sentinel-2 data

```python
from datetime import datetime
import warnings

from IPython.display import HTML
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import arcgis
from arcgis import GIS
from arcgis.raster.functions import *
from arcgis.geoanalytics.use_proximity import create_buffers
from arcgis.geoenrichment import enrich
from arcgis.features import SpatialDataFrame
from arcgis.raster.analytics import create_image_collection
from arcgis.raster.analytics import list_datastore_content

gis= GIS("home")
```

### Data Preparation

In this analysis, we will be using Sentinel-2 data.

Sentinel-2 is an Earth observation mission developed by ESA as part of the Copernicus Programme to perform terrestrial observations in support of services such as forest monitoring, land cover change detection, and natural disaster management.

In this analysis data downloaded from https://earthexplorer.usgs.gov/ is used for creating hosted image service.
We add the data to the datastore and we then run the create_image_collection function which creates a collection with the input_rasters specified and publishes the collection as an image service.

We use list_datasore_content() in order to see the contents in the rasterstore.

```python
list_datastore_content("/rasterStores/LocalRasterStore")
```

    ['/rasterStores/LocalRasterStore/Hosted_GeneratedRasterProduct_AANS52.crf/',
     '/rasterStores/LocalRasterStore/L1C_T10SEJ_A015697_20180624T190108.zip',
     '/rasterStores/LocalRasterStore/me.txt',
     '/rasterStores/LocalRasterStore/m_7013bfcd7273e0a4a779fce061167d5c/',
     '/rasterStores/LocalRasterStore/m_b5f745ad6ef601d5e6adf104c8b4ef70/',
     '/rasterStores/LocalRasterStore/m_d0221b176ab5ed2398828b8079d62ef8/',
     '/rasterStores/LocalRasterStore/pawnee_fire_multispectral/',
     '/rasterStores/LocalRasterStore/pool_chips_1/',
     '/rasterStores/LocalRasterStore/pool_chips_2/',
     '/rasterStores/LocalRasterStore/S2A_MSIL1C_20180624T184921_N0206_R113_T10SEJ_20180624T234856.SAFE/',
     '/rasterStores/LocalRasterStore/S2B_MSIL1C_20180622T185919_N0206_R013_T10SEJ_20180622T205930.SAFE/',
     '/rasterStores/LocalRasterStore/sentinel_data/']

```python
sentinel_collection = create_image_collection(image_collection="pawnee_fire_multispectral",
                      input_rasters=["/rasterStores/LocalRasterStore/S2A_MSIL1C_20180624T184921_N0206_R113_T10SEJ_20180624T234856.SAFE",
                                     "/rasterStores/LocalRasterStore/S2B_MSIL1C_20180622T185919_N0206_R013_T10SEJ_20180622T205930.SAFE"],
                      raster_type_name="Sentinel-2",
                      raster_type_params={"productType":"All","processingTemplate":"Multispectral"},
                      context={"image_collection_properties":{"imageCollectionType":"Satellite"},"byref":True}, gis = gis)
```

```python
sentinel = sentinel_collection.layers[0]
```

### Select before and after rasters

```python
aoi = {'spatialReference': {'latestWkid': 3857, 'wkid': 102100},
 'xmax': -13643017.100720055,
 'xmin': -13652113.10708598,
 'ymax': 4739654.477447927,
 'ymin': 4731284.622850712}
arcgis.env.analysis_extent = aoi
sentinel.extent = aoi
```

```python
selected = sentinel.filter_by(where="acquisitiondate BETWEEN timestamp '2018-06-15 00:00:00' AND timestamp '2018-06-24 19:59:59'",
                              geometry=arcgis.geometry.filters.intersects(aoi))

df = selected.query(out_fields="*", order_by_fields="OBJECTID ASC").df
df['AcquisitionDate'] = pd.to_datetime(df['AcquisitionDate'], unit='ms')
df.tail(40)
```

      AcquisitionDate
      Category
      CenterX
      CenterY
      CloudCover
      CreationTime
      Creator
      GroupName
      HighPS
      LowPS
      ...
      SOrder
      SensingOrbit
      SensorName
      Shape_Area
      Shape_Length
      StereoID
      Tag
      Thumbnail
      ZOrder
      SHAPE

      0
      2018-06-24 18:49:21.024
      1
      -1.362127e+07
      4.758219e+06
      0
      1548778472000
      rjackson
      MTD_MSIL1C
      60
      60
      ...
      None
      113
      Sentinel-2A
      2.008228e+10
      562920.108277
      None
      MS
      None
      None
      {'rings': [[[-13549665.930399999, 4828694.125]...

      1
      2018-06-22 18:59:19.024
      1
      -1.364188e+07
      4.762904e+06
      0
      1548778472000
      rjackson
      MTD_MSIL1C
      60
      60
      ...
      None
      13
      Sentinel-2B
      1.417943e+10
      488606.729731
      None
      MS
      None
      None
      {'rings': [[[-13611168.835900001, 4689604.0966...

2 rows × 26 columns

```python
prefire = sentinel.filter_by('OBJECTID=2')
midfire = sentinel.filter_by('OBJECTID=1')
```

## Visual Assessment

```python
truecolor = extract_band(midfire, [4,3,2])
truecolor
```

### Visualize Burn Scars

We extract the [13, 12, 4] bands to improve visibility of fire and burn scars. This band combination pushes further into the SWIR range of the electromagnetic spectrum, where there is less susceptibility to smoke and haze generated by a burning fire.

```python
extract_band(midfire, [13,12,4])

```

For comparison, the same area before the fire started shows no burn scar.

```python
extract_band(prefire, [13,12,4])

```

## Quantitative Assessment

The **Normalized Burn Ratio (NBR)** can be used to delineate the burnt areas and identify the severity of the fire.

The formula for the NBR is very similar to that of NDVI except that it uses near-infrared band 9 and the short-wave infrared band 13:
\begin{align}
{\mathbf{NBR}} = \frac{\mathbf{B9} - \mathbf{B13}}{\mathbf{B9} + \mathbf{B13} + \mathbf{WS}} \\
\end{align}

The NBR equation was designed to be calcualted from reflectance, but it can be calculated from radiance and digital_number_(dn) with changes to the burn severity table below. The WS parameter is used for water suppression, and is typically 2000.

For a given area, NBR is calculated from an image just prior to the burn and a second NBR is calculated for an image immediately following the burn. Burn extent and severity is judged by taking the difference between these two index layers:

\begin{align}
{\Delta \mathbf{NBR}} = \mathbf{NBR_{prefire}} - \mathbf{NBR_{postfire}} \\
\end{align}

The meaning of the ∆NBR values can vary by scene, and interpretation in specific instances should always be based on some field assessment. However, the following table from the USGS FireMon program can be useful as a first approximation for interpreting the NBR difference:

| \begin{align}{\Delta \mathbf{NBR}}  \end{align}      | Burn Severity |
| ------------- |:-------------:|
| 0.1 to 0.27   | Low severity burn |
| 0.27 to 0.44  | Medium severity burn |
| 0.44 to 0.66 | Moderate severity burn |
| > 0.66 | High severity burn |

[Source: http://wiki.landscapetoolbox.org/doku.php/remote_sensing_methods:normalized_burn_ratio]

### Use Band Arithmetic and Map Algebra

In order to perform raster analysis on raw pixel value, we filter out the scenes from the sentinel image service again and create new layers

```python
nbr_prefire  = band_arithmetic(prefire, "(b9 - b13) / (b9 + b13 + 2000)")
nbr_postfire = band_arithmetic(midfire, "(b9 - b13) / (b9 + b13 + 2000)")

nbr_diff = nbr_prefire - nbr_postfire
```

```python
burnt_areas = colormap(remap(nbr_diff,
                             input_ranges=[0.1,  0.27,  # low severity
                                           0.27, 0.44,  # medium severity
                                           0.44, 0.66,  # moderate severity
                                           0.66, 1.00], # high severity burn
                             output_values=[1, 2, 3, 4],
                             no_data_ranges=[-1, 0.1], astype='u8'),
                       colormap=[[4, 0xFF, 0xC3, 0], [3, 0xFA, 0x8E, 0], [2, 0xF2, 0x55, 0], [1, 0xE6, 0,    0]])
```

```python
# Visualize burnt areas
burnt_areas
```

With this, we have computed the NBR on scenes from before and after the burn, and computed the NBR difference to identify places that have been affected by the fire. We've also normalized the values to match a burn severity index, and applied a color map that brings out the extent of fire damage.

### Area calculation

```python
pixx = (aoi['xmax'] - aoi['xmin']) / 1200.0
pixy = (aoi['ymax'] - aoi['ymin']) / 450.0

res = burnt_areas.compute_histograms(aoi, pixel_size={'x':pixx, 'y':pixy})

numpix = 0
histogram = res['histograms'][0]['counts'][1:]
for i in histogram:
    numpix += i
```

### Report burnt area

```python
sqmarea = numpix * pixx * pixy # in sq. m
acres = 0.00024711 * sqmarea   # in acres

HTML('<h3>Fire has consumed <font color="red">{:,} acres</font>  till {}</h3>.' \
     .format(int(acres), df.iloc[-1]['AcquisitionDate'].date()))
```

Fire has consumed 3,569 acres  till 2018-06-22
.

```python
%matplotlib inline

plt.title('Distribution by severity', y=-0.1)
plt.pie(histogram, labels=['Low Severity', 'Medium Severity', 'Moderate Severity', 'High Severity']);
plt.axis('equal');
```

### Visualize burnt areas

```python
firemap = gis.map()
firemap.extent = aoi
firemap.add_layer([truecolor, burnt_areas])

firemap
```

### Persist the burnt areas layer in the GIS

If required, using the save(), we can persist the output in the gis as a new layer. This uses distributed raster analysis to perform the analysis at the source resolution.

```python
burnt_areas = burnt_areas.save()
```

## Raster to Feature layer conversion

Use Raster Analytics and Geoanalytics to convert the burnt area raster to a feature layer. The `to_features()` method converts the raster to a feature layer and `create_buffers()` fills holes in the features and dissolves them to output one feature that covers the extent of the Pawnee Fire.

```python
burnt_areas = burnt_areas.layers[0]
fire_item = burnt_areas.to_features(output_name='Pawnee_Fire_Feature_Layer', gis=gis)
fire_layer = fire_item.layers[0]

```

```python
fire = create_buffers(fire_layer, 100, 'Meters', dissolve_option='All', multipart=True, output_name='PawneeFireArea_Buffer')
fire = fire.layers[0]
```

## Visualize Feature Layer

```python
vectormap = gis.map()
vectormap.basemap = 'dark-gray'
vectormap.extent  = aoi

vectormap.add_layer(fire)
vectormap
```

## Impact Assessment

### Assess Human Impact

```python
from arcgis import geometry

sdf = SpatialDataFrame.from_layer(fire)

fire_geometry = sdf.iloc[0].SHAPE
sa_filter = geometry.filters.intersects(geometry=fire_geometry, sr=4326)

def age_pyramid(df):
    %matplotlib inline
    warnings.simplefilter(action='ignore', category=FutureWarning)
    pd.options.mode.chained_assignment = None
    plt.style.use('ggplot')

    df = df[[x for x in impacted_people.columns if 'MALE' in x or 'FEM' in x]]
    sf = pd.DataFrame(df.sum())
    age = sf.index.str.extract('(\d+)').astype('int64')
    f = sf[sf.index.str.startswith('FEM')]
    m = sf[sf.index.str.startswith('MALE')]
    sf = sf.reset_index(drop = True)
    f = f.reset_index(drop = True)
    m = m.reset_index(drop = True)
    sf['age'] = age
    f["age"] = age
    m["age"] = age
    f = f.sort_values(by='age', ascending=False).set_index('age')
    m = m.sort_values(by='age', ascending=False).set_index('age')

    popdf = pd.concat([f, m], axis=1)
    popdf.columns = ['F', 'M']
    popdf['agelabel'] = popdf.index.map(str) + ' - ' + (popdf.index+4).map(str)
    popdf.M = -popdf.M

    sns.barplot(x="F", y="agelabel", color="#CC6699", label="Female", data=popdf, edgecolor='none')
    sns.barplot(x="M",  y="agelabel", color="#008AB8", label="Male",   data=popdf,  edgecolor='none')
    plt.ylabel('Age group')
    plt.xlabel('Number of people');
    return plt;
```

### Age Pyramid of Affected Population

```python
impacted_people = enrich(sdf, 'Age')
age_pyramid(impacted_people);
```

# Conclusion

In this notebook example, we used Sentinel-2 data in order to perform remote sensing. For this we filtered out pre and post fire scenes. Using extract_band() we carried out visual assessment of the burnt area. We then computed the NBR on these scenes and computed the NBR difference to identify places that have been affected by the fire, using raster functions. We also normalized the values to match the burn severity index, applied a color map raster function that brings out the extent of fire damage and calculated the burnt area. Finally, we carried out a human impact assessment by plotting the age pyramid of affected population


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

It is sometimes very difficult to unequivocally attribute outcomes to causal
factors. For example, did your  experiment generate the outcome you were hoping
for or not?  Maybe something did happen, but the effect is not pronounced
enough
to separate it from inescapable measurement errors or other
factors in the
ambient environment?  Hypothesis testing is a powerful
statistical method to
address these questions.  Let's begin by again
considering our coin-tossing
experiment with unknown parameter $p$.  Recall
that the individual coin-flips
are Bernoulli distributed.  The first step is
to establish separate hypotheses.
First, $H_0$ is the so-called null
hypothesis. In our case this can be

$$
H_0 \colon \theta <  \frac{1}{2}
$$

 and the alternative hypothesis is then

$$
H_1 \colon \theta \geq  \frac{1}{2}
$$

 With this set up, the question now boils down to figuring out which
hypothesis
the data is most consistent with.  To choose between these, we need
a
statistical test that is a function, $G$, of the sample set
$\mathbf{X}_n=\left\{ X_i \right\}_n $ into the real line, where $X_i$ is the
heads or tails outcome ($X_i \in \lbrace 0,1 \rbrace$). In other words, we
compute $G(\mathbf{X}_n)$ and check if it exceeds a threshold $c$. If not, then
we declare $H_0$ (otherwise, declare $H_1$). Notationally, this is the
following:

$$
\begin{align*}
 G(\mathbf{X}_n) < c & \Rightarrow H_0   \\\
 G(\mathbf{X}_n)
\geq c & \Rightarrow H_1
\end{align*}
$$

 In summary, we have the observed data $\mathbf{X}_n$ and a function
$G$ that
maps that data onto the real line. Then, using the
constant $c$ as a threshold,
the inequality effectively divides the real line
into two parts, one
corresponding to each of the hypotheses.

Whatever this test $G$ is, it will
make mistakes of two types --- false
negatives and false positives. The false
positives arise from the case where we
declare $H_0$ when the test says we
should declare $H_1$.  This is
summarized in the Table [1](#tbl:decision).

$$
\begin{table}
\footnotesize
\centering
\begin{tabular}{l|p{1.3in}|p{1.3in}}
\multicolumn{1}{c}{ } & \multicolumn{1}{c}{Declare $H_0$ } & \multicolumn{1}{c}{
Declare $H_1$ } \\
\hline
$H_0\:$ True & Correct                        & False
positive (Type I error) \\
\hline
$H_1\:$ True & False negative (Type II error)
& Correct (true-detect)      \\
\hline
\end{tabular}
\caption{Truth table for
hypotheses testing.}
\label{tbl:decision} \tag{1}
\end{table}
$$

 For this example, here are the false positives (aka false alarms):

$$
P_{FA} = \mathbb{P}\left( G(\mathbf{X}_n) > c \mid \theta \leq \frac{1}{2}
\right)
$$

 Or, equivalently,

$$
P_{FA} = \mathbb{P}\left( G(\mathbf{X}_n) > c \mid H_0 \right)
$$

 Likewise, the other error is a false negative, which we can write
analogously
as

$$
P_{FN} = \mathbb{P}\left( G(\mathbf{X}_n) < c \vert H_1\right)
$$

 By choosing some acceptable values for either of these errors,
we can solve for
the other one.  The practice is usually to pick a value of
$P_{FA}$ and then
find the corresponding value of $P_{FN}$.  Note that it is
traditional in
engineering to speak about *detection probability*, which is
defined as

$$
P_{D} = 1- P_{FN} = \mathbb{P}\left( G(\mathbf{X}_n) > c \mid H_1\right)
$$

 In other words, this is the probability of declaring $H_1$ when the
test
exceeds the threshold. This is otherwise known as the *probability of a
true
detection* or *true-detect*.

## Back to the Coin Flipping Example

In our
previous maximum likelihood discussion, we wanted to derive an
estimator for the
*value* of the probability of heads for the coin
flipping experiment. For
hypthesis testing, we want to ask a softer
question: is the probability of heads
greater or less than $\nicefrac{1}{2}$? As we
just established, this leads to
the two hypotheses:

$$
H_0 \colon \theta < \frac{1}{2}
$$

 versus,

$$
H_1 \colon \theta > \frac{1}{2}
$$

 Let's assume we have five observations.  Now we need the $G$ function
and a
threshold $c$ to help pick between the two hypotheses. Let's count the
number of
heads observed in five observations as our
criterion. Thus, we have

$$
G(\mathbf{X}_5) := \sum_{i=1}^5 X_i
$$

 and, suppose further that we pick $H_1$  only if exactly five out of
five
observations are heads. We'll call this the *all-heads* test.

Now, because all
of the $X_i$ are random variables, so is $G$ and we must
find the corresponding
probability mass function for $G$.  Assuming the
individual coin tosses are
independent, the probability of five heads is $\theta^5$.
This means that the
probability of rejecting the $H_0$ hypothesis (and choosing
$H_1$, because there
are only two choices here) based on the unknown underlying
probability is
$\theta^5$. In the parlance, this is known and the *power function*
as in
denoted by $\beta$ as in

$$
\beta(\theta) = \theta^5
$$

 Let's get a quick plot this in [Figure](#fig:Hypothesis_testing_001).

```python
%matplotlib inline

from matplotlib.pylab import subplots
import numpy as np
fig,ax=subplots()
fig.set_size_inches((6,3))
xi = np.linspace(0,1,50)
_=ax.plot(xi, (xi)**5,'-k',label='all heads')
_=ax.set_xlabel(r'$\theta$',fontsize=22)
_=ax.plot(0.5,(0.5)**5,'ko')
fig.tight_layout()
fig.savefig('fig-statistics/Hypothesis_Testing_001.png')
```

Power function for the all-heads
test. The dark circle indicates the value of the function indicating
$\alpha$.

 Now, we have the following false alarm probability,

$$
P_{FA} = \mathbb{P}( G(\mathbf{X}_n)= 5 \vert H_0) =\mathbb{P}( \theta^5
\vert H_0)
$$

 Notice that this is a function of $\theta$, which means there are
many false
alarm probability values that correspond to this test. To be on the
conservative
side, we'll pick the supremum (i.e., maximum) of this function,
which is known
as the *size* of the test, traditionally denoted by $\alpha$,

$$
\alpha = \sup_{\theta \in \Theta_0} \beta(\theta)
$$

 with domain $\Theta_0 = \lbrace \theta < 1/2 \rbrace$ which in our case is

$$
\alpha = \sup_{\theta < \frac{1}{2}} \theta^5 = \left(\frac{1}{2}\right)^5 =
0.03125
$$

 Likewise, for the detection probability,

$$
\mathbb{P}_{D}(\theta) = \mathbb{P}( \theta^5 \vert H_1)
$$

 which is again a function of the parameter $\theta$. The problem with
this test
is that the $P_{D}$ is pretty low for most of the domain of
$\theta$. For
instance, values in the nineties for $P_{D}$
only happen when $\theta > 0.98$.
In other words, if the coin produces
heads 98 times out of 100, then we can
detect $H_1$ reliably. Ideally, we want
a test that is zero for the domain
corresponding to $H_0$ (i.e., $\Theta_0$) and
equal to one otherwise.
Unfortunately, even if we increase the length of the
observed sequence, we
cannot escape this effect with this test. You can try
plotting $\theta^n$ for
larger and larger values of $n$ to see this.

### Majority Vote Test

Due to the
problems with the detection probability in the all-heads test, maybe
we can
think of another test that will have the performance we want? Suppose we
reject
$H_0$ if the majority of the observations are heads. Then, using the
same
reasoning as above, we have

$$
\beta(\theta) = \sum_{k=3}^5 \binom{5}{k} \theta^k(1-\theta)^{5-k}
$$

[Figure](#fig:Hypothesis_testing_002) shows the power function
for both the
majority vote and the all-heads tests.

```python
fig,ax=subplots()
fig.set_size_inches((6,3))
from sympy.abc import theta,k # get some variable symbols
import sympy as S
xi = np.linspace(0,1,50)
expr=S.Sum(S.binomial(5,k)*theta**(k)*(1-theta)**(5-k),(k,3,5)).doit()
_=ax.plot(xi, (xi)**5,'-k',label='all heads')
_=ax.plot(xi, S.lambdify(theta,expr)(xi),'--k',label='majority vote')
_=ax.plot(0.5, (0.5)**5,'ko')
_=ax.plot(0.5, S.lambdify(theta,expr)(0.5),'ko')
_=ax.set_xlabel(r'$\theta$',fontsize=22)
_=ax.legend(loc=0)
fig.tight_layout()
fig.savefig('fig-statistics/Hypothesis_Testing_002.png')
```

Compares the power
function for the all-heads test with that of the majority-vote test.

In this case, the new test has *size*

$$
\alpha = \sup_{\theta < \frac{1}{2}} \theta^{5} + 5 \theta^{4} \left(- \theta
+ 1\right) + 10 \theta^{3} \left(- \theta + 1\right)^{2} = \frac{1}{2}
$$

 As before we only get to upwards of 90% for detection
probability only when the
underlying parameter $\theta > 0.75$.
Let's see what happens when we consider
more than five samples. For
example, let's suppose that we have $n=100$ samples
and we want to
vary the threshold for the majority vote test. For example, let's
have
a new test where we declare $H_1$ when $k=60$ out of the 100 trials
turns
out to be heads. What is the $\beta$ function in this case?

$$
\beta(\theta) = \sum_{k=60}^{100} \binom{100}{k} \theta^k(1-\theta)^{100-k}
$$

 This is too complicated to write by hand, but the statistics module
in Sympy
has all the tools we need to compute this.

```python
from sympy.stats import P, Binomial
theta = S.symbols('theta',real=True)
X = Binomial('x',100,theta)
beta_function = P(X>60)
print (beta_function.subs(theta,0.5)) # alpha
print (beta_function.subs(theta,0.70))
```

    0.0176001001088524
    0.979011423996075

These results are much better than before because the $\beta$
function is much
steeper. If we declare $H_1$ when we observe 60 out of 100
trials are heads,
then we wrongly declare heads approximately 1.8% of the
time.  Otherwise, if it
happens that the true value for $p>0.7$, we will
conclude correctly
approximately 97% of the time. A quick simulation can sanity
check these results
as shown below:

```python
from scipy import stats
rv=stats.bernoulli(0.5) # true p = 0.5
# number of false alarms ~ 0.018
print (sum(rv.rvs((1000,100)).sum(axis=1)>60)/1000.)
```

    0.015

The above code is pretty dense so let's unpack it. In the first line, we use
the `scipy.stats` module to define the
Bernoulli random variable for the coin
flip. Then, we use the `rvs` method of
the variable to generate 1000 trials of
the experiment where each trial
consists of 100 coin flips. This generates a
$1000 \times 100$ matrix where the
rows are the individual trials and the
columns are the outcomes of each
respective set of 100 coin flips. The
`sum(axis=1)` part computes the sum across the
columns. Because the values of
the embedded matrix are only `1` or `0` this
gives us the count of flips that
are heads per row. The next `>60` part
computes the boolean 1000-long vector of
values that are bigger than 60. The
final `sum` adds these up. Again, because
the entries in the array are `True`
or `False` the `sum` computes the count of
times the number of heads has
exceeded 60 per 100 coin flips in each of 1000
trials. Then, dividing this
number by 1000 gives a quick approximation of false
alarm probability we
computed above for this case where the true value of
$p=0.5$.

## Receiver Operating Characteristic

Because the majority vote test
is a binary test, we can compute the *Receiver
Operating Characteristic* (ROC)
which is the graph of the $(P_{FA},
P_D)$. The term comes from radar systems but
is a very general method for
consolidating all of these issues into a single
graph. Let's consider a typical
signal processing example with two hypotheses.
In $H_0$, there is noise but no
signal present at the receiver,

$$
H_0 \colon X =  \epsilon
$$

 where $\epsilon \sim \mathcal{N}(0,\sigma^2)$ represents additive
noise. In the
alternative hypothesis, there is a  deterministic signal at the receiver,

$$
H_1 \colon X = \mu + \epsilon
$$

 Again, the problem is to choose between these two hypotheses. For
$H_0$, we
have $X \sim \mathcal{N}(0,\sigma^2)$ and for $H_1$, we have $ X \sim
\mathcal{N}(\mu,\sigma^2)$.  Recall that we only observe values for $x$ and
must
pick either $H_0$ or $H_1$ from these observations. Thus, we need a
threshold,
$c$, to compare $x$ against in order to distinguish the two
hypotheses.
[Figure](#fig:Hypothesis_testing_003) shows the probability density
functions
under each of the hypotheses. The dark vertical line is the threshold
$c$. The
gray shaded area is the probability of detection, $P_D$ and the shaded
area is
the probability of false alarm, $P_{FA}$. The test evaluates every
observation
of $x$ and concludes $H_0$ if $x<c$ and $H_1$ otherwise.

The two density functions for the
$H_0$ and $H_1$ hypotheses. The shaded gray area is the detection probability
and the shaded dark gray area is the probability of false alarm. The vertical
line is the decision threshold.

**Programming Tip.**

The shading shown in [Figure](#fig:Hypothesis_testing_003)
comes from
Matplotlib's `fill_between` function. This function has a `where`
keyword
argument to specify which part of the plot to apply shading with
specified
`color` keyword argument. Note there is also a `fill_betweenx`
function that
fills horizontally.  The `text` function can place formatted
text
anywhere in the plot and can utilize basic \LaTeX{} formatting.

As we slide
the threshold left and right along the horizontal axis, we naturally change the
corresponding areas under
each of the curves shown in
[Figure](#fig:Hypothesis_testing_003) and thereby
change the values of $P_D$ and
$P_{FA}$. The contour that emerges from sweeping
the threshold this way is the
ROC as shown in [Figure](#fig:Hypothesis_testing_004). This figure also shows
the diagonal line which
corresponds to making decisions based on the flip of a
fair coin. Any
meaningful test must do better than coin flipping so the more the
ROC bows up
to the top left corner of the graph, the better. Sometimes ROCs are
quantified
into a single number called the *area under the curve* (AUC), which
varies from
0.5 to 1.0 as shown. In our example, what separates the two
probability density
functions is the value of $\mu$. In a real situation, this
would be determined
by signal processing methods that include many complicated
trade-offs. The key
idea is that whatever those trade-offs are, the test itself
boils down to the
separation between these two density functions --- good tests
separate the two
density functions and bad tests do not. Indeed, when there is
no separation, we
arrive at the diagonal-line coin-flipping situation we just
discussed.

What values for $P_D$ and $P_{FA}$ are considered *acceptable*
depends on the
application. For example, suppose you are testing for a fatal
disease. It could
be that you are willing to except a relatively high $P_{FA}$
value if that
corresponds to a good $P_D$ because the test is relatively cheap
to administer
compared to the alternative of missing a detection. On the other
hand,
may be a false alarm triggers an expensive response, so that minimizing
these alarms is more important than potentially missing a detection. These
trade-offs can only be determined by the application and design factors.

The Receiver Operating Characteristic
(ROC) corresponding to [Figure](#fig:Hypothesis_testing_003).

##
P-Values

There are a lot of moving parts in hypothesis testing. What we need
is
a way to consolidate the findings. The idea is that we want to find
the minimum
level at which the test rejects $H_0$. Thus, the p-value
is the probability,
under $H_0$, that the test-statistic is at least
as extreme as what was actually
observed.  Informally, this means
that smaller values imply that $H_0$ should be
rejected, although
this doesn't mean that large values imply that $H_0$ should
be
retained. This is because a large p-value can arise from either $H_0$
being
true or the test having low statistical power.

If $H_0$ is true, the p-value is
uniformly distributed in the interval $(0,1)$.
If $H_1$ is true, the
distribution of the p-value will concentrate closer to
zero. For continuous
distributions, this can be proven rigorously and implies
that if we reject $H_0$
when the corresponding p-value is less than $\alpha$,
then the probability of a
false alarm is $\alpha$. Perhaps it helps to
formalize this a bit before
computing it. Suppose $\tau(X)$ is a test
statistic that rejects $H_0$ as it
gets bigger. Then, for each sample $x$,
corresponding to the data we actually
have on-hand, we define

$$
p(x) = \sup_{\theta \in \Theta_0} \mathbb{P}_{\theta}(\tau(X) > \tau(x))
$$

 This equation states that the supremum (i.e., maximum)
probability that the
test statistic, $\tau(X)$, exceeds the value for
the test statistic on this
particular data ($\tau(x)$) over the
domain $\Theta_0$ is defined as the
p-value. Thus, this embodies a
worst-case scenario over all values of $\theta$.
Here's one way to think about this. Suppose you rejected $H_0$, and someone
says
that you just got *lucky* and somehow just drew data that happened to
correspond
to a rejection of $H_0$.   What p-values provide is a way to address
this by
capturing the odds of just a favorable data-draw.  Thus, suppose that
your
p-value is 0.05. Then, what you are showing is that the odds of just
drawing
that data sample, given $H_0$ is in force, is just 5%. This means that
there's a
5% chance that you somehow lucked out and got a favorable draw of
data.

Let's
make this concrete with an example. Given, the majority-vote rule above,
suppose
we actually do observe three of five heads. Given the $H_0$, the
probability of
observing this event is the following:

$$
p(x) =\sup_{\theta \in \Theta_0} \sum_{k=3}^5\binom{5}{k}
\theta^k(1-\theta)^{5-k} = \frac{1}{2}
$$

 For the all-heads test, the corresponding computation is the following:

$$
p(x) =\sup_{\theta \in \Theta_0} \theta^5 = \frac{1}{2^5} = 0.03125
$$

From just looking at these p-values, you might get the feeling that the second
test is better, but we still have the same detection probability issues we
discussed above; so, p-values help in summarizing some aspects of our
hypothesis
testing, but they do *not* summarize all the salient aspects of the
*entire*
situation.

## Test Statistics

As we have seen, it is difficult to derive good
test statistics for hypothesis
testing without a systematic process.  The
Neyman-Pearson Test is derived from
fixing a false-alarm value ($\alpha$) and
then maximizing  the detection
probability. This results in the Neyman-Pearson
Test,

$$
L(\mathbf{x}) = \frac{f_{X|H_1}(\mathbf{x})}{f_{X|H_0}(\mathbf{x})}
\stackrel[H_0]{H_1}{\gtrless} \gamma
$$

 where $L$ is the likelihood ratio and where the threshold
$\gamma$ is chosen
such that

$$
\int_{x:L(\mathbf{x})>\gamma} f_{X|H_0}(\mathbf{x}) d\mathbf{x}=\alpha
$$

 The Neyman-Pearson Test is one of a family of tests that use
the likelihood
ratio.

**Example.** Suppose we have a receiver and we want to distinguish
whether just noise ($H_0$) or signal pluse noise ($H_1$) is received.
For the
noise-only case, we have  $x\sim \mathcal{N}(0,1)$ and for the
signal pluse
noise case we have $x\sim \mathcal{N}(1,1)$.  In other
words, the mean of the
distribution shifts in the presence of the
signal. This is a very common problem
in signal processing and
communications. The Neyman-Pearson Test then boils down
to the
following,

$$
L(x)= e^{-\frac{1}{2}+x}\stackrel[H_0]{H_1}{\gtrless}\gamma
$$

 Now we have to find the threshold $\gamma$ that solves the
maximization problem
that characterizes the Neyman-Pearson Test. Taking
the natural  logarithm and
re-arranging gives,

$$
x\stackrel[H_0]{H_1}{\gtrless} \frac{1}{2}+\log\gamma
$$

 The next step is find $\gamma$ corresponding to the desired
$\alpha$ by
computing it from the following,

$$
\int_{1/2+\log\gamma}^{\infty} f_{X|H_0}(x)dx = \alpha
$$

 For example, taking $\alpha=1/100$, gives
$\gamma\approx 6.21$. To summarize
the test in this case, we have,

$$
x\stackrel[H_0]{H_1}{\gtrless} 2.32
$$

 Thus, if we measure $X$ and see that its value
exceeds the threshold above, we
declare $H_1$ and otherwise
declare $H_0$. The following code shows how to
solve
this example using Sympy and Scipy. First, we
set up the likelihood ratio,

```python
import sympy as S
from sympy import stats
s = stats.Normal('s',1,1) # signal+noise
n = stats.Normal('n',0,1) # noise
x = S.symbols('x',real=True)
L = stats.density(s)(x)/stats.density(n)(x)
```

Next, to find the $\gamma$ value,

```python
g = S.symbols('g',positive=True) # define gamma
v=S.integrate(stats.density(n)(x),
             (x,S.Rational(1,2)+S.log(g),S.oo))
```

**Programming Tip.**

Providing additional information regarding the Sympy
variable by using the
keyword argument `positive=True` helps the internal
simplification algorithms
work faster and better. This is especially useful when
dealing with complicated
integrals that involve special functions. Furthermore,
note that we used the
`Rational` function to define the `1/2` fraction, which is
another way of
providing hints to Sympy. Otherwise, it's possible that the
floating-point
representation of the fraction could  disguise the simple
fraction and
thereby miss internal simplification opportunities.

 We want to
solve for `g` in the above expression. Sympy has some
built-in numerical solvers
as in the following,

```python
print (S.nsolve(v-0.01,3.0)) # approx 6.21
```

    6.21116124253284

Note that in this situation it is better to use the numerical
solvers because
Sympy `solve` may grind along for a long time to
resolve this.

### Generalized
Likelihood Ratio Test

The likelihood ratio test can be generalized using the
following statistic,

$$
\Lambda(\mathbf{x})= \frac{\sup_{\theta\in\Theta_0}
L(\theta)}{\sup_{\theta\in\Theta}
L(\theta)}=\frac{L(\hat{\theta}_0)}{L(\hat{\theta})}
$$

 where $\hat{\theta}_0$ maximizes $L(\theta)$ subject to
$\theta\in\Theta_0$ and
$\hat{\theta}$ is the maximum likelihood estimator.
The intuition behind this
generalization of the Likelihood Ratio Test is that
the denomimator is the usual
maximum likelihood estimator and the numerator is
the maximum likelihood
estimator, but over a restricted domain ($\Theta_0$).
This means that the ratio
is always less than unity because the maximum
likelihood estimator over the
entire space will always be at least as maximal
as that over the more restricted
space. When this $\Lambda$ ratio gets small
enough, it means that the maximum
likelihood estimator over the entire domain
($\Theta$) is larger which means
that it is safe to reject the null hypothesis
$H_0$.  The tricky part is that
the statistical distribution of $\Lambda$ is
usually eye-wateringly difficult.
Fortunately, Wilks Theorem says that with
sufficiently large $n$, the
distribution of $-2\log\Lambda$ is approximately
chi-square with $r-r_0$ degrees
of freedom, where $r$ is the number of free
parameters for $\Theta$ and $r_0$ is
the number of free parameters in
$\Theta_0$.  With this result, if we want an
approximate test at level
$\alpha$, we can reject $H_0$ when $-2\log\Lambda \ge
\chi^2_{r-r_0}(\alpha)$
where $\chi^2_{r-r_0}(\alpha)$ denotes the $1-\alpha$
quantile of the
$\chi^2_{r-r_0}$ chi-square distribution.  However, the problem
with this
result is that there is no definite way of knowing how big $n$ should
be. The
advantage of this generalized likelihood ratio test is that it
can test
multiple hypotheses simultaneously, as illustrated
in the following example.
**Example.** Let's return to our coin-flipping example, except now we have
three
different coins. The likelihood function is then,

$$
L(p_1,p_2,p_3) =
\texttt{binom}(k_1;n_1,p_1)\texttt{binom}(k_2;n_2,p_2)\texttt{binom}(k_3;n_3,p_3)
$$

 where $\texttt{binom}$ is the binomial distribution with
the given parameters.
For example,

$$
\texttt{binom}(k;n,p) =\sum_{k=0}^n \binom{n}{k} p^k(1-p)^{n-k}
$$

  The null hypothesis is that all three coins have the
same probability of
heads, $H_0:p=p_1=p_2=p_3$. The alternative hypothesis is
that at least one of
these probabilites is different. Let's consider the
numerator of the $\Lambda$
first, which will give us the maximum likelihood
estimator of $p$. Because the
null hypothesis is that all the $p$ values are
equal, we can just treat this as
one big binomial distribution with
$n=n_1+n_2+n_3$ and $k=k_1+k_2+k_3$ is the
total number of heads observed for
any coin.  Thus, under the null hypothesis,
the distribution of $k$ is binomial
with parameters $n$ and $p$. Now, what is
the maximum likelihood estimator for
this distribution? We have worked this
problem before and have the following,

$$
\hat{p}_0= \frac{k}{n}
$$

 In other words, the maximum likelihood estimator under the null
hypothesis is
the proportion of ones observed in the sequence of $n$ trials
total. Now, we
have to substitute this in for the likelihood under the null
hypothesis to
finish the numerator of $\Lambda$,

$$
L(\hat{p}_0,\hat{p}_0,\hat{p}_0) =
\texttt{binom}(k_1;n_1,\hat{p}_0)\texttt{binom}(k_2;n_2,\hat{p}_0)\texttt{binom}(k_3;n_3,\hat{p}_0)
$$

For the denomimator of $\Lambda$, which represents the case of maximizing over
the entire space, the maximum likelihood estimator for each separate binomial
distribution is likewise,

$$
\hat{p}_i= \frac{k_i}{n_i}
$$

 which makes the likelihood in the denominator the following,

$$
L(\hat{p}_1,\hat{p}_2,\hat{p}_3) =
\texttt{binom}(k_1;n_1,\hat{p}_1)\texttt{binom}(k_2;n_2,\hat{p}_2)\texttt{binom}(k_3;n_3,\hat{p}_3)
$$

 for each of the $i\in \lbrace 1,2,3 \rbrace$ binomial distributions. Then, the
$\Lambda$ statistic is then the following,

$$
\Lambda(k_1,k_2,k_3) =
\frac{L(\hat{p}_0,\hat{p}_0,\hat{p}_0)}{L(\hat{p}_1,\hat{p}_2,\hat{p}_3)}
$$

 Wilks theorems states that $-2\log\Lambda$  is chi-square
distributed. We can
compute this example with the statistics tools in  Sympy and
Scipy.

```python
from scipy.stats import binom, chi2
import numpy as np
# some sample parameters
p0,p1,p2 = 0.3,0.4,0.5
n0,n1,n2 = 50,180,200
brvs= [ binom(i,j) for i,j in zip((n0,n1,n2),(p0,p1,p2))]
def gen_sample(n=1):
    'generate samples from separate binomial distributions'
    if n==1:
        return [i.rvs() for i in brvs]
    else:
        return [gen_sample() for k in range(n)]
```

**Programming Tip.**

Note the recursion in the definition of the `gen_sample`
function where a
conditional clause of the function calls itself. This is a
quick way to reusing
code and generating vectorized output. Using `np.vectorize`
is another way, but
the code is simple enough in this case to use the
conditional clause. In
Python, it is generally bad for performance to have code
with nested recursion
because of how the stack frames are managed.  However,
here we are only
recursing once so this is not an issue.

 Next, we compute
the logarithm of the numerator of the $\Lambda$
statistic,

```python
np.random.seed(1234)
```

```python
k0,k1,k2 = gen_sample()
print (k0,k1,k2)
pH0 = sum((k0,k1,k2))/sum((n0,n1,n2))
numer = np.sum([np.log(binom(ni,pH0).pmf(ki))
                  for ni,ki in
                      zip((n0,n1,n2),(k0,k1,k2))])
print (numer)
```

    12 68 103
    -15.545863836567879

Note that we used the null hypothesis estimate for the $\hat{p}_0$.
Likewise,
for the logarithm of the denominator we have the following,

```python
denom = np.sum([np.log(binom(ni,pi).pmf(ki))
                  for ni,ki,pi in
                     zip((n0,n1,n2),(k0,k1,k2),(p0,p1,p2))])
print (denom)
```

    -8.424106480792402

Now, we can compute the logarithm of the $\Lambda$ statistic as
follows and see
what the corresponding value is according to Wilks theorem,

```python
chsq=chi2(2)
logLambda =-2*(numer-denom)
print (logLambda)
print (1- chsq.cdf(logLambda))
```

    14.243514711550954
    0.0008073467083287156

Because the value reported above is less than the 5% significance
level, we
reject the null  hypothesis that all the coins have the same
probability of
heads. Note that there are two degrees of freedom because the
difference in the
number of parameters between the null hypothesis ($p$) and
the alternative
($p_1,p_2,p_3$) is two. We can build a quick Monte
Carlo simulation to check the
probability of detection for this example using
the following code, which is
just a combination of the last few code blocks,

```python
c= chsq.isf(.05) # 5% significance level
out = []
for k0,k1,k2 in gen_sample(100):
   pH0 = sum((k0,k1,k2))/sum((n0,n1,n2))
   numer = np.sum([np.log(binom(ni,pH0).pmf(ki))
                     for ni,ki in
                         zip((n0,n1,n2),(k0,k1,k2))])
   denom = np.sum([np.log(binom(ni,pi).pmf(ki))
                     for ni,ki,pi in
                        zip((n0,n1,n2),(k0,k1,k2),(p0,p1,p2))])
   out.append(-2*(numer-denom)>c)

print (np.mean(out)) # estimated probability of detection
```

    0.59

The above simulation shows the estimated probability of
detection, for this set
of example parameters. This relative low
probability of detection means that
while the test is unlikely (i.e.,
at the 5% significance level) to mistakenly
pick the null hypothesis,
it is likewise missing many of the $H_1$ cases (i.e.,
low probability
of detection). The trade-off between which is more important is
up to
the particular context of the problem. In some situations, we may
prefer
additional false alarms in exchange for missing fewer $H_1$
cases.

###
Permutation Test

The Permutation Test is good way to test whether or not
samples
samples come from the same distribution. For example, suppose that

$$
X_1, X_2, \ldots, X_m \sim F
$$

 and also,

$$
Y_1, Y_2, \ldots, Y_n \sim G
$$

 That is, $Y_i$ and $X_i$ come from different distributions. Suppose
we have
some test statistic, for example

$$
T(X_1,\ldots,X_m,Y_1,\ldots,Y_n)  = \vert\overline{X}-\overline{Y}\vert
$$

 Under the null hypothesis for which $F=G$, any of the
$(n+m)!$ permutations are
equally likely. Thus, suppose for
each of the $(n+m)!$ permutations, we have the
computed
statistic,

$$
\lbrace T_1,T_2,\ldots,T_{(n+m)!} \rbrace
$$

 Then, under the null hypothesis, each of these values is equally
likely. The
distribution of $T$ under the null hypothesis is the *permutation
distribution*
that puts weight $1/(n+m)!$ on each $T$-value. Suppose $t_o$ is
the observed
value of the test statistic and assume that large $T$ rejects the
null
hypothesis, then the p-value for the permutation test is the following,

$$
P(T>t_o)= \frac{1}{(n+m)!} \sum_{j=1}^{(n+m)!} I(T_j>t_o)
$$

 where $I()$ is the indicator function. For large $(n+m)!$, we can
sample
randomly from the set of all permutations to estimate this p-value.
**Example.** Let's return to our coin-flipping example from last time, but
now
we have only two coins. The hypothesis is that both coins
have the same
probability of heads. We can use the built-in
function in Numpy to compute the
random permutations.

```python
x=binom(10,0.3).rvs(5) # p=0.3
y=binom(10,0.5).rvs(3) # p=0.5
z = np.hstack([x,y]) # combine into one array
t_o = abs(x.mean()-y.mean())
out = [] # output container
for k in range(1000):
   perm = np.random.permutation(z)
   T=abs(perm[:len(x)].mean()-perm[len(x):].mean())
   out.append((T>t_o))

print ('p-value = ', np.mean(out))
```

    p-value =  0.0

Note that the size of total permutation space is
$8!=40320$ so we are taking
relatively few (i.e., 100) random
permutations from this space.

### Wald Test
The Wald Test is an asympotic test. Suppose we have $H_0:\theta=\theta_0$ and
otherwise $H_1:\theta\ne\theta_0$, the corresponding statistic is defined as
the
following,

$$
W=\frac{\hat{\theta}_n-\theta_0}{\texttt{se}}
$$

 where $\hat{\theta}$ is the maximum likelihood estimator and
$\texttt{se}$ is
the standard error,

$$
\texttt{se} = \sqrt{\mathbb{V}(\hat{\theta}_n)}
$$

 Under general conditions, $W\overset{d}{\to} \mathcal{N}(0,1)$.
Thus, an
asympotic test at level $\alpha$ rejects when $\vert W\vert>
z_{\alpha/2}$ where
$z_{\alpha/2}$ corresponds to $\mathbb{P}(\vert
Z\vert>z_{\alpha/2})=\alpha$
with $Z \sim \mathcal{N}(0,1)$.  For our favorite
coin-flipping example, if
$H_0:\theta=\theta_0$, then

$$
W = \frac{\hat{\theta}-\theta_0}{\sqrt{\hat{\theta}(1-\hat{\theta})/n}}
$$

 We can simulate this using the following code at the usual
5% significance
level,

```python
from scipy import stats
theta0 = 0.5 # H0
k=np.random.binomial(1000,0.3)
theta_hat = k/1000. # MLE
W = (theta_hat-theta0)/np.sqrt(theta_hat*(1-theta_hat)/1000)
c = stats.norm().isf(0.05/2) # z_{alpha/2}
print (abs(W)>c) # if true, reject H0
```

    True

This rejects $H_0$ because the true $\theta=0.3$ and the null hypothesis
is
that $\theta=0.5$.  Note that $n=1000$ in this case which puts us well inside
the
asympotic range of the result. We can re-do this example to estimate
the
detection probability for this example as in the following code,

```python
theta0 = 0.5 # H0
c = stats.norm().isf(0.05/2.) # z_{alpha/2}
out = []
for i in range(100):
   k=np.random.binomial(1000,0.3)
   theta_hat = k/1000. # MLE
   W = (theta_hat-theta0)/np.sqrt(theta_hat*(1-theta_hat)/1000.)
   out.append(abs(W)>c) # if true, reject H0

print (np.mean(out)) # detection probability
```

    1.0

## Testing Multiple Hypotheses

Thus far, we have focused primarily on two
competing hypotheses. Now, we
consider multiple comparisons. The general
situation is the following.  We test
the null hypothesis against  a sequence of
$n$ competing hypotheses $H_k$.  We
obtain p-values for each hypothesis so now
we have multiple p-values to
consider $\lbrace p_k \rbrace$. To boil this
sequence down to a single
criterion, we can make the following argument. Given
$n$ independent hypotheses
that are all untrue, the probability of getting at
least one false alarm is the
following,

$$
P_{FA} = 1-(1-p_0)^n
$$

 where $p_0$ is the individual p-value threshold (say, 0.05). The
problem here
is that $P_{FA}\rightarrow 1$ as $n\rightarrow\infty$.  If we want
to make many
comparisons at once and control the overall false alarm rate the
overall p-value
should be computed under the assumption that none of the
competing hypotheses is
valid. The most common way to address this is with the
Bonferroni correction
which says that the individual significance level should
be reduced to $p/n$.
Obviously, this  makes it much harder to declare
significance for any particular
hypothesis. The natural consequence of this
conservative restriction is to
reduce the statistical power of the experiment,
thus making it more likely the
true effects will be missed.

In 1995, Benjamini and Hochberg devised a simple
method that tells which
p-values are statistically significant. The procedure is
to sort the list of
p-values in ascending order, choose a false-discovery rate
(say, $q$), and then
find the largest p-value in the sorted list such that $p_k
\le k q/n$, where
$k$ is the p-value's position in the sorted list. Finally,
declare that $p_k$
value and all the others less than it statistically
significant. This procedure
guarantees that the proportion of false-positives is
less than $q$  (on
average). The Benjamini-Hochberg procedure (and its
derivatives) is fast and
effective and is widely used for testing hundreds of
primarily false hypotheses
when studying genetics or diseases. Additionally,
this
procedure provides better statistical power than the Bonferroni correction.

## Fisher Exact Test

$$
\begin{table}[]
\centering
\caption{Example Contingency Table}
\label{tab:contingencyTable} \tag{2}
\begin{tabular}{lllll}
\cline{1-4}
\multicolumn{1}{|l|}{}       & \multicolumn{1}{l|}{Infection} &
\multicolumn{1}{l|}{No infection} & \multicolumn{1}{l|}{Total} &  \\ \cline{1-4}
\multicolumn{1}{|l|}{Male}   & \multicolumn{1}{c|}{13}        &
\multicolumn{1}{c|}{11}           & \multicolumn{1}{c|}{24}    &  \\ \cline{1-4}
\multicolumn{1}{|l|}{Female} & \multicolumn{1}{c|}{12}        &
\multicolumn{1}{c|}{1}            & \multicolumn{1}{c|}{13}    &  \\ \cline{1-4}
\multicolumn{1}{|l|}{Total}  & \multicolumn{1}{c|}{25}        &
\multicolumn{1}{c|}{12}           & \multicolumn{1}{c|}{37}    &  \\ \cline{1-4}
\end{tabular}
\end{table}
$$

Contingency tables represent the partitioning of a sample population of two
categories between two different classifications as shown in the following
Table
[2](#tab:contingencyTable). The question is whether or not the observed
table
corresponds to a random partition of the sample population, constrained
by the
marginal sums. Note that because this is a two-by-two table, a change in
any of
the table entries automatically affects all of the other terms because
of the
row and column sum constraints. This means that equivalent questions
like "Under
a random partition, what is the probability that a particular
table entry is at
least as large as a given value?" can be meaningfully posed.

The Fisher Exact Test addresses this question. The idea is to compute the
probability of a particular entry of the table, conditioned upon the marginal
row and column sums,

$$
\mathbb{P}(X_{i,j}\vert r_1,r_2,c_1,c_2)
$$

 where $X_{i,j}$ is $(i,j)$ table entry,  $r_1$ represents the sum of
the first
row, $r_2$ represents the sum of the second row, $c_1$ represents the
sum of the
first column, and $c_2$ is the sum of the second column.  This
probability is
given by the *hypergeometric distribution*. Recall that the
hypergeometric
distribution gives the probability of sampling (without
replacement) $k$ items
from a population of $N$ items consisting of exactly two
different kinds of
items,

$$
\mathbb{P}(X=k) = \frac{\binom{K}{k}\binom{N-K}{n-k}}{\binom{N}{n}}
$$

 where $N$ is the population size, $K$ is the total number of possible
favorable
draws, $n$ is the number of draws, and $k$ is the number of observed
favorable
draws.  With the corresponding identification of variables, the
hypergeometric
distribution gives the desired conditional probability: $K=r_1,
k=x, n= c_1,
N=r_1+r_2$.

In the example of the Table [2](#tab:contingencyTable), the
probability for
$x=13$ male infections among a population of $r_1=24$ males in a
total
population of $c_1=25$ infected persons, including $r_2=13$ females. The
`scipy.stats` module has the Fisher Exact Test implemented as shown below,

```python
import scipy.stats
table = [[13,11],[12,1]]
odds_ratio, p_value=scipy.stats.fisher_exact(table)
print(p_value)
```

    0.02718387758955712

The default for `scipy.stats.fisher_exact` is the two-sided
test. The following
result is for the `less` option,

```python
import scipy.stats
odds_ratio, p_value=scipy.stats.fisher_exact(table,alternative='less')
print(p_value)
```

    0.018976707519532877

This means that the p-value is computed by summing  over the
probabilities of
contingency tables that are *less* extreme than the
given table. To undertand
what this means, we can use
the `scipy.stats.hypergeom` function to compute the
probabilities of
these with the number of infected men is less than or equal to
13.

```python
hg = scipy.stats.hypergeom(37, 24, 25)
probs = [(hg.pmf(i)) for i in range(14)]
print (probs)
print(sum(probs))
```

    [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0014597467322717626, 0.017516960787261115]
    0.018976707519532877

This is the same as the prior p-value result we obtained from
`scipy.stats.fisher_exact`. Another option is `greater` which derives from the
following analogous summation,

```python
odds_ratio, p_value=scipy.stats.fisher_exact(table,alternative='greater')
probs = [hg.pmf(i) for i in range(13,25)]
print(probs)
print(p_value)
print(sum(probs))
```

    [0.017516960787261115, 0.08257995799708828, 0.2018621195484381, 0.28386860561499044, 0.24045340710916852, 0.12467954442697629, 0.039372487713781906, 0.00738234144633414, 0.0007812001530512284, 4.261091743915799e-05, 1.0105355914424832e-06, 7.017608273906114e-09]
    0.9985402532677288
    0.9985402532677288

Finally, the two-sided version excludes those individual
table probabilities
that are less that of the given table

```python
_,p_value=scipy.stats.fisher_exact(table)
probs = [ hg.pmf(i) for i in range(25) ]
print(sum(i for i in probs if i<= hg.pmf(13)))
print(p_value)
```

    0.027183877589557117
    0.02718387758955712

Thus, for this particular contingency table, we
could reasonably conclude that
13 infected males in this total
population is statistically significant with a
p-value less than
five percent.

Performing this kind of analysis for tables
larger than `2x2` easily becomes
computationally challenging due to the nature
of the underlying combinatorics and
usually requires specialized
approximations.

In this section, we discussed the structure of statistical
hypothesis testing
and defined the various  terms that are commonly used for
this process, along
with the illustrations of what they mean in our running
coin-flipping example.
From an engineering standpoint, hypothesis testing is not
as common as
confidence-intervals and point estimates. On the other hand,
hypothesis testing
is very common in social and medical science, where one must
deal with
practical constraints that may limit the sample size or other aspects
of the
hypothesis testing rubric. In engineering, we can usually have much more
control over the samples and models we employ because they are typically
inanimate objects that can be measured repeatedly and consistently. This is
obviously not so with human studies, which generally have other ethical and
legal considerations.


# PHY321: Classical Mechanics 1
**Homework 7, due Monday  March 25**

Date: **Mar 16, 2022**

### Practicalities about  homeworks and projects

1. You can work in groups (optimal groups are often 2-3 people) or by yourself. If you work as a group you can hand in one answer only if you wish. **Remember to write your name(s)**!

2. Homeworks are available  the week before the deadline.

3. How do I(we)  hand in?  Due to the corona virus and many of you not being on campus, we recommend that you scan your handwritten notes and upload them to D2L. If you are ok with typing mathematical formulae using say Latex, you can hand in everything as a single jupyter notebook at D2L. The numerical exercise(s) should always be handed in as a jupyter notebook by the deadline at D2L.

### Introduction to homework 7

In this week's homework we will apply our insights about harmonic
oscillations. The relevant material to survey is chapter 5 of Taylor.
See also the slides from [week 11](https://mhjensen.github.io/Physics321/doc/pub/week10/html/week10.html).

We have also added an exercise (exercise 2) related to our discussion of two-body problems.
The relevant reading background for exercise 2 is given by sections 8.1-8.2 of Taylor.

### Exercise 1 (80 pts), the mathematical pendulum

Relevant reading here is Taylor chapter 5 and the lecture notes on oscillations from [week 11](https://mhjensen.github.io/Physics321/doc/pub/week10/html/week10.html).

The angular equation of motion of the pendulum is given by
Newton's equation and with no external force it reads

$$
\begin{equation}
  ml\frac{d^2\theta}{dt^2}+mgsin(\theta)=0,
\label{_auto1} \tag{1}
\end{equation}
$$

with an angular velocity and acceleration given by

$$
\begin{equation}
     v=l\frac{d\theta}{dt},
\label{_auto2} \tag{2}
\end{equation}
$$

and

$$
\begin{equation}
     a=l\frac{d^2\theta}{dt^2}.
\label{_auto3} \tag{3}
\end{equation}
$$

We do however expect that the motion will gradually come to an end
due a viscous drag torque acting on the pendulum.
In the presence of the drag, the above equation becomes

$$
\begin{equation}
   ml\frac{d^2\theta}{dt^2}+\nu\frac{d\theta}{dt}  +mgsin(\theta)=0,
\label{eq:pend1} \tag{4}
\end{equation}
$$

where $\nu$ is now a positive constant parameterizing the viscosity
of the medium in question. In order to maintain the motion against
viscosity, it is necessary to add some external driving force.
We choose here a periodic driving force. The last equation becomes then

$$
\begin{equation}
   ml\frac{d^2\theta}{dt^2}+\nu\frac{d\theta}{dt}  +mgsin(\theta)=Asin(\omega t),
\label{eq:pend2} \tag{5}
\end{equation}
$$

with $A$ and $\omega$ two constants representing the amplitude and
the angular frequency respectively. The latter is called the driving frequency.

* 1a (10pts)

Rewrite Eqs. ([4](#eq:pend1)) and ([5](#eq:pend2)) as dimensionless
equations in time.

* 1b (40pts)

Write then a code which solves Eq. ([4](#eq:pend1)) using the
Euler-Cromer method or for example the fourth-order Runge Kutta method. Perform
calculations for at least ten periods with $N=100$, $N=1000$ and
$N=10000$ mesh points and values of $\nu = 1$, $\nu = 5$ and $\nu
=10$.  Set $l=1.0$ m, $g=1$ m/s$^2$ and $m=1$ kg.  Choose as initial
conditions $\theta(0) = 0.2$ (radians) and $v(0) = 0$ (radians/s).
Make plots of $\theta$ (in radians) as function of time and phase
space plots of $\theta$ versus the velocity $v$.  Check the stability
of your results as functions of time and number of mesh points.  Which
case corresponds to damped, underdamped and overdamped oscillatory
motion?  Comment your results.

* 1c (30pts)

Now we switch to Eq. ([5](#eq:pend2)) for the rest of the exercise. Add
an external driving force and set $l=g=1$, $m=1$, $\nu = 1/2$ and
$\omega = 2/3$.  Choose as initial conditions $\theta(0) = 0.2$ and
$v(0) = 0$ and $A=0.5$ and $A=1.2$.  Make plots of $\theta$ (in
radians) as function of time for at least 300 periods and phase space
plots of $\theta$ versus the velocity $v$. Choose an appropriate time
step. Comment and explain the results for the different values of $A$.

* 1d **optional exercise** (20pts bonus)

Keep now the constants from the previous exercise fixed but
set now $A=1.35$, $A=1.44$ and $A=1.465$. Plot $\theta$ (in radians)
as function of time for at least 300 periods for these values of $A$
and comment your results.

* 1e **optional exercise** (20pts bonus)

We want to analyse further these results by making phase space plots
of $\theta$ versus the velocity $v$ using only the points where we
have $\omega t=2n\pi$ where $n$ is an integer. These are normally
called the drive periods.  This is an example of what is called a
Poincare section and is a very useful way to plot and analyze the
behavior of a dynamical system. Comment your results.

### Exercise 2 (20pt), Center-of-Mass and Relative Coordinates and Reference Frames

We define the two-body center-of-mass coordinate and relative coordinate by expressing the trajectories for
$\boldsymbol{r}_1$ and $\boldsymbol{r}_2$ into the center-of-mass coordinate
$\boldsymbol{R}_{\rm cm}$

$$
\boldsymbol{R}_{\rm cm}\equiv\frac{m_1\boldsymbol{r}_1+m_2\boldsymbol{r}_2}{m_1+m_2},
$$

and the relative coordinate

$$
\boldsymbol{r}\equiv\boldsymbol{r}_1-\boldsymbol{r_2}.
$$

Here, we assume the two particles interact only with one another, so $\boldsymbol{F}_{12}=-\boldsymbol{F}_{21}$ (where $\boldsymbol{F}_{ij}$ is the force on $i$ due to $j$.

* 2a (5pt) Show that the equations of motion then become $\ddot{\boldsymbol{R}}_{\rm cm}=0$ and $\mu\ddot{\boldsymbol{r}}=\boldsymbol{F}_{12}$, with the reduced mass $\mu=m_1m_2/(m_1+m_2)$.

The first expression simply states that the center of mass coordinate $\boldsymbol{R}_{\rm cm}$ moves at a fixed velocity. The second expression can be rewritten in terms of the reduced mass $\mu$.

* 2b (5pt) Show that the linear momenta for the center-of-mass $\boldsymbol{P}$ motion and the relative motion $\boldsymbol{q}$ are given by $\boldsymbol{P}=M\dot{\boldsymbol{R}}_{\rm cm}$ with $M=m_1+m_2$ and $\boldsymbol{q}=\mu\dot{\boldsymbol{r}}$.  The linear momentum of the relative motion is defined $\boldsymbol{q} = (m_2\boldsymbol{p}_1-m_1\boldsymbol{p}_2)/(m_1+m_2)$.

* 2c (5pt) Show then the kinetic energy for two objects can then be written as

$$
K=\frac{P^2}{2M}+\frac{q^2}{2\mu}.
$$

* 2d (5pt) Show that the total angular momentum for two-particles in the center-of-mass frame $\boldsymbol{R}=0$, is given by

$$
\boldsymbol{L}=\boldsymbol{r}\times \mu\dot{\boldsymbol{r}}.
$$

### Classical Mechanics Extra Credit Assignment: Scientific Writing and attending Talks

The following gives you an opportunity to earn **five extra credit
points** on each of the remaining homeworks and **ten extra credit points**
on the midterms and finals.  This assignment also covers an aspect of
the scientific process that is not taught in most undergraduate
programs: scientific writing.  Writing scientific reports is how
scientist communicate their results to the rest of the field.  Knowing
how to assemble a well written scientific report will greatly benefit
you in you upper level classes, in graduate school, and in the work
place.

The full information on extra credits is found at <https://github.com/mhjensen/Physics321/blob/master/doc/Homeworks/ExtraCredits/>. There you will also find examples on how to write a scientific article.
Below you can also find a description on how to gain extra credits by attending scientific talks.

This assignment allows you to gain extra credit points by practicing
your scientific writing.  For each of the remaining homeworks you can
submit the specified section of a scientific report (written about the
numerical aspect of the homework) for five extra credit points on the
assignment.  For the two midterms and the final, submitting a full
scientific report covering the numerical analysis problem will be
worth ten extra points.  For credit the grader must be able to tell
that you put effort into the assignment (i.e. well written, well
formatted, etc.).  If you are unfamiliar with writing scientific
reports, [see the information here](https://github.com/mhjensen/Physics321/blob/master/doc/Homeworks/ExtraCredits/IntroductionScientificWriting.md)

The following table explains what aspect of a scientific report is due
with which homework.  You can submit the assignment in any format you
like, in the same document as your homework, or in a different one.
Remember to cite any external references you use and include a
reference list.  There are no length requirements, but make sure what
you turn in is complete and through.  If you have any questions,
please contact Julie Butler at butler@frib.msu.edu.

  HW/Project 	 Due Date	 Extra Credit Assignment

   HW 3             	    2-8         	    Abstract

   HW 4             	    2-15        	    Introduction

   HW 5             	    2-22        	    Methods

   HW 6             	    3-1         	    Results and Discussion

   **Midterm 1**    	    **3-12**    	    *Full Written Report*

   HW 7             	    3-22        	    Abstract

   HW 8             	    3-29        	    Introduction

   HW 9             	    4-5         	    Results and Discussion

   **Midterm 2**    	    **4-16**    	    *Full Written Report*

   HW 10            	    4-26        	    Abstract

   **Final**        	    **4-30**    	    *Full Written Report*

You can also gain extra credits if you attend scientific talks.
This is described here.

### Integrating Classwork With Research

This opportunity will allow you to earn up to 5 extra credit points on a Homework per week. These points can push you above 100% or help make up for missed exercises.
In order to earn all points you must:

1. Attend an MSU research talk (recommended research oriented Clubs is  provided below)

2. Summarize the talk using at least 150 words

3. Turn in the summary along with your Homework.

Approved talks:
Talks given by researchers through the following clubs:
* Research and Idea Sharing Enterprise (RAISE): Meets Wednesday Nights Society for Physics Students (SPS): Meets Monday Nights

* Astronomy Club: Meets Monday Nights

* Facility For Rare Isotope Beam (FRIB) Seminars: Occur multiple times a week

If you have any questions please consult Jeremy Rebenstock, rebensto@msu.edu.

All the material on extra credits is at <https://github.com/mhjensen/Physics321/blob/master/doc/Homeworks/ExtraCredits/>.


Lambda School Data Science

*Unit 4, Sprint 2, Module 2*

---

# Train (Prepare)
__*Neural Network Foundations*__

## Learning Objectives
* Part 1: Student should be able to explain the intuition behind backpropagation and gradient descent
* Part 2: Student should be able to discuss the importance of batch size
* Part 3: Student should be able to discuss the importance of learning rate

## Summary of Yesterday

Yesterday, we learned about some of the principal components of Neural Networks: Neurons, Weights, Activation Functions, and layers (input, output, & hidden). Today, we will reinforce our understanding of those components and introduce the mechanics of training a neural network. Feed-forward neural networks, such as multi-layer perceptrons (MLPs), are almost always trained using some variation of gradient descent where the gradient has been calculated by backpropagation.

- There are three kinds of layers: input, hidden, and output layers.
- Each layer is made up of **n** individual neurons (aka activation units) which have a corresponding weight and bias.
- Signal is passed from layer to layer through a network by:
 - Taking in inputs from the training data (or previous layer)
 - Multiplying each input by its corresponding weight (think arrow/connecting line)
 - Adding a bias to this weighted some of inputs and weights
 - Activating this weighted sum + bias by squishifying it with sigmoid or some other activation function. With a single perceptron with three inputs, calculating the output from the node is done like so:
\begin{align}
 y = sigmoid(\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)
\end{align}
 - this final activated value is the signal that gets passed onto the next layer of the network.

## Training a Neural Network: *Formal Summary*

0. Pick a network architecture
   - No. of input units = No. of features
   - No. of output units = Number of Classes (or expected targets)
   - Select the number of hidden layers and number of neurons within each hidden layer
1. Randomly initialize weights
2. Implement forward propagation to get $h_{\theta}(x^{(i)})$ for any $x^{(i)}$
3. Implement code to compute a cost function $J(\theta)$
4. Implement backpropagation to compute partial derivatives $\frac{\delta}{\delta\theta_{jk}^{l}}{J(\theta)}$
5. Use gradient descent (or other advanced optimizer) with backpropagation to minimize $J(\theta)$ as a function of parameters $\theta\$
6. Repeat steps 2 - 5 until cost function is 'minimized' or some other stopping criteria is met. One pass over steps 2 - 5 is called an iteration or epoch.

# Backpropagation & Gradient Descent (Learn)

## Overview

Backpropagation is short for ["Backwards Propagation of errors"](https://en.wikipedia.org/wiki/Backpropagation) and refers to a specific (rather calculus intensive) algorithm for how weights in a neural network are updated in reverse order at the end of each training epoch. Our purpose today is to demonstrate the backpropagation algorithm on a simple Feedforward Neural Network and in so doing help you get a grasp on the main process. If you want to understand all of the underlying calculus of how the gradients are calculated then you'll need to dive into it yourself, [3Blue1Brown's video is a great starting place](https://www.youtube.com/watch?v=tIeHLnjs5U8). I also highly recommend this Welch Labs series [Neural Networks Demystified](https://www.youtube.com/watch?v=bxe2T-V8XRs) if you want a rapid yet orderly walk through of the main intuitions and math behind the backpropagation algorithm.

### What is a Gradient?

> In vector calculus, the gradient is a multi-variable generalization of the derivative.

The gradients that we will deal with today will be vector representations of the derivative of the activation function.

## Follow Along

In this section, we will again a simple neural network using base TensorFlow. We'll focus on using a __Feed Forward Neural Network__ to predict test scores.

### Generate some Fake Data

```python
import tensorflow as tf

# Imagine that our data is drawn from a linear function
# y = 3*hours_studying + 50

TRUE_W = 3.5
TRUE_b = 50.0
NUM_EXAMPLES = 1000

inputs = tf.random.normal(shape=[NUM_EXAMPLES])
noise = tf.random.normal(shape=[NUM_EXAMPLES])

outputs = inputs * TRUE_W + TRUE_b + noise
```

### Loss Function
Here we will use Mean Squared Error (MSE), because this is a regression problem. We are trying to predict a continuous target.

```python
def loss(target_y, predicted_y):
  "MSE"
  return tf.reduce_mean(tf.square(target_y - predicted_y))
```

### Neural Network Architecture
Lets create a Neural Network class called "Model" to contain this functionality. Note: This is essentially a linear regression whose coefficients are trained by gradient descent. In practice, gradient descent works on much more complex function like the multi-layer networks we constructed yesterday.

```python
class Model(object):

  def __init__(self):
    self.W = tf.Variable(8.0)
    self.b = tf.Variable(40.0)

  def __call__(self, x):
    return self.W * x + self.b

model = Model()

assert model(3.0).numpy() == 64.0
```

### Initial Weights
The initial weights in our model were arbitrary. In practice, weights are initialized randomly.

```python
import matplotlib.pyplot as plt

plt.scatter(inputs, outputs, c='b')
plt.scatter(inputs, model(inputs), c='r')
plt.show()

print('Current loss: %1.6f' % loss(model(inputs), outputs).numpy())
```

### Update Weights Based on Gradient

> *Assigning blame for bad predictions and delivering justice - repeatedly and a little bit at a time*

You should also know that with neural networks it is common to have gradients that are not convex (like what we saw when we applied gradient descent to linear regression).

Due to the high complexity of these models and their nonlinearity, it is common for gradient descent to get stuck in a local minimum, but there are ways to combat this:

1) Stochastic Gradient Descent

2) More advanced Gradient-Descent-based "Optimizers" - See Stretch Goals on assignment.

```python
 def train(model, inputs, outputs, learning_rate):
  with tf.GradientTape() as t:
     current_loss = loss(outputs, model(inputs))
  dW, db = t.gradient(current_loss, [model.W, model.b])
  model.W.assign_sub(learning_rate * dW)
  model.b.assign_sub(learning_rate * db)
```

### Train the Network

```python
model = Model()

# Store Some history of weights
Ws, bs = [], []
epochs = range(10)
for epoch in epochs:
  Ws.append(model.W.numpy())
  bs.append(model.b.numpy())
  current_loss = loss(outputs, model(inputs))

  train(model, inputs, outputs, learning_rate=0.1)
  print('Epoch %2d: W=%1.2f b=%1.2f loss=%2.5f' % (epoch, Ws[-1], bs[-1], current_loss))
```

    Epoch  0: W=8.00 b=40.00 loss=127.54656
    Epoch  1: W=6.97 b=42.05 loss=80.43711
    Epoch  2: W=6.17 b=43.68 loss=50.89282
    Epoch  3: W=5.56 b=44.98 loss=32.35456
    Epoch  4: W=5.08 b=46.02 loss=20.71602
    Epoch  5: W=4.71 b=46.84 loss=13.40525
    Epoch  6: W=4.43 b=47.50 loss=8.81047
    Epoch  7: W=4.21 b=48.02 loss=5.92111
    Epoch  8: W=4.04 b=48.44 loss=4.10316
    Epoch  9: W=3.91 b=48.77 loss=2.95870

```python
import matplotlib.pyplot as plt

plt.plot(epochs, Ws, 'r', epochs, bs, 'b')
plt.plot([TRUE_W] * len(epochs), 'r--',
         [TRUE_b] * len(epochs), 'b--')
plt.legend(['W', 'b', 'True W', 'True b'])
plt.show()
```

## Challenge

In the module project, you will be asked to explain the logic of backpropagation and gradient descent.

### In the following two sections we'll look at batch size and learning rate hyperparameters in isolation.
However, it's important to know that recent research found some interesting relationships between batch sizes and learning rates. The best available suggestion today is to scale batch size proportionally to a learning rate:
- https://openreview.net/pdf?id=B1Yy1BxCZ
- https://papers.nips.cc/paper/8398-control-batch-size-and-learning-rate-to-generalize-well-theoretical-and-empirical-evidence.pdf

# Batch Size (Learn)

## Overview

The What - Stochastic Gradient Descent calculates an approximation of the gradient over the entire dataset by reviewing the predictions of a random sample.

The Why - *Speed*. Calculating the gradient over the entire dataset is extremely expensive computationally.

### Batch Size
Batches are the number of observations our model is shown to make predictions and update the weights. Batches are selected randomly during epoch. All observations are considered when passing thru an epoch at some point.

### Baseline Model

```python
from tensorflow.keras.datasets import mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train / 255.
X_test = X_test / 255.

X_train = X_train.reshape((60000, 784))
X_test = X_test.reshape((10000, 784))
```

```python
# Our Model

```

## Follow Along
Let's run a series of experiments for a default, small, and large batch size.

### Default
Batch Size is 32

```python

```

### Small Batch Size
Batch Size is 8

```python

```

### Large Batch Size
Batch Size is 512

```python

```

### Visualization of Results

```python

```

## Challenge

You will be expected to experiment with batch size on today's assignment.

# Learning Rate (Learn)

## Overview

Learning Rate controls the size of the update to our weights that the optimization algorithm makes. VERY IMPORTANT hyperparameter.

* Too high of a learning rate causes unstable results
* Too Low of a learning rate the model will underfit
* Goldy Locks parameters - it needs be "just right"
* Scale of 0-1

## Follow Along

Same experiment with Batch but different learning rates:
* High Learning = .75
* Default Learning = .01
* Low Learning Rate = .0001

### Default Learning Rate

```python

```

### High Learning Rate

```python

```

### Low Learning Rate

```python

```

### Visualization of Results

```python

```

## Challenge

You will be expected to experiment with different learning rates today.

---