# Potential flow

## Potencial functions

**Uniform flow**:

$w = U\exp{(-i\alpha)}z$

**Line source**:

$w = \dfrac{\dot{Q}}{2\pi}\ln z$

## Velocity fields

$\dfrac{dw}{dz} = u + iv$

**Uniform flow:**

$\dfrac{dw}{dz} = U\left(\cos{\alpha}-i\sin{\alpha}\right)$

**Line source:**

$\dfrac{dw}{dz} = \dfrac{\dot{Q}}{2\pi}\dfrac{1}{z}$

## Combine potentials

Consider a potential flow with a sink at $(0,0)$ and a source at $(r,0)$, under the influence of a horizontal uniform flow going from left to right ($\alpha = \pi$). The potential that describes that system is given by:

$w = w_{\rm uniform} + w_{\rm source} + w_{\rm sink}$

Hence,

$w = U\exp{(-i\alpha)}z + \dfrac{\dot{Q_{\rm in}}}{2\pi}\ln (z-r) - \dfrac{\dot{Q_{\rm out}}}{2\pi}\ln z$

$w = -Uz + \dfrac{\dot{Q_{\rm in}}}{2\pi}\ln (z-r) - \dfrac{\dot{Q_{\rm out}}}{2\pi}\ln z$

The uniform flow component will represent the regional flow, which can be described using Darcy's law as $U=-K\dot{I}$, where $K$ is the hydraulic conductivity (m/s) and $\dot{I}$ is the water table gradient (m/m). The source and sink wells represent injection and extraction points, characterized by flow rates per unit depth $\dot{Q_{\rm in}}$ and $\dot{Q_{\rm out}}$ (m²/s). For an aquifer of depth $H$, the volumetric flow rate $Q$ is $\dot{Q}=Q/H$. Now, the extraction rate can be expressed as a proportion to the injection rate as $Q_{\rm out} = fQ_{\rm in}$. With all this replacements, the potential function can be rewritten as:

$w = K\dot{I}z + \dfrac{Q_{\rm in}}{2\pi H} \left( \ln(z-r) - f \ln z \right)$

Notice that in our case, the gradient $\dot{I}$ must be negative so the regional flow goes indeed from right to left. To avoid confusion, let's rewrite it in terms of the abs value of the gradient $I$:

$w = -KIz + \dfrac{Q_{\rm in}}{2\pi H} \left( \ln(z-r) - f \ln z \right)$

## Velocity field

$\dfrac{dw}{dz} = \dfrac{d}{dz}\left(-KIz + \dfrac{Q_{\rm in}}{2\pi H} \left( \ln(z-r) - f \ln z \right)\right)$

$\dfrac{dw}{dz} = -KI + \dfrac{Q_{\rm in}}{2\pi H} \dfrac{d}{dz}\left( \ln(z-r) - f \ln z \right)$

$\dfrac{dw}{dz} = -KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{z-r} - \dfrac{f}{z} \right)$

## Velocity at the midpoint

We'll pick the velocity in the midpoint between the extraction and injection points as the characteristic velocity of the system. This means that we want to evaluate the velocity field at $z = r/2 + 0i$

\begin{equation}
\begin{array}{rl}
    \dfrac{dw}{dz}\bigg\rvert_{z=r/2+0i} =& -KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{r/2-r} - \dfrac{f}{r/2} \right)\\
    =& -KI - \dfrac{Q_{\rm in}}{\pi H} \left(\dfrac{1+f}{r}\right)
\end{array}
\end{equation}

Notice that this velocity can be split into the contribution of the regional flow and the contribution from the wells:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{1}{\pi}\dfrac{Q_{\rm in} (1+f)}{r H}
\end{array}
\end{equation}

### Non-dimensional flow number ($\mathcal{F}_L$)

A comparison between the component in the velocity due regional flow and the component due the wells as a ratio between them two.

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}} \\
=& \dfrac{w'_{\rm regional}}{w'_{\rm wells}} \\
=& \dfrac{-KI}{-\dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1+f}{r}\right)} \\
=& \dfrac{\pi K I H r}{Q_{in}(1+f)}
\end{array}
\end{equation}

If $\mathcal{F}_L \to 0$, the flow induced by the wells rule the system, whereas if $\mathcal{F}_L \to \infty$, the regional flow rule the system.

## Minimum velocity along a streamline

The minimum velocity is found at:

\begin{equation}
\begin{array}{rl}
    \dfrac{d}{dx}\dfrac{dw}{dz}\bigg\rvert_{z=x+0i} =& 0 \\
    \dfrac{d}{dx}\left(-KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{x-r} - \dfrac{f}{x} \right)\right) =& 0\\
    \dfrac{Q_{\rm in}}{2 \pi H} \dfrac{d}{dx}\left(\dfrac{1}{x-r} - \dfrac{f}{x} \right) =& 0\\
    \dfrac{Q_{\rm in}}{2 \pi H} \left(\dfrac{f}{x^2} - \dfrac{1}{\left(x-r\right)^2}\right) =& 0\\
    \\
    \dfrac{f}{x^2} =& \dfrac{1}{\left(x-r\right)^2}\\
    x^2 =& f \left( x-r \right) ^2\\
    x =& \dfrac{\sqrt{f}}{\sqrt{f}\pm1}r
    \\
    x < r \Rightarrow x =& \dfrac{\sqrt{f}}{\sqrt{f}+1}r
\end{array}
\end{equation}

Defining $m = \dfrac{\sqrt{f}}{\sqrt{f}+1}$, evaluating at $z = mr + 0i$ gives the minimum velocity.

\begin{equation}
\begin{array}{rl}
    \dfrac{dw}{dz}\bigg\rvert_{z=mr+0i} =& -KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{mr-r} - \dfrac{f}{mr} \right)\\
    =& -KI + \dfrac{Q_{\rm in}}{2 \pi H} \left(\dfrac{m-fm+f}{m(m-1)}\right)\\
    =& -KI - \dfrac{Q_{\rm in}}{2 \pi H} \left(\dfrac{\left(\sqrt{f}+f\right)\left(\sqrt{f}+1\right)}{\sqrt{f}}\right)\\
    =& -KI - \dfrac{Q_{\rm in}}{2 \pi H} \left( \left( 1+ \sqrt{f}\right)\left( 1 + \dfrac{\sqrt{f}}{f} \right) \right)\\
    =& -KI - \dfrac{ Q_{\rm in}\left( 1 + \tfrac{\sqrt{f}}{f} \right) } {\pi H r}
\end{array}\\
\end{equation}

This velocity can also be split into the contribution of the regional flow and the contribution from the wells:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{Q_{\rm in} }{\pi r H}\left(1+\tfrac{\sqrt{f}}{f}\right)
\end{array}
\end{equation}

## Mean velocity along a streamline

Consider the streamline that connects the sink and source points. It cannot be measured at those points because those are poles in the system, hence, some separation $\delta$ from those points has to be considered:  the well radius should be the straighforward election so we'll adopt it and assume that both injection and extraction wells have the same diameter.

Now, with $z = x+iy$, the streamline exists at $y=0$ and $\delta<x<r-\delta$, hence, the mean velocity can be derived as follows:

\begin{equation}
\begin{array}{rl}
\bar{u} =& \dfrac{1}{r-2\delta} \int_{\delta}^{r-\delta}\dfrac{dw}{dz}dx\\
=& \dfrac{1}{r-2\delta} \int_{\delta}^{r-\delta} \left(-KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{z-r} - \dfrac{f}{z} \right)\right)dx\\
=& -KI + \dfrac{Q_{\rm in}}{2\pi (r-2\delta) H}\int_{\delta}^{r-\delta}\left(\dfrac{1}{x-r} - \dfrac{f}{x} \right)dx\\
=& -KI + \dfrac{Q_{\rm in}}{2\pi H}\left(\ln({x-r}) - f\ln{x} \right)\bigg\rvert_{x=\delta}^{x=r-\delta}\\
=& -KI + \dfrac{Q_{\rm in}}{2\pi H}\left[ (1+f)(\ln(\delta) -\ln(r-\delta)  \right]\\
=& -KI - \dfrac{Q_{\rm in}(1+f)}{2\pi H}\ln\left(\dfrac{r-\delta}{\delta}\right)\\
\end{array}
\end{equation}

Notice that this velocity can be split into the contribution of the regional flow and the contribution from the wells:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{Q_{\rm in}(1+f)}{2\pi H}\ln\left(\dfrac{r-\delta}{\delta}\right)
\end{array}
\end{equation}

### Non-dimensional flow number ($\mathcal{F}_L$)

A comparison between the component in the velocity due regional flow and the component due the wells as a ratio between them two.

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}} \\
=& \dfrac{w'_{\rm regional}}{w'_{\rm wells}} \\
=& \dfrac{-KI}{-\dfrac{Q_{\rm in}(1+f)}{2\pi H}\ln\left(\dfrac{r-\delta}{\delta}\right)} \\
=& \dfrac{2 \pi K I H (r-2\delta)}{Q_{in}(1+f)\ln\left(\frac{r-\delta}{\delta}\right)}
\end{array}
\end{equation}

If $\mathcal{F}_L \to 0$, the flow induced by the wells rule the system, whereas if $\mathcal{F}_L \to \infty$, the regional flow rule the system.

## Mass flow through an infinite line

The previous choice of characteristic velocity seems arbitrary in the sense it overestimates the effect of the potential induced by the wells on the system. This is because the streamline along the x-axis has the highest velocity magnitude (the sharpest potential gradient), if we were to pick a point elsewhere, the veloicty form the wells would be less. But in the same sense, for $f=1$, this point represents the lowest velocity along that streamline.

To avoid this apparent bias, let's consider the totality of the flow that crosses an infinite vertical line at $z=r/2+iy$. This won't give us a useful insight on the regional flow as that mass flow will diverge, but it will give us a point of comparison to the wells contribution to the total mass balance in the system.

So, the velocity field along that vertical line is

\begin{equation}
\begin{array}{rl}
\dfrac{dw}{dz}\bigg\rvert_{z=\tfrac{r}{2}+iy} =& -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{z-r} - \dfrac{f}{z} \right)\bigg\rvert_{z=\tfrac{r}{2}+iy}\\
=& - KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{\tfrac{r}{2}+iy-r} - \dfrac{f}{\tfrac{r}{2}+iy} \right)\\
=& -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{\tfrac{r}{2}(1+f) + iy(1-f)}{\tfrac{r^2}{4}-y^2}\right)
\end{array}
\end{equation}

The mass flow through that vertical line will require only the horizontal component of the velocity field $u$ at that point, meaning that we are only intereste in the real component of the function above:

$u = -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{\tfrac{r}{2}(1+f)}{\tfrac{r^2}{4}-y^2}\right) = -KI - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right) $

The mass flow will be described as:

\begin{equation}
\begin{array}{rl}
    \int_{-\infty}^{\infty} u \, dy =& \int_{-\infty}^{\infty}\left[ -KI - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right)\right] \, dy \\
    =& -\underbrace{\int_{-\infty}^{\infty}KI \, dy}_{\text{Diverges}} - \underbrace{\dfrac{Q_{\rm in}r(1+f)}{4\pi H} \int_{-\infty}^{\infty}\left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right) \, dy}_{\text{Converges}}\\
    =& -\int_{-\infty}^{\infty}KI \, dy - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{2}{r} \tan^{-1}{\left(\dfrac{2y}{r}\right)}\right]\bigg\rvert_{y \to -\infty}^{y \to \infty}\\
    =& -\int_{-\infty}^{\infty}KI \, dy - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{2\pi}{r} \right] \\
    =& -\int_{-\infty}^{\infty}KI \, dy - \dfrac{Q_{\rm in}(1+f)}{2 H}
\end{array}
\end{equation}

This end term, we can divide it by the lenght of the line $l_r$ in order to obtain a velocity, hence, the contribution from the wells are:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -\dfrac{1}{l_r}\int_{-\infty}^{\infty}KI \, dy \\
    w'_{\rm wells} =& -\dfrac{1}{2l_r}\dfrac{Q_{\rm in}(1+f)}{rH}
\end{array}
\end{equation}

### Non-dimensional flow number ($\mathcal{F}_L$)

In this case, this comparison don't make much sense as one of the integrals diverged.

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}} \\
=& \dfrac{w'_{\rm regional}}{w'_{\rm wells}} \\
=& \dfrac{-\dfrac{1}{l_r}\int_{-\infty}^{\infty}KI \, dy }{-\dfrac{1}{2l_r}\dfrac{Q_{\rm in}(1+f)}{rH}} \\
=& \dfrac{2rH \int_{-\infty}^{\infty}KI \, dy}{Q_{in}(1+f)}
\end{array}
\end{equation}

We could infer that in this approach, the regional flow will always dwarf the potential induced by the pair of wells. But this is not useful for our purpose :(

## Mass flow through an finite line

In order to obtain a convergent integral for the regional component, let's pick a vertical line that spans from $r/2(1-i)$ to $r/2(1+i)$, that way, the mass flow will be described as:

\begin{equation}
\begin{array}{rl}
    \int_{-r/2}^{r/2} u \, dy =& \int_{-r/2}^{r/2}\left[ -KI - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right)\right] \, dy \\
    =& \int_{-r/2}^{r/2}KI \, dy - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \int_{-r/2}^{r/2}\left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right) \, dy\\
    =& -KIy\bigg\rvert_{y = -r/2}^{y = r/2} - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{2}{r} \tan^{-1}{\left(\dfrac{2y}{r}\right)}\right]\bigg\rvert_{y = -r/2}^{y = r/2}\\
    =& -KIr - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{\pi}{r} \right] \\
    =& -KIr - \dfrac{Q_{\rm in}(1+f)}{4 H}
\end{array}
\end{equation}

Jusy as before, this end term can be divided by $r$ in order to obtain a velocity, hence, both contributions are:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{1}{4}\dfrac{Q_{\rm in}(1+f)}{rH}
\end{array}
\end{equation}

### Non-dimensional flow number (2) ($\mathcal{F}_L$)

From this analysis we could also draw a comparison between the two contributors of the system:

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}}\\
=& \dfrac{w'_{\rm regional}}{w'_{wells}}\\
=& \dfrac{-KI}{-\dfrac{Q_{\rm in}}{4 H} \left(\dfrac{1+f}{r}\right)} \\
=& \dfrac{4 K I H r}{Q_{in}(1+f)}
\end{array}
\end{equation}

There is not much of a difference in this "more conservative" approach, and it holds that if $\mathcal{F}_L \to 0$, the flow induced by the wells rule the system, whereas if $\mathcal{F}_L \to \infty$, the regional flow rule the system. We well conserve the first non-dimensional flow number definition, i.e.,

\begin{equation}
\mathcal{F}_L = \dfrac{4 K I H r}{Q_{in}(1+f)}
\end{equation}

## Characteristic velocity ($q_c$)

We found out before an expression for the velocity at the midpoint between the well couple. This will be used as our characteristic velocity, which can be rewritten in terms of the non-dimensional flow number:

\begin{equation}
\begin{array}{rl}
    q_c =& \dfrac{1}{r}\int_{-r/2}^{r/2}\dfrac{dw}{dz}dy\\
    =& -KI - \dfrac{Q_{\rm in}(1+f)}{4 r H}\\
    =& -KI \left( 1 + \dfrac{1}{\mathcal{F}_L} \right)\\
    =& - KI \left( \dfrac{\mathcal{F}_L}{\mathcal{F}_L + 1} \right)
\end{array}
\end{equation}

The actual pore-water velocity $u_c$ is found by dividing by the porosity $\theta$

\begin{equation}
\begin{array}{rl}
    u_c =& \dfrac{q_c}{\theta}\\
    =& - \dfrac{KI}{\theta} \left( \dfrac{\mathcal{F}_L}{\mathcal{F}_L + 1} \right)
\end{array}
\end{equation}

## Flow time scale ($\tau$)
Picking the half-distance between the two wells as our characteristic lenght $r/2$, a characteristic time $\tau$ can be derived from $u_c$

\begin{equation}
\begin{array}{rl}
    \tau =& \dfrac{r}{|u_c|}\\
    =& \dfrac{r\theta}{KI} \left( \dfrac{\mathcal{F}_L + 1}{\mathcal{F}_L} \right)\\
\end{array}
\end{equation}

## Decay rate and timescale comparison

Considering a tracer $C$ that is injected in the source point. This tracer follows a first-order decay reaction with a rate $\lambda$.

$\dfrac{dC}{dt} = -\lambda C$

With an initial condition $C(0) = C_0$, the concentration of that tracer as a function of space and time is,

$C(t) = C_{0}\exp{-\lambda t}$

$C_\tau = C_0 \exp{-\lambda \tau} =  C_0 \exp{\left(-\lambda\dfrac{r}{|u_c|}\right)}$

Hence, for our characteristic time ($t=\tau$), the relative concentration of that tracer will be:

$C_\tau = C_0\exp{ \left( - \lambda \dfrac{r \theta (\mathcal{F}_L + 1)}{KI \mathcal{F}_L} \right) }$

## Dilution effect

Assuming that perfect mixing has between the injection point and the distance between the wells, we could calculate a relative concentration that takes into account this mass balance. Notice that this analysis cannot be done directly from our previous assumptions because there is no mixing in potential flow. However, we can keep the characteristic velocity.

\begin{equation}
\begin{array}{rl}
    C_0 u_c \Delta y \Delta z \theta=& C_{\rm in} Q_{\rm in} + C_{\rm reg} Q_{\rm reg}\\
    C_0 u_c \Delta y \Delta z \theta=& C_{\rm in} Q_{\rm in}\\
    C_0 =& \dfrac{Q_{\rm in}}{u_c \Delta y \Delta z \theta} C_{in} \\
    =& \dfrac{Q_{\rm in}}{u_c \Delta y \Delta z \theta} C_{\rm in}\\
    =& \dfrac{Q_{\rm in}}{- \dfrac{KI}{\theta} \left( \dfrac{\mathcal{F}_L}{\mathcal{F}_L + 1} \right) \Delta y \Delta z \theta} C_{\rm in}\\
    =& \dfrac{Q_{\rm in}(\mathcal{F}_L + 1)}{KI \mathcal{F}_L \Delta y \Delta z} C_{\rm in}
\end{array}
\end{equation}

$\Delta y$ and $\Delta z$ are just lenghts around the arbitrary control volume. For our 2D case, $\Delta z = H$ and $\Delta y = $ size of the element in PFLOTRAN

$C_0 =
\dfrac{Q_{\rm in}(\mathcal{F}_L + 1)}{KI \mathcal{F}_L \Delta y H} C_{\rm in}$

### Which process is more dominant?

The effects of each process is more enhanced dependind of the system. For example, if $I$ is big, our characteristic time will be small, then, decay won't be as important as dilution could be in the aquifer. We are interested in finding which set of parameters maximize $C_\tau$, or minimize the log-reductions of the tracer concentration.

\begin{equation}
\begin{array}{rl}
C_\tau =& C_0 \exp{\left(-\lambda\dfrac{r}{|u_c|}\right)}\\
C_\tau =& \dfrac{C_{\rm in}Q_{\rm in}}{u_c \Delta y H \theta} \exp{\left(-\lambda\dfrac{r}{|u_c|}\right)}
\end{array}
\end{equation}

# Potential flow visualization

## Well flow is dominant

\begin{equation}
\begin{array}{rl}
\dfrac{dw}{dz} =& -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{f}{z} - \dfrac{1}{z-r}\right)\\
\\
\mathcal{F}_L =& \dfrac{K I}{\dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{f}{z} - \dfrac{1}{z-r}\right)}
\end{array}
\end{equation}

```python
%reset -f
import numpy as np
import matplotlib.pyplot as plt
import matplotlib

def ZFix(x,y):
    XX,YY = np.meshgrid(X,Y)
    return XX + (YY*1j)

def regionalFlow():
    return -K*I

def wellFlow(z):
    w = -(QIN/(2*PI*H))*(F/z - 1/(z-R))
    wAbs = np.abs(w)
    return (w, w.real, -w.imag, wAbs)

def totalFlow(z):
    reg  = regionalFlow()
    well = wellFlow(z)
    tot  = reg + well[0]
    totAbs = np.abs(tot)
    return (tot, tot.real,-tot.imag,totAbs)

def flowNumber(z):
    reg  = regionalFlow()
    well = wellFlow(z)[3]
    FL = np.abs(reg)/well
    return FL
```

```python
''' GLOBAL CONSTANTS '''
PI = 3.141592
K = 1.0E-2
#I = 6.6E-14
I = 1.0E-10
R = 40.
F = 1.5
QIN = 1.0/86400.
H = 1
#H = 20.

POINT_OUT = {"x":0,
             "y":0,
             "c":"r",
             "s":100,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Extraction"}

POINT_QIN = {"x":R,
             "y":0,
             "c":"b",
             "s":80,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Injection"}

X = np.linspace(-50,150,151)
Y = np.linspace(-100,100,33)
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)

fig, axs = plt.subplots(2,1,figsize=(12,10))
ax = axs[0]
st = ax.streamplot(X,Y,U,V,color="k",density=[0.8, 0.8])
fi = ax.pcolormesh(X,Y,C,shading='auto',vmin=0,vmax=5.0E-7,cmap="cool")
pi = ax.scatter(**POINT_QIN,zorder=3)
po = ax.scatter(**POINT_OUT,zorder=3)

cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
cbar.set_label(r'$\bf{|U|}$')

ax.legend(loc='lower right')

ax = axs[1]
fi = ax.pcolormesh(X,Y,np.log10(FL),shading='auto')
pi = ax.scatter(**POINT_QIN)
po = ax.scatter(**POINT_OUT)

ax.xaxis.set_tick_params(which="both",labelbottom=False,bottom=False)
ax.yaxis.set_tick_params(which="both",labelleft=False,left=False)
cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
ax.legend(loc='lower right')

plt.show()
```

```python
X = np.linspace(0.01,R-0.01,151)
Y = np.array([0.])
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)
fig, axs = plt.subplots(2,1,figsize=(8,6),sharex=True,\
                        gridspec_kw={"height_ratios":[3,1],"hspace":0})
ax = axs[0]
li = ax.plot(X,C[0],lw=5)
ax.set(xlim=[0,R],ylim=[0,1.0E-6],ylabel=r"$\bf{|U|}$ [m/s]")

ax = axs[1]
li = ax.plot(X,np.log10(FL[0]),lw=3,c="gray")
ax.set(ylim=[-7.5,-4.5],xlabel=r"$\bf{X}$ [m]",ylabel=r"$\bf{\log(\mathcal{F}_L)}$")

plt.show()
```

## Regional flow is dominant

```python
%reset -f
import numpy as np
import matplotlib.pyplot as plt
import matplotlib

def ZFix(x,y):
    XX,YY = np.meshgrid(X,Y)
    return XX + (YY*1j)

def regionalFlow():
    return -K*I

def wellFlow(z):
    w = -(QIN/(2*PI*H))*(F/z - 1/(z-R))
    wAbs = np.abs(w)
    return (w, w.real, -w.imag, wAbs)

def totalFlow(z):
    reg  = regionalFlow()
    well = wellFlow(z)
    tot  = reg + well[0]
    totAbs = np.abs(tot)
    return (tot, tot.real,-tot.imag,totAbs)

def flowNumber(z):
    reg  = regionalFlow()
    well = wellFlow(z)[3]
    FL = np.abs(reg)/well
    return FL
```

```python
''' GLOBAL CONSTANTS '''
PI = 3.141592
K = 1.0E-2
I = 6.6E-4
R = 40.
F = 10
QIN = 1.0/86400.
H = 20.

POINT_OUT = {"x":0,
             "y":0,
             "c":"r",
             "s":100,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Extraction"}

POINT_QIN = {"x":R,
             "y":0,
             "c":"b",
             "s":80,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Injection"}

X = np.linspace(-50,150,151)
Y = np.linspace(-100,100,33)
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)

fig, axs = plt.subplots(2,1,figsize=(12,10))
ax = axs[0]
st = ax.streamplot(X,Y,U,V,color="k",density=[0.8, 0.8])
fi = ax.pcolormesh(X,Y,C,shading='auto',vmin=0,vmax=2.0E-5,cmap="cool")
pi = ax.scatter(**POINT_QIN,zorder=3)
po = ax.scatter(**POINT_OUT,zorder=3)

cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
cbar.set_label(r'$\bf{|U|}$')

ax.legend(loc='lower right')

ax = axs[1]
fi = ax.pcolormesh(X,Y,np.log10(FL),shading='auto')
pi = ax.scatter(**POINT_QIN)
po = ax.scatter(**POINT_OUT)

ax.xaxis.set_tick_params(which="both",labelbottom=False,bottom=False)
ax.yaxis.set_tick_params(which="both",labelleft=False,left=False)
cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
ax.legend(loc='lower right')

plt.show()
```

```python
X = np.linspace(0.01,R-0.01,151)
Y = np.array([0.])
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)
fig, axs = plt.subplots(2,1,figsize=(8,6),sharex=True,\
                        gridspec_kw={"height_ratios":[3,1],"hspace":0})
ax = axs[0]
li = ax.plot(X,C[0],lw=5)
ax.set(xlim=[0,R],ylim=[0,1.0E-5],ylabel=r"$\bf{|U|}$ [m/s]")

ax = axs[1]
li = ax.plot(X,np.log10(FL[0]),lw=3,c="gray")
ax.set(xlabel=r"$\bf{X}$ [m]",ylabel=r"$\bf{\log(\mathcal{F}_L)}$")
ax.yaxis.set_tick_params(which="both",labelright=True,right=True,labelleft=False,left=False)

plt.show()
```

___
# Implementation to find worst cases

```python
%reset -f
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
from os import system
from matplotlib.gridspec import GridSpec

''' GLOBAL CONSTANTS '''
PI = 3.141592
THETA = 0.35
```

\begin{equation}
\begin{array}{rl}
    \mathcal{F}_L =& \dfrac{4 K I H r}{Q_{in}(1+f)}\\
    u_{c} =& \dfrac{-KI\mathcal{F}_L}{\theta \left(\mathcal{F}_L+1\right)}\\
    \tau =& -\dfrac{r}{|u_{c}|}\\
    C_{\tau,{\rm decay}}=& C_0 \exp{\left(-\lambda \tau \right)}\\
    C_{\tau,{\rm dilut}} =& C_{in} \left( \dfrac{Q_{in}}{u_c \Delta y \Delta z} \right)\\
    C_{\tau,{\rm both}} =& \dfrac{C_{\rm in}Q_{\rm in}}{u_c \Delta y H \theta} \exp{\left(-\lambda\dfrac{r}{|u_c|}\right)}
\end{array}
\end{equation}

```python
def flowNumber():
    return (4.0*K*I*H*r) / (Qin*(1+f))

def uChar():
    return -(K*I*flowNumber())/(THETA*(flowNumber() + 1))

def tChar():
    return -r/uChar()

def cDecay():
    return C0 * np.exp(-decayRate * tChar())

def cDilut():
    return (C0 * Qin) / (-uChar() * delY * delZ * THETA)

def cBoth():
    return (C0 * Qin) / (-uChar() * delY * delZ * THETA) * np.exp(-decayRate * tChar())

def findSweet():
    deltaConc = np.abs(cBoth() - np.max(cBoth()))
    return np.argmin(deltaConc)
```

**Some constant Parameters**

```python
K = 10**-2
Qin = 0.24/86400
f = 10
H = 20
r = 40
I = 0.001
C0 = 1.0
decayRate = 3.5353E-06
delY,delZ = 1.35,H
```

```python
print("Nondim Flow  = {:.2E}".format(flowNumber()))
print("Charac. Vel  = {:.2E} m/s".format(uChar()))
print("Charac. time = {:.2E} s".format(tChar()))
```

    Nondim Flow  = 1.05E+03
    Charac. Vel  = -2.85E-05 m/s
    Charac. time = 1.40E+06 s

```python
print("Rel concenc. due decay = {:.2E}".format(cDecay()))
print("Rel conc. due dilution = {:.2E}".format(cDilut()))
print("Rel conc. due both eff = {:.2E}".format(cBoth()))
```

    Rel concenc. due decay = 7.05E-03
    Rel conc. due dilution = 1.03E-02
    Rel conc. due both eff = 7.26E-05

```python
I = 10**np.linspace(-5,0,num=100)

c1 = cDecay()
c2 = cDilut()
c3 = cBoth()
i = findSweet()

#worstC = np.mean([c1[i],c2[i]])
worstC = c3[i]
worstI = I[i]
```

```python
print(I[i])
```

    0.0047508101621027985

```python
# create objects
fig, axs = plt.subplots(2,2,sharex=True, sharey=False,\
                        figsize=(12,8),gridspec_kw={"height_ratios":[1,3],"hspace":0.04,"wspace":0.35})

bbox = dict(boxstyle='round', facecolor='mintcream', alpha=0.90)

arrowprops = dict(
    arrowstyle="->",
    connectionstyle="angle,angleA=90,angleB=10,rad=5")

annotation = \
    r"$\bf{-\log(C/C_0)} = $" + "{:.1f}".format(-np.log10(worstC)) + \
    "\n@" + r" $\bf{I} = $" + "{:.1E}".format(worstI)

information = \
    r"$\bf{K}$" + " = {:.1E} m/s".format(K) + "\n"\
    r"$\bf{H}$" + " = {:.1f} m".format(H) + "\n"\
    r"$\bf{r}$" + " = {:.1f} m".format(r) + "\n"\
    r"$\bf{Q_{in}}$" + " = {:.2f} m³/d".format(Qin*86400) + "\n"\
    r"$\bf{f}$" + " = {:.1f}".format(f) + "\n"\
    r"$\bf{\lambda}$" + " = {:.2E} 1/s".format(decayRate)

# Ax1 - Relative concentration
ax = axs[1,0]
ax.plot(I,c1,label="Due decay",lw=3,ls="dashed",alpha=0.8)
ax.plot(I,c2,label="Due dilution",lw=3,ls="dashed",alpha=0.8)
#ax.plot(I,np.minimum(c1,c2),label="Overall effect",lw=3,c='k',alpha=0.9)
ax.plot(I,c3,label="Overall effect",lw=3,c='k',alpha=0.9)

ax.set_xscale("log")
ax.set_yscale("log")

ax.set_ylim(1.0E-10,1)
ax.set_xlim(1.0E-4,1.0E-1)

ax.set_xlabel("Water table gradient\n$I$ [m/m]")
ax.set_ylabel("Relative Concentration\n$C/C_0$ [-]")
ax.legend(loc="lower left",shadow=True)

ax.annotate(annotation,(worstI,worstC),
    xytext=(0.05,0.85), textcoords='axes fraction',
    bbox=bbox, arrowprops=arrowprops)

ax.text(0.65,0.05,information,bbox=bbox,transform=ax.transAxes)
ax.axvline(x=I[i], lw = 1, ls = "dashed", c = "red")

####################################
# Ax2 - log-removals
ax = axs[1,1]
ax.plot(I,-np.log10(c1),label="Due decay",lw=3,ls="dashed",alpha=0.8)
ax.plot(I,-np.log10(c2),label="Due dilution",lw=3,ls="dashed",alpha=0.8)
#ax.plot(I,-np.log10(np.minimum(c1,c2)),label="Overall effect",lw=3,c='k',alpha=0.9)
ax.plot(I,-np.log10(c3),label="Overall effect",lw=3,c='k',alpha=0.9)

ax.set_xscale("log")
ax.set_ylim(0,10)
ax.set_xlim(1.0E-4,1.0E-1)
ax.set_xlabel("Water table gradient\n$I$ [m/m]")
ax.set_ylabel("log-reductions\n$-\log(C/C_0)$ [-]")
ax.legend(loc="lower right",shadow=True)

ax.annotate(annotation,(worstI,-np.log10(worstC)),
    xytext=(0.05,0.10), textcoords='axes fraction',
    bbox=bbox, arrowprops=arrowprops)

ax.text(0.65,0.70,information,bbox=bbox,transform=ax.transAxes)
ax.axvline(x=I[i], lw = 1, ls = "dashed", c = "red")

####################################
#Ax3 - Flow number
ax = axs[0,1]
ax.plot(I,flowNumber(),label="flowNumber",lw=3,c="gray")
ax.axhline(y=1.0)
ax.set_xscale("log")
ax.set_yscale("log")
ax.xaxis.set_tick_params(which="both",labeltop='on',top=True,bottom=False)

ax.set_ylabel("Nondim. flow number\n$\mathcal{F}_L$ [-]")
ax.axvline(x=I[i], lw = 1, ls = "dashed", c = "red")

#Ax4 - Flow number
ax = axs[0,0]
ax.plot(I,flowNumber(),label="flowNumber",lw=3,c="gray")
ax.axhline(y=1.0)
ax.set_xscale("log")
ax.set_yscale("log")
ax.xaxis.set_tick_params(which="both",labeltop='on',top=True,bottom=False)

ax.set_ylabel("Nondim. flow number\n$\mathcal{F}_L$ [-]")
ax.axvline(x=I[i], lw = 1, ls = "dashed", c = "red")

plt.show()
```

____
# Find the worst case
## >> Geometric parameters $H$ and $r$

```python
from drawStuff import *
```

```python
K = 10**-2
Qin = 0.24/86400
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Harray = np.array([2.,5.,10.,20.,50.])
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Harray)])
Ii = np.zeros([len(rarray),len(Harray)])
FLi = np.zeros([len(rarray),len(Harray)])

for hi,H in enumerate(Harray):
    for ri,r in enumerate(rarray):
        i = findSweet()

        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,hi]  = worstC
        Ii[ri,hi]  = worstGradient
        FLi[ri,hi] = worstFlowNumber
```

```python
myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Aquifer thickness\n$\\bf{H}$ (m)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":FLi.T},\
               xlabel=Harray,ylabel=rarray,myLabels=myLabels)
```

## >>Well parameters

```python
K = 10**-2
H = 20
r = 40
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
f_array = np.array([1,10.,100.,1000.,10000.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(Qin_array),len(f_array)])
Ii = np.zeros([len(Qin_array),len(f_array)])
FLi = np.zeros([len(Qin_array),len(f_array)])

for fi,f in enumerate(f_array):
    for qi,Qin in enumerate(Qin_array):
        i = findSweet()
        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[qi,fi] = worstC
        Ii[qi,fi] = worstGradient
        FLi[qi,fi] = worstFlowNumber
```

```python
myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Extraction to injection ratio\n$\\bf{f}$ (-)",
          "X": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":FLi.T},\
               xlabel=f_array,ylabel=np.round(Qin_array*86400,decimals=2),myLabels=myLabels)
```

## Hydraulic conductivity

```python
Qin = 0.24/86400
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Karray = 10.**np.array([-1.,-2.,-3.,-4.,-5.])
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Karray)])
Ii = np.zeros([len(rarray),len(Karray)])
FLi = np.zeros([len(rarray),len(Karray)])

for ki,K in enumerate(Karray):
    for ri,r in enumerate(rarray):
        i = findSweet()

        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,ki] = worstC
        Ii[ri,ki] = worstGradient
        FLi[ri,ki] = worstFlowNumber
```

```python
myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Hydraulic conductivity\n$\\bf{K}$ (m/s)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":FLi.T},\
               xlabel=Karray,ylabel=rarray,myLabels=myLabels)
```

>> EXPERIMENTAL

```python

```

## r and Qin

```python
K = 10**-2
H = 20
f = 10
C0 = 1.0
decayRate = 3.5353E-06

Qin_array = np.array([0.24,1.0,10.0,100.])/86400.
rarray = np.array([5.,10.,40.,100.])
Iarray = 10**np.linspace(-5,0,num=100)

Ci = np.zeros([len(rarray),len(Qin_array)])
Ii = np.zeros([len(rarray),len(Qin_array)])
FLi = np.zeros([len(rarray),len(Qin_array)])

for qi,Qin in enumerate(Qin_array):
    for ri,r in enumerate(rarray):
        i = findSweet()

        worstC = -np.log10(cBoth()[i])
        worstGradient = Iarray[i]
        worstFlowNumber = flowNumber()[i]

        Ci[ri,qi] = worstC
        Ii[ri,qi] = worstGradient
        FLi[ri,qi] = worstFlowNumber
```

```python
myLabels={"Title": { 0: r"$\bf{-\log (C_{\tau}/C_0)}$",
                     1: r"$\bf{I}$ (%)",
                     2: r"$\log(\mathcal{F}_L)$"},
          "Y": "Injection flow rate\n$\\bf{Q_{in}}$ (m³/d)",
          "X": "Setback distance\n$\\bf{r}$ (m)"}

threeHeatplots(data={"I":Ii.T,"C":Ci.T,"FL":FLi.T},\
               ylabel=rarray,xlabel=np.round(Qin_array*86400,decimals=2),myLabels=myLabels)
```

```python

```


## Regression

$J=\displaystyle \sum_{i=1}^n(\vec{p}\cdot{\vec{\tilde{x_i}}}-y_i)^2$

$\displaystyle \vec{p}^* = arg \min_{\vec{p}} J(\vec{p})$

$ \displaystyle \forall i :\frac{\partial J}{\partial p_j}=0 \;\;\quad \frac{\partial J}{\partial p_j} =\sum_{i=1}^n 2(\vec{p}\cdot{\vec{\tilde{x_i}}}-y_i)\vec{\tilde{x_i}}$

solution is $\displaystyle \quad \vec{p}^*= (X^TX)^{-1}X^T\vec{y}$

**Over-determined**:

$\displaystyle \quad arg \min_{\vec{p}}(||A\vec{p}-\vec{b}||_2 +\lambda g(p)) $

**Under-determined**:

$\displaystyle \quad arg \min_{\vec{p}} g(p) \quad$ subject to  $||A\vec{p}-\vec{b}||_2\le \epsilon$

## Loss

$\displaystyle L(\hat{y},y)=\frac{1}{N} \sum_i (\hat{y}_i -y_i)^2 \quad$   MSE

$\displaystyle L(\hat{y},y)=-(y \,\text{log}\hat{y} +(1-y)\,\text{log}(1-\hat{y})) \quad $ Cross Entropy

$\displaystyle L(\hat{y},y)=-\sum_c(y_{o,c})\,\text{log}p_{o,c} \quad $ Cross Entropy Multi class

## Cost

$\displaystyle J(W,b) = \frac{1}{M}\sum_{i=1}^m L(\hat{y}^{(i)},y^{(i)})$

## Logistic Regression

We'll start with single neuron logistic regression using activation function **sigmoid**

* Loss Cross Entropy
* Maximum Likelihood
* Convex Optimization

### Computation Graph

$\hat{y} = \sigma(\vec{w}^T\vec{x} + b) \rightarrow $ cross entropy $\rightarrow L(\hat{y},y)$

## Multiple Examples Training set

* **Forward Propagation**: Computing the loss through forward pass for a single training example
* **Backward Propagation**: Computing gradients of parameters through backward pass for a single training example $\\$

* **Batch**: Traning set could be divided into smaller sets called batches
* **Iteration**: When an entire batch is passed both forward and backward
* **Epoch**: When an entire dataset is passed both forward and backward through the NN once

## Multiple Outputs
Sigmoid -> softmax with one hot encoding

$softmax (\hat{y})_i= \frac{e^{y_i}}{\sum_ie^{y_i}}$

## Other Loss/Activations
* Changing **activation** will change the gradients

* Changing **loss** will change the gradients and could make the composition unseparable

* Most activations we discussed have analytic gradients. It is possible that there is no analytic expression.

* Gradients could be evaluated numerically
    - When **analytic** expression is not available
    - When it is **faster** to evaluate them numerically

* Use central difference formula
* Check against analytic gradient for several examples

## Curriculum Learning
* Training machine learning models with particular order. Starting with easier subtasks and gradually increase the difficulty level of the tasks (For example, NLP problem learn words and then learn sentences)

* Both traning set and cost functions are updated accordingly

## Stochastic Gradient Descent

**Almost surely convergence**

* Performs an update for each training example $x^{(i)}$ and label $y^{(i)}$

* The values of the loss and parameters will fluctuate
    - (+) will discover better minimums
    - (-) convergence to chosen minimum will keep overshooting

* Learning rate plays a very important role

## Mini-batch GD

$$w_{k+1} =w_k -\alpha \cdot \nabla_wJ(w;x^{(i:i+n)};y^{(i:i+n)})$$

* Mini-batch GD is a hybrid method between GD and SGD.
* Performs an update for every mini-batch of n traning examples.

    - (+) reduces the variance of the parameter updates
    - (+) efficient in computing the gradient w.r.t a mini-batch
* mini-batch sizes range between 50-256

**Challenges**
* Chossing a proper learning rate can be difficult

* Learning rate smart schedule
    - Annealing
    - Change of J below threshold
* Variable learning for different parameters
* Suboptimal local (saddle points)

**Parameters**
* Model Parameters: W, b, activation, output, cost

* Hyper-parameters: Batch/minibatch size, learning parameters, external parameters

## Methods for choosing learning rate
**1. Learning rate decay**

* $\displaystyle \alpha = \frac{\alpha_0}{1+\text{decr}\cdot\text{epnum}}$

* $\displaystyle \alpha = d^{\text{epnum}}\cdot \alpha_0$

* $\displaystyle \alpha = \frac{d\cdot\alpha_0}{\sqrt{\text{epnum}}}$

**2. Momentum Method**

$$ v_{k+1} =\gamma v_k+\alpha\cdot\nabla_wJ(w_k)$$
$$w_{k+1} =w_k -v_{k+1}$$

<div class="verticalhorizontal">

</div>

**3. Nesterov Accelerated Gradient**

$$v_{k+1}=\gamma v_k +\alpha \cdot\nabla_wJ(w_k-\gamma v_k)$$
$$w_{k+1}=w_k-v_{k+1}$$

**4. Adagrad**
$$w_{k+1,j} =w_{k,j} -\frac{\alpha}{\sqrt{G_{k,jj}+\epsilon}}\cdot g_{k,j}$$

g is our gradient

* Adagrad uses a different learning rate for every parameter $w_j$ at every step $k$. $G$ is diagonal matrix of sum squared gradient values.

* Performs smaller update (i.e. low learning rates) for parameters associated with frequntly occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.

**5. RMSProp**

$$E[g^2]_k =\gamma E[g^2]_{k-1} + (1-\gamma)g_k^2$$
$$w_{k+1} =w_k -\frac{\eta}{\sqrt{E[g^2]_k+ \epsilon}}g_k$$

* Prevents accumulation by adding regularizing term in the running average (exponentially decaying)

* Beneficial for RNNs

**6. Adadelta**

\begin{align}
E[\Delta w^2]_k = \gamma E[\Delta &w^2]_{k-1} +(1-\gamma) \Delta w_k^2\\
\text{RMS}[\Delta w]_k &=\sqrt{E[\Delta w^2]_k +\epsilon}\\
                    \Delta w_k    &=-\frac{\text{RMS}[\Delta w]_{k-1}}{\text{RMS}[g]_k}g_k\\
                    w_{k+1}&=w_k+\Delta w_k
\end{align}

* Generalizes RMSProp /Adagard for considering RMS instead of accumulation of grad

* No learning rate parameter

**7. AdaM - Adaptive Moment Estimation**

\begin{align}
m_k &=\beta_1m_{k-1} + (1-\beta_1)g_k\\
v_k &=\beta_2v_{k-1} +(1-\beta_2)g_k^2\\
w_{k+1}&=w_k -\frac{\eta}{\sqrt{\hat{v}_k}+\epsilon}\hat{m}_k
\end{align}

* Keeps track of 2 moments: mean and variance
* Normalizes them to prevent biases
    - $\displaystyle \hat{m}_k =\frac{m_k}{1-\beta_1^k}$
    - $\displaystyle \hat{v}_k =\frac{v_k}{1-\beta_2^k}$

**Additional**
- AdaMax: Generalization of AdaM to L-infinity norm
- Nadam: Nesterov AdaM
- AMSgrad: Max normalization instead of exponential in AdaM

**Notes on Choosing Opimizers**

* RSMProp & AdaDelta adaptive
* AdaM adaptive + momentum  -> robust
* SGD as a first pass


```python
from IPython.display import display, Latex
from sympy import *
init_printing(use_latex='mathjax')

x, y = symbols('x y')
display(x,y)

int_x = Integral(cos(x)*exp(x), x)
result ="$${} = {}$$".format(latex(int_x), latex(int_x.doit()))
display(Latex(result))

derv_x = Derivative(cos(x)*exp(x), x)
result ="$${} = {}$$".format(latex(derv_x), latex(derv_x.doit()))
display(Latex(result))
```

$\displaystyle x$

$\displaystyle y$

$$\int e^{x} \cos{\left(x \right)}\, dx = \frac{e^{x} \sin{\left(x \right)}}{2} + \frac{e^{x} \cos{\left(x \right)}}{2}$$

$$\frac{d}{d x} e^{x} \cos{\left(x \right)} = - e^{x} \sin{\left(x \right)} + e^{x} \cos{\left(x \right)}$$

```python
from IPython.display import display

def show(a, fmt='png'):
    import PIL.Image
    from io import BytesIO
    import IPython.display
    import numpy as np
    f = BytesIO()
    PIL.Image.fromarray(np.uint8(a)).save(f, fmt)
    IPython.display.display(IPython.display.Image(data=f.getvalue()))

from sympy import *
init_printing()
3+4j, Rational(1,3), sqrt(8), sqrt(-1), pi, E**2, oo, -oo
import IPython.display
display(3+4j, Rational(1,3), sqrt(8), sqrt(-1), pi, E**2, oo, -oo)
# x = symbols('x')
# a = Integral(cos(x)*exp(x), x)
# Eq(a, a.doit())
```

```python
alpha, beta, nu = symbols('alpha beta nu')
display(alpha, beta, nu)
```

```python
alpha, beta
```

```python
3+4j, Rational(1,3), sqrt(8), sqrt(-1), pi, E**2, oo, -oo
```

    ((3+4j), 1/3, 2*sqrt(2), I, pi, exp(2), oo, -oo)

```python
diff(sin(x)*exp(x),x)
```

```python
integrate(exp(x)*sin(x)+exp(x)*cos(x),x)
```

```python
integrate(sin(x**2),(x,-oo,oo))
```

```python
import sympy
from sympy import init_printing, Integral, Symbol

x = Symbol('x')
expr1 = 2*x**2 + 3*x + 2

init_printing(use_latex='mathjax')
print("Here is an integral ")
Integral(expr1, x), sympy.integrate(expr1)
```

    Here is an integral

$\displaystyle \left( \int \left(2 x^{2} + 3 x + 2\right)\, dx, \  \frac{2 x^{3}}{3} + \frac{3 x^{2}}{2} + 2 x\right)$

```python
import sympy
from sympy import init_printing, Integral, Symbol

x = Symbol('x')
expr1 = 2*x**2 + 3*x + 2

init_printing(use_latex='mathjax')
print("Here is list")
expr = Limit(sin(x)/x,x,0)
[expr,expr.doit()]
```

    Here is list

$\displaystyle \left[ \lim_{x \to 0^+}\left(\frac{\sin{\left(x \right)}}{x}\right), \  1\right]$

```python
import sympy
from sympy import init_printing, Integral, Symbol

x = Symbol('x')
expr1 = 2*x**2 + 3*x + 2

init_printing(use_latex='mathjax')
print("Here is list")
expr = Limit(sin(x)/x,x,0)
expr,expr.doit()
```

    Here is list

$\displaystyle \left( \lim_{x \to 0^+}\left(\frac{\sin{\left(x \right)}}{x}\right), \  1\right)$

```python
solve(x**2+1, x)
```

$\displaystyle \left[ - i, \  i\right]$

```python
solve(x**2 - 2, x)
```

$\displaystyle \left[ - \sqrt{2}, \  \sqrt{2}\right]$

```python
y = Function('y')
t = symbols('t')
expr = Eq(y(t).diff(t,t) - y(t), exp(t))
expr,y(t)
```

$\displaystyle \left( - y{\left(t \right)} + \frac{d^{2}}{d t^{2}} y{\left(t \right)} = e^{t}, \  y{\left(t \right)}\right)$

```python
dsolve(expr,y(t))
```

$\displaystyle y{\left(t \right)} = C_{2} e^{- t} + \left(C_{1} + \frac{t}{2}\right) e^{t}$

```python
latex(expr)
```

    '- y{\\left(t \\right)} + \\frac{d^{2}}{d t^{2}} y{\\left(t \\right)} = e^{t}'

```python
from sympy.plotting import plot
x = symbols('x')
p = plot(2*x+3, 3*x+1, legend=True, show=False)
p.show()
```

```python
i = tensor.Idx('i',3)
j = tensor.Idx('j',3)
k = tensor.Idx('k',3)
l = tensor.Idx('l',3)

F = MatrixSymbol('F', 3, 3)
i,j,k,l,F
```

$\displaystyle \left( Idx\left(i, \left( 0, \  2\right)\right), \  Idx\left(j, \left( 0, \  2\right)\right), \  Idx\left(k, \left( 0, \  2\right)\right), \  Idx\left(l, \left( 0, \  2\right)\right), \  F\right)$

```python
from sympy.tensor.array import *
var("a,b,c,d,e,f")
X = Array([[a, b, c], [d, e, f]])
X
var("w1,w2,w3")
W = Array([w1, w2, w3])
W
tp = tensorproduct(X, W)
tp
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}a w_{1} & a w_{2} & a w_{3}\\b w_{1} & b w_{2} & b w_{3}\\c w_{1} & c w_{2} & c w_{3}\end{matrix}\right] & \left[\begin{matrix}d w_{1} & d w_{2} & d w_{3}\\e w_{1} & e w_{2} & e w_{3}\\f w_{1} & f w_{2} & f w_{3}\end{matrix}\right]\end{matrix}\right]$

```python
stc = sum(tensorcontraction(tp, (1, 2)))
stc
```

$\displaystyle a w_{1} + b w_{2} + c w_{3} + d w_{1} + e w_{2} + f w_{3}$

```python
derive_by_array(stc, W)
```

$\displaystyle \left[\begin{matrix}a + d & b + e & c + f\end{matrix}\right]$

```python
X = IndexedBase("X")
W = IndexedBase("W")
var("i,j,M,K", integer=True)
s = Sum(X[i, j]*W[j], (i, 1, M), (j, 1, K))
s
```

$\displaystyle \sum_{\substack{1 \leq i \leq M\\1 \leq j \leq K}} {W}_{j} {X}_{i,j}$

```python
s.diff(W[j])
```

$\displaystyle \sum_{\substack{1 \leq i \leq M\\1 \leq j \leq K}} {X}_{i,j}$

```python
s.diff(W[k])
```

$\displaystyle \sum_{\substack{1 \leq i \leq M\\1 \leq j \leq K}} \delta_{j Idx\left(k, \left( 0, \  2\right)\right)} {X}_{i,j}$

```python
# https://docs.sympy.org/latest/modules/tensor/array.html

Array([[[1,2,3],[2,3,4],[3,4,5]],[[1,2,3],[2,3,4],[3,4,5]]])
)
```

$\displaystyle \left[\begin{matrix}\left[\begin{matrix}1 & 2 & 3\\2 & 3 & 4\\3 & 4 & 5\end{matrix}\right] & \left[\begin{matrix}1 & 2 & 3\\2 & 3 & 4\\3 & 4 & 5\end{matrix}\right]\end{matrix}\right]$

```python
from sympy.physics.mechanics import *
a = Array([1,2,3,4])
b = Array([2,3,4,5])
c=tensorproduct(a,b)
d=tensorproduct(b,a)
#dot(a,b)
```

```python
expr = sin(x**2)
expr, integrate(expr,(x,-oo,oo))
```

$\displaystyle \left( \sin{\left(x^{2} \right)}, \  \frac{\sqrt{2} \sqrt{\pi}}{2}\right)$

```python
solve(x**2+1,x)
```

$\displaystyle \left[ - i, \  i\right]$

```python

```


# Verlet Algorithm

In molecular dynamics, the most commonly used time integration algorithm is probably the so-called Verlet algorithm [L. Verlet, Computer experiments on classical fluids. I. Thermodynamical properties of Lennard-Jones molecules, Physical Review 159, 98 (1967)]. The basic idea is to write two third-order Taylor expansions for the positions ${\bf r} (t)$, one forward and one backward in time. Calling $\bf v$ the velocities, $\bf a$ the accelerations, and $\bf b$ the third derivatives of ${\bf r}$ with respect to $t$, one has:
$$\begin{equation}
{\bf r} (t+\Delta t) = {\bf r} (t) + {\bf v} (t) \Delta t + \frac{1}{2} {\bf a}(t) \Delta t^2 + (1/6) {\bf b} (t) \Delta t^3
 + O(\Delta t^4) \end{equation}
$$
$$
\begin{equation}
{\bf r} (t-\Delta t) = {\bf r} (t) - {\bf v} (t) \Delta t + \frac{1}{2} {\bf a}(t) \Delta t^2 - (1/6) {\bf b} (t) \Delta t^3
 + O(\Delta t^4) \end{equation}
$$
Adding the two expressions gives
$$\begin{equation}
{\bf r} (t+\Delta t) = 2{\bf r} (t) - {\bf r} (t-\Delta t)
 + {\bf a} (t) \Delta t^2 + O(\Delta t^4) \end{equation}
$$
This is the basic form of the Verlet algorithm. Since we are integrating Newton's equations, ${\bf a} (t)$ is just the force divided by the mass, and the force is in turn a function of the positions ${\bf r} (t)$:
$$\begin{equation}
{\bf a} (t) = - \frac{1}{m} {\bf\nabla} V\left( {\bf r}(t) \right) \end{equation}
$$
As one can immediately see, the truncation error of the algorithm when evolving the system by $\Delta t$ is of the order of  $\Delta t^4$, even if third derivatives do not appear explicitly. This algorithm is at the same time simple to implement, accurate and stable, explaining its large popularity among molecular dynamics simulators.

While the velocities are not needed for the time evolution, their knowledge is sometimes necessary. Moreover, they are required to compute the kinetic energy $K$, whose evaluation is necessary to test the conservation of the total energy $E=K+V$. This is one of the most important tests to verify that a MD simulation is proceeding correctly. One could compute the velocities from the positions by subtracting the previous expression to obtain:

$$\begin{equation}
{\bf v} (t) = \frac { {\bf r}(t+\Delta t) - {\bf r}(t-\Delta t) }
 { 2 \Delta t } . \end{equation}
$$
However, the error associated to this expression is of order $\Delta t^2$ rather than $\Delta t^4$.

The main problem with the Verlet algorithm is that it is not self starting, and the first step needs to be computed by different means. An additional problem is that the new velocity is found by computing the difference between two quantities of the same order of magnitude. When using computers which always operate with finite numerical precision, such an operation results in a loss of numerical precision and may give rise to substantial roundoff error.

An even better implementation of the same basic algorithm is the so-called "**velocity Verlet scheme**", where positions, velocities and accelerations at time $t+\Delta t$ are obtained from the same quantities at time $t$ in the following way:

$$\begin{eqnarray}
{\bf r} (t + \Delta t) &=& {\bf r} (t) + {\bf v} (t) \Delta t + (1/2) {\bf a} (t) \Delta t^2 \\
{\bf v} (t + \Delta t/2) &=& {\bf v} (t) + (1/2) {\bf a} (t) \Delta t \\
{\bf a} (t + \Delta t) &=& - (1/m) {\bf\nabla} V \left( {\bf r}(t+\Delta t) \right) \\
{\bf v} (t + \Delta t) &=& {\bf v} (t + \Delta t/2) + (1/2) {\bf a} (t + \Delta t) \Delta t
\end{eqnarray}$$

Note how we need $9N$ memory locations to save the $3N$ positions, velocities and accelerations, but we never need to have simultaneously stored the values at two different times for any one of these quantities.

Here, we modify the code for particle2 implementing velocity Verlet:

```python
class particle2(object):

    def __init__(self, mass=1., x=0., y=0., vx=0., vy=0.):
        self.mass = mass
        self.x = x
        self.y = y
        self.vx = vx
        self.vy = vy

    def euler(self, fx, fy, dt):
        self.vx = self.vx + fx*dt
        self.vy = self.vy + fy*dt
        self.x = self.x + self.vx*dt
        self.y = self.y + self.vy*dt

    def get_force(self):  # returns force per unit of mass (acceleration)
        GM=4*math.pi*math.pi # We use astronomical units
        r = math.sqrt(self.x*self.x+self.y*self.y)
        r3 = r * r * r
        fx = -GM*self.x/r3
        fy = -GM*self.y/r3
        return (fx,fy)

    def verlet(self, dt):
        (fx,fy) = self.get_force() # before I move to the new position
        self.x += self.vx*dt + 0.5*fx*dt*dt
        self.y += self.vy*dt + 0.5*fy*dt*dt
        self.vx += 0.5*fx*dt
        self.vy += 0.5*fy*dt
        (fx,fy) = self.get_force() # after I move to the new position
        self.vx += 0.5*fx*dt
        self.vy += 0.5*fy*dt

```

### Challenge 2.3:

Use velocity verlet to simulate the mini-solar system from Challenge 2.2.
Can you come up with a different way to write the algorithm such that you call get_force only once per move?

```python

```


# Symbolically Understanding Christoffel Symbol and Riemann Curvature Tensor using EinsteinPy

```python
import sympy
from einsteinpy.symbolic import MetricTensor, ChristoffelSymbols, RiemannCurvatureTensor

sympy.init_printing()  # enables the best printing available in an environment
```

### Defining the metric tensor for 3d spherical coordinates

```python
syms = sympy.symbols('r theta phi')
# define the metric for 3d spherical coordinates
metric = [[0 for i in range(3)] for i in range(3)]
metric[0][0] = 1
metric[1][1] = syms[0]**2
metric[2][2] = (syms[0]**2)*(sympy.sin(syms[1])**2)
# creating metric object
m_obj = MetricTensor(metric, syms)
m_obj.tensor()
```

### Calculating the christoffel symbols

```python
ch = ChristoffelSymbols.from_metric(m_obj)
ch.tensor()
```

```python
ch.tensor()[1,1,0]
```

### Calculating the Riemann Curvature tensor

```python
# Calculating Riemann Tensor from Christoffel Symbols
rm1 = RiemannCurvatureTensor.from_christoffels(ch)
rm1.tensor()
```

```python
# Calculating Riemann Tensor from Metric Tensor
rm2 = RiemannCurvatureTensor.from_metric(m_obj)
rm2.tensor()
```

### Calculating the christoffel symbols for Schwarzschild Spacetime Metric
  - The expressions are unsimplified

```python
syms = sympy.symbols("t r theta phi")
G, M, c, a = sympy.symbols("G M c a")
# using metric values of schwarschild space-time
# a is schwarzschild radius
list2d = [[0 for i in range(4)] for i in range(4)]
list2d[0][0] = 1 - (a / syms[1])
list2d[1][1] = -1 / ((1 - (a / syms[1])) * (c ** 2))
list2d[2][2] = -1 * (syms[1] ** 2) / (c ** 2)
list2d[3][3] = -1 * (syms[1] ** 2) * (sympy.sin(syms[2]) ** 2) / (c ** 2)
sch = MetricTensor(list2d, syms)
sch.tensor()
```

```python
# single substitution
subs1 = sch.subs(a,0)
subs1.tensor()
```

```python
# multiple substitution
subs2 = sch.subs([(a,0), (c,1)])
subs2.tensor()
```

```python
sch_ch = ChristoffelSymbols.from_metric(sch)
sch_ch.tensor()
```

### Calculating the simplified expressions

```python
simplified = sch_ch.simplify()
simplified
```

docs/source/examples/Symbolically Understanding Christoffel Symbol and Riemann Curvature Tensor using EinsteinPy.ipynb


# Normal Equations

Real numbers only this time.

## Linear Systems of Equations

Linear equations are of the form $Ax = b$ where $A$ is a matrix and $x$ and $b$ are vectors. The rows of $A$ and $b$ form a system of equations that must be simultaneously satisfied by the entries of $x$. If $x,b\in\mathbb{R}^n$, then the solutions to the equation of a single row corresponds to an $n-1$-dimensional hyperplane. If the rows of $A$ are linearly independent, then solutions that that simultaneously satisfy the equations in $k$-rows correspond to the $n-k$-dimensional intersection of $k$ $n$-dimensional hyperplanes. The solution of $x$ that satisfies all $n$ equations is a $n-n = 0$-dimensional point, and so $x$ is uniquely determined. If any two rows of $A$ are not linearly independent, then the hyperplanes that correspond to values of $x$ that satisfy them overlap exactly, and their intersection is $n$ dimensional, rather than $n-1$ dimensional. In this case, the value of $x$ that satisfies all rows of $A$ is not narrowed down to a single point. The system of equations is said to be *underdetermined*:. This is equivalently the case when $A$ has $m<n$ rows.

### Square Matices $A\in\mathbb{R}^{nxn}$
If $A\in\mathbb{R}^{nxn}$, then the solution to the system is formally $x = A^{-1}b$, where $A^{-1}$ is the matrix inverse, satisfying $A^{-1}A=I$, where $I$ is the identity matrix.

Since $Ax=b$ is the same as expressing $b$ in terms of a linear combination of the columns of $A$, the entries of $x$ can be interpreted as the coefficients resulting from the projection of $b$ into the column space of $A$. Therefore, for an orthonormal matrix, the $A^-1$ is simply $A^T$

### Rectangular Matrices $A\in\mathbb{R}^{mxn}$
If $A\in\mathbb{R}^{mxn}$ with $m>n$ rows, then there need not be any point $x\in\mathbb{R}^n$ in which the $m$ hyperplanes all intersect. In that case, the system does not have a solution $x\in\mathbb{R}^n$, and the system is considered *overdetermined*. (The intersection of $m$ distinct hyperplanes in $n$ dimensional space would have negative dimension $(n-m)<0$ if $m>n$, which my feeble brain can't make sense of.)

## The Normal Equations
In the overdetermined case $A^{mxn}$ with $m>n$, the columns of $A$ do not span $\mathbb{R}^m$ and therefore $b\in\mathbb{R}^m$ may have some component $\epsilon$ that lies outside of the column space of $A$. In that case, no linear combination $x$ of the columns of $A$ can express $b$ perfectly, but we might look for approximate solutions $\hat{x}$ so that:

\begin{equation}
A\hat{x} + \epsilon = b
\end{equation}

A natural approach for picking an approximate solution $\hat{x}$ is to look for the projection of $b$ in the column space of $A$, which can be thought of as looking for the shadow of an $m$-dimensional vector in $n$ dimensional space.

The projection maximizes the dot product $(A\hat{x})\cdot b$, and hence minimizes the length of the difference vector $\epsilon$. In turn, the length of the difference vector $\epsilon$ is $\sqrt{\epsilon\cdot\epsilon}$, which is monotonic to $\epsilon\cdot\epsilon = \sum^m_i \epsilon_i^2$. That means that finding the projection of $b$ in the column space of $A$ minimizes the  $L_2$ norm of $\epsilon$ or *least squares error*.

There are two ways to go about finding $\hat{x}$.

### The Quick Way to $\hat{x}$

By construction, the vector $\epsilon$ is orthogonal to the column space of $A$. Which means:

\begin{equation}
\begin{array}{rl}
A^T\epsilon &= 0\\
A^T\left(A\hat{x}-b\right) &= 0\\
A^TA\hat{x} &= A^Tb\\
\hat{x} &= \left(A^TA\right)^{-1}A^Tb
\end{array}
\end{equation}

Making use of the fact that $A^TA$ is square and therefore hopefully invertible.

### The Long Way to $\hat{x}$

Loss functions play a central role in computational statistics (for example when regularization is introduced), and therefore it is of interest to approach finding $\hat{x}$ by instead minimizing the least square error. This requires:

\begin{equation}
\frac{d}{d\hat{x}}L_2(\epsilon) = 0
\end{equation}

where

\begin{equation}
\begin{array}{rl}
L_2(\epsilon) &= \left(A\hat{x}-b\right)^T\left(A\hat{x}-b\right)\\
&= \hat{x}^TA^TA\hat{x} - \hat{x}^TA^Tb - b^TA\hat{x} + b^Tb
\end{array}
\end{equation}

Useful factoids about taking derivatives with respect to vectors include:

\begin{equation}
\begin{array}{l}
\frac{d}{dx} \left(u^Tx\right) = \left[\frac{d}{dx_1}\left(\sum_i u_i x_i\right),...,\frac{d}{dx_n}\left(\sum_i u_i x_i\right)\right] = u^T\\
\\
\frac{d}{dx} \left(x^Tu\right) = \left[\frac{d}{dx_1}\left(\sum_i u_i x_i\right),...,\frac{d}{dx_n}\left(\sum_i u_i x_i\right)\right] = u^T\\
\\
\frac{d}{dx} \left(x^Tx\right) = \left[\frac{d}{dx_1}\left(\sum_i x_i^2\right),...,\frac{d}{dx_n}\left(\sum_i x_i^2\right)\right] = 2x^T\\
\\
\frac{d}{dx} \left(Ax\right) = \left[
\begin{array}{ccc}
\underbrace{\frac{d}{dx_1}\left(\sum_i A_1i x_i\right)}_{A_{11}} &...& \underbrace{\frac{d}{dx_n}\left(\sum_i A_1i x_i\right)}_{A_1n}\\
\vdots&\vdots&\vdots\\
\underbrace{\frac{d}{dx_1}\left(\sum_i A_ni x_i\right)}_{A_{n1}} &...& \underbrace{\frac{d}{dx_n}\left(\sum_i A_ni x_i\right)}_{A_{nn}}\\
\end{array}\right] = A\\
\end{array}
\end{equation}

It follows:

\begin{equation}
\begin{array}{l}
\frac{d}{d\hat{x}}\left(x^TA^T\underbrace{A\hat{x}}_{u(\hat{x})}\right) = \frac{d}{du}\left(u^Tu\right)\frac{d}{d\hat{x}}u = 2u^T\frac{d}{d\hat{x}}u = 2\hat{x}^TA^TA\\
\\
\frac{d}{d\hat{x}}\hat{x}^TA^Tb = b^TA\\
\\
\frac{d}{d\hat{x}}b^TA\hat{x} = b^TA\\
\\
\frac{d}{d\hat{x}}b^Tb = 0
\end{array}
\end{equation}

So that

\begin{equation}
\begin{array}{rl}
\frac{d}{dx}L_2(\epsilon) = 0 &= 2\hat{x}^TA^TA - 2b^TA\\
\hat{x}^TA^TA &= b^TA\\
A^TA\hat{x} &= A^Tb\\
\hat{x} &= \left(A^TA\right)^{-1}A^Tb
\end{array}
\end{equation}

### Projection Matrix

If $\hat{b}=A\hat{x}$ is the projection of $b$ in the column space of $A$, then, based on the result for $\hat{x}$, the projection matrix is $P = A\left(A^TA\right)^{-1}A^T$. In a fully determined system, $P=I$. Projection matrices have eigenvalues that are either 1 or 0, corresponding to dimensions that are kept or discarded during the projection operation.

### Underdetermined Case

For an underdetermined system with $m<n$, there is either no solution (if the $m$ hyperplanes don't intersect), or there are infinitely many possible solutions that lie on an $n-m$ dimensional hyperplane. One idea is to pick the solution that minimizes $||\hat{x}||_2$ based on the idea that it might generalize better. The least norm solution is $\hat{x} = A^T\left(AAT\right)^{-1}b$, which is the projection of $\vec{0}$ on the solution set.

Statistics - Normal Equations, Least Squares Estimator.ipynb