# Lecture 2: second-order ordinary differential equations

We now look at solving second-order ordinary differential equations using a computer algebra system.

To use SymPy, we first need to import it and call `init_printing()` to get nicely typeset equations:

```python
from sympy import *

# This initialises pretty printing
init_printing()
from IPython.display import display

# This command makes plots appear inside the browser window
%matplotlib inline
```

# Mass-spring-damper system

The differential equation that governs an unforced, single degree-of-freedom mass-spring-damper system is

$$
m \frac{d^{2}y}{dt^{2}} + \lambda \frac{dy}{dt} + ky = 0
$$

To solve this problem using SymPy, we first define the symbols $t$ (time), $m$ (mass), $\lambda$ (damper coefficient) and $k$ (spring stiffness), and the function $y$ (displacement):

```python
t, m, lmbda, k = symbols("t m lambda k")
y = Function("y")
```

Note that we mis-spell $\lambda$ as `lmbda` because `lambda` is a protected keyword in Python.

Next, we define the differential equation, and print it to the screen:

```python
eqn = Eq(m*Derivative(y(t), t, t) + lmbda*Derivative(y(t), t) + k*y(t), 0)
display(eqn)
```

Checking the order of the ODE:

```python
print("This order of the ODE is: {}".format(ode_order(eqn, y(t))))
```

    This order of the ODE is: 2

and now classifying the ODE:

```python
print("Properties of the ODE are: {}".format(classify_ode(eqn)))
```

    Properties of the ODE are: ('nth_linear_constant_coeff_homogeneous', '2nd_power_series_ordinary')

we see as expected that the equation is linear, constant coefficient, homogeneous and second order.

The `dsolve` function solves the differential equation:

```python
y = dsolve(eqn, y(t))
display(y)
```

The solution looks very complicated because we have not specified values for the constants $m$, $\lambda$ and $k$. The nature of the solution depends heavily on the relative values of the coefficients, as we will see later. We have four constants because the most general case the solution is complex, with two complex constants having four real coefficients.

Note that the solution is make up of expoential functions and sinusoidal functions. This is typical of second-order ODEs.

# Second order, constant coefficient equation

We'll now solve

$$
\frac{d^{2}y}{dx^{2}} + 2 \frac{dy}{dx} - 3 y = 0
$$

The solution for this problem will appear simpler because we have concrete values for the coefficients.

Entering the differential equation:

```python
y = Function("y")
x = symbols("x")
eqn = Eq(Derivative(y(x), x, x) + 2*Derivative(y(x), x) - 3*y(x), 0)
display(eqn)
```

Solving this equation,

```python
y1 = dsolve(eqn)
display(y1)
```

which is the general solution. As expected for a second-order equation, there are two constants.

Note that the general solution is of the form

$$
y = C_{1} e^{\lambda_{1} x} + C_{2} e^{\lambda_{2} x}
$$

The constants $\lambda_{1}$ and $\lambda_{2}$ are roots of the \emph{characteristic} equation

$$
\lambda^{2} + 2\lambda - 3 = 0
$$

This quadratic equation is trivial to solve, but for completeness we'll look at how to solve it using SymPy. We first define the quadratic equation:

```python
eqn = Eq(lmbda**2 + 2*lmbda -3, 0)
display(eqn)
```

and then compute the roots:

```python
solve(eqn)
```

which as expected are the two exponents in the solution to the differential equattion.


# Numerical Recipes Workshop 6
For the week of 28 October to 1 November, 2019.

This notebook will cover some boundary value problem solving and minimization.

```python
from matplotlib import pyplot as plt
%matplotlib inline
import numpy as np
```

```python
plt.rcParams['figure.figsize'] = (10, 6)
plt.rcParams['font.size'] = 14
```

## Solving Boundary Value Problems

SciPy's [solve_bvp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_bvp.html) function will solve a boundary value problem given a system of ordinary differential equations (ODEs). We will consider an example from the function's documentation.

### The Bratu Problem
This is defined as:

$
\begin{align}
\large
\frac{d^{2}y}{dx^{2}} + e^{y} = 0
\end{align}
$

$
\begin{align}
\large
y(0) = y(1) = 0
\end{align}
$

The setup is similar to that of [solve_ivp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp) for initial value problems. We need to define a function that returns the derivative of the system of equations. As with the parabolic motion example, we can decompse the nth order ODE into a system of n 1st order ODEs.

$
\begin{align}
\large
\frac{dy}{dx} = v(x)
\end{align}
$

$
\begin{align}
\large
\frac{dv}{dx} = -e^{y(x)}
\end{align}
$

The difference here is that both $y$ and $v$ will be arrays instead of single values.

```python
def derivatives(x, y):
    """
    y0 = f(x)  => dy0/dx = y1
    y1 = df/dx => dy1/dx = -e^y0
    """
    return np.vstack((y[1], -np.exp(y[0])))
```

It may be helpful to look up the documentation on `np.vstack` and to stick a print statement into the `derivatives` to understand what the arguments look like.

We also need to create a function to return the relevant boundary conditions. Similar to the events functions for `solve_ivp`, the `solve_bvp` function will look for solutions where the boundary conditions function returns 0s. For a systen of n ODEs, the boundary conditions function must n values, but they can be related to any one of the equations and be for either the left or right side.

The arguments of the boundary conditions function are `ya` and `yb`, the values of the system of equations on the left side ($x = a$) and the right side ($x = b$). For the Bratu problem, the relevant boundary conditions are $y(a) = 0$ and $y(b) = 0$, where $a = 0$ and $b = 1$.

```python
def bc(ya, yb):
    """
    ya is [f(x), df/dx] on the left (x = 0)
    yb is [f(x), df/dx] on the right (x = 1)

    For this problem, we want y(0) = 0 and y(1) = 0
    """
    return np.array([ya[0], # f(x) on left side
                     yb[0]]) # f(x) on right side

```

Now define the initial x-space over which to solve the problem.

```python
x = np.linspace(0, 1, 5)
```

Create an initial guess for the solution. The array must have the shape ($N_{equations}$, $N_x$).

```python
# start with all zeroes
y = np.zeros((2, x.size))
```

Now solve the BVP. The `verbose` keyword gives some additional output. Remove or set it to 0 to get rid of the output.

```python
from scipy.integrate import solve_bvp

sol = solve_bvp(derivatives, bc, x, y, verbose = 2)
```

       Iteration    Max residual  Max BC residual  Total nodes    Nodes added
           1          1.05e-04       0.00e+00           5              0
    Solved in 1 iterations, number of nodes 5.
    Maximum relative residual: 1.05e-04
    Maximum boundary residual: 0.00e+00

### What `solve_bvp` returns

Printing the return value of `solve_bvp` (in this case `sol`) shows a complicate object with multiple components.

```python
print(sol)
```

           message: 'The algorithm converged to the desired accuracy.'
             niter: 1
                 p: None
     rms_residuals: array([9.86500717e-05, 1.05360602e-04, 1.05360602e-04, 9.86500717e-05])
               sol: <scipy.interpolate.interpolate.PPoly object at 0x15229e09b0>
            status: 0
           success: True
                 x: array([0.  , 0.25, 0.5 , 0.75, 1.  ])
                 y: array([[ 0.00000000e+00,  1.04784145e-01,  1.40534773e-01,
             1.04784145e-01,  0.00000000e+00],
           [ 5.49349275e-01,  2.84320977e-01, -1.02436237e-17,
            -2.84320977e-01, -5.49349275e-01]])
                yp: array([[ 5.49349275e-01,  2.84320977e-01, -1.02436237e-17,
            -2.84320977e-01, -5.49349275e-01],
           [-1.00000000e+00, -1.11047088e+00, -1.15088910e+00,
            -1.11047088e+00, -1.00000000e+00]])

The most important thing returned is the `sol` attribute of the return value (in this case `sol.sol`). This is a function that will return the value of the solution for given $x$ values.

Note, a similar object is returned for `solve_ivp` when the `dense_output=True` keyword is given.

```python
x_sol = np.linspace(0, 1, 100)
y_sol = sol.sol(x_sol)
# sol.sol return values for the whole system [f(x), df/dx]
plt.plot(x_sol, y_sol[0])
```

The Bratu problem has two solutions. The second can be found by changing the initial guess slightly.

```python
# start with all zeroes
y = np.zeros((2, x.size))
# small change in initial guess
y[0] = 3
```

Now, solve the BVP again, compare the output and plot the two solutions.

```python
x = np.linspace(0, 1, 10)
# start with all zeroes
y = np.zeros((2, x.size))
from scipy.integrate import solve_bvp

sol = solve_bvp(derivatives, bc, x, y,)
```

```python
x_sol = np.linspace(0, 1, 100)
y_sol = sol.sol(x_sol)
# sol.sol return values for the whole system [f(x), df/dx]
plt.plot(x_sol, y_sol[0])
```

### Heat Exchanger Problem
(from the old bvp_solver tutorial)

Consider a device where heat is exchanged between two fluids. The hot fluid (with temperature $T_1$) enters from the left, moving to the right. The cold fluid (with temperature $T_2$) enters from the right, moving to the left. They exchange heat through a metal plate in between them. The cold fluid has twice the specific heat of the hot fluid. This system can be expressed as

$
\begin{align}
\large
q = U * (T_1 - T_2)
\end{align}
$

$
\begin{align}
\large
\frac{dT_1}{dx} = -q
\end{align}
$

$
\begin{align}
\large
\frac{dT_2}{dx} = \frac{-q}{2}
\end{align}
$

where $U$ is a coefficient of heat transfer.

The temperatures of the hot and cold fluids when they enter the device are known.

$
\begin{align}
\large
T_1(x = 0) = 200
\end{align}
$

$
\begin{align}
\large
T_2(x = L) = 50
\end{align}
$

Relevant constants are given below.

```python
T1_0 = 200
T2_L = 50
L = 5
U = 1
```

Implement the derivatives and bounary conditions functions that describe this system.

```python
def derivatives(x , y):
    """
    y[0] is T_1
    y[1] is T_2
    """
    q = U * (y[0] - y[1])
    return np.vstack((- q, - q / 2))

def bc(ya, yb):
    """
    ya is [T1, T2] on the left
    yb is [T1, T2] on the right
    """
    return np.array([ya[0]-200, yb[0]-50])

```

```python
derivatives(x, y).shape
```

    (2, 10)

Set an initial guess for the solution. Try $T_1(x) = T_1(0)$ and $T_2(x) = T_2(L)$.

```python
x = np.linspace(0, L, 10)
y = np.empty([2, x.size])
y[0] = T1_0
y[1] = T2_L
```

Now find the solution and plot $T_1(x)$ and $T_2(x)$.

```python
sol = solve_bvp(derivatives, bc, x, y,)
x_sol = np.linspace(0, L, 100)
y_sol = sol.sol(x_sol)
plt.plot(x_sol, y_sol[0])
plt.plot(x_sol, y_sol[1])
```

## Minimization

The [scipy.optimize](https://docs.scipy.org/doc/scipy/reference/optimize.html) module provides a number of options for finding function minima and maxima. The most relevant to our purposes is [minimize_scalar](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html#scipy.optimize.minimize_scalar).

Study the documentation of `minimize_scalar` and use it to find all local minima of the function below in the range [-10, 10]. Plot $f(x)$ to gain some intuition. Is there a more informative way to plot $f(x)$?

```python
def f(x):
    return (x - 4*np.euler_gamma) * 0.5 * x * (x + np.e)**2 - 5*np.sin(3*x) + 11
```

```python
from scipy.optimize import minimize_scalar
```

```python
x = np.arange(-10, 10, 0.1)
y = f(x)
#plt.ylim(0,100)
plt.semilogy(x,y)
#plt.plot(x,y)
```

```python
res = minimize_scalar(f, method = 'Brent', bracket = (-10, 0, 10))
sol_1 = res.x
res = minimize_scalar(f, method = 'Brent', bracket = (-5, -3, -2.5))
sol_2 = res.x
res = minimize_scalar(f, method = 'Brent', bracket = (0, 1, 2.5))
sol_3 = res.x
sol = np.array([sol_1, sol_2, sol_3])
print(sol)
```

    [-1.65627449 -3.3207098   0.73763619]

```python

```

```python

```


# Solving Ordinary Differential Equations using Numerical Methods

### The following shows different numerical methods in solving ODEs

## Euler Method (1st Order)

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def Euler(f, x0, y, x, deltax):

    X = []
    Y = []
    X.append(x0)
    Y.append(y)

    while x0 < x:
        deltax = min(deltax, x - x0)
        y = y + deltax*f(x,y)
        x0 = x0 + deltax
        X.append(x0)
        Y.append(y)

    return np.array(X), np.array(Y)
```

### Exponential Decay

Let $N_u$ be the number of $U^{235}$ nuclei present in the sample at a time $t$:
\begin{equation}
\frac{dN_U}{dt} = - \frac{N_U}{\tau}
\end{equation}
Initial condition: $N_U = N_{U,0}$ at $t = 0$ with an exact solution $N_U(t) = N_{U,0} e^{-t/\tau}$

```python
N = lambda t,y:  -y*tau

tau = 1
N0 = 100
t0, tf = 0, 5

deltat1 = Euler(N, x0=t0, y=N0, x=tf, deltax=0.5)
deltat2 = Euler(N, x0=t0, y=N0, x=tf, deltax=0.1)
deltat3 = Euler(N, x0=t0, y=N0, x=tf, deltax=0.05)

t1, N1 = deltat1[0], deltat1[1]
t2, N2 = deltat2[0], deltat2[1]
t3, N3 = deltat3[0], deltat3[1]

t_exact = np.linspace(0,5)
N_exact = N0*np.exp(-t_exact/tau)

plt.plot(t1,N1, 'b.', label = r'$\Delta t = 0.5$')
plt.plot(t2,N2, 'bx', label = r'$\Delta t = 0.1$')
plt.plot(t3,N3, 'b+', label = r'$\Delta t = 0.05$')
plt.plot(t_exact, N_exact, 'r', label = 'Exact')
plt.legend()
plt.xlabel('time')
plt.ylabel(r'$N_u$')
plt.title('Number of nuclei of $U^{235}$')
plt.show()
```

### Linear Motion with Air Drag

It is often the case that the frictional force on an object will increase as the object moves faster. An example is the motion of a falling parachutist; the role of the parachute is to produce a frictional force due to air drag, which is larger than would normally be the case without the parachute. Assume that the velocity of the parachutist is described by the equation:
\begin{equation}
\frac{dv}{dt} = g - Cv
\end{equation}
where $g$ and $C$ are constants. The constant $g$ is the acceleration due to gravity while $C$ is a drag coefficient. Provide a way of estimating $v$ using the Euler method.

Using Euler method, we could estimate the $v$ as:
\begin{equation}
v_{n+1} = v_n + (g - Cv_n)\Delta t
\end{equation}
where $\Delta t$ is small increments of time $t$

Let $C=1$, $g=9.8$, $v_0=100$, at $t=0$ to $t=5$ and $\Delta t=0.05$

```python
C = 1
g = 9.8
v0 = 100
t0, tf = 0, 5
deltat = 0.05

v = lambda t, y: g - C*y

v_deltat = Euler(v, x0=t0, y=v0, x=tf, deltax=deltat)
t, v = v_deltat[0], v_deltat[1]

vexact = np.exp(-C*t)*(v0*C-g+g*np.exp(C*t))/C

plt.plot(t, v, 'b.', label = r'$\Delta t = 0.05$')
plt.plot(t, vexact, 'r', label = 'Exact')
plt.legend()
plt.xlabel('time')
plt.ylabel('$v$')
plt.title('Velocity of the parachutist through time')
plt.show()
```

## Euler Method (2nd Order)

```python
def Euler2nd(f, t0, tf, y0, v0, dt, Cromer= 'False'):

    t = np.arange(t0, tf+dt, dt)
    n = len(t)

    y = np.zeros(n)
    v = np.zeros(n)

    y[0] = y0
    v[0] = v0

    for i in range(n-1):
        if Cromer == 'False':
            v[i+1] = v[i] + f(y[i],v[i],t)*dt
            y[i+1] = y[i] + v[i]*dt
        else:
            v[i+1] = v[i] + f(y[i],v[i],t[i])*dt
            y[i+1] = y[i] + v[i+1]*dt

    return t, y
```

### Simple Harmonic Motion

\begin{equation}
\frac{d^2y}{dt^2} = -ky
\end{equation}
where, $y_0 = 0.2$, $v_0 = 0$, $k = 1$, and $\Delta t = 0.01$

```python
y0 = 0.2
v0 = 0
k = 1
dt = 0.01
tf = 20.

f = lambda y,v,t: -k*y

sol = Euler2nd(f, t0=0, tf=tf, y0=y0, v0=v0, dt=dt, Cromer= 'False')
t, y = sol[0], sol[1]

yexact = y0*np.cos(t*np.sqrt(k))

plt.plot(t, y, 'b.', label = 'Euler')
plt.plot(t, yexact, 'r', label = 'Exact')
plt.title('Simple Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

## Euler-Cromer (2nd Order)

### Simple Harmonic Motion

\begin{equation}
\frac{d^2y}{dt^2} = -ky
\end{equation}
where, $y_0 = 0.2$, $v_0 = 0$, $k = 1$, and $\Delta t = 0.01$

```python
y0 = 0.2
v0 = 0
k = 1
dt = 0.01
tf = 20.

f = lambda y,v,t: -k*y

sol = Euler2nd(f, t0=0, tf=tf, y0=y0, v0=v0, dt=dt, Cromer= 'True')
t, y = sol[0], sol[1]

yexact = y0*np.cos(t*np.sqrt(k))

plt.plot(t, y, 'b.', label = 'Euler-Cromer')
plt.plot(t, yexact, 'r', label = 'Exact')
plt.title('Simple Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

### Idealized Projectile Motion

Consider a projectile such as a shell shot from a cannon. The equations of motion of the cannon shell are obtained from Newton’s 2nd law:

\begin{equation}
\frac{d^2x}{dt^2} = 0, \frac{d^2y}{dt^2} = -g
\end{equation}

Express each of these 2nd-order ODEs as 1st-order ODEs and write down the equations for the Euler method that will approximate the motion of the cannon shell.

\begin{equation}
\frac{dx}{dt} = v, \frac{dy}{dt} = -gt + v
\end{equation}

\begin{equation}
x = x_0 + vt, y = y_0 + vt - \frac{gt^2}{2}
\end{equation}

\begin{equation}
v_{i+1} = v_i \\
x_{i+1} = x_i + v_{i+1}\Delta t
\end{equation}

\begin{equation}
v_{i+1} = v_i - g\Delta t \\
y_{i+1} = y_i + v_{i+1}\Delta t
\end{equation}

Let $g=9.8$, $v_{0x}=0.1$, $v_{0y}=10$, at $t=0$ to $t=2$ and $\Delta t=0.01$

```python
g = 9.8
v0x, v0y = 0.1, 10
x0, y0 = 0, 0
t0, tf = 0, 2
dt = 0.01

fx = lambda y,v,t: 0
fy = lambda y,v,t: -g

solx = Euler2nd(f=fx, t0=t0, tf=tf, y0=x0, v0=v0x, dt=dt, Cromer= 'True')
soly = Euler2nd(f=fy, t0=t0, tf=tf, y0=y0, v0=v0y, dt=dt, Cromer= 'True')
x, y = solx[1], soly[1]
t = solx[0]

xexact = v0x*t
yexact = y0 + v0y*t - 0.5*g*(t**2.)

plt.plot(x, y, 'b.', label = 'Euler-Cromer')
plt.plot(xexact, yexact, 'r', label = 'Exact')
plt.title('Idealized Projectile Motion')
plt.legend()
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

### Damped Harmonic Oscillator

The motion of a damped harmonic oscillator can be described using Newton’s 2nd Law:

\begin{equation}
m\frac{d^2x}{dt^2} + C\frac{dx}{dt} + kx = 0
\end{equation}

where $m$ is the mass of the oscillator, $C$ is the drag coefficient that damps the motion, and $k$ is a constant that determines the restoring force. Provide a numerical scheme that will approximate the motion of the oscillator.

Let $m=1$, $C=1$, $k=10$, $v_0=0$, $x_0=1$, at $t=0$ to $t=20$ and $\Delta t=0.01$

```python
m = 1
C = 1
k = 10
v0 = 0
x0 = 1
t0, tf = 0, 20
dt = 0.01

f = lambda y,v,t: -((C/m)*v + (k/m)*y)

sol = Euler2nd(f, t0=t0, tf=tf, y0=x0, v0=v0, dt=dt, Cromer= 'True')
t, x = sol[0], sol[1]

xexact = (1./39.)*np.exp(-t/2.)*(39*np.cos(0.5*t*np.sqrt(39)) + \
         np.sqrt(39)*np.sin(0.5*t*np.sqrt(39)))

plt.plot(t, x, 'b.', label = 'Euler-Cromer')
plt.plot(t, xexact, 'r', label = 'Exact')
plt.title('Damped Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('x')
plt.show()
```

### Pendulum

Consider a pendulum of length $L$ and mass $m$ which is acted upon by a frictional force and an external driving force. Let the driving force be a sinusoidally varying force with constant amplitude $F_D$ and driving frequency $\Omega_D$. Assume that the frictional force is proportional to the pendulum’s velocity with a constant damping factor $q$. (1) Write down the differential equation that describes the motion of the damped, driven pendulum. From the differential equation, write down the equations that approximate the solution using (2) Euler-Cromer, (3) 2nd-order Runge-Kutta, and (4) Verlet algorithms.

\begin{equation}
\frac{d^2y}{dt^2} + q\frac{dy}{dt} + \frac{g}{l}\sin{y} = F_D\cos{\Omega_D}
\end{equation}

\begin{equation}
v_{i+1} = v_i + (- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt})\Delta t \\
y_{i+1} = y_i + (v_i + (- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt})\Delta t)\Delta t
\end{equation}

Let $L=1$, $Fd=0.5$, $\Omega_D=10$, $g=9.8$, $q=0.1$, $v_0=0$, $y_0=10$, at $t=0$ to $t=20$ and $\Delta t=0.005$

```python
L = 1
Fd = 0.5
omegaD = 10
g = 9.8
q = 0.1
v0, y0 = 0, 10
t0, tf = 0, 20
dt = 0.005

f = lambda y,v,t: -q*v-(g/L)*np.sin(y)+Fd*np.cos(omegaD*t)

solEu = Euler2nd(f=f, t0=t0, tf=tf, y0=y0, v0=v0, dt=dt, Cromer= 'True')
t, y = solEu[0], solEu[1]

plt.plot(t, y, 'b.', label = 'Euler-Cromer')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

## 2nd - Order Runge Kutta

```python
def RK2(f, t0, tf, y0, v0, dt):

    t = np.arange(t0, tf+dt, dt)
    n = len(t)

    y = np.zeros(n)
    v = np.zeros(n)

    y[0] = y0
    v[0] = v0

    for i in range(n-1):
        v[i+1] = v[i] + f(y[i],v[i],t[i])*dt
        y[i+1] = y[i] + v[i+1]*dt

    return t, v, y
```

### Linear Motion with Air Drag

Use RK2 to approximate the motion of the falling parachutist:

\begin{equation}
\frac{dv}{dt} = g - Cv
\end{equation}

The constants are the same with the Linear Motion problem stated previously under the Euler Method section

```python
C = 1
g = 9.8
y0, v0 = 0, 100
t0, tf = 0, 5
dt = 0.05

f = lambda y,v,t: g-C*(v + (dt/2.)*(g-C*v))

sol = RK2(f, t0, tf, y0, v0, dt)
t, v = sol[0], sol[1]

vexact = np.exp(-C*t)*(v0*C-g+g*np.exp(C*t))/C

plt.plot(t, v, 'b.', label = 'RK2')
plt.plot(t, vexact, 'r', label = 'Exact')
plt.title('Linear Motion with Air Drag')
plt.legend()
plt.xlabel('time')
plt.ylabel('v')
plt.show()
```

### Damped Harmonic Oscillator

Use the RK2 method to estimate the position of the damped harmonic oscillator at any time t.

\begin{equation}
m\frac{d^2x}{dt^2} + C\frac{dx}{dt} + kx = 0
\end{equation}

The constants used are the same with the problem on the Euler-Cromer section

```python
m = 1
C = 1
k = 10
v0 = 0
x0 = 1
t0, tf = 0, 20
dt = 0.01

f = lambda y,v,t: -((C/m)*(v - (dt/2.)*((C/m)*v + (k/m)*y)) + (k/m)*y)

sol = RK2(f=f, t0=t0, tf=tf, y0=x0, v0=v0, dt=dt)
t, x = sol[0], sol[2]

xexact = (1./39.)*np.exp(-t/2.)*(39*np.cos(0.5*t*np.sqrt(39)) + \
         np.sqrt(39)*np.sin(0.5*t*np.sqrt(39)))

plt.plot(t, x, 'b.', label = 'RK2')
plt.plot(t, xexact, 'r', label = 'Exact')
plt.title('Damped Harmonic Oscillator')
plt.legend()
plt.xlabel('time')
plt.ylabel('x')
plt.show()
```

### Pendulum

Consider a pendulum of length $L$ and mass $m$ which is acted upon by a frictional force and an external driving force. Let the driving force be a sinusoidally varying force with constant amplitude $F_D$ and driving frequency $\Omega_D$. Assume that the frictional force is proportional to the pendulum’s velocity with a constant damping factor $q$. (1) Write down the differential equation that describes the motion of the damped, driven pendulum. From the differential equation, write down the equations that approximate the solution using (2) Euler-Cromer, (3) 2nd-order Runge-Kutta, and (4) Verlet algorithms.

\begin{equation}
\frac{d^2y}{dt^2} + q\frac{dy}{dt} + \frac{g}{l}\sin{y} = F_D\cos{\Omega_D}
\end{equation}

\begin{equation}
v_{i+1} = v_i + (- q(v_i + \frac{\Delta t}{2}(- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt}) ) - \frac{g}{l}\sin{y} + F_D\cos{\Omega_D(t + \frac{dt}{2}}))\Delta t \\
y_{i+1} = y_i + (v_i + (- q(v_i + \frac{\Delta t}{2}(- qv - \frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt}) ) - \frac{g}{l}\sin{y} + F_D\cos{\Omega_D(t + \frac{dt}{2}}))\Delta t)\Delta t
\end{equation}

The constants used are the same as that of the Pendulum problem under the Euler-Cromer section

```python
L = 1
Fd = 0.5
omegaD = 10
g = 9.8
q = 0.1
v0, y0 = 0, 10
t0, tf = 0, 20
dt = 0.005

f = lambda y,v,t: -q*(v + (dt/2.)*(-q*v - (g/L)*np.sin(y) + Fd*np.cos(omegaD*t))) - \
                 (g/L)*np.sin(y) + Fd*np.cos(omegaD*(t+dt/2.))

solRK = RK2(f=f, t0=t0, tf=tf, y0=y0, v0=v0, dt=dt)
t, y = solRK[0], solRK[2]

plt.plot(t, y, 'b.', label = 'RK2')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

## Verlet Method

### Pendulum

Consider a pendulum of length $L$ and mass $m$ which is acted upon by a frictional force and an external driving force. Let the driving force be a sinusoidally varying force with constant amplitude $F_D$ and driving frequency $\Omega_D$. Assume that the frictional force is proportional to the pendulum’s velocity with a constant damping factor $q$. (1) Write down the differential equation that describes the motion of the damped, driven pendulum. From the differential equation, write down the equations that approximate the solution using (2) Euler-Cromer, (3) 2nd-order Runge-Kutta, and (4) Verlet algorithms.

\begin{equation}
\frac{d^2y}{dt^2} + q\frac{dy}{dt} + \frac{g}{l}\sin{y} = F_D\cos{\Omega_D}
\end{equation}

\begin{equation}
y_{i+1} = (2y_i - y_{i-1} + \frac{q}{2}y_{i-1}\Delta t + (-\frac{g}{l}\sin{y} + F_D\cos{\Omega_Dt})\Delta t^2)(1 - \frac{q}{2}\Delta t)
\end{equation}

The constants used are the same with the Pendulum problem under the Euler Cromer section

```python
def Verlet(f1, f2, f3, t0, tf, y0, v0, dt):

    t = np.arange(t0, tf+dt, dt)
    n = len(t)

    y = np.zeros(n)
    v = np.zeros(n)

    v1 = v0 + f1*dt
    y1 = y0 + v0*dt

    y[0], y[1] = y0, y1
    v[0], v[1] = v0, v1

    for i in range(1,n-1):
        y[i+1] = (2.*y[i] - y[i-1] + f3*y[i-1]*dt + (f2(y[i],t[i]))*(dt**2))*(1. - f3*dt)

    return t, y
```

```python
L = 1
Fd = 0.5
omegaD = 10
g = 9.8
q = 0.1
v0, y0 = 0, 10
t0, tf = 0, 20
dt = 0.005

f1 = (-q*v0-(g/L)*np.sin(y0)+Fd*np.cos(omegaD*t0))
f2 = lambda y, t: (-(g/L)*np.sin(y) + Fd*np.cos(omegaD*t))
f3 = q/2.

solVer = Verlet(f1, f2, f3, t0, tf, y0, v0, dt)
t, y = solVer[0], solVer[1]

plt.plot(t, y, 'b.', label = 'Verlet')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```

```python
t1, y1 = solEu[0], solEu[1]
t2, y2 = solRK[0], solRK[2]
t3, y3 = solVer[0], solVer[1]

plt.plot(t1, y1, 'b', label = 'Euler-Cromer')
plt.plot(t2, y2, 'r--', label = 'RK2')
plt.plot(t3, y3, 'g:', label = 'Verlet')
plt.title('Pendulum')
plt.legend()
plt.xlabel('time')
plt.ylabel('y')
plt.show()
```


# Contitional probability

Let us assume that we have two random variables $X$ and $Y$.  Both are generated by rolling fair six-sided dice, so the possible outcomes are:

\begin{align}
Pr(X=1) & = \frac{1}{6} & Pr(Y=1) & = \frac{1}{6} \\
Pr(X=2) & = \frac{1}{6} & Pr(Y=2) & = \frac{1}{6} \\
Pr(X=3) & = \frac{1}{6} & Pr(Y=3) & = \frac{1}{6} \\
Pr(X=4) & = \frac{1}{6} & Pr(Y=4) & = \frac{1}{6} \\
Pr(X=5) & = \frac{1}{6} & Pr(Y=5) & = \frac{1}{6} \\
Pr(X=6) & = \frac{1}{6} & Pr(Y=6) & = \frac{1}{6}
\end{align}

Further, we assume that $X$ and $Y$ are independent.

So, if one defines notation $Pr(X=x \cap Y=y)$ to mean the probability that $X=x$ and $Y=y$
then $X$ and $Y$ being independent means that $Pr(X=x \cap Y=y) = Pr(X=x)Pr(Y=y)$.

For example, $Pr(X=1 \cap Y=2) = Pr(X=1)Pr(Y=2) = \frac{1}{6} \frac{1}{6} = \frac{1}{36}$.

With this notation in mind, we define the conditional probality of $X=x$ given $Y=y$, denoted by $Pr(X=x | Y=y)$, as:

$$
Pr(X=x | Y=y) = \frac{Pr(X=x \cap Y=y)}{Pr(Y=y)}
$$

If $X$ and $Y$ are independent then the conditional probability is

$$
Pr(X=x | Y=y) = \frac{Pr(X=x \cap Y=y)}{Pr(Y=y)} = \frac{Pr(X=x) Pr(Y=y)}{Pr(Y=y)} = Pr(X=x).
$$

In other words, knowing $Y=y$ does not provide any information about $Pr(X=x)$.

Note, since

$$
Pr(X=x | Y=y) = \frac{Pr(X=x \cap Y=y)}{Pr(Y=y)}
$$

we also have that

$$
Pr(X=x | Y=y)Pr(Y=y) = Pr(X=x \cap Y=y).
$$

So, we have the following identity

$$
Pr(X=x | Y=y)Pr(Y=y) = Pr(X=x \cap Y=y) = Pr(Y=y \cap X=x) = Pr(Y=y | X=x)Pr(X=x).
$$

# Example

Things get more interesting when you have dependent random variables so let us define the random variables $Z$ by setting $Z=X+Y$.  All of the possible values for $Z$ as a function of $Y$ and $Z$ can be seen in the following table.

x\Y  |  1 | 2 | 3 | 4 | 5 | 6
-----|:---|:--|:--|:--|:--|:--
1    |  2 | 3 | 4 | 5 | 6 | 7
2    |  3 | 4 | 5 | 6 | 7 | 8
3    |  4 | 5 | 6 | 7 | 8 | 9
4    |  5 | 6 | 7 | 8 | 9 | 10
5    |  6 | 7 | 8 | 9 | 10| 11
6    |  7 | 8 | 9 | 10| 11| 12

Let us compute the

$$
Pr(Z=5 \cap X=2).
$$

Note there are two ways to proceed since

$$
Pr(Z=5 \cap X=2) = Pr(Z=5 | X=2) Pr(X=2) = Pr(X=2 | Z=5) Pr(Z=5)
$$

Doing it the first way we have that

\begin{align}
Pr(Z=5 \cap X=2) &= Pr(Z=5 | X=2) Pr(X=2) \\
                 &= Pr(Y=3) Pr(X=2) \\
                 &= \frac{1}{6} \frac{1}{6} \\
                 &= \frac{1}{36}
\end{align}

We use the face that $Pr(Z=5 | X=2) = Pr(Y=3)$ since, given that $X=2$, $Z=5$ if, and only if, $Y=3$.

Doing it the second way we have that

\begin{align}
Pr(X=2 \cap Z=5) &= Pr(X=2 | Z=5) Pr(Z=5) \\
                 &= \frac{1}{4} \frac{4}{36} \\
                 &= \frac{1}{36}
\end{align}

We use the face that $Pr(Z=5)$ in 4 of the 36 entries in the table above giving us that
$Pr(Z=5) = \frac{4}{36}$.  We also know that the four places where  $Z=5$ occur when

\begin{align}
X=1 ,\;& Y = 4 \\
X=2 ,\;& Y = 3 \\
X=3 ,\;& Y = 2 \\
X=4 ,\;& Y = 1
\end{align}

and $X=2$ in exactly one of those places, so $Pr(X=2 | Z=5) = \frac{1}{4}$.

The two ways match!

```python

```

lectures/05 Basic Statistics, Probability, and Linear Algebra/ConditionalProbabilty.ipynb


# 2.9 Model Selection and the Bias-Variance Tradeoff

All the models described have a *smoothing* or *complexity* parameter that has to be determined:

- the multiplier of the penalty term;

- the width of the kernel

- or the number of basis functions

We cannont use RSS on the training data to determine these parameters, since we would always pick those that gave interpolating fits and have zero residuals.

The kNN regression fit $\hat{f_k}(x_0)$ illustrates the competing forces that effect the predictive ability of such approximations. Suppose the data arise from a model $Y = f(X) + \varepsilon$ with $E(\varepsilon)=0$ and $Var(\varepsilon) = \sigma^2$. We assume that the values of $x_i$ in the sample are fixed. The EPE at $x_0$:

$$
\begin{align}
EPE_k(x_0) &= E[(Y - \hat{f_k}(x_0))^2|X=x_0]\\
&=\sigma^2 + [Bias^2(\hat{f_k}(x_0)) + Var_\tau(\hat{f_k}(x_0))]\\
&=\sigma^2 + \left[f(x_0) - \frac{1}{k}\sum_{l=1}^k{f(x_{(l)})} \right]^2 + \frac{\sigma^2}{k}
\end{align}
$$

The subscripts in parentheses ($l$)  indicates the sequence of nearest neighbors to $x_0$. There are three terms in this expression:

1. $\sigma^2$ is the *irreducible* error - is beyond our control, even if we know the true $f(x_0)$.

2. The bias term and the expected value of the estimate - $[E_\tau(\hat{f_k}(x_0))-f(x_0)]^2$ - where the expected averages the randomness in the training data. This term increases with $k$ if the function is smooth.

3. The variance term and it decreases as the inverse of k. The expected value of the variance is:

$$
\begin{align}
Var_\tau(\hat{f_k}(x_0)) &= E_\tau\left[\hat{f_k}(x_0) - E_\tau(\hat{f_k}(x_0))\right]^2\\
&= E_\tau\left[\frac{1}{k}\sum_{l=1}^k (f(x_{(l)}) + \varepsilon_l) - \frac{1}{k}\sum_{l=1}^k{f(x_{(l)})}\right]^2\\
&= E_\tau\left[\frac{1}{k}\sum_{l=1}^k \varepsilon_l\right]^2\\
&= \frac{1}{k^2}E_\tau\left[\sum_{l=1}^k \varepsilon_l\right]^2\\
&= \frac{1}{k^2}E_\tau\left[\sum_{l=1}^k \varepsilon_l^2\right]\\
&= \frac{\sigma^2}{k}
\end{align}
$$

As the *model complexity* of our procedure is increased, the variance tends to increase and the squared bias tends to decrease. The opposite behavior occurs as the model complexity is decreased.


# Frequency folding
## The relationship between the Fourier transform of a function and the Fourier transform of its sampled function
Consider the function $f(t)$. The function is sampled with the sampling period $h$ (correspoding to sampling frequency $\omega_s = \frac{2\pi}{h}$) to obtain the sequence $f(kh)$.
If $F(\omega)$ is the Fourier transform of the function $f(t)$ and $F_s(\omega)$ is the Fourier transform of the sampled signal $f(kh)$, then
\begin{equation}
F_s(\omega) = \frac{1}{h} \sum_{k=-\infty}^{\infty} F(\omega + k\omega_s)
\end{equation}

From this we see that the spectrum of a sampled signal is periodic, since the function $F_s(\omega)$ is periodic with period $\omega_s$, i.e.
\begin{equation}
F_s(\omega + m\omega_s) = \frac{1}{h} \sum_{k=-\infty}^{\infty} F(\omega + k\omega_s + m\omega_s) = \frac{1}{h} \sum_{k=-\infty}^{\infty} F(\omega + (k+m)\omega_s) = F_s(\omega)
\end{equation}

We also see that the power of the signal $f(kh)$ at a frequency $\omega_1$, $0 \le \omega \le \omega_N$ contains contributions from all frequencies of the original signal at the frequencies
\begin{equation}
\omega = \omega_1 + k\omega_s, \; k=-\infty, \ldots, 0, \ldots, \infty
\end{equation}
and
\begin{equation}
\omega = -\omega_1 + k\omega_s, \; k=-\infty, \ldots, 0, \ldots, \infty
\end{equation}
We say that $\omega_1$ is the alias of all these frequencies. The lowest such alias frequency is the frequency
\begin{equation}
\omega = -\omega_1 + \omega_s,
\end{equation}
which can be written
\begin{equation}
\omega = |\omega_1 - \omega_s| = | \omega_1 + \omega_N - \omega_N - \omega_s | = | (\omega_a + \omega_N) - \omega_s - \omega_N| = | (\omega_1 + \omega_N)\, \mathrm{mod}\, \omega_s - \omega_N|.
\end{equation}

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt

def spectrum(x,h):
    """ Computes spectrum using np.fft.fft  Returns frequency in rad/s from -\omega_N to \omega_N"""
    wN = np.pi/h # The Nyquist frequency
    N = len(x)
    X = np.fft.fft(x) # Computes the Fourier transform
    Xpos = X[:N/2] # Positive part of the spectrum
    Xneg = X[N/2:] # Negative part. Obs: for frequencies wN up to ws
    wpos = np.linspace(0, wN, N/2) # Positive frequencies, goes from 0 to wN

    W = np.hstack((-wpos[::-1], wpos))
    XX = np.hstack((Xneg, Xpos))

    return (XX, W)

# Assume too slow sampling of signal consisting of two high-frequency sinusoids
ws = 16 # Sampling frequency in rad/s
wN = ws/2
h = np.pi/wN

w1 = 10 # rad/s
w2 = w1+1*ws # rad/s

w1Alias = np.abs( (w1+wN) % ws - wN )
w2Alias = np.abs( (w1+wN) % ws - wN )

M = 4000 # Number of samples in the over-sampled ("continuous") signal
t = np.linspace(0, 60*2*np.pi/w1, M) # 60 periods of the slowest sinusoid
y = np.sin(w1*t) + np.sin(w2*t) # Continuous time (sort of) signal

N = 400 # Number of samples to take
ts = np.arange(N)*h
ys = np.sin(w1*ts) + np.sin(w2*ts) # Sampled signal

(Y,W) = spectrum(y,t[1]-t[0]) # get spectrum (from FFT) of the "continuous" signal
(Ys, Ws) = spectrum(ys, h) # Spectrum of discrete signal

plt.figure(figsize=(10,7))
plt.subplot(2,1,1)
plt.plot(W, np.real(Y))
plt.plot(Ws, np.real(Ys))
plt.plot([wN, wN], [-50, 150], 'k--')
plt.plot([-wN, -wN], [-50, 150], 'k--')
plt.xlim((-1.2*w2, 1.2*w2))
plt.xticks((-w2, -w1, -wN, -w1Alias, w1Alias, wN, w1, w2))
plt.ylim((-500, 500))
plt.ylabel('Real part')
plt.subplot(2,1,2)
plt.plot(W, np.imag(Y))
plt.plot(Ws, np.imag(Ys))
plt.plot([wN, wN], [-1500, 1500], 'k--')
plt.plot([-wN, -wN], [-1500, 1500], 'k--')
plt.xlim((-1.2*w2, 1.2*w2))
plt.xticks((-w2, -w1, -wN, -w1Alias, w1Alias, wN, w1, w2))

plt.ylim((-400, 400))
#plt.xticks((-10, -5, -1, 0, 1, 5, 10))
plt.ylabel('Imaginary part')
plt.xlabel(r'$\omega$  [rad/s]')

plt.legend(('Continuous', 'Sampled', 'Nyquist frequency'), loc=1, borderaxespad=0.)

```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x7f102ccdf510>

```python
plt.figure(figsize=(10,5))
plt.plot(t,y, color=(0.7,0.7,1))
plt.stem(ts[:20], ys[:20],linefmt='r--', markerfmt='ro', basefmt = 'r-')
plt.xlim((0,7.8))
plt.xlabel('t  [s]')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7f102e1b38d0>

```python

```


# Truncation Error Analysis via sympy

Copyright (C) 2020 Andreas Kloeckner

<details>
<summary>MIT License</summary>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
</details>

```python
import sympy as s
s.init_printing()
```

Establish some variables that we'll need:

```python
u = s.Function("u")
a, x, t, h_x, h_t = s.symbols("a, x, t, h_x, h_t")
xi_1, xi_2, tau = s.symbols("xi1, xi2, tau")
```

`taylor` is a utility function that spits out a taylor expansion for $f(x+h)$, optionally including a remainder term, with all variables under our control.

```python
def taylor(f, x, h, n, remainder_variable=None):
    result = sum(f.diff(x, i)*h**i/s.factorial(i) for i in range(n))
    if remainder_variable:
        result += f.diff(x, n).subs(x, remainder_variable)*h**n/s.factorial(n)
    return result
```

- Try it out by expanding $u(x+h_x,t)$
- Vary the order
- Expand $u(x,t+h_t)$ instead

```python
#clear
taylor(u(x,t), x, h_x, 3, xi)
```

Assign the PDE we're solving to `pde`:

```python
#clear
pde = u(x, t).diff(t) + a * u(x, t).diff(x)
pde
```

Write out the scheme we're analyzing, in this case ETCS:

```python
etcs = (
    (u(x, t+h_t) - u(x, t))/h_t
    +
    a*(u(x+h_x, t) - u(x-h_x, t))/(2*h_x))
etcs
```

Follow this general pattern:
```
etcs
.subs(u(x, t+h_t), taylor(u(x,t), t, h_t, 2, tau))
```
to arrive at the truncation error.

⚠ Make sure to keep the two $x$ remainder terms separate.

```python
#clear
etcs_taylor = (
    etcs
    .subs(u(x, t+h_t), taylor(u(x,t), t, h_t, 2, tau))
    .subs(u(x+h_x, t), taylor(u(x,t), x, h_x, 3, xi_1))
    .subs(u(x-h_x, t), taylor(u(x,t), x, -h_x, 3, xi_2))
)
sp.simplify(etcs_taylor - pde)
```

```python

```

demos/fd-tdep/Truncation Error Analysis via sympy.ipynb


# Lesson 01 - From Machine Learning to Deep Learning

## Training Logistic classifier
- Is a Linear Classifier

```
W X + b = Y
```
- W, X are matrices
- X is input
- W (Weights) and b (bias) are found by training the model
- Y vector
    - contains output prediction score for each possible output
    - also known as Logits for Logistic Regression
    - turned into probabilities (by using SoftMax function)

```python
scores = [3.0, 1.0, 0.2]
```

```python
import numpy as np

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)
```

```python
import matplotlib.pyplot as plt

x = np.arange(-2.0, 6.0, 0.1)
scores = np.vstack([x, np.ones_like(x), 0.2 * np.ones_like(x)])
scores.shape
```

    (3, 80)

```python
plt.plot(x, softmax(scores).T, linewidth = 2)
plt.legend(['x', '1', '0.2'])
plt.show()
```

If we increase the magnitude of scores then classifier becomes more confident but if we decrease the size of the outputs then the classifier becomes less confident

## One Hot encoding
- Each label would be represented by a vector which is as long as the number of labels and has value 1.0 for the label and 0 for other labels
- Works not well when we have tens of thousands of labels or more. We have lots of zeros at that time
- We can find how well we are doing by simply comparing 2 vectors

Putting all pieces together we have the following Multinomial Logistic Classification

```
D(S(W * X + b), L)
```

## Minimizing Cross Entropy
- Now how do we find the W and b so that our classifier works i.e. D(A, a) is less but D(A, b) is high
- We can try and minimize the loss when summed over all training data
\begin{align}
L = (1/ N) *  \sum_{i} D(S(WX_{i} + b), L_{i})
\end{align}

- We need to calculate the derivative w.r.t to parameters and follow derivative to solution. This is called gradient descent

## Numerical stability
- You have to worry abut calculating values that are too big or too small whenever doing numerical computation

```python
a = 10 ** 9
for i in range(10 ** 6):
    a += 10 ** (-6)

a - 10 ** 9
```

    0.95367431640625

- For numerical stability the values involved in our Loss function L should not get too big or too small.
- A good principle is
    - 0 mean
    - equal variance

- For images it is simple to normalize it by
\begin{align}
(R/G/B - 128) / 128
\end{align}

- The weights and bias should also be good enough for gradient descent to proceed
- Many many but a simple way is to take from gaussian distribution with mean 0 and small variance. Higher variance would mean that the our model is opinioninated in the beginning which we don't want. Our model should be uncertain in the beginning and learn from data

- Validation sets are needed because as we iterate we are exposing the test set to classifier through our decisions
- Using 30000 examples as validation set is good. But if that is a lot of data then we could possibly use cross validation

- Logistic Regression's biggest problem is that it is very difficult to scale
    - Depends on all of data
    - Usually computing the gradient uses 3 times the compute of Loss function
    - In gradient descent we iterate
    - So instead of using all of data we use a small random sample and iterate more. This is called Stochastic Gradient Descent (SGD)
    - SGD is important because it scales well with big data and big models

## Helping SGD
- Momentum
    - Instead of using the current direction we use the knowledge of the general direction in which we are going to decide the current direction in which we want to go
    -
- Learning Rate average
    - Learing rate should be decreased over time


# Alice's Rose

## Dependencies

The `import` statement is special... it imports programmer's wisdom!
The common usage is to acquire access to python packages.

```python
import sympy
import math
```

## A Romantic Introduction to Matematical Optimization and to Python

Note: this story was originally told in the book [Optimization](https://press.princeton.edu/books/hardcover/9780691102870/optimization) and the protagonist there is called Samantha, but we stick to the alphabetical order...

As the story goes, Alice receives a beautiful rose. She has nothing but a lemonade glass to hold the rose and becomes very distressed when the ensemble falls down.

Adding a bit of water helps! Not only that helps the rose, but it also helps the stability: glass, with some water, and rose stands!

Alice thinks: if a bit of water helps, the let us fill the glass!

However, it tilts and falls, as in the beginning, just much more wet.

Alice has a problem to solve: what is the _optimal_ level of water for her rose on a lemonade glass?

She learns from [Archimedes]( https://en.wikipedia.org/wiki/Archimedes) how to compute the _center of gravity_ of the glass with water, which has height
$h = \frac{m_w}{m_w+m_g} h_w + \frac{m_g}{m_w+m_g} h_g$ with:

* $m_w$ the mass of water
* $m_g$ the mass of glass
* $h_w$ the height of the center of gravity of the water in the glass
* $h_g$ the height of the center of gravity of the glass without water

Since Alice's glass is $20$ cm tall, $4$ cm wide and weighs $100$ gram, Alice may fill the glass with water up to height $x$ cm, provided that $0 \leq x \leq 20$ since the water must fit in the glass.

The volume of water is $\pi r^2 x$ with $r$ the radius of the base, i.e. $r=2$.
The volume is therefore $4\pi x$ cubic centimetres.

Since the density of water can be [taken](https://en.wikipedia.org/wiki/Gram_per_cubic_centimetre) as being $1$ gram per cubic centimeter we have:

* $m_w = 4\pi x$
* $m_g = 100$
* $h_w = \frac{x}{2}$
* $h_g = \frac{20}{2} = 10$

And from here we finally obtain the following formula for the height of the center of gravity of the glass with water:

$$
h = \frac{4\pi x}{4\pi x + 100} \frac{x}{2} + \frac{100}{4\pi x + 100} 10 = \frac{4\pi x^2 + 2000}{8\pi x + 200}
$$

Alice's problem is therefore:

$$
\begin{array}{rl}
\min    & \frac{4\pi x^2 + 2000}{8\pi x + 200}  \\
s.t.    & x \geq 0   \\
        & x \leq 20  \\
\end{array}
$$

## Analytical solution

Alice learns from [Fermat]( https://en.wikipedia.org/wiki/Pierre_de_Fermat) that for a function to reach its highest and lowest points inside its domain the derivative must vanish.

This is a good moment to play with symbolic mathematics in python, we will use [sympy](https://www.sympy.org/en/index.html).

### With $\pi$ as a number

```python
# x is a symbol and pi is a number
x = sympy.Symbol('x')
pi = math.pi

# h is a function of x, and hprime its derivative
h = (4*pi*x**2 + 2000)/(8*pi*x+200)
hprime = sympy.diff(h, x)

# sol is(are) the value(s) of x that solve hprime(x) == 0
sol = sympy.solveset(hprime, x)
sol
```

$\displaystyle \left\{-22.8735335189926, -7.95774715459477, 6.95803920980307\right\}$

Above we see that the equation $h^\prime(x) = 0$ has two solutions: one negative and one positive.
Obviously, only the positive may be feasible for Alice.
And, since its value is between $0$ and $20$, it is indeed feasible.

You may recall that the sign of the second derivative tells you whether the root of the first derivative is a *maximum*, a *minimum* or a *saddle point*.

```python
opt = max(sol)
sympy.diff(hprime, x).subs(x,opt).evalf()
```

$\displaystyle 0.0670430626699561$

Since $h^{\prime\prime}(\text{opt}) > 0$ it is indeed a (local) **minimum**.

### With $\pi$ as a symbol

```python
# now pi is a symbol, just like x
pi = sympy.Symbol('pi')

# we redefine h using the same right-hand-side code as before,
# but now with x and pi as symbols
h = (4*pi*x**2 + 2000)/(8*pi*x + 200)

# to have the drivative on the symbol pi we need it from the new version of h
hprime = sympy.diff(h, x)

solution = sympy.solveset(sympy.diff(h, x), x )
solution
```

$\displaystyle \left\{\frac{- 5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}, \frac{5 \sqrt{5} \sqrt{4 \pi + 5} - 25}{\pi}\right\}$

### From a symbolic $\pi$ to a numeric $\pi$

```python
s = max(solution.subs(pi, math.pi).evalf())
print(s)
```

    6.95803920980307

### A picture says more than thousand words

```python
import matplotlib.pyplot as plt
import numpy as np

def plot_alice(h, s, start, stop, width=18, height=8):

    plt.rcParams["figure.figsize"] = (18,8)

    x = sympy.Symbol('x')
    f = sympy.lambdify(x, h.subs( pi, math.pi))

    x = np.linspace(start=start,stop=stop,num=100)
    y = f(x)

    plt.plot(x,y,label='$'+sympy.latex(h)+'$',linewidth=3)
    plt.plot(s,f(s), 'ro', label='optimum', markersize=12)

    plt.legend()
    plt.show()
```

```python
plot_alice( h, s, 0, 20 )
```

## What if we only care about the numerical solution?

### Introducing `pyomo`

This is the moment to meet:
 * mathematical models expressed in `python`, using `pyomo`,
 * powerful numerical optimization algorithms and how to use them.

We will see that `pyomo` completely separates modeling from solving, which allows us to switch solver without recoding!

### Notebook dependencies requiring installation on `colab`

Note that [this notebook](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb) explains how to run `Pyomo` on Google Colab.
For a complete overview please check the [cookbook](https://jckantor.github.io/ND-Pyomo-Cookbook/).

```python
import shutil
if not shutil.which('pyomo'):
    !pip install -q pyomo
    assert(shutil.which('pyomo'))
```

```python
import pyomo.environ as pyo

alice = pyo.ConcreteModel('Alice')
alice.h = pyo.Var(bounds=(0,20))

@alice.Objective(sense=pyo.minimize)
def cog(m):
    return (4*math.pi*alice.h**2 + 2000)/(8*math.pi*alice.h + 200)

alice.pprint()
```

    1 Var Declarations
        h : Size=1, Index=None
            Key  : Lower : Value : Upper : Fixed : Stale : Domain
            None :     0 :  None :    20 : False :  True :  Reals

    1 Objective Declarations
        cog : Size=1, Index=None, Active=True
            Key  : Active : Sense    : Expression
            None :   True : minimize : (12.566370614359172*h**2 + 2000)/(25.132741228718345*h + 200)

    2 Declarations: h cog

We will use `ipopt`. We refer again to [this notebook](https://nbviewer.jupyter.org/github/jckantor/ND-Pyomo-Cookbook/blob/master/notebooks/01.02-Running-Pyomo-on-Google-Colab.ipynb) explains how to run `Pyomo` **and how to install solvers** on Google Colab. For a complete overview please check the [cookbook](https://jckantor.github.io/ND-Pyomo-Cookbook/).

```python
import sys
if 'google.colab' in sys.modules:
    !wget -N -q 'https://ampl.com/dl/open/ipopt/ipopt-linux64.zip'
    !unzip -o -q ipopt-linux64
```

```python
results = pyo.SolverFactory('ipopt').solve(alice)
print(results.solver.status, results.solver.termination_condition )
alice.display()
```

    ok optimal
    Model Alice

      Variables:
        h : Size=1, Index=None
            Key  : Lower : Value            : Upper : Fixed : Stale : Domain
            None :     0 : 6.95803921230998 :    20 : False : False :  Reals

      Objectives:
        cog : Size=1, Index=None, Active=True
            Key  : Active : Value
            None :   True : 6.95803920980307

      Constraints:
        None

## Conclusion

This notebook shows how to solve Alice's problem: finding the most stable amount of water in a vase.

The notebook shows how to solve the problem analytically with `sympy`, how to use `matplotlib` to visualize the function and the optimum. And how to model Alice's problem on `pyomo` and solve it with `ipopt` both at [neos](https://neos-server.org/neos/solvers/index.html) and "locally" at your own Colab session.

## Last remarks

This notebook deferred installation of the packages needed to the moment that we needed them. This was deliberate, but subsequent notebooks will normally list all dependencies on their top part, which we often call the _preamble_. Furthermore, the `colab' dependencies will be streamlined in future notebooks.

```python

```


# Solving Linear Systems: Direct Methods

This notebook by Xiaozhou Li is licensed under a Creative Commons Attribution 4.0 International License.
All code examples are also licensed under the [MIT license](http://opensource.org/licenses/MIT).

```python
%matplotlib inline
import numpy as np
from scipy.linalg import lu
```

## Gaussian Elimination

The simplest version of Gaussian Elimination only involves two operations as follows:
* Add or substrct a muliple of one equation from another.
* Multiply an equation by a nonzero constant.

### Solving a linear system by Gaussian elimination
The general form of a linear system for $n$ equations in $n$ unknown can be wriiten as
\begin{equation}
\begin{pmatrix}{a_{11}} & {a_{12}} & {\cdots} & {a_{1 n}} \\ {a_{21}} & {a_{22}} & {\cdots} & {a_{2 n}} \\ {\vdots} & {\vdots} & {} & {\vdots} \\ {a_{n 1}} & {a_{n 2}} & {\cdots} & {a_{n n}}\end{pmatrix}\begin{pmatrix}{x_{1}} \\ {x_{2}} \\ {\vdots} \\ {x_{n}}\end{pmatrix}=\begin{pmatrix}{b_{1}} \\ {b_{2}} \\ {\vdots} \\ {b_{n}}\end{pmatrix}
\end{equation}

To solve this linear system by Gaussian elimination
1. Using the allowed row operations to eliminate the system to an upper triangular system
\begin{equation}
\begin{pmatrix}{a_{11}} & {a_{12}} & {\cdots} & {a_{1 n}} & {b_{1}} \\ {} & {a_{22}^{(1)}} & {\cdots} & {a_{2 n}^{(1)}} & {b_{2}^{(1)}} \\ {} & {} & {\ddots} & {\vdots} & {\vdots} \\ {} & {} & {} & {a_{n n}^{(n-1)}} & {b_{n}^{(n-1)}}\end{pmatrix}
\end{equation}
2. Using the back substituion (backsolving) to solve the upper triangular system,
\begin{align}
x_{n}=& \frac{b_{n}^{(n-1)}}{a_{n-1}^{(n-1)}}  \\ x_{k}=& \frac{b_{k}^{(k-1)}-\sum_{j=k+1}^{n} a_{k j}^{(k-1)} x_{j} }{a_{k k}^{(k-1)}}, \quad k=n-1, \ldots, 1.
\end{align}
Here, assuming $a^{(k-1)}_{kk} \neq 0,\,\, k = 1,\ldots,n$

**Example**

Using Gaussian elimination to solve the following linear system
\begin{align}
 10^{-20}x_1 + x_2 & = 1 \\
 x_1 + 2x_2 & = 4
\end{align}
The argumented matrix reads
\begin{pmatrix}
10^{-20} & 1 & 1 \\
1 & 2 & 4
\end{pmatrix}

1. Elimination: $\text{Row}_2 = \text{Row}_2 - 10^{20}\times\text{Row}_1$

```python
A = np.array([[1e-20, 1, 1], [1, 2, 4]])
A[1] = A[1] - 10**20*A[0]
print (A[1])
```

    [ 0.e+00 -1.e+20 -1.e+20]

2. The echelon form of the argumented matrix reads
\begin{pmatrix}
10^{-20} & 1 & 1 \\
0 & -10^{20} & -10^{20}
\end{pmatrix}
3. Using the back substituion, the solution is
$$ x_2 = 1,\qquad x_1 = 0.$$

## The $PA = LU$ factorization
### Partial pivoting
At the start of classical Gaussian elimination of $n$ equations in $n$ unknowns, the first step is to use the diagonal element $a_{11}$ as a pivot to eliminate the first column.  The partial pivoting protocol consists of comparing numbers before carrying out each elimination step. The largest entry of the first column is located, and its row is swapped with the pivot row, in this case the top row.

**Example**

Solve the following linear system
\begin{align}
 10^{-20}x_1 + x_2 & = 1 \\
 x_1 + 2x_2 & = 4
\end{align}

The argumented matrix reads
\begin{pmatrix}
10^{-20} & 1 & 1 \\
1 & 2 & 4
\end{pmatrix}

1. Interchange $\text{Row}_1$ with $\text{Row}_2$
\begin{pmatrix}
1 & 2 & 4 \\
10^{-20} & 1 & 1
\end{pmatrix}
2. Elimination: $\text{Row}_2 = \text{Row}_2 - 10^{-20}\times\text{Row}_1$

```python
A = np.array([[1, 2, 4], [1e-20, 1, 1]])
A[1] = A[1] - 10**(-20)*A[0]
print (A[1])
```

    [0. 1. 1.]

3. The echelon form of the argumented matrix reads
\begin{pmatrix}
1 & 2 & 4 \\
0 & 1 & 1
\end{pmatrix}
4. Using the back substituion, the solution is
$$ x_2 = 1,\qquad x_1 = 2.$$

* Solving this equation by the build-in linear solver in numpy:

```python
A = np.array([[1e-20, 1], [1, 2]])
b = np.array([1, 4])

x = np.linalg.solve(A, b)
print (x)
```

    [2. 1.]

### Implementation
#### The $LU$ factorization

```python
def LU_factor(A):
    A = A.astype(np.float)
    print (A)
    n = len(A)
    for j in range(0,n-1):
        if A[j,j] != 0:
            for i in range(j+1,n):
                lam = A[i,j]/A[j,j]
                A[i,j+1:n] = A[i,j+1:n] - lam*A[j,j+1:n]
                A[i,j] = lam
    return A
```

**Example**
Find the LU factorization of matrix
$$
A=\begin{pmatrix}{2} & {4} & {4} & {2} \\ {3} & {3} & {12} & {6} \\ {2} & {4} & {-1} & {2} \\ {4} & {2} & {1} & {1}\end{pmatrix}
$$
and
$$
A=\begin{pmatrix}{1} & {1} & {3} & {4} \\ {2} & {4} & {1} & {3} \\ {3} & {2} & {1} & {1} \\ {2} & {1} & {2} & {1}\end{pmatrix}
$$

```python
A = np.array([[2, 4, 4, 2], [3, 3, 12, 6], [2, 4, -1, 2], [4, 2, 1, 1]])
print (LU_factor(A))
```

    [[ 2.  4.  4.  2.]
     [ 3.  3. 12.  6.]
     [ 2.  4. -1.  2.]
     [ 4.  2.  1.  1.]]
    [[ 2.   4.   4.   2. ]
     [ 1.5 -3.   6.   3. ]
     [ 1.  -0.  -5.   0. ]
     [ 2.   2.   3.8 -9. ]]

```python
A = np.array([[1, 1, 3, 4], [2, 4, 1, 3], [3, 2, 1, 1], [2, 1, 2, 1]])
print (LU_factor(A))
```

    [[1. 1. 3. 4.]
     [2. 4. 1. 3.]
     [3. 2. 1. 1.]
     [2. 1. 2. 1.]]
    [[  1.           1.           3.           4.        ]
     [  2.           2.          -5.          -5.        ]
     [  3.          -0.5        -10.5        -13.5       ]
     [  2.          -0.5          0.61904762  -1.14285714]]

#### The $PA = LU$ factorization

All of the techniques described so far are implemented in Python (Scipy). The most sophisticated form of Gaussian elimination we have discussed is the $PA=LU$ (or $A = PLU$) factorization. Scipy package has the command accepts a square coefficient matrix $A$ and returns $P$ , $L$, and $U$, such that $A = PLU$.

```python
from scipy.linalg import lu
A = np.array([[2, 1, 5], [4, 4, -4], [1, 3, 1]])
P, L, U = lu(A)
print (P)
print (L)
print (U)
```

    [[0. 0. 1.]
     [1. 0. 0.]
     [0. 1. 0.]]
    [[ 1.    0.    0.  ]
     [ 0.25  1.    0.  ]
     [ 0.5  -0.5   1.  ]]
    [[ 4.  4. -4.]
     [ 0.  2.  2.]
     [ 0.  0.  8.]]

```python
A = np.array([[2, 4, 4, 2], [3, 3, 12, 6], [2, 4, -1, 2], [4, 2, 1, 1]])
P, L, U = lu(A)
print (P)
print (L)
print (U)
```

    [[0. 0. 0. 1.]
     [0. 0. 1. 0.]
     [0. 1. 0. 0.]
     [1. 0. 0. 0.]]
    [[1.         0.         0.         0.        ]
     [0.5        1.         0.         0.        ]
     [0.75       0.5        1.         0.        ]
     [0.5        1.         0.41666667 1.        ]]
    [[ 4.     2.     1.     1.   ]
     [ 0.     3.    -1.5    1.5  ]
     [ 0.     0.    12.     4.5  ]
     [ 0.     0.     0.    -1.875]]

```python
print (np.dot(P, np.dot(L, U)) - A)
```

    [[0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]]

## Erros, Conditions Number

**Example** (Hilbert Matrix)

The $n\times n$ Hilbert Matrix $H_n$ is defined as follows:
$$
H_n = \begin{pmatrix}{1} & {1 / 2} & {1 / 3} & {\cdots} & {1 / n} \\ {1 / 2} & {1 / 3} & {1 / 4} & {\cdots} & {1 /(n+1)} \\ {1 / 3} & {1 / 4} & {1 / 5} & {\cdots} & {1 /(n+2)} \\ {\cdots} & {\cdots} & {\cdots} & {\cdots} & {\cdots} \\ {1 / n} & {1 /(n+1)} & {1 /(n+2)} & {\cdots} & {1 /(2 n-1)}\end{pmatrix}
$$

Solving the linear system
$$ H_n x = b$$
with
$$ b = H_n \cdot\begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix} $$
for $n = 5, 10, 20$
* The exact solution is $x = \begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix} $

```python
def hil(n):
    A = np.empty([n, n])
    for i in range(n):
        for j in range(n):
            A[i,j] = 1/(i+j+1)
    return A

def hil_example(n):
    A = hil(n)
    x_exact = np.zeros(n) + 1.
    b = np.dot(A, x_exact)
    x = np.linalg.solve(A, b)

    print ("The Hilbert Example")
    print ("Exact Solution: ", x_exact)
    print ("Numerical Solution: ", x)
    print ("Max Norm Error: ", np.max(x_exact - x))
```

```python
n = 30
#A = hil(n)
#print (A, '\n')

hil_example(n)
```

    The Hilbert Example
    Exact Solution:  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 1. 1. 1. 1.]
    Numerical Solution:  [  0.99999999   1.00000549   0.9995934    1.01063208   0.86492543
       1.95654753  -2.98980522  10.66985353 -10.64549422   1.40496892
      15.91216488  -9.67086839  -3.19672645   5.14709635  -5.92419006
      16.63359268   3.83136369 -13.36494379  -3.48351104  -2.37205148
      14.37850702  22.52283623 -37.62581185  20.90624367  -8.37575394
       1.62476071  10.16821532  -4.99742976   1.20141018   1.41386896]
    Max Norm Error:  38.62581184885538

```python
#n = 5
A = hil(n)
print (np.linalg.cond(A, np.inf))
```

    6.598338740631285e+18

**Example**

Solve the following linear system
\begin{align}
 (1 + 10^{-20})x_1 + x_2 & = 2 + 10^{-20} \\
 x_1 + x_2 & = 2
\end{align}

```python
A = np.array([[1e-20, 1], [1, 1]])
b = np.array([2+1e-20, 2])

x = np.linalg.solve(A, b)
print (x)
```

    [0. 2.]

```python

```


# Tensor Manipulation: Psi4 and NumPy manipulation routines
Contracting tensors together forms the core of the Psi4Julia project. First let us consider the popluar [Einstein Summation Notation](https://en.wikipedia.org/wiki/Einstein_notation) which allows for very succinct descriptions of a given tensor contraction.

For example, let us consider a [inner (dot) product](https://en.wikipedia.org/wiki/Dot_product):
$$c = \sum_{ij} A_{ij} * B_{ij}$$

With the Einstein convention, all indices that are repeated are considered summed over, and the explicit summation symbol is dropped:
$$c = A_{ij} * B_{ij}$$

This can be extended to [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication):
\begin{align}
\rm{Conventional}\;\;\;  C_{ik} &= \sum_{j} A_{ij} * B_{jk} \\
\rm{Einstein}\;\;\;  C &= A_{ij} * B_{jk} \\
\end{align}

Where the $C$ matrix has *implied* indices of $C_{ik}$ as the only repeated index is $j$.

However, there are many cases where this notation fails. Thus we often use the generalized Einstein convention. To demonstrate let us examine a [Hadamard product](https://en.wikipedia.org/wiki/Hadamard_product_(matrices)):
$$C_{ij} = \sum_{ij} A_{ij} * B_{ij}$$

This operation is nearly identical to the dot product above, and is not able to be written in pure Einstein convention. The generalized convention allows for the use of indices on the left hand side of the equation:
$$C_{ij} = A_{ij} * B_{ij}$$

Usually it should be apparent within the context the exact meaning of a given expression.

Finally we also make use of Matrix notation:
\begin{align}
{\rm Matrix}\;\;\;  \bf{D} &= \bf{A B C} \\
{\rm Einstein}\;\;\;  D_{il} &= A_{ij} B_{jk} C_{kl}
\end{align}

Note that this notation is signified by the use of bold characters to denote matrices and consecutive matrices next to each other imply a chain of matrix multiplications!

## Tensor Operations

To perform most operations we turn to tensor packages (here we use [TensorOperations.jl](https://github.com/Jutho/TensorOperations.jl)). Those allow Einstein convention as an input. In addition to being much easier to read, manipulate, and change, it has (usually) optimal performance.
First let us import our normal suite of modules:

```julia
using PyCall: pyimport
psi4 = pyimport("psi4")
np   = pyimport("numpy")
using TensorOperations: @tensor
```

We can then use conventional Julia loops or `@tensor` to perform the same task.

```julia
using BenchmarkTools: @btime, @belapsed
```

With `@btime`/`@belapsed` we average time over several executions to have more reliable timings than `@time`/`@elapsed` (single execution).

**WARNING: We are using Julia's global variables, and those are known to be less efficient than local variables. It is better to wrap code inside function.**

To begin let us consider the construction of the following tensor (which you may recognize):
$$G_{pq} = 2.0 * I_{pqrs} D_{rs} - 1.0 * I_{prqs} D_{rs}$$

Keep size relatively small as these 4-index tensors grow very quickly in size.

```julia
dims = 20

@assert dims <= 30 "Size must be smaller than 30."
D = rand(dims, dims)
I = rand(dims, dims, dims, dims)

# Build the Fock matrix using loops, while keeping track of time
println("Time for loop G build:")
Gloop = @btime begin
   Gloop = np.zeros((dims, dims))
   @inbounds for ind in CartesianIndices(I)
       p, q, r, s = Tuple(ind)
       Gloop[p, q] += 2I[p, q, r, s] * D[r, s]
       Gloop[p, q] -=  I[p, r, q, s] * D[r, s]
   end
   Gloop
end

# Build the Fock matrix using einsum, while keeping track of time
println("Time for @tensor G build:")
G = @btime @tensor G[p,q] := 2I[p,q,r,s] * D[r,s] - I[p,r,q,s] * D[r,s]

# Make sure the correct answer is obtained
println("Loop and einsum builds of the Fock matrix match?    ", np.allclose(G, Gloop))
println()
# Print out relative times for explicit loop vs einsum Fock builds
#println("G builds with einsum are $(g_loop_time/einsum_time) times faster than Julia loops!")
```

    Time for loop G build:
      81.372 ms (2400038 allocations: 61.04 MiB)
    Time for @tensor G build:
      264.983 μs (46 allocations: 5.75 KiB)
    Loop and einsum builds of the Fock matrix match?    true

As you can see, the `@tensor` macro can be considerably faster than plain Julia loops.

## Matrix multiplication chain/train

Now let us turn our attention to a more canonical matrix multiplication example such as:
$$D_{il} = A_{ij} B_{jk} C_{kl}$$

Matrix multiplication is an extremely common operation in all branches of linear algebra. Thus, these functions have been optimized to be extremely efficient. `@tensor` uses it. The matrix product will explicitly compute the following operation:
$$C_{ij} = A_{ij} * B_{ij}$$

This is Julia's matrix multiplication method `*` for matrices.

```julia
dims = 200
A = rand(dims, dims)
B = rand(dims, dims)
C = rand(dims, dims)

# First compute the pair product
tmp_dot = A * B
@tensor tmp_tensor[i,k] := A[i,j] * B[j,k]
println("Pair product allclose? ", np.allclose(tmp_dot, tmp_tensor))
```

    Pair product allclose? true

Now that we have proved exactly what `*` product does, let us consider the full chain and do a timing comparison:

```julia
D_dot = A * B * C
@tensor D_tensor[i,l] := A[i,j] * B[j,k] * C[k,l]
println("Chain multiplication allclose? ", np.allclose(D_dot, D_tensor))
```

    Chain multiplication allclose? true

```julia
println()
println("* time:")
@btime A * B * C

println()
println("@tensor time:")
@btime @tensor D_tensor[i,l] := A[i,j] * B[j,k] * C[k,l];
```

    * time:
      467.984 μs (4 allocations: 625.16 KiB)

    @tensor time:
      471.215 μs (34 allocations: 314.20 KiB)

Both have similar timings, and both call [Basic Linear Algebra Subprograms (BLAS)](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms). The BLAS routines are highly optimized and threaded versions of the code.
 - The `@tensor` code will factorize the operation by default; Thus, the overall cost is not ${\cal O}(N^4)$ (as there are four indices) rather it is the factored $(\bf{A B}) \bf{C}$ which runs ${\cal O}(N^3)$.

Therefore you do not need to factorize the expression yourself (sometimes you might need):

```julia
println("@tensor factorized time:")
@btime @tensor begin
   tmp[i,k]  := A[i,j] * B[j,k]
   tmp2[i,l] := tmp[i,k] * C[k,l]
end
nothing
```

    @tensor factorized time:
      474.136 μs (8 allocations: 625.25 KiB)

On most machines the three have similar timings. The BLAS usage is usually recommended. Thankfully, in Julia its syntax is very clear. The Psi4Julia project tends to lean toward usage of tensor packages but if Julia's built-in matrix multiplication is significantly cleaner/faster we would use it. The real value of tensor packages will become tangible for more complicated expressions.

## Complicated tensor manipulations
Let us consider a popular index transformation example:
$$M_{pqrs} = C_{pi} C_{qj} I_{ijkl} C_{rk} C_{sl}$$

Here, a naive loop implementation would scale like $\mathcal{O}(N^8)$ which translates to an extremely costly computation for all but the smallest $N$. A smarter implementation (factorizing the whole expression) would scale
like $\mathcal{O}(N^5)$.

**WARNING: First execution is slow because of compilation time. Successive are more honest to the running time.**

```julia
# Grab orbitals
dims = 15
@assert dims <= 15 || "Size must be smaller than 15."

C = rand(dims, dims)
I = rand(dims, dims, dims, dims)

# @tensor full transformation.
print("\nStarting @tensor full transformation...")
n8_time = @elapsed @tensor MO_n8[I,J,K,L] := C[p,I] * C[q,J] * I[p,q,r,s] * C[r,K] * C[s,L]
print("complete in $n8_time s\n")

# @tensor factorized N^5 transformation.
print("\nStarting @tensor factorized N^5 transformation with einsum ... ")
n5_time = @elapsed @tensor begin
   MO_n5[A,q,r,s] := C[p,A] * I[p,q,r,s]
   MO_n5[A,B,r,s] := C[q,B] * MO_n5[A,q,r,s]
   MO_n5[A,B,C,s] := C[r,C] * MO_n5[A,B,r,s]
   MO_n5[A,B,C,D] := C[s,D] * MO_n5[A,B,C,s]
end
print("complete in $n5_time s \n")
println("  @tensor factorized is $(n8_time/n5_time) faster than full @tensor algorithm!")
println("  Allclose? ", np.allclose(MO_n8, MO_n5))

# Julia's GEMM N^5 transformation.
# Try to figure this one out!
print("\nStarting Julia's factorized transformation with * ... ")
dgemm_time = @elapsed begin
   MO = C' * reshape(I, dims, :)
   MO = reshape(MO, :, dims) * C
   MO = permutedims(reshape(MO, dims, dims, dims, dims), (2, 1, 4, 3))

   MO = C' * reshape(MO, dims, :)
   MO = reshape(MO, :, dims) * C
   MO = permutedims(reshape(MO, dims, dims, dims, dims),(2, 1, 4, 3))
end
print("complete in $dgemm_time s \n")
println("  * factorized is $(n8_time/dgemm_time) faster than full @tensor algorithm!")
println("  Allclose? ", np.allclose(MO_n8, MO))

# There are still several possibilities to explore:
# @inbounds, @simd, LinearAlgebra.LAPACK calls, Einsum.jl, Tullio.jl, ...
```

    Starting @tensor full transformation...complete in 0.008222635 s

    Starting @tensor factorized N^5 transformation with einsum ... complete in 0.000705554 s
      @tensor factorized is 11.65415404065458 faster than full @tensor algorithm!
      Allclose? true

    Starting Julia's factorized transformation with * ... complete in 0.001960656 s
      * factorized is 4.193818293469126 faster than full @tensor algorithm!
      Allclose? true

None of the above algorithms is $\mathcal{O}(N^8)$. `@tensor` factorizes the expression to achieve better performance. There is a small edge in doing the factorization manually. Factorized algorithms have similar timings, although it is clear that with `@tensor` is easier than with Julia's built-in `*`. To use the usual matrix multiplication with tensors we have to reshape and permute their dimensions, subtracting appeal to the simple `*` syntax.


# First Post
> Gotta start somewhere

A big part of the workflow students in my python-based mathematics classes is to create clear, beautiful documents with Jupyter.  For that reason, I'll use Jupyter to generate all the content in PythonMathClassroom.

The decision to host the blog from githup fastpages came down to the ease with which Jupyter content can go up on that platform without any intermediate fuss.

The first post ought to have some python mathematics, so here we go:

Let's use the sympy library to compute and plot some functions related to $x^2e^{-x}$

```python
from sympy import *
x, y = symbols("x y")

ii=integrate(x**2*exp(-x),x)
ii

```

$\displaystyle \left(- x^{2} - 2 x - 2\right) e^{- x}$

```python
diff(ii,x)
```

$\displaystyle \left(- 2 x - 2\right) e^{- x} - \left(- x^{2} - 2 x - 2\right) e^{- x}$

```python
expand(_)
```

$\displaystyle x^{2} e^{- x}$

```python
solve(x**2*exp(-x)- 3/10 ,x)
```

    [-0.439637356954377, 0.829068989148422, 3.95284287457532]

```python
plot(x**2*exp(-x),3/10,(x,-.6,10))
```

```python

```


```
import numpy as np
import scipy as sp
import scipy.signal
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

The linear interpolation is a convolution of the samples with a triangular pulse $h_l(t)$. The triangular pulse is the convolution of two rectangular pulses $ h_l(t) = (p_{\tau} \ast p_{\tau}) (t) $. The frequency response $ H_l(f) $ is therefore
\begin{equation}
H_l(f) = (\tau \cdot \text{sinc}(f \tau))^2.
\end{equation}

For the linear interpolator $ \tau = T_S $ where $ T_S $ is the sampling time of the input discrete signal.

```
# sampling period & frequency ofthe input discrete signal
Ts = 1
Fs = 1/Ts

# analytic frequency response of the interpolating filter
N = 40
res = 100
f = np.linspace(-N*Fs,N*Fs, 2*N*res)
tau = Ts
H_l = tau**2 * np.sinc(f * tau)**2
```

```
fig, ax = plt.subplots(1, 1, figsize=(13,4))
ax.plot(f, 20 * np.log10( np.abs(H_l) ))
ax.set_ylabel('Amplitude [dB]', fontsize=12)
ax.set_xlabel('Frequency [Hz]', fontsize=12)

n = np.arange(0,9,1)
ax.set_xticks(Fs * n)
labels = ["$%d F_S$" % y for y in n[1:]]
labels.insert(0,"0")
ax.set_xticklabels(labels, fontsize=12)
ax.grid()

ax.axis('tight');
ax.set_ylim([-100, 10]); # semicolon suppresses the output
ax.set_xlim([0, n[-1]*Fs]);
```

With this operation I get a continuous signal from a discrete one. If I want to output a discrete signal, I must sample the output. This is the same as both:
 - sampling the impulse response of the linear interpolation filter,
 - interpolating zeros to the input signal

with sampling time
\begin{equation} T_{S1} = L \cdot T_S \qquad L \in \mathbb{N}. \end{equation}

This produces spectral copies of the original frequency response, at $ n F_{S1} $

```
L = 6 # Fs1 is L times Fs
Fs1 = L * Fs

# view
f_min = 0
f_max = 8*Fs

fig, ax = plt.subplots(1, 1, figsize=(12,6))

# plot some spectral copies
f = np.linspace(f_min, f_max, ((f_max-f_min)/Fs)*res)
tau = Ts
H_l_tot = np.zeros((len(f),))
for i in range(-10,10):
    H_l = tau**2 * np.sinc((f + i*Fs1) * tau)**2
    ax.plot(f, 20 * np.log10( np.abs(H_l) ), '--b', alpha=0.4)
    H_l_tot = H_l_tot + H_l

# plot the sum of all the spectral copies
ax.plot(f, 20 * np.log10( np.abs(H_l_tot)), 'b')

ax.set_ylabel('Amplitude [dB]', fontsize=12)
ax.set_xlabel('Frequency [Hz]', fontsize=12)

n = np.arange(0,9,1)
ax.set_xticks(Fs * n)
labels = ["$%d F_S$" % y for y in n[1:]]
labels.insert(0,"0")
ax.set_xticklabels(labels, fontsize=12)
ax.grid()

ax.axis('tight');
ax.set_ylim([-100, 10]); # semicolon suppresses the output
ax.set_xlim([f_min, f_max]);
```

We show that by fourier transforming the impulse response of the linear interpolator, sampled at $F_{S1}$, we obtain the same result.

```
nfft = 1024
h_l_sampled = np.hstack((np.linspace(0,1-1/L,L), np.linspace(1,0+1/L,L)))
H_l_sampled = np.fft.rfft(h_l_sampled, nfft, axis=-1)
f_sampled = Fs1 * np.linspace(0,0.5,(nfft/2)+1)
```

```
fig, ax = plt.subplots(1, 1, figsize=(6,4))

#plt.title('Digital filter frequency response')

# highest frequency (pi) corresponds to M * Fs/2
ax.plot(f_sampled, 20 * np.log10(abs(H_l_sampled)), 'b',  label=r"$ALIAS_{10}$")
ax.set_ylabel('Amplitude [dB]', fontsize=12)
ax.set_xlabel('Frequency [Hz]', fontsize=12)
ax.grid()

n = np.arange(0,9,1)
ax.set_xticks(Fs * n)
labels = ["$%d F_S$" % y for y in n[1:]]
labels.insert(0,"0")
ax.set_xticklabels(labels, fontsize=12)
ax.set_xlim([0, L*Fs/2]);
```

Lets normalize them to compare the frequency response of the sampled version with respect to the continuous version

```
# frequency response of the sampled (at L*Fs) interpolator
L = 6
Fs1 = L * Fs

# fft computation
nfft = 1024
h_l_sampled = np.hstack((np.linspace(0,1-1/L,L), np.linspace(1,0+1/L,L)))
H_l_sampled = np.fft.rfft(h_l_sampled, nfft, axis=-1)
f_sampled = Fs1 * np.linspace(0,0.5,(nfft/2)+1) # [0, .5 Fs1]

#Normalization
H_l_sampled = H_l_sampled/np.max(np.abs(H_l_sampled))

# frequency response of the continuous-time interpolator
f = np.linspace(0*Fs,(L/2)*Fs, (L/2)*100) # we care only up to (L/2)*Fs
tau = Ts
H_l = tau**2 * np.sinc(f * tau)**2

#Normalization
H_l = H_l / np.max(H_l)
```

```
fig, ax = plt.subplots(1, 1, figsize=(12,4))

# highest frequency (pi) corresponds to M * Fs/2
ax.plot(f_sampled, 20 * np.log10(abs(H_l_sampled)), 'b',  label=r"$ALIAS_{%d}$" % L)
ax.plot(f, 20 * np.log10(abs(H_l)), 'g',  label=r"$sinc$")
ax.set_ylabel('Amplitude [dB]', fontsize=12)
ax.set_xlabel('Frequency [Hz]', fontsize=12)
ax.grid()
ax.legend(loc=0)

n = np.arange(0,9,1)
ax.set_xticks(Fs * n)
labels = ["$%d F_S$" % y for y in n[1:]]
labels.insert(0,"0")
ax.set_xticklabels(labels, fontsize=12)
ax.set_xlim([0, L*Fs/2]);
ax.set_ylim([-100, 10]);
```


# Parameter Learning

## 1. Gradient Descent

We may summarize our steps in the solution of our housing prices regression problem as follows:

- We proposed a hypothesis function:
  $$h_{\theta}(x) = \theta_0 + \theta_1 x.$$

- Whose parameters are:
  $$\theta_0, \theta_1.$$

- Then, we had the cost function:
  $$J(\theta_0, \theta_1) = \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})^2,$$
  which accounts for the square of the vertical distance between the hypothesis function and the training examples.

- Our goal is to find the parameters $\theta_0$ and $\theta_1$ that solve the following optimization problem:
  $$\min_{\theta_0, \theta_1} J(\theta_0, \theta_1)$$

### How do we do this?:

- Start with some $\theta_0, \theta_1.$
- Keep changing $\theta_0, \theta_1$ to reduce $J(\theta_0, \theta_1)$ until we hopefully end up at a minimum.

The **gradient descent** algorithm, which we will explore in a moment, actually applies to cost functions with an arbitrary number of parameters:

$$J(\theta_0, \theta_1, \dots, \theta_n),$$

with $n\in\mathbb{N}$.

### Gradient descent algorithm

The gradient descent algorithm can be described as follows:

- Initialize $\theta_j$, for $j\in\{0, 1,\dots, n\}$.

- repeat until convergence {
  $$\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\theta_0, \theta_1, \dots, \theta_n); \qquad \text{ for } j\in\{0, 1,\dots, n\}$$
  }

Or, in a vector form, with $\boldsymbol{\theta} = \left[\theta_0, \theta_1, \dots, \theta_n\right]^T\in\mathbb{R}^{n+1}$:

- Initialize $\boldsymbol{\theta}$.

- repeat until convergence {
  $$\boldsymbol{\theta} := \boldsymbol{\theta} - \alpha \frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta})$$
  }

where $\frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta})$ is the gradient of the function $J$.

> The *hyperparameter* $\alpha>0$ is called the **learning rate**. It basically controls how big/small is the step to be taken in the direction of the gradient $\frac{\partial}{\partial \boldsymbol{\theta}} J(\boldsymbol{\theta})$.

To gain some intuition with this algorithm, let's consider the function (of only one parameter)

$$J(\theta) = (\theta - 3)^2 + 1,$$

and its derivative

$$\frac{d}{d \theta} J(\theta) = 2(\theta - 3).$$

```python
# Import libraries
from matplotlib import pyplot as plt
import numpy as np
%matplotlib inline
```

```python
# Cost function
def J(t):
    return (t - 3)**2 + 1
# Derivative of the cost function
def dJ(t):
    return 2 * (t - 3)
```

```python
t = np.linspace(0, 6, 101)
```

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
```

Now, let's suppose that our initialization for the parameter is $\theta=5$. In this case, one iteration of the gradient descent algorithm, with $\alpha = 0.1$ would produce:

\begin{align}
    \theta & := \theta - 2\alpha (\theta - 3) \\
           &  = 5 - 0.1 \times 2 \times 2
\end{align}

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.plot(5, J(5), 'or', ms=10, label='$(5, J(5))$')
plt.arrow(x=5, y=J(5), dx=(5 - 0.1 * 2 * 2 - 5), dy=(J(5 - 0.1 * 2 * 2) - J(5)), width=0.1)
plt.legend(loc='best')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
```

The update of $\theta=4.6$ is in the direction of the minimum.

Let's look at the behavior of the gradient descent algorithm until its convergence:

```python
from grad_desc import grad_desc
```

```python
steps = grad_desc(grad=dJ, x0=5, alpha=0.1, grad_tol=1e-3)
```

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.plot(steps, J(steps), 'or', ms=5, label='Gradient descent iterations')
plt.plot(steps[-1], J(steps[-1]), 'ok', ms=5, label='Iteration at convergence')
plt.legend(loc='best')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
```

Now, let's try initializing $\theta=0$:

```python
steps = grad_desc(grad=dJ, x0=0, alpha=0.1, grad_tol=1e-3)
```

```python
steps
```

    array([0.        , 0.6       , 1.08      , 1.464     , 1.7712    ,
           2.01696   , 2.213568  , 2.3708544 , 2.49668352, 2.59734682,
           2.67787745, 2.74230196, 2.79384157, 2.83507326, 2.8680586 ,
           2.89444688, 2.91555751, 2.93244601, 2.9459568 , 2.95676544,
           2.96541235, 2.97232988, 2.97786391, 2.98229113, 2.9858329 ,
           2.98866632, 2.99093306, 2.99274645, 2.99419716, 2.99535772,
           2.99628618, 2.99702894, 2.99762316, 2.99809852, 2.99847882,
           2.99878306, 2.99902644, 2.99922116, 2.99937692, 2.99950154])

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.plot(steps, J(steps), 'or', ms=5, label='Gradient descent iterations')
plt.plot(steps[-1], J(steps[-1]), 'ok', ms=5, label='Iteration at convergence')
plt.legend(loc='best')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
```

### What about $\alpha$?

If $\alpha$ is too small, gradient descent can be slow:

```python
steps = grad_desc(grad=dJ, x0=0, alpha=1e-4, grad_tol=1e-3)
```

```python
len(steps)
```

    43495

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.plot(steps, J(steps), 'or', ms=5, label='Gradient descent iterations')
plt.plot(steps[-1], J(steps[-1]), 'ok', ms=5, label='Iteration at convergence')
plt.legend(loc='best')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
```

If $\alpha$ is too large, gradient descent can fail to converge, or even diverge:

```python
steps = grad_desc(grad=dJ, x0=2.5, alpha=1.5, grad_tol=1e-3, max_iter=10)
```

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.plot(steps, J(steps), 'or', ms=5, label='Gradient descent iterations')
dsteps = np.diff(steps)
plt.arrow(steps[0], J(steps[0]), dsteps[0], J(steps[1]) - J(steps[0]), width=0.1)
plt.arrow(steps[1], J(steps[1]), dsteps[1], J(steps[2]) - J(steps[1]), width=0.1)
plt.arrow(steps[2], J(steps[2]), dsteps[2], J(steps[3]) - J(steps[2]), width=0.1)
plt.legend(loc='best')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
plt.axis([-1, 7, -1, 10])
```

### If we initialize already at a local optimum, the gradient descent will stuck at that point

```python
steps = grad_desc(grad=dJ, x0=3, alpha=0.1, grad_tol=1e-3, max_iter=10)
```

```python
len(steps)
```

    1

```python
plt.figure(figsize=(6, 4))
plt.plot(t, J(t), 'b--', lw=2, label=r'$J(\theta)$')
plt.plot(steps, J(steps), 'or', ms=5, label='Gradient descent iterations')
plt.plot(steps[-1], J(steps[-1]), 'ok', ms=5, label='Iteration at convergence')
plt.legend(loc='best')
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
```

## 2. Gradient descent for linear regression

Given that we proposed the hypothesis function:
  $$h_{\theta}(x) = \theta_0 + \theta_1 x,$$

with the cost function:
  $$J(\theta_0, \theta_1) = \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})^2,$$

what are the partial derivatives of $J$?

\begin{align}
\frac{\partial}{\partial \theta_j} J(\theta_0, \theta_1) & = \frac{\partial}{\partial \theta_j}\frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})^2 \\
& = \frac{\partial}{\partial \theta_j}\frac{1}{2m}\sum_{i=1}^{m}(\theta_0 + \theta_1 x^{(i)} - y^{(i)})^2, \qquad \text{ for } j=0,1
\end{align}

Thus, for $j=0$:

$$
\frac{\partial}{\partial \theta_0} J(\theta_0, \theta_1) = \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)}),
$$

and, for $j=1$:

$$
\frac{\partial}{\partial \theta_1} J(\theta_0, \theta_1) = \frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})x^{(i)}.
$$

Equivalently, the gradient of $J$ is:

$$
\frac{\partial}{\partial \boldsymbol{\theta}} J(\theta_0, \theta_1) = \frac{1}{m}\left[
\begin{array}{c}
\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)}) \\
\sum_{i=1}^{m}(h_{\theta}(x^{(i)}) - y^{(i)})x^{(i)}
\end{array}
\right].
$$

As we saw in the previous notebook, this is a (strictly) convex function. Thus, it has at most one local/global minimum. Under this conditions, the gradient descent algorithm is guaranteed to converge to that point.

In our housing prices example:

```python
import pandas as pd
```

```python
# Read the data
data = pd.read_csv("house_pricing.csv", low_memory=False)
```

```python
# Data (scaled)
x = data['size'].values / 1000
y = data['price'].values / 100000
m = len(x)
```

```python
# Hypothesis function (as a function of the params)
def h(t):
    return t[0] + t[1] * x
# Cost function
def J(t):
    return ((h(t) - y)**2).sum() / (2 * m)
# Gradient of the cost function
def dJ(t):
    return np.array([(h(t) - y).sum(), ((h(t) - y) * x).sum()]) / m
```

```python
# Generate theta0, theta1 grid
t0 = np.linspace(-10, 30, 100)
t1 = np.linspace(-10, 10, 1000)
T0, T1 = np.meshgrid(t0, t1)
```

```python
# Compute cost function in the theta0, theta1 grid
# TODO: find a more efficient way to evaluate this function
cost_fcn = np.zeros(T0.shape)
for i in range(T0.shape[0]):
    for j in range(T0.shape[1]):
        cost_fcn[i, j] = J([T0[i, j], T1[i, j]])
```

```python
steps = grad_desc(grad=dJ, x0=np.array([30., 10.]), alpha=0.1, grad_tol=1e-3)
```

```python
# Import libraries
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
```

```python
# 3D plot and gradient descent steps
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(T0, T1, cost_fcn, rstride=8, cstride=8, alpha=0.5)
ax.scatter(steps[::50, 0], steps[::50, 1], [J(step) for step in steps[::50]], c='r', s=5,
           label='Gradient descent iterations')
ax.scatter(steps[-1, 0], steps[-1, 1], J(steps[-1]), c='k',
           label=r'Convergence: $\theta_0=${}, $\theta_1=${}'.format(np.round(steps[-1, 0], 2),
                                                                     np.round(steps[-1, 1],2)))
ax.set_xlabel(r'$\theta_0$')
ax.set_ylabel(r'$\theta_1$')
ax.set_zlabel(r'$J(\theta_0, \theta_1)$')
ax.legend()
```

```python
# Plot the corresponding lines
plt.figure(figsize=(6, 4))
plt.plot(data['size'], data['price'], 'xr', ms=5, label='Housing prices data')
x = np.linspace(500, 5000)
y = 100000 * (steps[-1, 0] + steps[-1, 1] * x / 1000)
plt.plot(x, y, 'g', lw=3,
         label=r'$\theta_0=${}, $\theta_1=${}'.format(np.round(steps[-1, 0], 2),
                                                      np.round(steps[-1, 1], 2)))
plt.axhline(y=0, c='k', lw=2)
plt.axvline(x=0, c='k', lw=2)
plt.xlabel('Size in $ft^2$')
plt.ylabel('Price ($USD)')
plt.legend(loc='best')
```

This version of the *gradient descent algorithm* is often called **batch gradient descent algorithm**, beacuse is uses all the training examples at each step of the process.

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez. Based on the content of the Machine Learning course offered through coursera by Prof. Andrew Ng.
</footer>


## Ackley

The Ackley function is widely used for testing optimization algorithms. In its two-dimensional form, as shown in the plot above, it is characterized by a nearly flat outer region, and a large hole at the centre. The function poses a risk for optimization algorithms, particularly hillclimbing algorithms, to be trapped in one of its many local minima.

**Definition**

\begin{align}
\begin{split}
f(x) &=& \,-a \exp{ \Bigg[ -b \, \sqrt{ \frac{1}{n} \sum_{i=1}^{n}{x_i}^2 } \Bigg]} - \exp{ \Bigg[ \frac{1}{n}\sum_{i=1}^{n}{cos(c x_i)} \Bigg] } + a + e, \\[2mm]
&& a = \;20, \quad b = \; \frac{1}{5}, \quad c = \;2 \pi \\[2mm]
&&-32.768 \leq x_i \leq 32.768, \quad i=1 \ldots,n \\[4mm]
\end{split}
\end{align}

**Optimum**

$$f(x^*) = 0 \; \text{at} \; x^* = (0,\ldots,0) $$

**Contour**

```python
import numpy as np
from pymoo.factory import get_problem, get_visualization

problem = get_problem("ackley", n_var=2, a=20, b=1/5, c=2 * np.pi)
get_visualization("fitness-landscape", problem, angle=(45, 45), _type="surface").show()
get_visualization("fitness-landscape", problem, _type="contour", colorbar=True).show()
```


# 6. Similarity Functions

- **Created by Andrés Segura Tinoco**
- **Created on May 20, 2019**
- **Updated on Mar 19, 2021**

In statistics and related fields, a **similarity measure** or similarity function is a real-valued function that quantifies the similarity between two objects. In short, a similarity function quantifies how much alike two data objects are [1].

## 6.1. Common similarity functions

```python
# Load the Python libraries
from math import *
from decimal import Decimal
from scipy import stats as ss
import sklearn.metrics.pairwise as sm
import math
```

\begin{align}
    similarity(X, Y) = d(X, Y) = \sqrt{\sum_{i=1}^n (X_i - Y_i)^2} \tag{1}
\end{align}

```python
# (1) Euclidean distance function
def euclidean_distance(x, y):
    return sqrt(sum(pow(a-b,2) for a, b in zip(x, y)))
```

\begin{align}
    similarity(X, Y) = d(X, Y) = \sum_{i=1}^n |X_i - Y_i| \tag{2}
\end{align}

```python
# (2) manhattan distance function
def manhattan_distance(x, y):
    return sum(abs(a-b) for a,b in zip(x,y))
```

\begin{align}
    similarity(X, Y) = d(X, Y) = (\sum_{i=1}^n |X_i - Y_i|^p)^\frac{1}{p} \tag{3}
\end{align}

```python
# (3) Minkowski distance function
def _nth_root(value, n_root):
    root_value = 1/float(n_root)
    return round(Decimal(value) ** Decimal(root_value),3)

def minkowski_distance(x, y, p = 3):
    return float(_nth_root(sum(pow(abs(a-b), p) for a,b in zip(x, y)), p))
```

\begin{align}
    similarity(X, Y) = cos(\theta) = \frac{\vec{X}.\vec{Y}}{\|\vec{X}\|.\|\vec{Y}\|} = \frac{\sum_{i=1}^n X_i.Y_i}{\sqrt{\sum_{i=1}^n X_i^2}.\sqrt{\sum_{i=1}^n Y_i^2}} \tag{4}
\end{align}

```python
# (4) Cosine similarity function
def _square_rooted(x):
    return round(sqrt(sum([a*a for a in x])),3)

def cosine_similarity(x, y):
    numerator = sum(a*b for a,b in zip(x,y))
    denominator = _square_rooted(x) * _square_rooted(y)
    return round(numerator/float(denominator),3)
```

\begin{align}
    similarity(X, Y) = \frac{cov(X, Y)}{\sigma_X . \sigma_Y} = \frac{\sum_{i=1}^n (X_i - \bar{X}).(Y_i - \bar{Y})}{\sqrt{\sum_{i=1}^n (X_i - \bar{X})^2 . (Y_i - \bar{Y})^2}} \tag{5}
\end{align}

```python
# (5) Pearson similarity function
def _avg(x):
    assert len(x) > 0
    return float(sum(x)) / len(x)

def pearson_similarity(x, y):
    assert len(x) == len(y)
    n = len(x)
    assert n > 0
    avg_x = _avg(x)
    avg_y = _avg(y)
    diffprod = 0
    xdiff2 = 0
    ydiff2 = 0
    for idx in range(n):
        xdiff = x[idx] - avg_x
        ydiff = y[idx] - avg_y
        diffprod += xdiff * ydiff
        xdiff2 += xdiff * xdiff
        ydiff2 += ydiff * ydiff

    return diffprod / math.sqrt(xdiff2 * ydiff2)
```

\begin{align}
    similarity(X, Y) = J(X, Y) = \frac{|X \cap Y|}{|X \cup Y|} = \frac{|X \cap Y|}{|X| + |Y| - |X \cap Y|} \tag{6}
\end{align}

```python
# (6) Jaccard similarity function
def jaccard_similarity(x, y):
    intersection_cardinality = len(set.intersection(*[set(x), set(y)]))
    union_cardinality = len(set.union(*[set(x), set(y)]))
    return intersection_cardinality / float(union_cardinality)
```

## 6.2. Manual examples

```python
# Vectors
x = [-4.593481, -5.478033, 1.127111,  1.252885, -2.286953]  # Messi
y = [-4.080334, -3.406618, 4.334073, -0.485612, -2.817897]  # CR
z = [-4.048185, -5.546171, 0.505673,  0.616553, -1.730906]  # Neymar
```

### Euclidean distance

```python
euclidean_distance(x, y)
```

    4.259455195846412

```python
euclidean_distance(x, z)
```

    1.1841800466723797

### Manhattan distance

```python
manhattan_distance(x, y)
```

    8.060965

```python
manhattan_distance(x, z)
```

    2.4272509999999996

### Minkowski distance

```python
minkowski_distance(x, y)
```

    3.619

```python
minkowski_distance(x, z)
```

    0.941

### Cosine similarity

```python
cosine_similarity(x, y)
```

    0.842

```python
cosine_similarity(x, z)
```

    0.99

### Pearson similarity

```python
pearson_similarity(x, y)
```

    0.8214001476231276

```python
pearson_similarity(x, z)
```

    0.9888645775446726

### Jaccard similarity

```python
a = [0,  1, 2, 3, 4, 5]
b = [-1, 1, 2, 0, 3, 5]
```

```python
jaccard_similarity(a, b)
```

    0.7142857142857143

## 6.3. Sklearn examples

```python
corr = sm.euclidean_distances([x], [y])
float(corr[0])
```

    4.259455195846413

```python
corr = sm.manhattan_distances([x], [y])
float(corr[0])
```

    8.060965

```python
corr = sm.cosine_similarity([x], [y])
float(corr[0])
```

    0.841904969009294

```python
corr, p_value = ss.pearsonr(x, y)
corr
```

    0.8214001476231275

## Reference

[1] Wikipedia - Similarity measure.

---
« Home


# Importance Sampling

The expectation value of a function $H({\bf x})$ is defined as

$$
\begin{align}
 \langle H({\bf x})\rangle_{f} = \int d{\bf x} H({\bf x}) f({\bf x; u}),
\end{align}
$$

where $f({\bf x})$ is the probability distribution. The unbiased estimator of this expectation value is

$$
\begin{align}
\langle H({\bf x})\rangle_{f} \approx \frac{1}{N}\sum_{i=1} H({\bf X_i}),
\end{align}
$$
where the sample points $X_i$ are drawn from the distribution $f$. This method may be a computationally inneficient estimate of the expectation value of $H$. We would like to modify our previous estimator to the form
$$
\begin{align}
\langle H({\bf x})\rangle_{g} \approx \frac{1}{N}\sum_{i=1} w(X_i)H({\bf X_i}),
\end{align}
$$
where $w(X_i)$ is an unknown weight factor and $X_i$ are drawn from a new probability distribution $g({\bf x})$ that samples $H(x)$ more effectively.

We would like to relate how the weight functions $w(x)$ relate to the new distribution $g(x)$. This relation is very simple
$$
\begin{align}
 \langle H({\bf x})\rangle_{f} &= \int d{\bf x} H({\bf x}) f({\bf x; u}),\\
  &= \int d{\bf x} H({\bf x}) \frac{f({\bf x; u})}{g({\bf x})} g({\bf x}) \\
  &=  \int d{\bf x} H({\bf x}) w({\bf x; u}) g({\bf x}) \\
  &\equiv \langle H({\bf x})\rangle_{g}.
\end{align}
$$

Now we would like to choose the best distribution function $g^*({\bf x})$ to sample $H({\bf x})$. This can be done by choosing the $g({\bf x})$ function that minimizes the variance of the estimate $\langle H({\bf x})\rangle_g$

The variance of this estimate is given by the integral
$$
\begin{align}
\sigma^2 = \int d{\bf x} \left( w({\bf x})H({\bf x})-\mu \right)^2 g({\bf x}),
\end{align}
$$
where $\mu$ is the expectation value of the function $H({\bf x})$. Minimizing the variance of this expression with respect to $w({\bf x})$, we get

$$
\begin{align}
\frac{d\sigma^2}{dw} = 0 &= \int d{\bf x} \left( w({\bf x})H({\bf x})-\mu \right)H(x)\delta({\bf x-y})  g({\bf x}) \\
&\Rightarrow \\
w({\bf y}) &= \frac{\mu}{H({\bf y})}
\end{align}
$$

Re-writing $w({\bf y})$ in terms of $f$ and $g$ gives,

$$
\begin{align}
g^*({\bf y}) = \frac{H({\bf y})f({\bf y; u})}{\mu}.
\end{align}
$$

This last expression for $g^*$ represents the optimal choice of the importance sampling function $g$. If we want to find the best function $g({\bf y})$ to carry out importance sampling, we must choose $g=g^*$ given above.

# Cross Entropy Estimation

Let us return to the problem of estimating the expectation value
$$
\begin{align}
\langle H({\bf x})\rangle_{f} = \int d{\bf x} H({\bf x}) f({\bf x; u}).
\end{align}
$$
The best importance sampling function to estimate this integral is
$$
\begin{align}
g^*({\bf x}) = \frac{H({\bf x})f({\bf x;u})}{\mu}.
\end{align}
$$

Instead of using this function directly with the unknown parameter $\mu$, we approximate the function $g^*(x)$ using our parametrized function $f({\bf x;v})$. This approximation can be carried out using the Kullback-Leibler divergence

$$
\begin{align}
D(g^*,f({\bf x;v})) = \int dx \ g^*({\bf x}) {\rm ln}(g^*({\bf x})) - \int dx \ g^*({\bf x}) {\rm ln}(f({\bf x;v})).
\end{align}
$$

The above expression vanishes when the two functions $g^*$ and $f$ are equal. To minimize the above expression in terms of the parameter ${\bf v}$, we must maximize the expression
$$
\begin{align}
D({\bf v}) &= \int dx \ g^*({\bf x}) {\rm ln}(f({\bf x; v})),\\
 &= \frac{1}{\mu}\int dx \ H({\bf x}) f({\bf x; u}) {\rm ln}(f({\bf x; v})),\\
  &= \frac{1}{\mu}\int dx \ H({\bf x}) \frac{f({\bf x; u})}{f({\bf x,w})} {\rm ln}(f({\bf x; v})) f({\bf x,w}),\\
  &= \frac{1}{\mu}\int dx \ H({\bf x}) w({\bf x;u,w}) {\rm ln}(f({\bf x; v})) f({\bf x,w}).
\end{align}
$$
Therefore, The optimal choice of ${\bf v}$ is given by solving the expression
$$
\begin{align}
\nabla_{\bf v}D({\bf v}) &= 0 \\
\int dx \ H({\bf x}) w({\bf x;u,w}) {\bf \nabla}_{\bf v}{\rm ln}(f({\bf x; v})) f({\bf x,w})&=0.
\end{align}
$$
When sampling this integral, this last expression will be

$$
\begin{align}
\frac{1}{N}\sum_i H({\bf X_i})w({\bf X_i;u,w})  \nabla_{\bf v}{\rm ln}(f({\bf X_i;v})) = 0
\end{align}
$$
where
$$
X_i \sim  f({\bf x,w}).
$$

## Using Cross Entropy for Optimization

To optimize the function $H({\bf x})$ using cross entropy optimization we will consider the integral that represents that the function $H({\bf x})$ is greater or equal to the maximum of the function $H({\bf x^*})=\gamma^*$.

$$
P(H >= \gamma^*) = \int d{\bf x} \ I_{ \lbrace H({\bf x})  >= \gamma^* \rbrace }(x) f({\bf x; u})
$$

Note that we dont know ahead of time what $x^*$ and $\gamma^*$ are yet. These will be updated iteratively.

Using the ideas of importance sampling and minimizing the KL-divergence from the previous section,
$$
\begin{align}
\frac{1}{N}\sum_i H({\bf X_i})\nabla_{\bf v}{\rm ln}(f({\bf X_i;v})) = 0
\end{align}
$$
where $X_i \sim  f({\bf x,u})$. The algorithm to do this iteratively is below

#### Algorithm

* Choose $\rho$
* Choose $N$
* Initilize ${\bf u}$

1. Sample $N$ points,  $X_i \sim f({\bf x,u})$, compute $H(X_i)$
2. Order $(X_i,H(X_i))$ in decending order according to the values $H(X_i)$
3. Using the $N_\rho = N\rho$ elite samples determine the solution ${\bf v}$ to

$$
\frac{1}{N_\rho}\sum\limits_{i=1}^{N_\rho} H({\bf X_i})\nabla_{\bf v}{\rm ln}(f({\bf X_i;v})) = 0
$$

let this new solution replace the previous result, ${\bf v} \rightarrow {\bf u}$

4. Go to step 1, repeat until convergence criteria is met

#### References

[ [1] ](http://web.mit.edu/6.454/www/www_fall_2003/gew/CEtutorial.pdf) **A Tutorial on the Cross-Entropy Method** Pieter-Tjerk de Boer

[[2]](https://link.springer.com/article/10.1007/s11009-006-9753-0)  D. P. Kroese, S. Porotsky and R. Y. Rubinstein, **The Cross-Entropy Method for Continuous
Multi-Extremal Optimization**, Methodol. Comput. Appl. Probab. 8 (2006).

[[3]](https://www.sciencedirect.com/science/article/pii/S0968090X1100088X?via%3Dihub) M. Maher, R. Liu, D. Ngoduy, **Signal optimization using the cross entropy method**, Transportation Research C. 27, (2013).

[[4]](https://www.sciencedirect.com/science/article/pii/B9780444538598000035) Zdravko I. Botev, Dirk P. Kroese, Reuven Y. Rubinstein, Pierre L’Ecuyer, **Chapter 3 - The Cross-Entropy Method for Optimization**, Handbook of Statistics, Elsevier, 31, 35-59, (2013).

# Let us now implement the above algorithm as a simple example

```python
# Simple Implementation

import numpy as np
import matplotlib.pyplot as plt
```

```python
def H(x):
    '''
    This function has a global maximum at x=10
    and local maximum at x=-2
    '''
    return np.exp(-(x-10)**2)+0.8*np.exp(-(x+2)**2)

x = np.linspace(-20,20,num=1000)

plt.plot(x,H(x))
plt.show()
```

```python

N = 1000
rho = 0.1
mu, sigma = -20.0,1000.0
N_rho = int(N*rho)

steps =200

for k in range(0,steps):
    x_samples = np.random.normal(loc=mu,scale=sigma,size=N)
    H_samples = H(x_samples)
    data = np.array((x_samples,H_samples)).T

    # Now sort these tuples according to H(x_i) values
    sorted_data = np.array(sorted(data,key= lambda x: x[1],reverse=True))
    x_elite_samples = sorted_data[0:N_rho,0]

    # Update the Gaussian distribution parameters with the elite samples
    mu = np.mean(x_elite_samples)
    sigma = np.std(x_elite_samples)

print('Result from cross-entropy optimization: {0:0=.2f}'.format(mu))
print('Global maximum: {}'.format(10.0))
```

    Result from cross-entropy optimization: 10.00
    Global maximum: 10.0

```python

```