# Differential Algebraic Equation / Partial Differentiation Equation solvers
_By Dhruv Jain_

### **Objective: Implementation of fintie differentiation to solve various types of PDE**

```python
# Key libraries: Numpy(for mathematical procedures) and matplotlib(to create plots)
import numpy as np
import matplotlib.pyplot as plt
import copy
```

## 1D Heat Equation
$\frac{\partial h}{\partial t} = k\frac{\partial^2 h}{\partial x^2}$; $0 \leq t < \infty $, $0 \leq x \leq X_{limit}$

$h(0,t) = 0$

$h(X_{limit},t) = f(t)$

$h(x,0) = 0$

We notice that the DE is a PDE with derivatives with respect to t and x.

We discretize $x$ as $x_i$ where i = 0, 1, ....., $n_x$, $\Delta x = \frac{X_{limit} - 0}{n_x}$

$t$ as $t_k$ where k = 0,1,...., $n_t$, $\Delta t = \frac{max(t) - 0}{n_t}$

Using Finite Forward Differencing of first-order and Central Differencing of second-order accuracy, we approximate the derivative terms as,

(This setup is first-order accurate in time and second-order accurate in space)

\begin{equation}
   \frac{\partial h}{\partial t} \approx \frac{h_{i}^{k+1}-h_{i}^{k}}{\Delta t} \notag{}
\end{equation}

\begin{equation}
   \frac{\partial^2 h}{\partial x^2} \approx \frac{h_{i+1}^k-2h_i^k+h_{i-1}^k}{\Delta x^2} \notag{}
\end{equation}

where, $h(x_i, t_k) = h_i^k$
Subsituting and discretizing the other pieces in the given PDE,

\begin{equation}
    \frac{h_{i}^{k+1}-h_{i}^{k}}{\Delta t} = k\frac{h_{i+1}^k-2h_i^k+h_{i-1}^k}{\Delta x^2} \notag{}
\end{equation}

Rearranging the terms, we get the discretized PDE:

\begin{equation}
    h_{i}^{k+1} = h_{i}^{k} + k\frac{\Delta t}{\Delta x^2}(h_{i+1}^k-2h_i^k+h_{i-1}^k) \notag{}
\end{equation}

With B.C.:

$h_0^k = 0$

$h_{X_{limit}}^t = f_k(t)$

$h_i^0 = 0$

```python
def PDE_solve_1Dheat(tmax, xmax, hx, ht, bc0t, funcx, kconst):
    """"Dhruv Jain, 22 Nov 2021
    Obj: Solve 1D heat equation of the from
        h_t = k*h_xx; t ->[0,tmax), x -> [a,b]
        h(a,t) = bc0t
        h(x,0) = funcx
        dh/dx(b,t) = 0
    Args:
        tmax: end value of time interval, float
        xmax: end value of space interval, float
        hx: step size in space, float
        ht: step size in time, float
        bc0t: condition 1, Dirichlet condition, float
        funct: function that defines h(x,0)
        kconst: constant, float
    Output:
        soln: dict, evaluated y values ['y'] and respective time steps ['t'], ndarray
    """

    xi = np.arange(0, xmax+1e-15, hx)    # mesh points in space
    ti = np.arange(0, tmax+1e-12,ht)     # time discretization
    Nx = len(xi)-1
    Nt = len(ti)-1

    D = kconst*ht/hx**2

    yi = np.zeros((Nt+1, Nx+1))

    yi[0,:] = funcx(xi) #h(x,0), B.C. bottom
    yi[:,0] = bc0t*np.ones(Nt+1) #h(0,t), B.C. left

    for k in range(1,Nt):
        for i in range(1,Nx+1):
            if i == Nx:
                yi[k,i] = yi[k-1,i] + D*(- 2*yi[k-1,i] + 2*yi[k-1,i-1]) #  Neuman Condition on B.C. right
            else:
                yi[k,i] = yi[k-1,i] + D*(yi[k-1,i+1] - 2*yi[k-1,i] + yi[k-1,i-1]) # Basic Setup for 1D Heat Equation

    fval = {}
    fval['y'] = yi # y(x,t)
    fval['x'] = xi
    fval['t'] = ti

    return fval
```

# #Example 1
$\frac{\partial T}{\partial t} = c\frac{\partial^2 T}{\partial x^2}$; $0 < x < 4 m$

$T(0,t) = 273$

$T_x(4,t) = 0$

$T(x,0) = 273 + 25sin(3x)$

```python
def p1_bcx0(xi):
    """"Dhruv Jain, 11 Dec, 2021
    Obj: h(xi,0) expression for 1D Heat Equation
    Args:
        xi: xi's, ndarray, float
    Output:
        hi: ndarray
    """
    hi = 273 + 25*np.sin(3*xi)
    return hi

tmax = 31
xmax = 4
delta_x = 0.1
delta_t = 0.025
kconst = 1/5
bc0t = 273

fval_p1 = PDE_solve_1Dheat(tmax, xmax,delta_x, delta_t, bc0t, p1_bcx0, kconst)

# Find index of array for the required x values
xneeded = [0.2, 0.5 ,0.8]
xindex = []
tlim = []
for i in range(len(xneeded)):
        xindex.append(np.argmin(abs(fval_p1['x']-xneeded[i])))

# T(x,t) computed till 31 sec, get index for t till 4 sec
tlim = np.argmin(abs(fval_p1['t']-4))

for i in range(len(xindex)):
    plt.figure(i)
    plt.title('Plot of T(x,t) vs t @ x ='+str(xneeded[i])+' m ')
    plt.plot(fval_p1['t'][:tlim], fval_p1['y'][:tlim,xindex[i]],marker='.')
    plt.ylabel('T(x,t) @ x ='+str(xneeded[i])+' m')
    plt.xlabel('t (s)')
    plt.grid()
```

## Advect Equation
Finite Forward Differnce Method

$c\frac{\partial u}{\partial t} + k\frac{\partial u}{\partial x} = qx$, $u(x,0) = sin(2\pi x/L)$, u(0,t) = 0

Discretized PDE using first-order accuracy finite forward differencing method in time and backward differencing method in space:

\begin{equation}
    c\frac{u_{i}^{k+1}-u_{i}^{k}}{\Delta t} + k\frac{u_{i}^{k}-u_{i-1}^{k}}{\Delta x}  = qx_i \notag{}
\end{equation}

where, $u(x_i, t_k) = u_i^k$

$u(x_i,0) = sin(2\pi x_i/L)$

$u(0,t_k) = 0$

\begin{equation}
    u_{i}^{k+1} = u_{i}^{k} - \frac{k\Delta t}{c\Delta x}(u_{i}^{k}-u_{i-1}^{k}) + \frac{qx_i\Delta t}{c} \notag{}
\end{equation}

```python
def advect_solve_forwdiff_dir_dir(del_x, del_t, kconst, q, L, xmax, tmax):
    """"Dhruv Jain, 22 Nov, 2021
    Obj: Uses Finite Forward Difference Method of first-order accuracy in time and Backward Difference Method in space to solve BVP
        of Advect form: du/dt + kdu/dx = r
        Dirichlet BC left and bottom
    Args:
        del_x: spatial discrization, float
        del_t: temporal discretization, float
        k: coeffeicient of du/dx, float
        q: coefficient on rhs, float
        L: constant in B.C., float
        xmax: U.L. of x, float
        tmax: U.L. of t, float
    Output:
        soln: dict, evaluated y values ['y'] and respective time steps ['t'], ndarray
    """
    xi = np.arange(0,xmax+1e-12,del_x)
    ti = np.arange(0,tmax+1e-12,del_t)

    y = np.zeros((len(ti),len(xi))) # B.C. left = 0

    y[0,:] = np.sin(2*np.pi*xi/L) # Set B.C. bottom

    lam = kconst/5*del_t/del_x
    grhs = del_t*q*xi/5
    for k in range(1,len(ti)):
        for i in range(1,len(xi)):
            y[k,i] = (1-lam)*y[k-1,i] + lam*y[k-1,i-1] + grhs[i]

    fval = {}
    fval['y'] = y
    fval['t'] = ti
    fval['xi'] = xi
    return fval
```

# #Example 2

```python
delta_x = 0.05
delta_t = 0.005
k = 1 #m/s
q = 1 #1/m
L = 1 #m
xmax = 3 #m
tmax = 3 #sec

sol_p1 = advect_solve_forwdiff_dir_dir(delta_x, delta_t, k, q, L, xmax, tmax)

xindex = np.argmin(abs(sol_p1['xi']-3))

index = []
for i in range(4):
    index.append(np.argmin(abs(sol_p1['t']-i)))

plt.figure(1)
plt.title('Plot of u(x,t) ')
for i in range(0,len(index)):
    plt.plot(sol_p1['xi'][:-1],sol_p1['y'][index[i],:xindex],marker='.',label='t='+str(i))
plt.xlabel('x')
plt.ylabel('u(x,ti)')
plt.grid()
plt.legend()
```

## Wave Equation
$\frac{\partial^2 u}{\partial t^2} = c^2\frac{\partial^2 u}{\partial x^2}$

$u(0,t) = u(l,t) = 0$

$u_t(x,0) = g(x)$

$u(x,0) = f(x)$

```python
def wave_solve_dir_neu(c, m, l, tsteps, xsteps, tmax, funcf, funcg):
    """"Dhruv Jain, 7 Dec, 2021
    Obj: Uses Finite Central Difference Method of second-order accuracy to solve BVP
        of wave form: d^2u/dt^2  = c^2 d^2u/dx^2
        Dirichlet BC
    Args:
        c: coeffeicient of d^2u/dx^2, float
        m: constant in B.C., float
        L: constant in B.C. and U.L. of x, float
        tsetps: number of time steps, int
        xsteps: number of spatial steps, int
        tmax: U.L. of t, float
        funcf: function to compute f(x) for u(x,0)
        funcg: function to compute g(x) for u_t(x,0)
    Output:
        soln: dict, evaluated y values ['y'] and respective time steps ['t'], ndarray
    """
    xi = np.linspace(0,l,xsteps+1)
    del_x = xi[1]-xi[0]
    ti = np.linspace(0,tmax,tsteps+1)
    del_t = ti[1]-ti[0]

    D = (c*del_t/del_x)**2

    y = np.zeros((len(ti),len(xi)))
    # B.C. left and right: u(0,ti) = u(l,ti) = 0

    y[0,:] = funcf(xi, l)

    gi = funcg(xi)

    # As u_t(x,0) is known
    for i in range(1,len(xi)-1):
        y[1,i] = y[0,i] + del_t*gi[i] + 1/2 * D*(y[0,i+1] - 2*y[0,i] + y[0,i-1])

    #Using u(0,i) and u(1,i) compute u(k,i) for k>1 and i lies between 1 and xi-1
    for k in range(1,len(ti)-1):
        for i in range(1,len(xi)-1):
            y[k+1,i] = 2*y[k,i] - y[k-1,i] + D*(y[k,i+1] - 2*y[k,i] + y[k,i-1])

    fval = {}
    fval['y'] = y
    fval['t'] = ti
    fval['xi'] = xi
    return fval
```

# #Example 3

```python
def p2_g(xi):
    """"Dhruv Jain, 6 Dec
    Obj: g(x) expression
    Args:
        xi: xi's, ndarray, float
    Output:
        gi: g(xi), ndarray
    """
    gi = xi*0
    return gi

def p2_f(xi, l):
    """"Dhruv Jain, 6 Dec
    Obj: f(x) expression
    Args:
        xi: xi's, ndarray, float
    Output:
        fi: f(xi), ndarray
    """

    fi = np.zeros((len(xi)))

    #Compute B.C. bottom: u(xi,0)
    for i in range(len(xi)):
        if xi[i] <= l/2 and xi[i] >= 0:
            fi[i] = 2*m*xi[i]/l
        elif xi[i] <= l and xi[i] > l/2:
            fi[i] = 2*m*(l-xi[i])/l

    return fi

tmax = 10
tsteps = 500
xsteps = 500
l = 10
c = 1
m = 1
sol_p2 = wave_solve_dir_neu(c, m, l, tsteps, xsteps, tmax, p2_f, p2_g)

t_req = [0, 2.5, 5, 7.5, 10]
index = []
for i in t_req:
    index.append(np.argmin(abs(sol_p2['t']-i)))

```

```python
for i in range(len(index)):
    plt.figure(i)
    plt.title('Plot of u(x,t) @ t = '+str(t_req[i]))
    plt.plot(sol_p2['xi'], sol_p2['y'][index[i],:],marker='.',label='t='+str(t_req[i]))
    plt.ylabel('u(x,'+str(t_req[i])+')')
    plt.xlabel('x')
    if max(abs(sol_p2['y'][index[i],:])) < 1e-12: # To avoid plotting near machine precision values of u(x,y)
        plt.ylim([-0.1, 0.1])
    plt.grid()
    plt.legend()
```

## Laplace Equation

```python
def laplace_solve_neumleft_neubottom(h,xmin,xmax, ymin, ymax, funcf, funcg):
    """"Dhruv Jain, 12 Dec
    Obj: Uses Finite Central Difference Method of second-order accuracy to solve BVP
        of laplace form: d^2u/dy^2  + d^2u/dx^2 = f(x,y)
        Dirichlet BC on top and right = g(x,y)
        Neuman BC on left(=0) and bottom(=0)
    Args:
        h: step size, dx=dy=h, float
        xmin: L.L. of x, float
        xmax: U.L. of x, float
        ymin: L.L. of y, float
        ymax: U.L. of y, float
        funcf: function to compute f(x,y) for u(x,y)
        funcg: function to compute g(x,y) for u(x,y) # B.C. top and right
    Output:
        soln: dict, ndarray
        {u: u(x,y) values
        xi: xi values
        yi: yi values}
    """
    xi = np.arange(xmin,xmax+1e-12,h)
    yi = np.arange(ymin,ymax+1e-12,h)

    u = np.zeros((len(yi),len(xi)))
    B = np.zeros((len(yi)-1,len(xi)-1))# B should not contain nodes on top and right

    u[0,-1] = funcg(-1,0,xi[-1],yi[0]) #u_0,-1
    for i in range(len(yi)-1):
        B[i,-1] = funcg(-1,i,xi[-1],yi[i]) # Add B.C. right to all u_-1,i

        u[i+1,-1] = funcg(-1,i+1,xi[-1],yi[i+1]) #u_-1,-1

    u[-1,0] += funcg(0,-1,xi[0],yi[-1])#u_-1,0
    for i in range(len(xi)-1):
        B[-1,i] = funcg(i,-1,xi[i],yi[-1]) # Add B.C. top to all u_i,-1
        u[-1,i+1] += funcg(i+1,-1,xi[i+1],yi[-1]) #u_-1,1 -> u_-1,-1

    for i in range(len(yi)-1):
        for j in range(len(xi)-1):
            B[i,j] += -h**2*funcf(xi[i],yi[j]) # Add -h^2*fij terms

    B = B.flatten()
    A = np.zeros((len(B),len(B)))

    D = np.zeros((len(xi)-1,len(xi)-1))

    for i in range(len(xi)-1): # Only difference in D from typical Dirichlet setup is -2 on D[1,1] instead of -1 to account for Neumann on left
        D[i,i] = 4
        for j in range(len(xi)-1):
            if j == i+1:
                D[i,j] = -1
                if i == 0:
                    D[i,j] = -1*2
            if j == i-1:
                D[i,j] = -1

    nx = len(xi)-1
    for i in range(len(yi)-1): # Sim
        A[nx*i:nx*(i+1),nx*i:nx*(i+1)] = D
        for j in range(len(yi)-1):
            if j == i+1 and j == 1:
                A[nx*i:nx*(i+1),nx*j:nx*(j+1)] = -2*np.identity(nx) # -2*I to account for Neumann on bottom
            if j == i+1 and j > 1:
                A[nx*i:nx*(i+1),nx*j:nx*(j+1)] = -np.identity(nx)
            if j == i-1:
                A[nx*i:nx*(i+1),nx*j:nx*(j+1)] = -np.identity(nx)

    uij = np.matmul(np.linalg.inv(A),B)

    for i in range(len(yi)-1):
        u[i,0:len(xi)-1] = uij[nx*i:nx*(i+1)]

    fval = {}
    fval['u'] = u
    fval['yi'] = yi
    fval['xi'] = xi
    return fval
```

# #Example 4
dT/dx(-10,y) = 0, dT/dy(x,-10) = 0, T(x,10) = 0, T(10,y) = 0

```python
def p2_f(xi,yj):
    """"Dhruv Jain, 11 Dec, 2021
    Obj: f(x,y) expression
    Args:
        xi: x value at i, float
        yj: y value at j, float
    Output:
        fi: f(i,j), float
    """
    if xi**2 + yj**2 <= 10:
        fij = -25
    else:
        fij = 0
    return fij

def p2_g_c(i,j,xi,yj):
    """"Dhruv Jain, 11 Dec, 2021
    Obj: g(x,y) expression
    Args:
        i: x index, int
        j: y index, int
        xi: x value at i, float
        yj: y value at j, float
    Output:
        gi: g(i,j), float
    """
    if i == -1 and j != -1: # Right BC
        gi = 0
    elif j == -1:# Top BC
        gi = 0
    return gi

# Setup
h = 0.5
xmin = -10
xmax = 10
ymin = -10
ymax = 10
sol_p2c = laplace_solve_neumleft_neubottom(h,xmin,xmax, ymin, ymax, p2_f, p2_g_c)

# Pot
xindex = np.argmin(abs(sol_p2c['xi']-0))
yindex = np.argmin(abs(sol_p2c['yi']-0))

# Contour Plot
fig,ax=plt.subplots(1,1)
cp = ax.contourf(sol_p2c['xi'],sol_p2c['yi'],sol_p2c['u'])
fig.colorbar(cp)
ax.set_title('Final Exam, P2(c): Contour plot of T(x,y)')
ax.set_xlabel('x (m)')
ax.set_ylabel('y (m)')
plt.show()

```


# Cours 3 : Récursivité

## Rappel : la pile d'exécution

Au cours de l'exécution d'un programme les variables sont stockées dans la mémoire sous forme d'une **pile**. A chaque appel de fonction, un espace mémoire est réservé pour les variables de la fonction.

**Exemple**

```python
def fonction1(a):
    c = a + 1
    return c

def fonction2(b):
    return fonction1(b+1)

a = fonction2(3)
a
```

    5

Voici une image simplifiée de l'évolution de la pile lors de l'exécution

On va utiliser le petit bout de code suivant pour suivre en direct les appels de fonctions, exécutez la cellule qui définit la fonction `printAppelsFonctions` puis regardons comment on l'applique sur notre petit bout de code.

```python
import functools

def printAppelsFonctions(func):
    """Print the function signature and return value"""
    @functools.wraps(func)
    def wrapper_debug(*args, **kwargs):
        args_repr = [repr(a) for a in args]
        kwargs_repr = [f"{k}={v!r}" for k, v in kwargs.items()]
        signature = ", ".join(args_repr + kwargs_repr)
        print(f"Appel de {func.__name__}({signature})")
        value = func(*args, **kwargs)
        print(f"valeur de retour {value!r}")
        print(f"Fin de {func.__name__}({signature})")
        return value
    return wrapper_debug
```

```python
@printAppelsFonctions
def fonction1(a):
    c = a + 1
    return c

@printAppelsFonctions
def fonction2(b):
    return fonction1(b+1)

a = fonction2(3)
a
```

    Appel de fonction2(3)
    Appel de fonction1(4)
    valeur de retour 5
    Fin de fonction1(4)
    valeur de retour 5
    Fin de fonction2(3)

    5

**Exemple**

que se passe-t-il dans le cas suivant ?

```python
def fonction1(a):
    fonction1(a)

fonction1(3)
```

```python
@printAppelsFonctions
def fonction1(a):
    fonction1(a)

fonction1(3)
```

## Définition

Un algorithme *récursif* est un algorithme qui fait appel à lui même. Lorsqu'un algorithme n'est pas récursif, on dit qu'il est *itératif*.

Pour éviter une boucle infinie, il faudra impérativement définir **une condition d'arrêt**.

**Exemple**

Le calcul de $n! = n \times (n-1) \times \dots \times 1$. Tout d'abord la version itérative

```python
def factorielleIterative(n):
    r = 1
    for i in range(1,n+1):
        r = r*i
    return r
```

```python
factorielleIterative(3)
```

    6

```python
factorielleIterative(5)
```

    120

La version récursive se base sur le principe suivant :

$n! = \begin{cases}
1 & \text{si } n = 0 \\
n \times (n-1)! & \text{sinon}
\end{cases}$

```python
def factorielleRecursive(n):
    if n == 0:
        return 1
    return n * factorielleRecursive(n-1)
```

```python
factorielleRecursive(3)
```

    6

```python
factorielleRecursive(5)
```

    120

Que se passe-t-il au niveau de la pile ?

```python
@printAppelsFonctions
def factorielleRecursive(n):
    if n == 0:
        return 1
    return n * factorielleRecursive(n-1)

```

```python
factorielleRecursive(3)
```

    Appel de factorielleRecursive(3)
    Appel de factorielleRecursive(2)
    Appel de factorielleRecursive(1)
    Appel de factorielleRecursive(0)
    valeur de retour 1
    Fin de factorielleRecursive(0)
    valeur de retour 1
    Fin de factorielleRecursive(1)
    valeur de retour 2
    Fin de factorielleRecursive(2)
    valeur de retour 6
    Fin de factorielleRecursive(3)

    6

Attention, sur certaines valeurs, on obtient des appels récurisfs à l'infini. Ici, la *fonction termine* sur l'ensemble des entiers positifs mais pas sur les négatifs

```python
factorielleRecursive(-2)
```

## Comment définir une fonction récursive ?

Le principe d'une fonction récursive est assez similaire à la notion de **récurrence** mathématique. On ne cherche pas à calculer le problème dans son ensemble mais simplement à le définir en fonction d'un problème de taille plus petite. Tout comme en mathématique le fait de prouver l'état initial et une seule étape permet de prouver la propriété, dans un algorithme, le fait de calculer l'état initial et une étape permet de calculer n'importe quelle valeur.

Un algorithme récursif s'écrira toujours de cette façon :
Input : des paramètres
Processus:
    Si les paramètres correspondent à la condition d'arret (cas simple):
        Actions cas d'arret
    Sinon
        Appel(s) récursifs sur de nouveaux paramètres
Pour être sûr que l'algorithme termine, les nouveaux paramètres doivent se rapprocher de la condition d'arrêt.

## Exercices

### Exercice 1

1. Donner un algorithme récursif qui affiche les entiers de 1 à $n$
1. Même chose de $n$ à 1
3. Puis-je modifier ces algorithmes pour qu'ils rajoutent "fin" à la fin de la liste

```python
@printAppelsFonctions
def affiche1n(n):
    if n == 0:
        print("fin")
        return
    affiche1n(n-1)
    print(n)
```

```python
affiche1n(10)
```

    Appel de affiche1n(10)
    Appel de affiche1n(9)
    Appel de affiche1n(8)
    Appel de affiche1n(7)
    Appel de affiche1n(6)
    Appel de affiche1n(5)
    Appel de affiche1n(4)
    Appel de affiche1n(3)
    Appel de affiche1n(2)
    Appel de affiche1n(1)
    Appel de affiche1n(0)
    fin
    valeur de retour None
    Fin de affiche1n(0)
    1
    valeur de retour None
    Fin de affiche1n(1)
    2
    valeur de retour None
    Fin de affiche1n(2)
    3
    valeur de retour None
    Fin de affiche1n(3)
    4
    valeur de retour None
    Fin de affiche1n(4)
    5
    valeur de retour None
    Fin de affiche1n(5)
    6
    valeur de retour None
    Fin de affiche1n(6)
    7
    valeur de retour None
    Fin de affiche1n(7)
    8
    valeur de retour None
    Fin de affiche1n(8)
    9
    valeur de retour None
    Fin de affiche1n(9)
    10
    valeur de retour None
    Fin de affiche1n(10)

```python
def affichen1(n):
    if n == 0:
        print("fin")
        return
    print(n)
    affichen1(n-1)

```

```python
affichen1(10)
```

    10
    9
    8
    7
    6
    5
    4
    3
    2
    1
    fin

```python
def concat1n(n):
    if n == 0:
        return ""
    return concat1n(n-1) + " " + str(n) + " fin "
```

```python
concat1n(10)
```

    ' 1 fin  2 fin  3 fin  4 fin  5 fin  6 fin  7 fin  8 fin  9 fin  10 fin '

```python
def concatn1(n):
    if n == 0:
        return ""
    return str(n) + " " + concatn1(n-1)
```

```python
concatn1(10)
```

    '10 9 8 7 6 5 4 3 2 1 '

```python

```

### Exercice 2

Pour chacun des algorithmes suivants :

1. Déterminez pour quelles valeurs d'entrée l'algorithme termine (= ne fait pas de boucles infinies)
2. Calculez un exemple à la main
3. Expliquez ce que calcule l'algorithme

```python
def fonction1(n):
    if n == 0:
        return 1
    return fonction1(n+1)
```

La fonction va terminer sur les valeurs négatives et renvoie toujours 1
Boucle infinie sur les strictements positifs

```python
fonction1(-6)
```

    1

```python
def fonction2(n):
    if n == 0:
        return 0
    return fonction2(n-1)+n
```

Termine pour les nombres positifs (boucle infinie sur les strictement négatifs)
Calcule la somme des nombres de 1 à n

```python
fonction2(5)
```

    15

```python
def fonction3(n):
    if n == 0:
        return 0
    return fonction3(n-1) - n
```

Termine pour les nombres positifs (boucle infinie sur les strictement négatifs)
Calcule -(la somme des nombres de 1 à n)

```python
fonction3(5)
```

    -15

```python
def fonction4(n):
    if n == 0:
        return 0
    if n < 0:
        return n + fonction4(-n)
    return n + fonction4(-n+1)
```

Termine pour tous les entiers
Calcule pour les positifs :

exemple 5

$5 + -4 + 4 + -3 + 3 + -2 + 2 + -1 + 1 + 0 = 5$

Pour les négatifs : 0

```python
fonction4(-10)
```

    0

```python
def fonction5(n):
    if n <= 1:
        return 0
    return 1 + fonction5(n-2)
```

Termine pour tous les entiers

Calcule n//2

```python
fonction5(11)
```

    5

### Exercice 3

La fonction de fibonacci est définie par :

$U_0 = 1$

$U_1 = 1$

$U_n = U_{n-1} + U_{n-2}$ si $n \geq 2$

Ces premières valeurs sont donc : 1, 1, 2, 3, 5, 8, 13, ...

Donner deux algorithmes, un itératif et un récursif, qui calculent la valeur $n$ de la suite.

```python
def fiboIteratif(n):
    u0 = 1
    u1 = 1
    for i in range(2,n+1):
        u0, u1 = u1, u0 + u1
    return u1
```

```python
[fiboIteratif(i) for i in range(10)]
```

    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

```python
def fiboRec(n):
    if n <= 1:
        return 1
    return fiboRec(n-1) + fiboRec(n-2)
```

```python
fiboRec(5)
```

    8

```python
[fiboRec(i) for i in range(10)]
```

    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

### Exercice 4

On rappelle sur un exemple le principe de l'algorithme d'Euclide du calcul du pgcd (plus grand diviseur commun).

Calcul du pgcd de 2145 et 630 : On commence par effectuer la division euclidienne de 2145 par 630

$2145 = 630 * 3 + 255$

puis on effectue la division de 630 par le reste obtenu 255

$630 = 255 * 2 + 120$

On continue jusqu'à ce que l'on trouve un reste nul.

$\begin{align*}
255 &= 120 *2 + 15 \\
120 &= 15 * 8
\end{align*}$

Le pgcd est le dernier reste non nul, c'est-à-dire **15**.

Donner un algorithme récursif qui calcule le pgcd de deux nombres.

```python

```

```python

```

```python

```

### Exercice 5

Reprendre l'algorithme de recherche dichotomique dans un tableau trié vu dans le TD1 et donner une version récursive.

```python
T = [1,2,3,3,4]
T
```

    [1, 2, 3, 3, 4]

```python
T[0]
```

    1

```python
len(T)
```

    5

```python
def dicho(T, deb, fin, x):
    if deb == fin:
        return False
    m = (deb + fin)//2
    if x == T[m]:
        return True
    if x < T[m]:
        return dicho(T,deb,m,x)
    return dicho(T,m+1,fin,x)
```

```python
T = [1,2,3,3,4]
dicho(T,0,len(T),2)
```

    True

```python
dicho(T,0,len(T),3.5)
```

    False

```python
dicho(T,0,len(T),6)
```

    False

```python
dicho(T,0,len(T),-3)
```

    False

### Exemple avancé : les Tours de Hanoï

Problème : on possède trois pics sur lesquels sont empilés des disques de tailles décroissantes (tous sur le premier pic). On souhaite déplacer la pile sur le second pic en suivant les règles suivantes:

* on ne peut déplacer qu'un seul disque à la fois, celui du haut de la pile,
* on ne peut poser un disque que sur un disque plus grand.

On suppose donc que l'on dispose d'une seule action possible :
Fonction Déplacer
Input :
    - le pic de départ
    - le pic d'arrivée
Processus :
    Si départ n'est pas vide:
        D <- disque retiré au sommet de départ
    Sinon:
        Erreur "Départ est vide"
    Si D < disque au sommet d'arrivée:
        Poser D au sommet d'arrivée
    Sinon:
        Erreur "Action impossible"
En python, on utilisera l'interface suivante :

```python
class Towers:

    def __init__(self,n):
        self.towers = (list(range(n,0,-1)),[],[])
        self.size = n

    def move(self,i,j):
        t1,t2 = self.towers[i], self.towers[j]
        if len(t1) == 0 or (len(t2) != 0 and t1[-1]>t2[-1]):
            raise ValueError("Invalid action")
        print("Move from Tower " + str(i) + " to Tower " + str(j))
        t2.append(t1.pop())

    def __repr__(self):
        res = ""
        n = self.size
        for i in range(n):
            line = ""
            for t in self.towers:
                if len(t) > i:
                    v = t[i]
                else:
                    v = 0
                stars = v*"*"
                spaces = (n-v)*" "
                line+= spaces + stars + "|" + stars + spaces + " "
            line+="\n"
            res = line + res
        return res
```

```python
T = Towers(3)
T
```

      *|*      |       |
     **|**     |       |
    ***|***    |       |

```python
T.move(0,1)
T
```

    Move from Tower 0 to Tower 1

       |       |       |
     **|**     |       |
    ***|***   *|*      |

```python
T.move(0,2)
T
```

    Move from Tower 0 to Tower 2

       |       |       |
       |       |       |
    ***|***   *|*    **|**

```python
T.move(1,2)
T
```

    Move from Tower 1 to Tower 2

       |       |       |
       |       |      *|*
    ***|***    |     **|**

```python
T.move(0,1)
T
```

    Move from Tower 0 to Tower 1

       |       |       |
       |       |      *|*
       |    ***|***  **|**

```python
T.move(2,0)
T
```

    Move from Tower 2 to Tower 0

       |       |       |
       |       |       |
      *|*   ***|***  **|**

```python
T.move(2,1)
T
```

    Move from Tower 2 to Tower 1

       |       |       |
       |     **|**     |
      *|*   ***|***    |

```python
T.move(0,1)
T
```

    Move from Tower 0 to Tower 1

       |      *|*      |
       |     **|**     |
       |    ***|***    |

```python

```

```python

```

**Comment résoudre le problème ?**

Pour résoudre un problème de façon récursive, il faut répondre à plusieurs questions :

1. Quelle est la taille de mon problème ?
2. Quelle est la plus petite taille possible ? (condition d'arrêt)
3. Comment résoudre le problème dans ce cas là ? (action d'arrêt)
4. Si je suppose que je sais résoudre le problème pour toutes les tailles $k < n$, comment le résoudre pour la taille $n$ ?

```python
@compteAppels
def hanoi(T,n,dep, arrivee, inter):
    if n == 1:
        T.move(dep,arrivee)
        print(T)
    else:
        hanoi(T,n-1,dep,inter,arrivee)
        T.move(dep,arrivee)
        print(T)
        hanoi(T,n-1,inter,arrivee,dep)
```

```python
hanoi(Towers(3),3,0,1,2)
```

    Move from Tower 0 to Tower 1
       |       |       |
     **|**     |       |
    ***|***   *|*      |

    Move from Tower 0 to Tower 2
       |       |       |
       |       |       |
    ***|***   *|*    **|**

    Move from Tower 1 to Tower 2
       |       |       |
       |       |      *|*
    ***|***    |     **|**

    Move from Tower 0 to Tower 1
       |       |       |
       |       |      *|*
       |    ***|***  **|**

    Move from Tower 2 to Tower 0
       |       |       |
       |       |       |
      *|*   ***|***  **|**

    Move from Tower 2 to Tower 1
       |       |       |
       |     **|**     |
      *|*   ***|***    |

    Move from Tower 0 to Tower 1
       |      *|*      |
       |     **|**     |
       |    ***|***    |

    7 appels à la fonction hanoi

```python

```

```python

```

## Complexité d'un algorithme récursif

### méthode de calcul

Reprenons le schéma de base d'un algorithme récursif  que l'on précise légèrement:
Fonction R
Input : des paramètres
Processus:
    Si les paramètres correspondent à la condition d'arret (cas simple):
        Actions cas d'arret
    Sinon
        Actions itératives et
        k Appel(s) récursifs: R(p1), R(p2), ..., R(pk)
On veut compter le nombre d'actions de base. Chaque appel de fonction est une action de base. Soit $f$ la complexité de l'algorithme. Pour simplifier l'écriture, on suppose que les paramètres de l'algorithme correspondent à la taille $n$ du problème. La condition d'arrêt est obtenue quand $n=0$ et correspond à une complexité constante $c_0$. On obtient une définition récursive de $f$

$\begin{align}
f(0) &= c_0 \\
f(n) &= g(n) + \sum_{i=0}^k f(p_i)
\end{align}$

où $g$ est la complexité de la partie itérative. La complexité dépend donc du **nombre d'appels récursifs** à chaque étape et de la **taille des paramètres**. Il peut être plus ou moins simple de *développer* la fonction $f$ pour obtenir une formule close (non récursive).

### Cas classiques

On va regarder quelques cas classiques et compter le nombre d'appels de fonction. Pour cela, on utilise la fonctionnalité suivante (exécuter la cellule)

```python
import functools

COMPTEUR = 0
STACK = 0

def compteAppels(func):
    c = 0
    """Print the function signature and return value"""
    @functools.wraps(func)
    def wrapper_debug(*args, **kwargs):
        global COMPTEUR, STACK
        COMPTEUR +=1
        STACK += 1
        value = func(*args, **kwargs)
        STACK -=1
        if STACK == 0:
            print(f"{COMPTEUR} appels à la fonction {func.__name__}")
            COMPTEUR = 0
        return value
    return wrapper_debug
```

**Exemple : cas linéaire**

On reprend la fonction factorielle vue précédemment.

```python
@compteAppels
def factorielle(n):
    if n <= 0:
        return 1
    return n*factorielle(n-1)
```

```python
factorielle(10)
```

    11 appels à la fonction factorielle

    3628800

Lorsqu'on appelle `factorielle(n)` on génère $n+1$ appels de fonctions. On est sur le schéma suivant :
Fonction R
Input : un entier n
Processus:
    Si n=0:
        Actions cas d'arret
    Sinon
        Appel de R(n-1)
Dans ce cas, si $f$ est le nombre d'appels, on a

$\begin{align}
f(0) &= 1 \\
f(n) &= 1 + f(n-1)
\end{align}$

Lorsqu'on développe, on obtient

$\begin{equation}
f(n) = 1 + 1 + 1 + \dots + 1 = n+1
\end{equation}$

$n+1$ fois.

Conclusion : la complexité est en $O(n)$.

**Exemple : cas exponentiel**

Reprenons l'exemple des Tours de Hanoï. Combien d'appels de fonctions sont réalisés pour la taille $n$ ?

```python
@compteAppels
def Hanoi(n, T, depart, arrivee, inter):
    if n == 0:
        return
    Hanoi(n-1, T, depart, inter, arrivee)
    T.move(depart, arrivee)
    Hanoi(n-1, T, inter, arrivee, depart)
```

```python
n = 3
T = Towers(n)
Hanoi(n,T,0,1,2)
T
```

    Move from Tower 0 to Tower 1
    Move from Tower 0 to Tower 2
    Move from Tower 1 to Tower 2
    Move from Tower 0 to Tower 1
    Move from Tower 2 to Tower 0
    Move from Tower 2 to Tower 1
    Move from Tower 0 to Tower 1
    15 appels à la fonction Hanoi

       |      *|*      |
       |     **|**     |
       |    ***|***    |

On est sur le schéma suivant :
Fonction R
Input : un entier n
Processus:
    Si n=0:
        Actions cas d'arret
    Sinon:
        2 appels de R(n-1)
Dans ce cas, la complexité est donnée par le nombre d'appels $f$ avec

$\begin{align}
f(0) &= 1 \\
f(n) &= 1 + 2 \times f(n-1).
\end{align}$

Par exemple, pour $n=3$ :

$\begin{align}
f(3) &= 1 + 2f(2) = 1 + 2(1 + 2f(1)) = 1 + 2(1 + 2(1 + 2))\\
 &= 15 = 2^4 - 1.
\end{align}$

On prouve facilement par récurrence que
$\begin{equation}
f(n) = 2^{n+1} - 1.
\end{equation}$

En effet, $f(0) = 1 = 2^1 - 1$ et si on suppose la formule vraie pour $n$, on obtient pour $n+1$

$f(n+1) = 1 + 2 f(n) = 1 + 2(2^{n+1} -1) = 1 + 2^{n+2} - 2 = 2^{n+2} - 1$

On est dans le cas d'une complexité exponentielle en $O(2^n)$.

**Exemple : cas logarithmique**

Reprenons l'exemple de la fonction de recherche dichotomique. Combien d'appels sont réalisés ici ?

```python
@compteAppels
def rechercheDich(T,v,deb, fin):
    m = (deb+fin)//2
    if deb == fin:
        return False
    if T[m] == v:
        return True
    if v < T[m]:
        return rechercheDich(T,v,deb,m)
    return rechercheDich(T,v,m+1,fin)
```

```python
n = 10000000
T = list(range(n))
v = 0.5
rechercheDich(T,v,0,n)
```

    25 appels à la fonction rechercheDich

    False

On est sur le schéma suivant
Fonction R
Input : un entier n
Processus:
    Si n=0:
        Actions cas d'arret
    Sinon:
        Appel de R(n/2)
Dans ce cas, si $f$ est le nombre d'appels, on a

$\begin{align}
f(0) &= 1 \\
f(n) &= 1 + f(\lfloor n/2 \rfloor).
\end{align}$

Développons sur un exemple :

$\begin{align}
f(10) &= 1 + f(5) = 2 + f(2) = 3 + f(1) = 5 \\
&= \log(10) + 2.
\end{align}$

(on prend le $\log$ en base 2)

On prouve par récurrence
$\begin{equation}
f(n) = \lfloor \log(n) \rfloor + 2.
\end{equation}$

En effet,

$f(1) = 1 + f(0) = 2 = \log(1) + 2$

En supposant la formule correcte pour toute valeur inférieure stricte à $n$, on obtient

$\begin{align}
f(n) &= 1 + f(\lfloor n/2 \rfloor) \\
&= 1 + \lfloor \log(\lfloor  n/2 \rfloor) \rfloor + 2
\end{align}$

Supposons $n$ pair, si $k = \lfloor \log(n/2) \rfloor$, cela signifie

$2^k \leq \frac{n}{2} < 2^{k+1}$, c'est à dire

$2^{k+1} \leq n < 2^{k+2}$

On peut obtenir une inégalité similaire dans le cas de $n$ impair. Cela signifie que :

$\lfloor \log(\lfloor  n/2 \rfloor) \rfloor = \lfloor \log(n) \rfloor - 1$

et donc

$f(n) = 1 + \lfloor \log(n) \rfloor - 1 + 2 = \lfloor \log(n) \rfloor + 2$

On a donc une complexité en $O(log(n))$.

### Conclusion : calcul de complexité

Si l'on simplifie le problème aux cas où la partie itérative de la fonction est en $O(1)$ (comme dans les exemples précédents). La complexité est donnée entièrement par le nombre d'appels récursifs, $f$ on a 3 cas de figures. ($v \geq 1$ et $k \geq 2$ sont des constantes qui ne dépendent pas de $n$)

$f(n) =  1 + f(n-v) \rightarrow$ complexité linéaire $O(n)$

$f(n) = 1 + k f(n - v) \rightarrow$ complexité exponentielle $O(k^n)$

$f(n) = 1 + f(n/k) \rightarrow$ complexité logarithmique $O(\log(n))$ (la constante $k$ agit sur la base dans laquelle on prend le logarithme, ce qui ne change pas la classe de complexité)

## Exercices

### Exercice 6

Dans la fonction suivante du calcul des nombres de Fibonacci, combien d'appels de fonctions sont réalisés pour un $n$ donné ? Quelle est la complexité ?

```python
@compteAppels
def fibo(n):
    if n <= 1:
        return 1
    return fibo(n-1) + fibo(n-2)
```

Le nombre d'appels est donné par

$f(0) = f(1) = 1$

$f(n) = 1 + f(n-1) + f(n-2)$

```python
fibo(20)
```

    21891 appels à la fonction fibo

    10946

```python
def fiboIteratif(n):
    u0 = 1
    u1 = 1
    for i in range(2,n+1):
        u0, u1 = u1, u0 + u1
    return u1
```

```python
fiboIteratif(100)
```

    573147844013817084101

```python

```

```python

```

### Exercice 7

Il existe deux définitions récursives de la fonction puissance :

$\begin{align*}
a^b &= \begin{cases}
1 &\text{si }b=0 \\
a \times a^{b-1} &\text{sinon}
\end{cases} \\
a^b &= \begin{cases}
1 &\text{si }b=0 \\
a \times a^{b-1} &\text{si }b\text{ est impair} \\
a^{\frac{b}{2}}a^{\frac{b}{2}} &\text{si }b\text{ est pair}
\end{cases}
\end{align*}$

Pour chacune des définitions, donner l'algorithme récursif correspondant ainsi que sa complexité.

```python
@compteAppels
def puissance1(a,b):
    if b == 0:
        return 1
    return a * puissance1(a,b-1)
```

```python
puissance1(2,100)
```

    101 appels à la fonction puissance1

    1267650600228229401496703205376

```python
@compteAppels
def puissance2(a,b):
    if b == 0:
        return 1
    if b%2 == 1:
        res = puissance2(a,(b-1)//2)
        return a * res * res
    res = puissance2(a,b//2)
    return res * res
    #return puissance2(a,b//2) * puissance2(a,b//2)
```

```python
puissance2(2,10000)
```

    15 appels à la fonction puissance2

    19950631168807583848837421626835850838234968318861924548520089498529438830221946631919961684036194597899331129423209124271556491349413781117593785932096323957855730046793794526765246551266059895520550086918193311542508608460618104685509074866089624888090489894838009253941633257850621568309473902556912388065225096643874441046759871626985453222868538161694315775629640762836880760732228535091641476183956381458969463899410840960536267821064621427333394036525565649530603142680234969400335934316651459297773279665775606172582031407994198179607378245683762280037302885487251900834464581454650557929601414833921615734588139257095379769119277800826957735674444123062018757836325502728323789270710373802866393031428133241401624195671690574061419654342324638801248856147305207431992259611796250130992860241708340807605932320161268492288496255841312844061536738951487114256315111089745514203313820202931640957596464756010405845841566072044962867016515061920631004186422275908670900574606417856951911456055068251250406007519842261898059237118054444788072906395242548339221982707404473162376760846613033778706039803413197133493654622700563169937455508241780972810983291314403571877524768509857276937926433221599399876886660808368837838027643282775172273657572744784112294389733810861607423253291974813120197604178281965697475898164531258434135959862784130128185406283476649088690521047580882615823961985770122407044330583075869039319604603404973156583208672105913300903752823415539745394397715257455290510212310947321610753474825740775273986348298498340756937955646638621874569499279016572103701364433135817214311791398222983845847334440270964182851005072927748364550578634501100852987812389473928699540834346158807043959118985815145779177143619698728131459483783202081474982171858011389071228250905826817436220577475921417653715687725614904582904992461028630081535583308130101987675856234343538955409175623400844887526162643568648833519463720377293240094456246923254350400678027273837755376406726898636241037491410966718557050759098100246789880178271925953381282421954028302759408448955014676668389697996886241636313376393903373455801407636741877711055384225739499110186468219696581651485130494222369947714763069155468217682876200362777257723781365331611196811280792669481887201298643660768551639860534602297871557517947385246369446923087894265948217008051120322365496288169035739121368338393591756418733850510970271613915439590991598154654417336311656936031122249937969999226781732358023111862644575299135758175008199839236284615249881088960232244362173771618086357015468484058622329792853875623486556440536962622018963571028812361567512543338303270029097668650568557157505516727518899194129711337690149916181315171544007728650573189557450920330185304847113818315407324053319038462084036421763703911550639789000742853672196280903477974533320468368795868580237952218629120080742819551317948157624448298518461509704888027274721574688131594750409732115080498190455803416826949787141316063210686391511681774304792596709376

## Pour finir : itératif ou récursif ?

En terme de complexité, la récursivité ne permet pas par défaut d'obtenir des algorithmes plus efficaces. Quand elle est mal utilisée, elle peut même donner des complexité **plus mauvaise** : exemple de Fibonacci. Par ailleurs, on peut prouver qu'il existe **toujours** une version itérative d'un algorithme récursif : il suffit de déplier la pile !

**Qu'en est-il de la complexité mémoire ?** Reprendre l'exemple de la fonction exponentielle en itératif et en récursif. La complexité mémoire de l'algorithme récursif est en $O(n)$ tandis qu'il est en $O(1)$ pour l'itératif.

**Alors pourquoi on fait du récursif ?** Sur de nombreux problèmes, les algorithmes récursifs sont beaucoup plus simples à concevoir que les algorithmes itératifs. C'est le cas de l'algorithme des Tour de Hanoi. Ils donnent des codes plus courts et plus lisibles. Par ailleurs, certaines **structure de données** ont elles-mêmes des **définition récursives** et sont particulièrement adaptés aux algorithmes récursifs : les arbres, les graphes, les listes chaînées.

```python

```


# Límites de la Precisión

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
sym.init_printing()
from scipy import optimize
```

```python
def bisection(f, a, b, n, tol=1e-16):
    fa = f(a)
    fb = f(b)
    x = np.empty(n)

    # Just checking if the sign is not negative => not root  necessarily
    if np.sign(f(a)*f(b)) >= 0:
        print('f(a)f(b)<0 not satisfied!')
        return None
    i = 0
    while (b-a) / 2 > tol and i < n:
        c = (a + b) / 2.
        x[i] = c
        fc = f(c)
        if fc == 0:
            x = x[:i+1]
            break
        elif np.sign(fa*fc) < 0:
            b = c
            fb = fc
        else:
            a = c
            fa = fc
        i += 1

    return x
```

```python
def fpi(g, x0, n, tol=1e-10):
    x = np.empty(n+1)
    x[0] = x0
    for i in range(n):
        x[i+1] = g(x[i])
        if np.abs(x[i+1] - x[i]) < tol:
            x = x[:i+2]
            break
    return x
```

```python
def secant(f, x0, x1, n, tol=1e-10):
    x = np.zeros(n + 2)
    x[0] = x0
    x[1] = x1
    for i in range(1, n + 1):
        x[i+1] = x[i] - (f(x[i]) * (x[i] - x[i-1])) / (f(x[i]) - f(x[i-1]))
        if np.abs(x[i+1] - x[i]) < tol:
            x = x[:i+1]
            break
    return x
```

```python
def newton_raphson(f, fp, x0, n, m=1, tol=1e-10):
    x = np.empty(n + 1)
    x[0] = x0
    for i in range(n):
        x[i+1] = x[i] - f(x[i]) / fp(x[i])
        if np.abs(x[i+1] - x[i]) < tol or f(x[i+1]) == 0:
            x = x[:i+2]
            break
    return x
```

# Backward y Forward Error

Analicemos la función
\begin{equation}
    f(x) = \left(x-\frac{2}{3}\right)^3 = x^3 - 2x^2 + \frac{4}{3}x-\frac{8}{27}
\end{equation}

donde su raíz es $x=2/3$ con multiplicidad $3$.

```python
f = lambda x: (x - 2 / 3) ** 3
fp = lambda x: 3 * (x - 2 / 3) ** 2
```

```python
error = lambda x, r: np.abs(x - r)
```

```python
def plotError(f, a, r):
    plt.figure(figsize=(12, 6))
    plt.plot(error(f(a), 0), 'r-o', label='Backward Error')
    plt.plot(error(a, r), 'b-o', label="Forward Error")
    plt.yscale('log')
    plt.xlabel("# Iteraciones")
    plt.ylabel("Error")
    plt.grid(True)
    plt.legend()
    plt.show()
```

```python
x = np.linspace(-1, 2, 100)
plt.plot(x, f(x))
plt.xlabel(r"$x$")
plt.ylabel(r"$f(x)$")
plt.grid(True)
plt.show()
```

```python
# Solución
r = 2/3
```

### Bisección

```python
a, b = .5, .7
n_b = 50
x_b = bisection(f, a, b, n_b)
x_b[-1]
```

```python
plotError(f, x_b, r)
```

### Secante

```python
x0 = .6
x1 = .7
n = 50
x_s = secant(f, x0, x1, n)
x_s[-1]
```

```python
plotError(f, x_s, r)
```

### Newton-Raphson

```python
x0 = .6
n = 50
x_n = newton_raphson(f, fp, x0, n, 3)
x_n[-1]
```

```python
plotError(f, x_n, r)
```

De acuerdo a lo revisado en clases y analizando los resultados, el **backward error** $|f(x_a)|$ decae rápidamente pero esto no implica que el **forward error** $|r-x_a|$ lo haga a la misma velocidad. Cuando un problema no está bien condicionado, no podemos asegurar que el comportamiento de ambos errores esté directamente relacionado.

# Wilkinson Polynomial

Este polinomio tiene un comportamiento interesante. Si bien sus raíces son evidentes, al utilizar los métodos revisados no vamos a obtener exactamente sus raíces.

```python
# Construccion de W(x)
x = sym.symbols('x', reals=True)
Wf = np.prod(x - np.arange(1, 21))
Wf # W(x)
```

```python
# Expansión del polinomio
We = sym.expand(Wf)
We
```

```python
# Derivada version expandida
Wep = sym.diff(We, x)
Wep
```

```python
# Derivada version factorizada
Wd = sym.diff(Wf, x)
Wd
```

```python
# Transformar a funcion lambda para podes evaluar con nuestros métodos
# Version expandida
P = sym.lambdify(x, We) # def P(x): ... o P = lambda x: ...
Pp = sym.lambdify(x, Wep)
# Factorizada
W = sym.lambdify(x, Wf)
Wp = sym.lambdify(x, Wd)
```

Solo para visualizar...

```python
x1 = np.linspace(0.9, 20.1, 500)
px = W(x1)
plt.figure(figsize=(12, 6))
plt.plot(x1, px)
plt.xticks(np.arange(1, 21))
plt.grid(True)
plt.show()
```

Con el gráfico $\log(f(x))$ no se verán los valores negativos pero verificamos que sus raíces sean las que corresponden.

¿Qué pasa con la raíz $x=16$? Notar que ocurre en un vecindario a $x=16$...

```python
P(16 - .1), P(16.0), P(16), P(16 + .1)
```

```python
W(16 - .1),  W(16), W(16.0), W(16 + .1)
```

¿Por qué la diferencia del polinomio expandido y el factorizado?

## Bisección

```python
r_w = 16
```

```python
a, b = r_w - 0.1, r_w + 0.1
n_b = 100
x_b = bisection(P, a, b, n_b, 1e-16)
x_b[-1]
```

```python
x_b[-1], P(x_b[-1]), W(x_b[-1])
```

```python
plotError(P, x_b, r_w)
```

## Secante

```python
x0 = r_w - .1
x1 = r_w + .1
n_s = 1000
x_s = secant(P, x0, x1, n_s, 1e-16)
x_s[-1]
```

```python
plotError(P, x_s, r_w)
```

```python
x0 = r_w - .1
n_n = 100
x_n = newton_raphson(P, Pp, x0, n_n, 1e-16)
x_n[-1]
```

```python
plotError(P, x_n, r_w)
```

```python
x_b[-1]-16
```

```python
print(error(x_b[-1], r_w), error(x_s[-1], r_w), error(x_n[-1], r_w))
```

    0.0033284982507311156 0.004299420643381069 0.017405068783038047

En clases derivamos que $\Delta r \approx \pm 0.0136$ que se acerca más o menos a los resultados experimentales. ¿Podemos concluir con la información del **backward error**?, ¿Es un problema bien o mal condicionado?

---

# Comentario Iteración Punto Fijo!

Recordemos el teorema de Taylor y tomando una idea similar al *Método de Newton-Raphson*, realicemos la siguiente expansión de $f(x)$ en torno a la raíz $r$:

\begin{equation}
    \begin{split}
        f(x) & = f(r) + f'(r)(x-r) + O(x^2) \\
        f(x) & = 0 + f'(r)(x-r) + O(x^2) \quad \frac{1}{f'(r)}, f'(r)\neq 0 \\
        \frac{f(x)}{f'(r)} & = 0 + (x-r) + O(x^2) \\
        r = x - \frac{f(x)}{f'(r)} + O(x^2).
    \end{split}
\end{equation}

Despreciando los términos $O(x^2)$ construyamos una iteración del tipo:
\begin{equation}
    \begin{split}
        x_{i+1} & = x_i + \alpha \, f(x_i) \\
         & = g(x_i),
    \end{split}
\end{equation}

donde $\alpha = -\frac{1}{f'(r)}$ es un parámetro conveniente que nos asegure $|g'(r)| < 1$.

Si analizamos la expresión:
\begin{equation}
    \begin{split}
        g(x) & = x + \alpha f(x) \\
        g'(x) & = 1 + \alpha f'(x),
    \end{split}
\end{equation}

la idea es que $|1 + \alpha f'(x)| < 1$ cuando $x\to r$.

```python
G = lambda x, f, a: x + a * f(x)
```

Ejemplos...

```python
f1 = lambda x: 2 - x ** 2
```

Probemos para distintos $\alpha$:

```python
g1 = lambda x: G(x, f1, .5)
g2 = lambda x: G(x, f1, .1)
```

```python
x0 = 1
n = 100
```

```python
x_g_1 = fpi(g1, x0, n)
```

```python
x_g_1, x_g_1.shape
```

    (array([1.        , 1.5       , 1.375     , 1.4296875 , 1.40768433,
            1.41689675, 1.41309855, 1.41467479, 1.41402241, 1.41429272,
            1.41418077, 1.41422714, 1.41420794, 1.41421589, 1.4142126 ,
            1.41421396, 1.4142134 , 1.41421363, 1.41421353, 1.41421357,
            1.41421356, 1.41421356, 1.41421356, 1.41421356, 1.41421356,
            1.41421356, 1.41421356]),
     (27,))

```python
x_g_2 = fpi(g2, x0, n)
```

```python
x_g_2, x_g_2.shape
```

    (array([1.        , 1.1       , 1.179     , 1.2399959 , 1.28623692,
            1.32079638, 1.34634607, 1.3650813 , 1.3787366 , 1.38864514,
            1.39581161, 1.4009826 , 1.40470738, 1.4073871 , 1.40931325,
            1.41069687, 1.4116903 , 1.41240335, 1.41291503, 1.41328214,
            1.4135455 , 1.41373441, 1.41386991, 1.4139671 , 1.4140368 ,
            1.4140868 , 1.41412265, 1.41414836, 1.4141668 , 1.41418003,
            1.41418951, 1.41419632, 1.41420119, 1.41420469, 1.4142072 ,
            1.414209  , 1.41421029, 1.41421122, 1.41421188, 1.41421236,
            1.4142127 , 1.41421294, 1.41421312, 1.41421324, 1.41421333,
            1.4142134 , 1.41421344, 1.41421348, 1.4142135 , 1.41421352,
            1.41421353, 1.41421354, 1.41421355, 1.41421355, 1.41421355,
            1.41421356, 1.41421356, 1.41421356, 1.41421356, 1.41421356,
            1.41421356, 1.41421356, 1.41421356, 1.41421356, 1.41421356,
            1.41421356]),
     (66,))

```python
r1 = np.sqrt(2)
print(np.abs(x_g_1[-1] - r1), np.abs(x_g_2[-1] - r1))
```

    2.4623636463161347e-11 2.1262236415964253e-10

```python
plt.figure(figsize=(12,6))
plt.plot(error(x_g_1, r1), 'b.', label=r"$\alpha=0.5$")
plt.plot(error(x_g_2, r1), 'r.', label=r"$\alpha=0.1$")
plt.yscale('log')
plt.xlabel("# iteraciones")
plt.ylabel("Error")
plt.legend()
plt.grid(True)
plt.show()
```

Según el $\alpha$ que utilicemos podemos asegurar convergencia, incluso mejorar la tasa!

Otro ejemplo!

```python
f2 = lambda x: x ** 2 - x - 1
```

```python
g3 = lambda x: G(x, f2, -.4)
g4 = lambda x: G(x, f2, -.8)
```

Recordar que $\alpha \approx -1/f'(r)$, y $f'(x) =2x-1$. Como $f'(x) > 0$ cerca de $r$ entonces $\alpha < 0$. Esa es la razón de utilizar valores negativos...

```python
x0 = 1
n = 100
```

```python
x_g_3 = fpi(g3, x0, n)
```

```python
x_g_3, x_g_3.shape
```

    (array([1.        , 1.4       , 1.576     , 1.6128896 , 1.6174803 ,
            1.61797541, 1.6180278 , 1.61803334, 1.61803392, 1.61803398,
            1.61803399, 1.61803399, 1.61803399]),
     (13,))

```python
x_g_4 = fpi(g4, x0, n)
```

```python
x_g_4, x_g_4.shape
```

    (array([1.        , 1.8       , 1.448     , 1.7290368 , 1.52061164,
            1.68729315, 1.55956113, 1.6614253 , 1.58229832, 1.6452026 ,
            1.59601141, 1.6350186 , 1.60440482, 1.62863682, 1.60957997,
            1.6246458 , 1.61278326, 1.622154  , 1.61477032, 1.62060002,
            1.61600449, 1.61963167, 1.61677161, 1.61902855, 1.61724863,
            1.61865303, 1.61754535, 1.61841926, 1.61772995, 1.61827376,
            1.6178448 , 1.6181832 , 1.61791626, 1.61812685, 1.61796073,
            1.61809177, 1.6179884 , 1.61806995, 1.61800562, 1.61805637,
            1.61801634, 1.61804791, 1.618023  , 1.61804265, 1.61802715,
            1.61803938, 1.61802973, 1.61803734, 1.61803134, 1.61803608,
            1.61803234, 1.61803529, 1.61803296, 1.6180348 , 1.61803335,
            1.61803449, 1.61803359, 1.6180343 , 1.61803374, 1.61803418,
            1.61803384, 1.61803411, 1.61803389, 1.61803406, 1.61803393,
            1.61803404, 1.61803395, 1.61803402, 1.61803397, 1.61803401,
            1.61803397, 1.618034  , 1.61803398, 1.618034  , 1.61803398,
            1.61803399, 1.61803399, 1.61803399, 1.61803399, 1.61803399,
            1.61803399, 1.61803399, 1.61803399, 1.61803399, 1.61803399,
            1.61803399, 1.61803399, 1.61803399, 1.61803399, 1.61803399,
            1.61803399, 1.61803399, 1.61803399, 1.61803399, 1.61803399,
            1.61803399]),
     (96,))

```python
r2 = (1 + np.sqrt(5)) / 2
```

```python
plt.figure(figsize=(12,6))
plt.plot(error(x_g_3, r2), 'b.', label=r"$\alpha=-0.4$")
plt.plot(error(x_g_4, r2), 'r.', label=r"$\alpha=-0.8$")
plt.yscale('log')
plt.xlabel("# iteraciones")
plt.ylabel("Error")
plt.legend()
plt.grid(True)
plt.show()
```

Quizás esta herramienta pueda ser útil para construir iteraciones de punto fijo...


# Training the Neural Network

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}$

```python
# init
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import tensorboard as tb
import torch
import torch.optim as optim
from torch import Tensor, nn
from torch.nn import functional as F
from torch.utils.tensorboard import SummaryWriter

%load_ext tensorboard
%load_ext jdc
%matplotlib inline

SEED = 0

# create samples
XY_rng = np.random.default_rng(SEED)
rho = 1 - 0.19 * XY_rng.random()
mean, cov, n = [0, 0], [[1, rho], [rho, 1]], 1000
XY = XY_rng.multivariate_normal(mean, cov, n)

XY_ref_rng = np.random.default_rng(SEED)
cov_ref, n_ = [[1, 0], [0, 1]], n
XY_ref = XY_ref_rng.multivariate_normal(mean, cov_ref, n_)
```

    The tensorboard extension is already loaded. To reload it, use:
      %reload_ext tensorboard
    The jdc module is not an IPython extension.

We will train a neural network with `torch` and use GPU if available:

```python
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

if DEVICE == "cuda":  # print current GPU name if available
    print("Using GPU:", torch.cuda.get_device_name(torch.cuda.current_device()))
```

When GPU is available, you can use [GPU dashboards][gpu] on the left to monitor GPU utilizations.

[gpu]: https://github.com/rapidsai/jupyterlab-nvdashboard

**How to train a neural network by gradient descent?**

We will first consider a simple implementation followed by a more practical implementation.

## A simple implementation of gradient descent

Consider solving for a given $z\in \mathbb{R}$,

$$ \inf_{w\in \mathbb{R}} \overbrace{e^{w\cdot z}}^{L(w):=}.$$

We will train one parameter, namely, $w$, to minimize the loss $L(w)$.

**Exercise**

What is the solution for $z=-1$?

**Solution**

With $z=-1$,

$$
L(w) = e^{-w} \geq 0
$$

which is achievable with equality as $w\to \infty$.

**How to implement the loss function?**

We will define the loss function using tensors:

```python
z = Tensor([-1]).to(DEVICE)  # default tensor type on a designated device

def L(w):
    return (w * z).exp()

L(float("inf"))
```

    tensor([0.])

The function `L` is vectorized because `Tensor` operations follow the [broadcasting rules of `numpy`](https://numpy.org/doc/stable/user/basics.broadcasting.html):

```python
ww = np.linspace(0, 10, 100)
ax = sns.lineplot(
    x=ww,
    y=L(Tensor(ww).to(DEVICE)).cpu().numpy(),  # convert to numpy array for plotting
)
ax.set(xlabel=r"$w$", title=r"$L(w)=e^{-w}$")
ax.axhline(L(float("inf")), ls="--", c="r")
plt.show()
```

**What is gradient descent?**

A gradient descent algorithm updates the parameter $w$ iteratively starting with some initial $w_0$:

$$w_{i+1} = w_i - s_i \nabla L(w_i) \qquad \text{for }i\geq 0,$$

where $s_i$ is the *learning rate* (*step size*).

**How to compute the gradient?**

With $w=0$,

$$\nabla L(0) = \left.-e^{-w}\right|_{w=0}=-1,$$

which can be computed using `backward` ([backpropagation][bp]):

[bp]: https://en.wikipedia.org/wiki/Backpropagation

```python
w = Tensor([0]).to(DEVICE).requires_grad_()  # requires gradient calculation for w
L(w).backward()  # calculate the gradient by backpropagation
w.grad
```

    tensor([-1.])

Under the hood, the function call `L(w)`

- not only return the loss function evaluated at `w`, but also
- updates a computational graph for calculating the gradient since `w` `requires_grad_()`.

**How to implement the gradient descent?**

With a learning rate of `0.001`:

```python
for i in range(1000):
    w.grad = None  # zero the gradient to avoid accumulation
    L(w).backward()
    with torch.no_grad():  # updates the weights in place without gradient calculation
        w -= w.grad * 1e-3

print("w:", w.item(), "\nL(w):", L(w).item())
```

    w: 0.6933202147483826
    L(w): 0.4999134838581085

**What is `torch.no_grad()`?**

It sets up a context where the computational graph will not be updated. In particular,

```Python
w -= w.grad * 1e-3
```

should not be differentiated in the subsequent calculations of the gradient.

[no_grad]: https://pytorch.org/docs/stable/generated/torch.no_grad.html

**Exercise**

Repeatedly run the above cell until you get `L(w)` below `0.001`. How large is the value of `w`? What is the limitations of the simple gradient descent algorithm?

**Solution**

The value of `w` needs to be smaller than `6.9`. The convergence can be slow, especially when the learning rate is small. Also, `w` can be far away from its optimal value even if `L(w)` is close to its minimum.

## A practical implementation

For a neural network to approximate a sophisticated function, it should have many parameters (*degrees of freedom*).

**How to define a neural network?**

The following code [defines a simple neural network][define] with 3 fully-connected (fc) hidden layers:

where

- $\M{W}_l$ and $\M{b}_l$ are the weight and bias respectively for the linear transformation $\M{W}_l \M{a}_l + \M{b}_l$ of the $l$-th layer; and
- $\sigma$ for the first 2 hidden layers is an activation function called the [*exponential linear unit (ELU)*](https://pytorch.org/docs/stable/generated/torch.nn.ELU.html).

[define]: https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#define-the-network

```python
class Net(nn.Module):
    def __init__(self, input_size=2, hidden_size=100, sigma=0.02):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)  # fully-connected (fc) layer
        self.fc2 = nn.Linear(hidden_size, hidden_size)  # layer 2
        self.fc3 = nn.Linear(hidden_size, 1)  # layer 3
        nn.init.normal_(self.fc1.weight, std=sigma)  #
        nn.init.constant_(self.fc1.bias, 0)
        nn.init.normal_(self.fc2.weight, std=sigma)
        nn.init.constant_(self.fc2.bias, 0)
        nn.init.normal_(self.fc3.weight, std=sigma)
        nn.init.constant_(self.fc3.bias, 0)

    def forward(self, z):
        a1 = F.elu(self.fc1(z))
        a2 = F.elu(self.fc2(a1))
        t = self.fc3(a2)
        return t

torch.manual_seed(SEED)  # seed RNG for PyTorch
net = Net().to(DEVICE)
print(net)
```

    Net(
      (fc1): Linear(in_features=2, out_features=100, bias=True)
      (fc2): Linear(in_features=100, out_features=100, bias=True)
      (fc3): Linear(in_features=100, out_features=1, bias=True)
    )

The neural network is also a vectorized function. E.g., the following call `net` once to plots the density estimate of all $t(\R{Z}_i)$'s and $t(\R{Z}'_i)$'s.

```python
Z = Tensor(XY).to(DEVICE)
Z_ref = Tensor(XY_ref).to(DEVICE)

tZ = (
    net(torch.cat((Z, Z_ref), dim=0))  # compute t(Z_i)'s and t(Z'_i)
    # output needs to be converted back to an array on CPU for plotting
    .cpu()  # copy back to CPU
    .detach()  # detach from current graph (no gradient calculation)
    .numpy()  # convert output back to numpy
)

tZ_df = pd.DataFrame(data=tZ, columns=["t"])
sns.kdeplot(data=tZ_df, x="t")
plt.show()
```

For 2D sample $(x,y)\in \mc{Z}$, we can plot the neural network $t(x,y)$ as a heatmap. The following code adds a method `plot` to `Net` using [`jdc`](https://alexhagen.github.io/jdc):

```python
%%add_to Net
def plot(net, xmin=-5, xmax=5, ymin=-5, ymax=5, xgrids=50, ygrids=50, ax=None):
    """Plot a heat map of a neural network net. net can only have two inputs."""
    x, y = np.mgrid[xmin : xmax : xgrids * 1j, ymin : ymax : ygrids * 1j]
    xy = np.concatenate((x[:, :, None], y[:, :, None]), axis=2)
    with torch.no_grad():
        z = (
            net(
                torch.cat(
                    [
                        Tensor(x.reshape(-1, 1)).to(DEVICE),
                        Tensor(y.reshape(-1, 1)).to(DEVICE),
                    ],
                    dim=-1,
                )
            )
            .reshape(x.shape)
            .cpu()
        )
    if ax is None:
        ax = plt.gca()
    im = ax.pcolormesh(x, y, z, cmap="RdBu_r", shading="auto")
    ax.figure.colorbar(im)
    ax.set(xlabel=r"$x$", ylabel=r"$y$", title=r"Heatmap of $t(z)$ for $z=(x,y)$")
```

To plot the heatmap:

```python
net.plot()
```

**Exercise**

Why are the values of $t(\R{Z}_i)$'s and $t(\R{Z}'_i)$'s concentrated around $0$?

**Solution**

The neural network parameters are all very close to $0$ as we have set a small variance `sigma=0.02` to initialize them randomly. Hence:

- The linear transformation $\M{W}_l (\cdot) + \M{b}_l$ is close to $0$ for when the weight and bias are close to $0$.
- The ELU activation function $\sigma$ is also close to $0$ if its input is close to $0$.

**How to implements the divergence estimate?**

We decompose the approximate divergence lower bound in {eq}`avg-DV` as follows:

$$
\begin{align}
\op{DV}(\R{Z}^n,\R{Z'}^{n'},\theta) &:= \underbrace{\frac1{n} \sum_{i\in [n]} t(\R{Z}_i)}_{\text{(a)}} - \underbrace{\log \frac1{n'} \sum_{i\in [n']} e^{t(\R{Z}'_i)}}_{ \underbrace{\log \sum_{i\in [n']} e^{t(\R{Z}'_i)}}_{\text{(b)}} - \underbrace{\log n'}_{\text{(c)}}}
\end{align}
$$

where $\theta$ is a tuple of parameters (weights and biases) of the neural network that computes $t$:

$$
\theta := (\M{W}_l,\M{b}_l|l\in [3]).
$$

```python
def DV(Z, Z_ref, net):
    avg_tZ = net(Z).mean()  # (a)
    log_avg_etZ_ref = net(Z_ref).logsumexp(dim=0) - np.log(Z_ref.shape[0])  # (b) - (c)
    return avg_tZ - log_avg_etZ_ref

DV_estimate = DV(Z, Z_ref, net)
```

**Exercise**

Why is it preferrable to use `logsumexp(dim=0)` instead of `.exp().sum().log()`? Try running

```Python
Tensor([100]).exp().log(), Tensor([100]).logsumexp(0)
```

in a separate console.

**Solution**

`logsumexp(dim=0)` is numerically more stable than `.exp().mean().log()` especially when the output of the exponential function is too large to be represented with the default floating point precision.

To calculate the gradient of the divergence estimate with respect to $\theta$:

```python
net.zero_grad()  # zero the gradient values of all neural network parameters
DV(Z, Z_ref, net).backward()  # calculate the gradient
a_param = next(net.parameters())
```

`a_param` is a (module) parameter in $\theta$ retrieved from the parameter iterator `parameters()`.

**Exercise**

Check that the value of `a_param.grad` is non-zero. Is `a_param` a weight or a bias?

**Solution**

It should be the weight matrix $\M{W}_1$ because the shape is `torch.Size([100, 2])`.

**How to gradient descend?**

We will use the [*Adam's* gradient descend algorithm][adam] implemented as an optimizer [`optim.Adam`][optimAdam]:

[adam]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent#cite_note-Adam2014-28
[optimAdam]: https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam

```python
net = Net().to(DEVICE)
optimizer = optim.Adam(
    net.parameters(), lr=1e-3
)  # Allow Adam's optimizer to update the neural network parameters
optimizer.step()  # perform one step of the gradient descent
```

To alleviate the problem of overfitting, the gradient is often calculated on randomly chosen batches:

$$
\begin{align}
\R{L}(\theta) := - \bigg[\frac1{\abs{\R{B}}} \sum_{i\in \R{B}} t(\R{Z}_i) - \log \frac1{\abs{\R{B}'}} \sum_{i\in \R{B}'} e^{t(\R{Z}'_i)} - \log \abs{\R{B}'} \bigg],
\end{align}
$$

which is the negative lower bound of the VD formula in {eq}`DV` but on the minibatches

$$\R{Z}_{\R{B}}:=(\R{Z}_i\mid i\in \R{B})\quad \text{and}\quad \R{Z}'_{\R{B}'}$$

where $\R{B}$ and $\R{B}'$ are batches of uniformly randomly chosen indices from $[n]$ and $[n']$ respectively.

The neural network parameter is updated

$$
\theta_{j+1} := \theta_j - s_j \nabla \R{L}_j(\theta_j),
$$

starting with a randomly initialized $\theta_0$
where $s_j>0$ is the learning rate and $\R{L}_j$ is the loss evaluated on the $j$-th randomly chosen batches $\R{B}_j$ and $\R{B}'_j$.

The different batches are often obtained by
- permuting the samples first, and then
- partitioning the samples into batches.

This is illustrated by the figure below:

```python
n_iters_per_epoch = 10  # ideally a divisor of both n and n'
batch_size = int((Z.shape[0] + 0.5) / n_iters_per_epoch)
batch_size_ref = int((Z_ref.shape[0] + 0.5) / n_iters_per_epoch)
```

We will use `tensorboard` to show the training logs.
Rerun the following to start a new log, for instance, after a change of parameters.

```python
if input("New log?[Y/n] ").lower() != "n":
    n_iter = n_epoch = 0  # keep counts for logging
    writer = SummaryWriter()  # create a new folder under runs/ for logging
```

    New log?[Y/n]

The following code carries out Adam's gradient descent on batch loss:

```python
if input("Train? [Y/n]").lower() != "n":
    for i in range(10):  # loop through entire data multiple times
        n_epoch += 1

        # random indices for selecting samples for all batches in one epoch
        idx = torch.randperm(Z.shape[0])
        idx_ref = torch.randperm(Z_ref.shape[0])

        for j in range(n_iters_per_epoch):  # loop through multiple batches
            n_iter += 1
            optimizer.zero_grad()

            # obtain a random batch of samples
            batch_Z = Z[idx[j : Z.shape[0] : n_iters_per_epoch]]
            batch_Z_ref = Z_ref[idx_ref[j : Z_ref.shape[0] : n_iters_per_epoch]]

            # define the loss as negative DV divergence lower bound
            loss = -DV(batch_Z, batch_Z_ref, net)
            loss.backward()  # calculate gradient
            optimizer.step()  # descend

        writer.add_scalar("Loss/train", loss.item(), global_step=n_epoch)

    # Estimate the divergence using all data
    with torch.no_grad():
        estimate = DV(Z, Z_ref, net).item()
        writer.add_scalar("Estimate", estimate, global_step=n_epoch)
        net.plot()
        print("Divergence estimation:", estimate)
```

Run the following to show the losses and divergence estimate in `tensorboard`. You can rerun the above cell to train the neural network more.

```python
%tensorboard --logdir=runs
```

The ground truth is given by

$$D(P_{\R{Z}}\|P_{\R{Z}'}) = \frac12 \log(1-\rho^2) $$

where $\rho$ is the randomly generated correlation in the previous notebook.

**Exercise**

Compute the ground truth using the formula above.

```python
### BEGIN SOLUTION
ground_truth = -0.5 * np.log(1 - rho ** 2)
### END SOLUTION
ground_truth
```

    0.7405246745135301

**Exercise**

See if you can get an estimate close to this value by training the neural network repeatedly as shown below.

## Encapsulation

It is a good idea to encapsulate the training by a class, so multiple configurations can be run without interfering each other:

```python
class DVTrainer:
    """
    Neural estimator for KL divergence based on the sample DV lower bound.

    Estimate D(P_Z||P_Z') using samples Z and Z' by training a network t to maximize
        avg(t(Z)) - log avg(e^t(Z'))

    Parameters:
    ----------

    Z, Z_ref : Tensors with first dimension indicing the samples of Z and Z' respect.
    net : The neural network t that take Z as input and output a real number for each sample.
    n_iters_per_epoch : Number of iterations per epoch.
    writer_params : Parameters to be passed to SummaryWriter for logging.
    """

    # constructor
    def __init__(self, Z, Z_ref, net, n_iters_per_epoch, writer_params={}, **kwargs):
        self.Z = Z
        self.Z_ref = Z_ref
        self.net = net
        self.n_iters_per_epoch = n_iters_per_epoch  # ideally a divisor of both n and n'

        # set optimizer
        self.optimizer = optim.Adam(net.parameters(), **kwargs)

        # logging
        self.writer = SummaryWriter(
            **writer_params
        )  # create a new folder under runs/ for logging
        self.n_iter = self.n_epoch = 0  # keep counts for logging

    def step(self, epochs=1):
        """
        Carries out the gradient descend for a number of epochs and returns
        the divergence estimate evaluated over the entire data.

        Loss for each epoch is recorded into the log, but only one divergence
        estimate is computed/logged using the entire dataset. Rerun the method,
        using a loop, to continue to train the neural network and log the result.

        Parameters:
        ----------
        epochs : number of epochs
        """
        for i in range(epochs):
            self.n_epoch += 1

            # random indices for selecting samples for all batches in one epoch
            idx = torch.randperm(self.Z.shape[0])
            idx_ref = torch.randperm(self.Z_ref.shape[0])

            for j in range(self.n_iters_per_epoch):
                self.n_iter += 1
                self.optimizer.zero_grad()

                # obtain a random batch of samples
                batch_Z = self.Z[idx[i : self.Z.shape[0] : self.n_iters_per_epoch]]
                batch_Z_ref = self.Z_ref[
                    idx_ref[i : self.Z_ref.shape[0] : self.n_iters_per_epoch]
                ]

                # define the loss as negative DV divergence lower bound
                loss = -DV(batch_Z, batch_Z_ref, self.net)
                loss.backward()  # calculate gradient
                self.optimizer.step()  # descend

                self.writer.add_scalar(
                    "Loss/train", loss.item(), global_step=self.n_iter
                )

        with torch.no_grad():
            estimate = DV(Z, Z_ref, self.net).item()
            self.writer.add_scalar("Estimate", estimate, global_step=self.n_epoch)
            return estimate
```

To use the above class to train, we first create an instance:

```python
torch.manual_seed(SEED)
net = Net().to(DEVICE)
trainer = DVTrainer(Z, Z_ref, net, n_iters_per_epoch=10)
```

Next, run `step` iteratively to train the neural network:

```python
if input("Train? [Y/n]").lower() != "n":
    for i in range(10):
        print("Divergence estimate:", trainer.step(10))
    net.plot()
```

```python
%tensorboard --logdir=runs
```

    Reusing TensorBoard on port 6006 (pid 4485), started 0:14:03 ago. (Use '!kill 4485' to kill it.)

## Clean-up

It is important to release the resources if it is no longer used. You can release the memory or GPU memory by `Kernel->Shut Down Kernel`.

To clear the logs:

```python
if input('Delete logs? [y/N]').lower() == 'y':
    !rm -rf ./runs
```

    Delete logs? [y/N] y

To kill a tensorboard instance without shutting down the notebook kernel:

```python
tb.notebook.list() # list all the running TensorBoard notebooks.
while (pid := input('pid to kill? (press enter to exit)')):
    !kill {pid}
```

    Known TensorBoard instances:
      - port 6006: logdir runs (started 0:00:37 ago; pid 9479)

    pid to kill? (press enter to exit) 9479
    pid to kill? (press enter to exit)


## Problem Sheet Question 7b

The general form of the population growth differential equation
$$ y^{'}-y+x=0, \ \ (0 \leq x \leq 1) $$
with the initial condition
$$y(0)=0$$
For h=0.2.
# Midpoint method Solution
\begin{equation}
\frac{w_{i+1}-w_i}{h}=f(x_i+\frac{h}{2},w_i+\frac{h}{2}f(x_i,w_i))
\end{equation}
Rearranging
\begin{equation}
w_{i+1}=w_i+hf(x_i+\frac{h}{2},w_i+\frac{h}{2}f(x_i,w_i))
\end{equation}
\begin{equation}
w_{i+1}=w_i+h(k_2)
\end{equation}
\begin{equation}
k_1=w_i-x_i+2
\end{equation}
\begin{equation}
k_2=w_i+\frac{h}{2}k_1-(x_i+\frac{h}{2})+2)
\end{equation}

```python
import numpy as np
import math
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

```python
def myfun_xy(x,y):
    return y-x+2

#PLOTS
def Midpoint_Question2(N,IC):

    x_start=0
    INTITIAL_CONDITION=IC
    h=0.2
    N=N+1
    x=np.zeros(N)
    w=np.zeros(N)
    k_mat=np.zeros((2,N))
    k=0
    w[0]=INTITIAL_CONDITION
    x[0]=x_start

    for k in range (0,N-1):
        k_mat[0,k]=myfun_xy(x[k],w[k])
        k_mat[1,k]=myfun_xy(x[k]+h/2,w[k]+h/2*k_mat[0,k])
        w[k+1]=w[k]+h*(k_mat[1,k])
        x[k+1]=x[k]+h

    fig = plt.figure(figsize=(10,4))
    plt.plot(x,w,'-.o',color='blue')
    plt.title('Numerical Solution h=%s'%(h))

    # --- title, explanatory text and save
    fig.suptitle(r"$y'=y-x+2$", fontsize=20)
    plt.tight_layout()
    plt.subplots_adjust(top=0.85)
    print('x')
    print(x)
    print('k1')
    print(k_mat[0,:])
    print('k2')
    print(k_mat[1,:])
    print('w')
    print(w)
```

```python
# Midpoint_Question2(N,IC)
Midpoint_Question2(5,1)
```

```python

```

```python

```

Chapter 03 - Runge Kutta/Supplementary/.ipynb_checkpoints/02_RK Mid point Example - Review Question 7b-checkpoint.ipynb


# Exponentials, Radicals, and Logs
Up to this point, all of our equations have included standard arithmetic operations, such as division, multiplication, addition, and subtraction. Many real-world calculations involve exponential values in which numbers are raised by a specific power.

## Exponentials
A simple case of using an exponential is squaring a number; in other words, multipying a number by itself. For example, 2 squared is 2 times 2, which is 4. This is written like this:

\begin{equation}2^{2} = 2 \cdot 2 = 4\end{equation}

Similarly, 2 cubed is 2 times 2 times 2 (which is of course 8):

\begin{equation}2^{3} = 2 \cdot 2 \cdot 2 = 8\end{equation}

In Python, you use the ****** operator, like this example in which **x** is assigned the value of 5 raised to the power of 3 (in other words, 5 x 5 x 5, or 5-cubed):

```python
x = 5**3
print(x)
```

    125

Multiplying a number by itself twice or three times to calculate the square or cube of a number is a common operation, but you can raise a number by any exponential power. For example, the following notation shows 4 to the power of 7 (or 4 x 4 x 4 x 4 x 4 x 4 x 4), which has the value:

\begin{equation}4^{7} = 16384 \end{equation}

In mathematical terminology, **4** is the *base*, and **7** is the *power* or *exponent* in this expression.

## Radicals (Roots)
While it's common to need to calculate the solution for a given base and exponential, sometimes you'll need to calculate one or other of the elements themselves. For example, consider the following expression:

\begin{equation}?^{2} = 9 \end{equation}

This expression is asking, given a number (9) and an exponent (2), what's the base? In other words, which number multipled by itself results in 9? This type of operation is referred to as calculating the *root*, and in this particular case it's the *square root* (the base for a specified number given the exponential **2**). In this case, the answer is 3, because 3 x 3 = 9. We show this with a **√** symbol, like this:

\begin{equation}\sqrt{9} = 3 \end{equation}

Other common roots include the *cube root* (the base for a specified number given the exponential **3**). For example, the cube root of 64 is 4 (because 4 x 4 x 4 = 64). To show that this is the cube root, we include the exponent **3** in the **√** symbol, like this:

\begin{equation}\sqrt[3]{64} = 4 \end{equation}

We can calculate any root of any non-negative number, indicating the exponent in the **√** symbol.

The **math** package in Python includes a **sqrt** function that calculates the square root of a number. To calculate other roots, you need to reverse the exponential calculation by raising the given number to the power of 1 divided by the given exponent:

```python
import math

# Calculate square root of 25
x = math.sqrt(25)
print (x)

# Calculate cube root of 64
cr = round(64 ** (1. / 3))
print(cr)
```

    5.0
    4

The code used in Python to calculate roots other than the square root reveals something about the relationship between roots and exponentials. The exponential root of a number is the same as that number raised to the power of 1 divided by the exponential. For example, consider the following statement:

\begin{equation} 8^{\frac{1}{3}} = \sqrt[3]{8} = 2 \end{equation}

Note that a number to the power of 1/3 is the same as the cube root of that number.

Based on the same arithmetic, a number to the power of 1/2 is the same as the square root of the number:

\begin{equation} 9^{\frac{1}{2}} = \sqrt{9} = 3 \end{equation}

You can see this for yourself with the following Python code:

```python
import math

print (9**0.5)
print (math.sqrt(9))
```

## Logarithms
Another consideration for exponential values is the requirement occassionally to determine the exponent for a given number and base. In other words, how many times do I need to multiply a base number by itself to get the given result. This kind of calculation is known as the *logarithm*.

For example, consider the following expression:

\begin{equation}4^{?} = 16 \end{equation}

In other words, to what power must you raise 4 to produce the result 16?

The answer to this is 2, because 4 x 4 (or 4 to the power of 2) = 16. The notation looks like this:

\begin{equation}log_{4}(16) = 2 \end{equation}

In Python, you can calculate the logarithm of a number using the **log** function in the **math** package, indicating the number and the base:

```python
import math

x = math.log(16, 4)
print(x)
```

The final thing you need to know about exponentials and logarithms is that there are some special logarithms:

The *common* logarithm of a number is its exponential for the base **10**. You'll occassionally see this written using the usual *log* notation with the base omitted:

\begin{equation}log(1000) = 3 \end{equation}

Another special logarithm is something called the *natural log*, which is a exponential of a number for base ***e***, where ***e*** is a constant with the approximate value 2.718. This number occurs naturally in a lot of scenarios, and you'll see it often as you work with data in many analytical contexts. For the time being, just be aware that the natural log is sometimes written as ***ln***:

\begin{equation}log_{e}(64) = ln(64) = 4.1589 \end{equation}

The **math.log** function in Python returns the natural log (base ***e***) when no base is specified. Note that this can be confusing, as the mathematical notation *log* with no base usually refers to the common log (base **10**). To return the common log in Python, use the **math.log10** function:

```python
import math

# Natural log of 29
print (math.log(29))

# Common log of 100
print(math.log10(100))
```

    3.367295829986474
    2.0

## Solving Equations with Exponentials
OK, so now that you have a basic understanding of exponentials, roots, and logarithms; let's take a look at some equations that involve exponential calculations.

Let's start with what might at first glance look like a complicated example, but don't worry - we'll solve it step-by-step and learn a few tricks along the way:

\begin{equation}2y = 2x^{4} ( \frac{x^{2} + 2x^{2}}{x^{3}} ) \end{equation}

First, let's deal with the fraction on the right side. The numerator of this fraction is x2 + 2x2 - so we're adding two exponential terms. When the terms you're adding (or subtracting) have the same exponential, you can simply add (or subtract) the coefficients. In this case, x2 is the same as 1x2, which when added to 2x2 gives us the result 3x2, so our equation now looks like this:

\begin{equation}2y = 2x^{4} ( \frac{3x^{2}}{x^{3}} ) \end{equation}

Now that we've condolidated the numerator, let's simplify the entire fraction by dividing the numerator by the denominator. When you divide exponential terms with the same variable, you simply divide the coefficients as you usually would and subtract the exponential of the denominator from the exponential of the numerator. In this case, we're dividing 3x2 by 1x3: The coefficient 3 divided by 1 is 3, and the exponential 2 minus 3 is -1, so the result is 3x-1, making our equation:

\begin{equation}2y = 2x^{4} ( 3x^{-1} ) \end{equation}

So now we've got rid of the fraction on the right side, let's deal with the remaining multiplication. We need to multiply 3x-1 by 2x4. Multiplication, is the opposite of division, so this time we'll multipy the coefficients and add the exponentials: 3 multiplied by 2 is 6, and -1 + 4 is 3, so the result is 6x3:

\begin{equation}2y = 6x^{3} \end{equation}

We're in the home stretch now, we just need to isolate y on the left side, and we can do that by dividing both sides by 2. Note that we're not dividing by an exponential, we simply need to divide the whole 6x3 term by two; and half of 6 times x3 is just 3 times x3:

\begin{equation}y = 3x^{3} \end{equation}

Now we have a solution that defines y in terms of x. We can use Python to plot the line created by this equation for a set of arbitrary *x* and *y* values:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 3*df['x']**3

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that the line is curved. This is symptomatic of an exponential equation: as values on one axis increase or decrease, the values on the other axis scale *exponentially* rather than *linearly*.

Let's look at an example in which x is the exponential, not the base:

\begin{equation}y = 2^{x} \end{equation}

We can still plot this as a line:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 2.0**df['x']

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that when the exponential is a negative number, Python reports the result as 0. Actually, it's a very small fractional number, but because the base is positive the exponential number will always positive. Also, note the rate at which y increases as x increases - exponential growth can be be pretty dramatic.

**So what's the practical application of this?**

Well, let's suppose you deposit $100 in a bank account that earns 5% interest per year. What would the balance of the account be in twenty years, assuming you don't deposit or withdraw any additional funds?

To work this out, you could calculate the balance for each year:

After the first year, the balance will be the initial deposit ($100) plus 5% of that amount:

\begin{equation}y_1 = 100 + (100 \cdot 0.05) \end{equation}

Another way of saying this is:

\begin{equation}y_1 = 100 \cdot 1.05 \end{equation}

At the end of year two, the balance will be the year one balance plus 5%:

\begin{equation}y_2 = 100 \cdot 1.05 \cdot 1.05 \end{equation}

Note that the interest for year two, is the interest for year one multiplied by itself - in other words, squared. So another way of saying this is:

\begin{equation}y_2 = 100 \cdot 1.05^{2} \end{equation}

It turns out, if we just use the year as the exponent, we can easily calculate the growth after twenty years like this:

\begin{equation}y_{20} = 100 \cdot 1.05^{20} \end{equation}

Let's apply this logic in Python to see how the account balance would grow over twenty years:

```python
import pandas as pd

# Create a dataframe with 20 years
df = pd.DataFrame ({'Year': range(1, 21)})

# Calculate the balance for each year based on the exponential growth from interest
df['Balance'] = 100 * (1.05**df['Year'])

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.Year, df.Balance, color="green")
plt.xlabel('Year')
plt.ylabel('Balance')
plt.show()
```

MathsToML/Module01-Equations, Graphs, and Functions/01-04-Exponentials Radicals and Logarithms.ipynb


### 적분
- 부정적분, 정적분

- 부정적분
    - 미분과 반대되는 개념, 반-미분
    - 도함수 미분되기 전의 원래 함수 찾는 과정

- 편미분의 부정적분
    - 편미분을 한 도함수에서 원래의 함수를 찾을 수 있다.
    - C(y) 가 y의 함수일 수 있다. 상수일 수도 있다.

- 다차 도함수와 다중적분
    - 변수 순서대로 적분

```python
import sympy
x = sympy.symbols('x')
f = x * sympy.exp(x) + sympy.exp(x)
f
```

$\displaystyle x e^{x} + e^{x}$

```python
sympy.integrate(f)
```

$\displaystyle x e^{x}$

```python
x, y = sympy.symbols('x y')
f = 2 * x + y
f
```

$\displaystyle 2 x + y$

```python
sympy.integrate(f, x)
```

$\displaystyle x^{2} + x y$

```python
x = sympy.symbols('x')
f = 3 * x ** 2
sympy.integrate(f)
```

$\displaystyle x^{3}$

```python
x = sympy.symbols('x')
f = 3 * x ** 2 - 6 * x + 1
sympy.integrate(f)
```

$\displaystyle x^{3} - 3 x^{2} + x$

```python
x = sympy.symbols('x')
f = 2 + 6 * x + 4 * sympy.exp(x) + 5 / x
sympy.integrate(f)
```

$\displaystyle 3 x^{2} + 2 x + 4 e^{x} + 5 \log{\left(x \right)}$

```python
x = sympy.symbols('x')
f = 2 * x / (x ** 2 - 1)
sympy.integrate(f)
```

$\displaystyle \log{\left(x^{2} - 1 \right)}$

```python
x, y = sympy.symbols('x y')
f = 1 + x * y
sympy.integrate(f, x)
```

$\displaystyle \frac{x^{2} y}{2} + x$

```python
x, y = sympy.symbols('x y')
f = x * y * sympy.exp(x ** 2 + y ** 2)
sympy.simplify(sympy.integrate(f, x))
```

$\displaystyle \frac{y e^{x^{2} + y^{2}}}{2}$

- 정적분
    - 독립변수 x가 어떤 구간 [a, b] 사이일 때 그 구간에서 함수 f(x)의 값과 수평선(x 축)이 이루는 면적을 구하는 방법

```python
from matplotlib.patches import Polygon

def f(x):
    return x ** 3 - 3 * x ** 2 + x + 6

a, b = 0, 2
x = np.linspace(a - 0.5, b + 0.5, 50)
y = f(x)

ax = plt.subplot(111)
plt.title("정적분의 예")
plt.plot(x, y, 'r', linewidth=2)
plt.ylim(bottom=0)
ix = np.linspace(a, b)
iy = f(ix)
verts = [(a, 0)] + list(zip(ix, iy)) + [(b, 0)]
poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
ax.add_patch(poly)
plt.text(0.5 * (a + b), 0.2 * (f(a) + f(b)), r"$\int_a^b f(x)dx$",
         horizontalalignment='center', fontsize=20)
plt.figtext(0.9, 0.05, '$x$')
plt.figtext(0.1, 0.9, '$y$')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.xaxis.set_ticks_position('bottom')
ax.set_xticks((a, b))
ax.set_xticklabels(('$a$', '$b$'))
ax.set_yticks([])
ax.set_xlim(-2, 4)
ax.set_ylim(0, 8)
plt.show()
```

```python
x, y = sympy.symbols('x y')
f = x ** 3 - 3 * x ** 2 + x + 6
f
```

$\displaystyle x^{3} - 3 x^{2} + x + 6$

```python
F = sympy.integrate(f)
F
```

$\displaystyle \frac{x^{4}}{4} - x^{3} + \frac{x^{2}}{2} + 6 x$

```python
(F.subs(x, 2) - F.subs(x, 0)).evalf()
```

$\displaystyle 10.0$

```python
def f(x):
    return x ** 3 - 3 * x ** 2 + x + 6

sp.integrate.quad(f, 0, 2)
```

    (10.0, 1.1102230246251565e-13)

```python
sympy.symbols('x')
f = 3 * x ** 2 - 6 * x + 1
F = sympy.integrate(f)
(F.subs(x, 1) - F.subs(x, 0))
```

$\displaystyle -1$

```python
def f(x):
    return 3 * x ** 2 - 6 * x + 1

sp.integrate.quad(f, 0, 1)
```

    (-1.0, 1.3085085171449517e-14)

```python
sympy.symbols('x')
f = 2 + 6 * x + 4 * sympy.exp(x) + (5 / x)
F = sympy.integrate(f)
(F.subs(x, 10) - F.subs(x, 1))
```

$\displaystyle - 4 e + 5 \log{\left(10 \right)} + 315 + 4 e^{10}$

```python
def f(x):
    return 2 + 6 * x + 4 * sympy.exp(x) + (5 / x)

sp.integrate.quad(f, 1, 10)
```

    (88421.50297737827, 1.5276890734473408e-06)

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
_x = np.arange(12) / 2 + 2
_y = np.arange(12) / 2
X, Y = np.meshgrid(_x, _y)
x, y = X.ravel(), Y.ravel()
z = x * x - 10 * x + y + 50
z0 = np.zeros_like(z)
ax.bar3d(x, y, z0, 0.48, 0.48, z)
ax.set_xlim(0, 10)
ax.set_ylim(-2, 10)
ax.set_zlim(0, 50)
ax.set_xlabel("x")
ax.set_ylabel("y")
plt.title("f(x, y)")
plt.show()
```

```python
def f(x, y):
    return np.exp(-x * y) / y**2

sp.integrate.dblquad(f, 1, np.inf, lambda x: 0, lambda x: np.inf)
```

    (0.4999999999999961, 1.068453874338024e-08)

```python
def f(x, y):
    return 1 + x * y

sp.integrate.dblquad(f, -1, 1, lambda x: -1, lambda x: 1)
```

    (4.0, 4.440892098500626e-14)

- 다차원 함수의 단일 정적분
    - 2차원 함수이지만 단일 정적분을 하는 경우


# Poisson Distribution - Waiting Time

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Background

> A certain call centre receives on average **5** calls per minute.

## Question A

> What is the probability of receiving **3** calls in any given minute?

\begin{equation}
\lambda = 5
\end{equation}

\begin{equation}
\begin{split}
P(X = 3) &= \frac{e^{-5} \times {5}^{3}}{3!} \\
    &= 0.1404
\end{split}
\end{equation}

Use the `dpois` function.

```R
dpois(x=3, lambda=5)
```

0.140373895814281

## Question B

> What is the probability of receiving *no* calls in **2** minutes?

\begin{equation}
\lambda = 5 \times 2 = 10
\end{equation}

\begin{equation}
\begin{split}
P(X = 0) &= \frac{e^{-10} \times {10}^{0}}{0!} \\
    &= e^{-10}
\end{split}
\end{equation}

```R
dpois(x=0, lambda=10)
```

4.53999297624849e-05

Or use the `exp` function directly.

```R
exp(-10)
```

4.53999297624849e-05

## Question C

> What is the probability of having to wait *less* than **1** minute for a call?

\begin{equation}
\begin{split}
P(X \geq 1) &= 1 - P(X = 0) \\
    &= 1 - \frac{e^{-5} \times {5}^{0}}{0!} \\
    &= 1 - e^{-5}
\end{split}
\end{equation}

```R
1 - exp(-5)
```

0.993262053000915

## Question D

> Write down an equation for the time $t$ in which you could be **90%** certain of receiving a call.

\begin{align}
1 - P(X = 0) = 0.9 \\
P(X = 0) = 0.1
\end{align}

\begin{equation}
\lambda = 5 \cdot t
\end{equation}

\begin{equation}
\begin{split}
P(X = 0) &= \frac{e^{-\lambda} \times {\lambda}^{0}}{0!} \\
    &= e^{-\lambda} \\
    &= 0.1
\end{split}
\end{equation}

Then we get

\begin{align}
\ln 0.1 = -2.3 \\
\lambda = 2.3
\end{align}

```R
log(0.1)
```

-2.30258509299405

\begin{equation}
t = \lambda \div 5 = 0.46
\end{equation}

exercises/Poisson Distribution - Waiting Time.ipynb


# Computing and plotting PDFs of discrete data

So let's investigate how to compute and plot probability distributions.

First, let's make some data according to a normal distribution. We use `numpy.random.normal` for this. The parameters are not well named. `loc` is the mean of the distribution, and `scale` is the standard deviation. We can call this function to create an arbitrary number of data points that are distributed according to that mean and std.

```python
import numpy as np
import numpy.random as random

mean = 3
std = 2

data = random.normal(loc=mean, scale=std, size=50000)
print(len(data))
print(data.mean())
print(data.std())
```

    50000
    3.00043200502
    2.01072423406

As you can see from the print statements we got 5000 points that have a mean very close to 3, and a standard deviation close to 2.

We can plot this Gaussian by using `scipy.stats.norm` to create a frozen function that we will then use to compute the pdf (probability distribution function) of the Gaussian.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.stats as stats

def plot_normal(xs, mean, std, **kwargs):
    norm = stats.norm(mean, std)
    plt.plot(xs, norm.pdf(xs), **kwargs)

xs = np.linspace(-5, 15, num=200)
plot_normal(xs, mean, std, color='k')
```

But we really want to plot the PDF of the discrete data, not the idealized function.

There are a couple of ways of doing that. First, we can take advantage of `matplotlib`'s `hist` method, which computes a histogram of a collection of data. Normally `hist` computes the number of points that fall in a bin, like so:

```python
plt.hist(data, bins=200)
plt.show()
```

that is not very useful to us - we want the PDF, not bin counts. Fortunately `hist` includes a `density` parameter which will plot the PDF for us.

```python
plt.hist(data, bins=200, normed=True)
plt.show()
```

I may not want bars, so I can specify the `histtype` as 'step' to get a line.

```python
plt.hist(data, bins=200, normed=True, histtype='step')
plt.show()
```

To be sure it is working, let's also plot the idealized Gaussian in black.

```python
plt.hist(data, bins=200, normed=True, histtype='step')
norm = stats.norm(mean, std)
plt.plot(xs, norm.pdf(xs), color='k', lw=2)
plt.show()
```

There is another way to get the approximate distribution of a set of data. There is a technique called *kernel density estimate* that uses a kernel to estimate the probability distribution of a set of data. NumPy implements it with the function `gaussian_kde`. Do not be mislead by the name - Gaussian refers to the type of kernel used in the computation. This works for any distribution, not just Gaussians. In this section we have a Gaussian distribution, but soon we will not, and this same function will work.

```python
kde = stats.gaussian_kde(data)

xs = np.linspace(-5, 15, num=200)
plt.plot(xs, kde(xs))
plt.show()
```

## Monte Carlo Simulations

We (well I) want to do this sort of thing because I want to use monte carlo simulations to compute distributions. It is easy to compute Gaussians when they pass through linear functions, but difficult to impossible to compute them analytically when passed through nonlinear functions. Techniques like particle filtering handle this by taking a large sample of points, passing them through a nonlinear function, and then computing statistics on the transformed points. Let's do that.

We will start with the linear function $f(x) = 2x + 12$ just to prove to ourselves that the code is working. I will alter the mean and std of the data we are working with to help ensure the numbers that are output are unique It is easy to be fooled, for example, if the formula multipies x by 2, the mean is 2, and the std is 2. If the output of something is 4, is that due to the multication factor, the mean, the std, or a bug? It's hard to tell.

```python
def f(x):
    return 2*x + 12

mean = 1.
std = 1.4
data = random.normal(loc=mean, scale=std, size=50000)

d_t = f(data) # transform data through f(x)

plt.hist(data, bins=200, normed=True, histtype='step')
plt.hist(d_t, bins=200, normed=True, histtype='step')

plt.ylim(0, .35)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

This is what we expected. The input is the Gaussian $\mathcal{N}(\mu=1, \sigma=1.4)$, and the function is $f(x) = 2x+1$. Therefore we expect the mean to be shifted to $f(\mu) = 2*1+12=14$. We can see from the plot and the print statement that this is what happened.

Before I go on, can you explain what happened to the standard deviation? You may have thought that the new $\sigma$ should be passed through $f(x)$ like so $2(1.4) + 12=14.81$. But that is not correct - the standard deviation is only affected by the multiplicative factor, not the shift. If you think about that for a moment you will see it makes sense. We multiply our samples by 2, so they are twice as spread out as before. Standard deviation is a measure of how spread out things are, so it should also double. It doesn't matter if we then shift that distribution 12 places, or 12 million for that matter - the spread is still twice the input data.

## Nonlinear Functions

Now that we believe in our code, lets try it with nonlinear functions.

```python
def f2(x):
    return (np.cos((1.5*x + 2.1))) * np.sin(0.3*x) - 1.6*x

d_t = f2(data)
plt.subplot(121)
plt.hist(d_t, bins=200, normed=True, histtype='step')

plt.subplot(122)
kde = stats.gaussian_kde(d_t)
xs = np.linspace(-10, 10, 200)
plt.plot(xs, kde(xs), 'k')
plot_normal(xs, d_t.mean(), d_t.std(), color='g', lw=3)
plt.show()
print('mean = {:.2f}'.format(d_t.mean()))
print('std  = {:.2f}'.format(d_t.std()))
```

Here I passed the data through the nonlinear function $f(x) = \cos(1.5x+2.1)\sin(\frac{x}{3}) - 1.6x$. That function is quite close to linear, but we can see how much it alters the pdf of the sampled data.

There is a lot of computation going on behind the scenes to transform 50,000 points and then compute their PDF. The Extended Kalman Filter (EKF) gets around this by linearizing the function at the mean and then passing the Gaussian through the linear equation. We saw above how easy it is to pass a Gaussian through a linear function. So lets try that.

We can linearize this by taking the derivative of the function at x. We can use sympy to get the derivative.

```python
import sympy
x = sympy.symbols('x')
f = sympy.cos(1.5*x+2.1) * sympy.sin(x/3) - 1.6*x
dfx = sympy.diff(f, x)
dfx
```

    -1.5*sin(x/3)*sin(1.5*x + 2.1) + cos(x/3)*cos(1.5*x + 2.1)/3 - 1.6

We can now compute the slope of the function by evaluating the derivative at the mean.

```python
m = dfx.subs(x, mean)
m
```

    -1.66528051815545

The equation of a line is $y=mx+b$, so the new standard deviation should be $~1.67$ times the input std. We can compute the new mean by passing it through the original function because the linearized function is just the slope of f(x) evaluated at the mean. The slope is a tangent that touches the function at $x$, so both will return the same result. So, let's plot this and compare it to the results from the monte carlo simulation.

```python
plt.hist(d_t, bins=200, normed=True, histtype='step')
plot_normal(xs, f2(mean), abs(float(m)*std), color='k', lw=3, label='EKF')
plot_normal(xs, d_t.mean(), d_t.std(), color='r', lw=3, label='MC')
plt.legend()
plt.show()
```

We can see from this that the estimate from the EKF (in red) is not exact, but it is not a bad approximation either.