# Python - Symbolic Mathematics (`sympy`)

```python
%matplotlib inline

import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

```python
sp.init_printing()
```

### `sympy` treats stuff fundementally different than `numpy`

```python
np.sqrt(8)
```

```python
sp.sqrt(8)
```

```python
np.pi
```

```python
sp.pi
```

#### Adding `.n()` to the end of a sympy expression will `evaluate` expression

```python
sp.pi.n()
```

```python
sp.pi.n(100)
```

#### `nsimplify()` will sort-of do the reverse

```python
sp.nsimplify(0.125)
```

```python
sp.nsimplify(4.242640687119286)
```

```python
sp.nsimplify(sp.pi, tolerance=1e-2)
```

```python
sp.nsimplify(sp.pi, tolerance=1e-5)
```

```python
sp.nsimplify(sp.pi, tolerance=1e-6)
```

### ... to $\infty$ and beyond

```python
sp.oo
```

```python
sp.oo + 3
```

```python
1e9 < sp.oo
```

### Primes

```python
list(sp.primerange(0,100))
```

```python
sp.nextprime(2018)
```

```python
sp.factorint(11192018)
```

# Symbolic

### You have to explicitly tell `SymPy` what symbols you want to use.

```python
x, y, z = sp.symbols('x y z')
a, b, c = sp.symbols('a b c')
mu, rho = sp.symbols('mu rho')
```

### Expressions are then able use these symbols

```python
my_equation = 2*x + y

my_equation
```

```python
my_equation + 3
```

```python
my_equation - x
```

```python
my_equation / x
```

```python
my_greek_equation = mu**2 / rho * (a + b)

my_greek_equation
```

### `SymPy` has all sorts of ways to manipulates symbolic equations

```python
sp.simplify(my_equation / x)
```

```python
another_equation = (x + 2) * (x - 3)

another_equation
```

```python
sp.expand(another_equation)
```

```python
long_equation = 2*y*x**3 + 12*x**2 - x + 3 - 8*x**2 + 4*x + x**3 + 5 + 2*y*x**2 + x*y

long_equation
```

```python
sp.collect(long_equation,x)
```

```python
sp.collect(long_equation,y)
```

### You can evaluate equations for specific values

```python
trig_equation = a*sp.sin(2*x + y) + b*sp.cos(x + 2*y)

trig_equation
```

```python
trig_equation.subs({a:2, b:3, x:4, y:5})
```

```python
trig_equation.subs({a:2, b:3, x:4, y:5}).n()
```

```python
sp.expand(trig_equation, trig=True)
```

```python
sp.collect(sp.expand(trig_equation, trig=True),sp.cos(x))
```

#### You can evaluate/simplify equations sybolically

```python
my_equation_xyz = sp.sqrt((x * (y - 4*x)) / (z * (y - 3*x)))

my_equation_xyz
```

```python
my_equation_x = (3 * a * y) / (9 * a - y)

my_equation_x
```

```python
my_new_xyz = my_equation_xyz.subs(x, my_equation_x)

my_new_xyz
```

```python
sp.simplify(my_new_xyz)
```

# System of equations

$$
\begin{array}{c}
9x - 2y = 5 \\
-2x + 6y = 10 \\
\end{array}
\hspace{3cm}
\left[
\begin{array}{cc}
9 & -2 \\
-2 & 6 \\
\end{array}
\right]
\left[
\begin{array}{c}
x\\
y
\end{array}
\right]
=
\left[
\begin{array}{c}
5\\
10
\end{array}
\right]
$$

```python
a_matrix = sp.Matrix([[9, -2],
                      [-2, 6]])

b_matrix = sp.Matrix([[5],
                      [10]])
```

```python
a_matrix, b_matrix
```

```python
a_matrix.inv()
```

```python
a_matrix.inv() * a_matrix
```

```python
a_matrix.inv() * b_matrix
```

# Solving equations - `solve`

```python
yet_another_equation = x**3 + x + 10

yet_another_equation
```

```python
sp.solve(yet_another_equation,x)
```

#### ... complex numbers

```python
sp.I
```

```python
a_complex_number = 2 + 3 * sp.I

a_complex_number
```

```python
sp.re(a_complex_number), sp.im(a_complex_number)
```

### ... solving more symbolically

```python
symbolic_equation = a*x**2 + b*x +c

symbolic_equation
```

```python
sp.solve(symbolic_equation, x)
```

## Calculus

```python
symbolic_equation
```

```python
sp.diff(symbolic_equation,x)
```

```python
sp.diff(symbolic_equation,x,2)
```

```python
sp.integrate(symbolic_equation,x)
```

```python
sp.integrate(symbolic_equation,(x,0,5))   # limits x = 0 to 5
```

```python
sp.integrate(symbolic_equation,(x,0,5)).subs({a:2, b:7, c:3}).n()
```

```python
trig_equation
```

```python
sp.diff(trig_equation,x)
```

```python
sp.integrate(trig_equation,x)
```

### Ordinary differential equation - `dsolve`

```python
f = sp.Function('f')
```

```python
f(x)
```

```python
sp.Derivative(f(x),x,x)
```

```python
equation_ode = sp.Derivative(f(x), x, x) + 9*f(x)

equation_ode
```

```python
sp.dsolve(equation_ode, f(x))
```

### Limits

```python
limit_equation = (1 + (1 / x)) ** x

limit_equation
```

$$\lim _{x\to 5 }\left(1+{\frac {1}{x}}\right)^{x}$$

```python
sp.limit(limit_equation, x, 5)
```

```python
sp.limit(limit_equation, x, 5).n()
```

$$\lim _{x\to \infty }\left(1+{\frac {1}{x}}\right)^{x}$$

```python
sp.limit(limit_equation, x, sp.oo)
```

```python
sp.limit(limit_equation, x, sp.oo).n()
```

### Summation

$$ \sum{\frac {x^{a}}{a!}} $$

```python
sum_equation = x**a / sp.factorial(a)

sum_equation
```

$$ \sum _{a=0}^{3}{\frac {x^{a}}{a!}} $$

```python
sp.summation(sum_equation, [a, 0, 3])
```

```python
sp.summation(sum_equation.subs({x:1}), [a, 0, 3])
```

```python
sp.summation(sum_equation.subs({x:1}), [a, 0, 3]).n()
```

$$ \sum _{a=0}^{10}{\frac {x^{a}}{a!}} $$

```python
sp.summation(sum_equation.subs({x:1}), [a, 0, 10]).n()
```

$$ \sum _{a=0}^{\infty}{\frac {x^{a}}{a!}} $$

```python
sp.summation(sum_equation, [a, 0, sp.oo])
```

## Let's do some graphing stuff ...

$$
\large y_1 = \frac{x^3}{4} - 3x
$$

```python
my_np_x = np.linspace(-2*np.pi,2*np.pi,200)
```

```python
my_np_y1 = my_np_x ** 3 / 4 - 3 * my_np_x
```

```python
fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,4)

fig.tight_layout()

ax.set_ylim(-7,7)
ax.set_xlim(-np.pi,np.pi)

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

ax.plot(my_np_x, my_np_y1, color='r', marker='None', linestyle='-', linewidth=4);
```

### Fourier Series

```python
my_sp_y1 = x ** 3 / 4 - 3 * x

my_sp_y1
```

```python
my_fourier = sp.fourier_series(my_sp_y1, (x, -sp.pi, sp.pi))

my_fourier
```

```python
my_fourier.truncate(3).n(2)
```

```python
my_np_1term = -4.1 * np.sin(my_np_x)
my_np_2term = -4.1 * np.sin(my_np_x) + 0.91 * np.sin(2*my_np_x)
my_np_3term = -4.1 * np.sin(my_np_x) + 0.91 * np.sin(2*my_np_x) - 0.47 * np.sin(3*my_np_x)
```

```python
fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,4)

fig.tight_layout()

ax.set_ylim(-7,7)
ax.set_xlim(-np.pi,np.pi)

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

ax.plot(my_np_x, my_np_y1, color='r', marker='None', linestyle='-', linewidth=8)

ax.plot(my_np_x, my_np_1term, color='b', marker='None', linestyle='--', label="1-term")
ax.plot(my_np_x, my_np_2term, color='g', marker='None', linestyle='--', label="2-term")
ax.plot(my_np_x, my_np_3term, color='k', marker='None', linestyle='--', label="3-term")

ax.legend(loc = 0);
```

### Another Function

$$
\large y_2 = 2\,\sin(5x) \ e^{-x}
$$

```python
my_np_y2 = 2 * np.sin(5 * my_np_x) * np.exp(-my_np_x)
```

```python
fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,4)

fig.tight_layout()

ax.set_ylim(-10,10)
ax.set_xlim(-np.pi,np.pi)

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

ax.plot(my_np_x, my_np_y2, color='r', marker='None', linestyle='-', linewidth=4);
```

### Taylor Expansions

```python
my_sp_y2 = 2 * sp.sin(5 * x) * sp.exp(-x)
my_sp_y2
```

```python
my_taylor = sp.series(my_sp_y2, x, x0 = 0)

my_taylor
```

```python
my_taylor.removeO()
```

```python
my_taylor.removeO().n(2)
```

## General Equation Solving - `nsolve`

$$
\large y_1 = \frac{x^3}{4} - 3x\\
\large y_2 = 2\,\sin(5x) \ e^{-x}
$$

### Where do they cross? - The graph

```python
fig,ax = plt.subplots(1,1)
fig.set_size_inches(10,4)

fig.tight_layout()

ax.set_ylim(-5,5)
ax.set_xlim(-np.pi,4)

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

ax.plot(my_np_x, my_np_y1, color='b', marker='None', linestyle='--', linewidth = 4)
ax.plot(my_np_x, my_np_y2, color='r', marker='None', linestyle='-', linewidth = 4);
```

### Where do they cross? - The `sympy` solution

```python
my_sp_y1, my_sp_y2
```

```python
my_guess = 3.3

sp.nsolve(my_sp_y1 - my_sp_y2, x, my_guess)
```

```python
all_guesses = (3.3, 0, -0.75)

for val in all_guesses:
    result = sp.nsolve(my_sp_y1 - my_sp_y2, x, val)
    print(result)
```

### Your guess has to be (somewhat) close or the solution will not converge:

```python
my_guess = -40

sp.nsolve(my_sp_y1 - my_sp_y2, x, my_guess)
```

# `SymPy` can do *so* much more. It really is magic.

## Complete documentation can be found [here](http://docs.sympy.org/latest/index.html)


# Final project
The Allen–Cahn equation (after John W. Cahn and Sam Allen) is a reaction–diffusion equation of mathematical physics which describes the process of phase separation in multi-component alloy systems, including order-disorder transitions.

The equation describes the time evolution of a scalar-valued state variable $\eta$  on a domain $\Omega=[0,1]$  during a time interval $[0,T]$, and is given (in one dimension) by:

$$
\frac{\partial \eta}{\partial t} - \varepsilon^2 \eta'' + f'(\eta) = 0, \qquad \eta'(0, t) = \eta'(1, t) = 0,\qquad\eta(x,0) = \eta_0(x)
$$

where $f$ is a double-well potential, $\eta_0$ is the initial condition, and $\varepsilon$ is the characteristic width of the phase transition.

This equation is the L2 gradient flow of the Ginzburg–Landau free energy functional, and it is closely related to the Cahn–Hilliard equation.

A typical example of double well potential is given by the following function

$$
f(\eta) = \eta^2(\eta-1)^2
$$

which has two minima in $0$ and $1$ (the two wells, where its value is zero), one local maximum in $0.5$, and it is always greater or equal than zero.

The two minima above behave like "attractors" for the phase $\eta$. Think of a solid-liquid phase transition (say water+ice) occupying the region $[0,1]$. When $\eta = 0$, then the material is liquid, while when $\eta = 1$ the material is solid (or viceversa).

Any other value for $\eta$ is *unstable*, and the equation will pull that region towards either $0$ or $1$.

Discretisation of this problem can be done by finite difference in time. For example, a fully explicity discretisation in time would lead to the following algorithm.

We split the interval $[0,T]$ in `n_steps` intervals, of dimension `dt = T/n_steps`. Given the solution at time `t[k] = k*dt`, it i possible to compute the next solution at time `t[k+1]` as

$$
\eta_{k+1} = \eta_{k} + \Delta t \varepsilon^2 \eta_k'' - \Delta t f'(\eta_k)
$$

Such a solution will not be stable. A possible remedy that improves the stability of the problem, is to treat the linear term $\Delta t \varepsilon^2 \eta_k''$ implicitly, and keep the term $-f'(\eta_k)$ explicit, that is:

$$
\eta_{k+1} - \Delta t \varepsilon^2 \eta_k'' = \eta_{k} - \Delta t f'(\eta_k)
$$

Grouping together the terms on the right hand side, this problem is identical to the one we solved in the python notebook number 9, with the exception of the constant $\Delta t \varepsilon^2$ in front the stiffness matrix.

In particular, given a set of basis functions $v_i$, representing $\eta = \eta^j v_j$ (sum is implied), we can solve the problem using finite elements by computing

$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} = \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_k)\big)
$$
where a sum is implied over $j$ on both the left hand side and the right hand side. Let us remark that while writing this last version of the equation we moved from a forward Euler scheme to a backward Euler scheme for the second spatial derivative term: that is, we used $\eta^j_{k+1}$ instead of $\eta^j_{k}$.

This results in a linear system

$$
A x = b
$$

where

$$
A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij} = \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big)
$$

and

$$
b_i = M_{ij} \big(\eta_k^j - \Delta t f'(\eta_k^j)\big)
$$

where we simplified the integration on the right hand side, by computing the integral of the interpolation of $f'(\eta)$.

## Step 1

Write a finite element solver, to solve one step of the problem above, given the solution at the previous time step, using the same techniques used in notebook number 9.

In particular:

1. Write a function that takes in input a vector representing $\eta$, an returns a vector containing $f'(\eta)$. Call this function `F`.

2. Write a function that takes in input a vector of support points of dimension `ndofs` and the degree `degree` of the polynomial basis, and returns a list of basis functions (piecewise polynomial objects of type `PPoly`) of dimension `ndofs`, representing the interpolatory spline basis of degree `degree`

3. Write a function that, given a piecewise polynomial object of type `PPoly` and a number `n_gauss_quadrature_points`, computes the vector of global_quadrature_points and global_quadrature_weights, that contains replicas of a Gauss quadrature formula with `n_gauss_quadrature_points` on each of the intervals defined by `unique(PPoly.x)`

4. Write a function that, given the basis and the quadrature points and weights, returns the two matrices $M$ and $K$

## Step 2

Solve the Allen-Cahan equation on the interval $[0,1]$, from time $t=0$ and time $t=1$, given a time step `dt`, a number of degrees of freedom `ndofs`, and a polynomial degree `k`.

1. Write a function that takes the initial value of $\eta_0$ as a function, eps, dt, ndofs, and degree, and returns a matrix of dimension `(int(T/dt), ndofs)`  containing all the coefficients $\eta_k^i$ representing the solution, and the set of basis functions used to compute the solution

2. Write a function that takes all the solutions `eta`, the basis functions, a stride number `s`, and a resolution `res`, and plots on a single plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$, computed on `res` equispaced points between zero and one

## Step 3

Solve the problem for all combinations of

1. eps = [01, .001]

2. ndofs = [16, 32, 64, 128]

3. degree = [1, 2, 3]

3. dt = [.25, .125, .0625, .03125, .015625]

with $\eta_0 = \sin(2 \pi x)+1$.

Plot the final solution at $t=1$ in all cases. What do you observe? What happens when you increase ndofs and keep dt constant?

## Step 4 (Optional)

Instead of solving the problem explicitly, solve it implicitly, by using backward euler method also for the non linear term. This requires the solution of a Nonlinear problem at every step. Use scipy and numpy methods to solve the non linear iteration.

# Project implementation
First we import the needed python libraries

```python
%pylab inline
import sympy as sym
import scipy
from scipy.interpolate import *
from scipy.integrate import *

%matplotlib inline
from matplotlib import cm
import matplotlib.pyplot
from IPython.display import Image
from IPython.display import display, clear_output
import time
```

    Populating the interactive namespace from numpy and matplotlib

*The Allen–Cahn equation $
\frac{\partial \eta}{\partial t} - \varepsilon^2 \eta'' + f'(\eta) = 0
$ is a reaction–diffusion equation which describes the process of phase separation in multi-component alloy systems, including order-disorder transitions.*

*The equation describes the time evolution of a scalar-valued state variable $\eta$  on a domain $\Omega=[0,1]$  during a time interval $[0,T]$. *

*The double well potential is given by the following function*
$$
f(\eta) = \eta^2(\eta-1)^2
$$
Before to start, let's represent graphically this function over it's interval of definition

```python
def f(eta):    # This is the double well potential function
    return eta**2*(eta-1)**2
```

To represent f over it's whole interval of definition, we split the interval $\Omega=[0,1]$ in `n_space` intervals, of dimension `dl = T/n_space`.

```python
n_space=500
Omega=linspace(0,1,n_space)
dl=1/n_space
```

```python
_ = plot(Omega,f(Omega))
_ = plt.ylabel(r"$f ( \eta )$", fontsize = 12)
_ = plt.xlabel(r"$\eta \in \Omega$", fontsize = 12)
```

As announced, $f$ "*has two minima in $0$ and $1$ (the two wells, where its value is zero), one local maximum in $0.5$, and it is always greater or equal than zero.*"

In preparation of the exercise to be done, we define and split the interval $[0,T]$ in `n_steps` intervals, of dimension `dt = T/n_steps`.

```python
T=1
n_steps=100
dt=T/n_steps
Time=linspace(0,T,n_steps)
```

## Step 1

*Write a finite element solver, to solve one step of the problem above, given the solution at the previous time step*
#### 1.1 Double potential derivative
*Write a function that takes in input a vector representing $\eta$, an returns a vector containing $f'(\eta)$. Call this function `F`.*

Given the definition of the double well potential $
f(\eta) = \eta^2(\eta-1)^2
$, it's derivative is the following function:
$$
f'(\eta) = F(\eta)= \frac{d(\eta^2)} {d \eta} \times(\eta-1)^2+ \eta^2\times\frac{d (\eta-1)^2}{d\eta}\\
=2\eta\times(\eta-1)^2 + \eta^2\times2(\eta-1)
$$

```python
def F(eta): # derivative of the double well potential function
    return 2*eta*(eta-1)**2+eta**2*2*(eta-1)
_ = plot(Omega,F(Omega))
_ = plt.ylabel(r"$F(\eta) = f'(\eta) $", fontsize = 12)
_ = plt.xlabel(r"$\eta \in \Omega$", fontsize = 12)
```

#### 1.2. Basis functions
*Write a function that takes in input a vector of support points of dimension `ndofs` and the degree `degree` of the polynomial basis, and returns a list of basis functions (piecewise polynomial objects of type `PPoly`) of dimension `ndofs`, representing the interpolatory spline basis of degree `degree`.*

```python
def compute_basis_functions(support_points, degree):
    # Insert here what was in notebook 9
    basis = []
    dbasis = []
    for i in range(len(support_points)):
        c = support_points*0 # c has same shape as support_points but is null
        c[i] = 1 # c is null everywhere except in one of the support_points
        bi = PPoly.from_spline(splrep(support_points,c,k=degree))
        basis.append(bi)# append base basis function to basis matrix
    return basis
```

*The basis functions are constructed from the spline interpolation by computing the piecewise interpolation of a function that has value one in one of the `support_points` and zero in all other support points.*

There are as many basis functions as support points, that is, `ndofs`. They define a basis for a piecewise polynomial space of dimension `ndofs`.

```python
ndofs=16
degree=2
```

Lets visualize this basis functions and use them to plot a piecewise polynomial interpolation of $f$ in $\Omega$

```python
support_points=linspace(0,1,ndofs)
BasisFunctions=compute_basis_functions(support_points,degree)
OmegaBasisMat=zeros((n_space,ndofs))
for i in range(len(support_points)):
    OmegaBasisMat[:,i]=BasisFunctions[i](Omega)

fig, ax = plt.subplots(nrows=1, ncols=2,figsize=(10,4))

ylim(-0.01,max(f(Omega)))
for i in range(ndofs):
    BasisFunc=BasisFunctions[i]
    ax[0].plot(Omega,BasisFunc(Omega),color=cm.gist_rainbow(i/ndofs)) # plot this basis function
    ax[1].plot(Omega,OmegaBasisMat[:,i:i+1].dot(f(support_points[i:i+1])),color=cm.gist_rainbow(i/ndofs)) # plot this basis function
    ax[1].scatter(support_points[i],f(support_points)[i:i+1],color=cm.gist_rainbow(i/ndofs))
    ax[1].plot(support_points[:i+1],f(support_points)[:i+1],'k',dashes=[2,2,2,2])
    clear_output()
    display(fig)
    time.sleep(2.0/float(ndofs))
clear_output()
display(fig)
time.sleep(0.5)
_ = ax[1].plot(Omega,f(Omega),'k',alpha=0.8)
clear_output()
display(fig)
clear_output()
```

.

.

.

.

.

.

.

#### 1.3. Global quadrature
*Write a function that, given a piecewise polynomial object of type `PPoly` and a number `n_gauss_quadrature_points`, computes the vector of global_quadrature_points and global_quadrature_weights, that contains replicas of a Gauss quadrature formula with `n_gauss_quadrature_points` on each of the intervals defined by `unique(PPoly.x)`*

 Gauss quadrature uses the function values evaluated at the `n_gauss_quadrature_points` and corresponding weights to approximate the integral by a weighted sum. Gauss quadrature deals with integration for $x \in [-1,1]$,  consequently we have to rescale the points and weights to work from zero to one.

```python
# Step 1.3

def compute_global_quadrature(basis, n_gauss_quadrature_points):
    # Create a Gauss quadrature formula with n_gauss_quadrature_points,
    # extract the intervals from basis (i.e., unique(basis.x)), and
    # create len(x)-1 shifted
    # and scaled Gauss quadrature formulas
    # that can be used to integrate on each interval.
    # Put all of these
    # together, and return the result

    # The intervals are stored as `x`  (with some repeated entries)
    # in the `PPoly` object. Thats why we use unique() to make sure
    # that every interval border is taken only once
    intervals = unique(basis[0].x)

    # and make sure we can integrate exactly the product
    # of two basis functions
    qp, w = numpy.polynomial.legendre.leggauss(n_gauss_quadrature_points)

    # Rescale the points and weights to work from zero to one
    qp = (qp+1)/2
    w /= 2

    # Now replicate these points and weights in all the intervals
    h = diff(intervals) # 1st order discrete difference
    Q = array([intervals[i]+h[i]*qp for i in range(len(h))]).reshape((-1,))
    W = array([w*h[i] for i in range(len(h))]).reshape((-1,))

    # return global_quadrature, global_weights
    return Q,W
```

```python
Q,W= compute_global_quadrature(BasisFunctions,degree +1)
plot(Q,W,'k',dashes=[2,2,2,2],alpha=0.3)
plot(Q,W,'xk')
xlabel("global quadrature points")
ylabel("global weights")
```

#### 1.4 Compute M and K
*Write a function that, given the basis and the quadrature points and weights, returns the two matrices $M$ and $K$ *

```python
# Step 1.4

def compute_system_matrices(basis, global_quadrature, global_weights):
    # Compute the matrices M_ij = (v_i, v_j) and K_ij = (v_i', v_j') and return them

    #compute the 1st derivative of the basis functions
    dbasis = []
    nB=len(basis)
    for i in range(nB):
        dbasis.append( basis[i].derivative(1) )

    Bq = array([basis[i](global_quadrature) for i in range(nB)]).T
    dBq = array([dbasis[i](global_quadrature) for i in range(nB)]).T
    M = einsum('qi, q, qj', Bq, global_weights, Bq)
    K = einsum('qi, q, qj', dBq, global_weights, dBq)
    # return M, Kn-th order discrete difference
    return M,K
```

```python
M,K=compute_system_matrices(BasisFunctions, Q,W)
shape(support_points)
fig,ax=plt.subplots(nrows=1,ncols=2,figsize=(10,4))
ax[0].imshow(M,cm.jet,vmin=0,vmax=0.02)
ax[0].set_title("M matrix")
ax[1].imshow(K,cm.jet,vmin=0,vmax=2)
ax[1].set_title("K matrix")
```

## Step 2

Solve the Allen-Cahan equation on the interval $[0,1]$, from time $t=0$ and time $t=1$, given a time step `dt`, a number of degrees of freedom `ndofs`, and a polynomial degree `k`.

1. Write a function that takes the initial value of $\eta_0$ as a function, eps, dt, ndofs, and degree, and returns a matrix of dimension `(int(T/dt), ndofs)`  containing all the coefficients $\eta_k^i$ representing the solution, and the set of basis functions used to compute the solution

```python
# Step 2.1

def solve_allen_cahan(eta_0_function, eps, dt, ndofs, degree):
    # put together all the above functions, loop over time, and produce
    # the result matrix eta, containing the solution at all points
    support_points=linspace(0,1,ndofs)

    basis=compute_basis_functions(support_points, degree)
    Q,W= compute_global_quadrature(basis,degree +1)
    M,K=compute_system_matrices(basis, Q,W)

    #A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij}
    A = M + dt * eps**2 * K

    ntime=int(1.0/dt)

    eta=zeros((ntime, ndofs),dtype=float)
    # store initial values in eta
    eta[0,:]=eta_0_function(support_points)

    for k in range(ntime-1):
        # b_i = M_{ij} (eta_k^j - Delta_t f'(eta_k^j) )
        b = M.dot( eta[k] - dt *F(eta[k]) )
        # A.eta_k+1 = b
        eta[k+1,:]=np.linalg.solve(A,b)

    return eta, basis
```

#### 2.2. plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$
Write a function that takes all the solutions `eta`, the basis functions, a stride number `s`, and a resolution `res`, and plots on a single plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$, computed on `res` equispaced points between zero and one

```python
# Step 2.2

def plot_solution(eta, basis, stride, resolution):
    x = linspace(0,1,resolution)
    BasisMatrix=np.zeros((resolution,len(basis)))
    for i in range(len(basis)):
        BasisMatrix[:,i]=basis[i](x)

    fig,ax=plt.subplots(figsize=(8,4))
    for k in range(0,len(eta)):#,stride):
        label="t="+str(k/len(eta))
        if( (k/len(eta))%0.1!=0):label=''
        ax.plot(x , eta[k,:].dot(BasisMatrix.T),color=cm.jet(k/len(eta)),label=label)
        ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
        clear_output()
        display(fig)
        time.sleep(2.0*1.0/float(len(eta)))
    ax.plot(x ,  eta[-1,:].dot(BasisMatrix.T),color='k',label="t=1",dashes=[2,2,2,2])
    ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    display(fig)
    clear_output()

```

## Step 3

*Solve the problem for all combinations of*

*1. eps = [01, .001]*

*2. ndofs = [16, 32, 64, 128]*

*3. degree = [1, 2, 3]*

*3. dt = [.25, .125, .0625, .03125, .015625]*

*with $\eta_0 = \sin(2 \pi x)+1$.*

*Plot the final solution at $t=1$ in all cases. What do you observe? What happens when you increase ndofs and keep dt constant? *

```python
def eta_0(x):
    return sin(2*pi*x)+1
```

```python
eps=.001
degree=2
ndofs=64
dt=0.05
eta,basis=solve_allen_cahan(eta_0, eps, dt, ndofs, degree)
stride=4
plot_solution(eta,basis,stride,200)
plt.title("Degree="+str(degree)+" ndofs="+str(ndofs)+" Small eps = "+str(eps)+" and small dt= "+str(dt))
```

.

.

.

.

.

.

```python
eps=.01
degree=3
ndofs=128
dt=0.25
eta,basis=solve_allen_cahan(eta_0, eps, dt, ndofs, degree)
stride=1
plot_solution(eta,basis,stride,ndofs)
plt.title("Degree="+str(degree)+" ndofs="+str(ndofs)+" Larger eps = "+str(eps)+" and Larger dt = "+str(dt))
```

.

.

.

.

.

.

The plots below show the influence of the different parameters within the ranges requested. We can see the convergence of the approximated solution increases with the increase of the number of degrees of freedom and the reduction of the timestep.

More precisely :

- the **degree** is represented by the different versions of dashes on the plot below:
 - degree **1** corrisponds to the curves with small hashes,
 - degree **2** to the dong dashes,
 - degree **3** to the continuous lines.

 The continuous and dashed lines are almost always superposed, indicating that the degrees used are almost not influent on the result at t=1. A very light effect can be visible only for the smallest numbers of degree of freedom **ndofs** or for large timesteps **dt**.

- the largest is the characteristic width of the phase transition **eps**, the smaller is the **dt** required to observe convergence of the approximated solution towards the exact solution .
In facts for the smallest values of the phase transition $\varepsilon$, the convergence of the problem modeled in this exercise requires $dt \le 0.0625$, while for the largest values of $\varepsilon$ a timestep $dt\le 0.125$ is almost sufficient. Choosing too large **dt** the solution experiences oscillations.

- increasing the number of degrees of freedom **ndofs** while keeping **dt** constant leads to smoother functions, removing some "local" variations of the solution respecto to the converged solution (specially for the smallest values of $\varepsilon$ that induce a sharper solution). But this doesn't change the global shape of the result, except for very large **dt** but for the largest **dt** the approximation is anyway far to be good.

```python
def plot_t1(eta, basis, stride, resolution,ax,color,dashes):
    x = linspace(0,1,resolution)
    BasisMatrix=np.zeros((resolution,len(basis)))
    for i in range(len(basis)):
        BasisMatrix[:,i]=basis[i](x)
    if(dashes!=[1,0,1,0]):label=''
    ax.plot(x , eta[-1,:].dot(BasisMatrix.T),color=color,dashes=dashes)
    #ax.legend(loc='lower right')
    #ax.set_title(title)
    ax.set_ylim(-1,1.5)

epslist=[0.1,0.04,0.01,.001]
deglist=[1,2,3]
ndofslist=[16, 32, 64, 128]
dtlist=[.25, .125, .0625, .03125, .015625]
fig,ax=plt.subplots(nrows=5,ncols=4,figsize=(20,20))
for col,ndofs in enumerate(ndofslist):
    for row,dt in enumerate(dtlist):
        ax[row,col].text(.5,.9," nodfs="+str(ndofs)+" dt="+str(dt),
            horizontalalignment='center',
            transform=ax[row,col].transAxes,fontsize=14)
        for eps,color in zip(epslist,cm.jet([0.1,.30,0.45,0.9])):
            ax[row,col].plot(0,0,color=color,label=" eps="+str(eps))
            ax[row,col].legend(loc='lower right')
            for degree,dash in zip(deglist,([3,3,3,3],[6,4,6,4],[1,0,1,0])):
                eta,basis=solve_allen_cahan(eta_0, eps, dt, ndofs, degree)
                plot_t1(eta,basis,stride,200,ax[row,col],color,dash)
```


# Finding Roots of Equations

## Calculus review

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy as scipy
from scipy.interpolate import interp1d
```

Let's review the theory of optimization for multivariate functions. Recall that in the single-variable case, extreme values (local extrema) occur at points where the first derivative is zero, however, the vanishing of the first derivative is not a sufficient condition for a local max or min.  Generally, we apply the second derivative test to determine whether a candidate point is a max or min (sometimes it fails - if the second derivative either does not exist or is zero).  In the multivariate case, the first and second derivatives are *matrices*.  In the case of a scalar-valued function on $\mathbb{R}^n$, the first derivative is an $n\times 1$ vector called the *gradient* (denoted $\nabla f$). The second derivative is an $n\times n$ matrix called the *Hessian* (denoted $H$)

Just to remind you, the gradient and Hessian are given by:

$$\nabla f(x) = \left(\begin{matrix}\frac{\partial f}{\partial x_1}\\ \vdots \\\frac{\partial f}{\partial x_n}\end{matrix}\right)$$

$$H = \left(\begin{matrix}
  \dfrac{\partial^2 f}{\partial x_1^2} & \dfrac{\partial^2 f}{\partial x_1\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_1\,\partial x_n} \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_2\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_2^2} & \cdots & \dfrac{\partial^2 f}{\partial x_2\,\partial x_n} \\[2.2ex]
  \vdots & \vdots & \ddots & \vdots \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_n\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_n\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_n^2}
\end{matrix}\right)$$

One of the first things to note about the Hessian - it's symmetric. This structure leads to some useful properties in terms of interpreting critical points.

The multivariate analog of the test for a local max or min turns out to be a statement about the gradient and the Hessian matrix.  Specifically, a function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ has a critical point at $x$ if $\nabla f(x) = 0$ (where zero is the zero vector!).  Furthermore, the second derivative test at a critical point is as follows:

* If $H(x)$ is positive-definite ($\iff$ it has all positive eigenvalues), $f$ has a local minimum at $x$
* If $H(x)$ is negative-definite ($\iff$ it has all negative eigenvalues), $f$ has a local maximum at $x$
* If $H(x)$ has both positive and negative eigenvalues, $f$ has a saddle point at $x$.

If you have $m$ equations with $n$ variables, then the $m \times n$ matrix of first partial derivatives is known as the Jacobian $J(x)$. For example, for two equations $f(x, y)$ and $g(x, y)$, we have

$$
J(x) = \begin{bmatrix}
\frac{\delta f}{\delta x} & \frac{\delta f}{\delta y} \\
\frac{\delta g}{\delta x} & \frac{\delta g}{\delta y}
\end{bmatrix}
$$

We can now express the multivariate form of Taylor polynomials in a familiar format.

$$
f(x + \delta x) = f(x) + \delta x \cdot J(x) + \frac{1}{2} \delta x^T H(x) \delta x + \mathcal{O}(\delta x^3)
$$

## Main Issues in Root Finding in One Dimension

* Separating close roots
* Numerical Stability
* Rate of Convergence
* Continuity and Differentiability

## Bisection Method

The bisection method is one of the simplest methods for finding zeros of a non-linear function.  It is guaranteed to find a root - but it can be slow. The main idea comes from the intermediate value theorem:  If $f(a)$ and $f(b)$ have different signs and $f$ is continuous, then $f$ must have a zero between $a$ and $b$.  We evaluate the function at the midpoint, $c = \frac12(a+b)$. $f(c)$ is either zero, has the same sign as $f(a)$ or the same sign as $f(b)$.  Suppose $f(c)$ has the same sign as $f(a)$ (as pictured below).  We then repeat the process on the interval $[c,b]$.

```python
def f(x):
    return x**3 + 4*x**2 -3

x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

a = -3.0
b = -0.5
c = 0.5*(a+b)

plt.text(a,-1,"a")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([a,b,c], [f(a), f(b),f(c)], s=50, facecolors='none')
plt.scatter([a,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

```python
x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

d = 0.5*(b+c)

plt.text(d,-1,"d")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([d,b,c], [f(d), f(b),f(c)], s=50, facecolors='none')
plt.scatter([d,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

We can terminate the process whenever the function evaluated at the new midpoint is 'close enough' to zero. This method is an example of what are known as 'bracketed methods'. This means the root is 'bracketed' by the end-points (it is somewhere in between). Another class of methods are 'open methods' - the root need not be somewhere in between the end-points (but it usually needs to be close!)

## Secant Method

The secant method also begins with two initial points, but without the constraint that the function values are of opposite signs.  We use the secant line to extrapolate the next candidate point.

```python
def f(x):
    return (x**3-2*x+7)/(x**4+2)

x = np.arange(-3,5, 0.1);
y = f(x)

p1=plt.plot(x, y)
plt.xlim(-3, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
t = np.arange(-10, 5., 0.1)

x0=-1.2
x1=-0.5
xvals = []
xvals.append(x0)
xvals.append(x1)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--']
while (notconverge==1 and count <  3):
    slope=(f(xvals[count+1])-f(xvals[count]))/(xvals[count+1]-xvals[count])
    intercept=-slope*xvals[count+1]+f(xvals[count+1])
    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(f(nextval)) < 0.001:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

plt.show()
```

The secant method has the advantage of fast convergence.  While the bisection method has a linear convergence rate (i.e. error goes to zero at the rate that $h(x) = x$ goes to zero, the secant method has a convergence rate that is faster than linear, but not quite quadratic (i.e. $\sim x^\alpha$, where $\alpha = \frac{1+\sqrt{5}}2 \approx 1.6$) however, the trade-off is that the secant method is not guaranteed to find a root in the brackets.

A variant of the secant method is known as the **method of false positions**. Conceptually it is identical to the secant method, except that instead of always using the last two values of $x$ for linear interpolation, it chooses the two most recent values that maintain the bracket property (i.e $f(a) f(b) < 0$). It is slower than the secant, but like the bisection, is safe.

## Newton-Raphson Method

We want to find the value $\theta$ so that some (differentiable) function $g(\theta)=0$.
Idea: start with a guess, $\theta_0$.  Let $\tilde{\theta}$ denote the value of $\theta$ for which $g(\theta) = 0$ and define $h = \tilde{\theta} - \theta_0$.  Then:

$$
\begin{eqnarray*}
g(\tilde{\theta}) &=& 0 \\\\
&=&g(\theta_0 + h) \\\\
&\approx& g(\theta_0) + hg'(\theta_0)
\end{eqnarray*}
$$

This implies that

$$ h\approx \frac{g(\theta_0)}{g'(\theta_0)}$$

So that

$$\tilde{\theta}\approx \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

Thus, we set our next approximation:

$$\theta_1 = \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

and we have developed an iterative procedure with:

$$\theta_n = \theta_{n-1} - \frac{g(\theta_{n-1})}{g'(\theta_{n-1})}$$

#### Example

Let $$g(x) = \frac{x^3-2x+7}{x^4+2}$$

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Example Function')
plt.show()
```

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Good Guess')
t = np.arange(-5, 5., 0.1)

x0=-1.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

```

From the graph, we see the zero is near -2.  We make an initial guess of $$x=-1.5$$

We have made an excellent choice for our first guess, and we can see rapid convergence!

```python
funval
```

    0.007591996330867034

In fact, the Newton-Raphson method converges quadratically.  However, NR (and the secant method) have a fatal flaw:

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Bad Guess')
t = np.arange(-5, 5., 0.1)

x0=-0.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge = 0
    else:
        xvals.append(nextval)
    count = count+1
```

We have stumbled on the horizontal asymptote.  The algorithm fails to converge.

### Convergence Rate

The following is a derivation of the convergence rate of the NR method:

Suppose $x_k \; \rightarrow \; x^*$ and $g'(x^*) \neq 0$.  Then we may write:

$$x_k = x^* + \epsilon_k$$.

Now expand $g$ at $x^*$:

$$g(x_k) = g(x^*) + g'(x^*)\epsilon_k + \frac12 g''(x^*)\epsilon_k^2 + ...$$
$$g'(x_k)=g'(x^*) + g''(x^*)\epsilon_k$$

We have that

\begin{eqnarray}
\epsilon_{k+1} &=& \epsilon_k + \left(x_{k-1}-x_k\right)\\
&=& \epsilon_k -\frac{g(x_k)}{g'(x_k)}\\
&\approx & \frac{g'(x^*)\epsilon_k + \frac12g''(x^*)\epsilon_k^2}{g'(x^*)+g''(x^*)\epsilon_k}\\
&\approx & \frac{g''(x^*)}{2g'(x^*)}\epsilon_k^2
\end{eqnarray}

## Gauss-Newton

For 1D, the Newton method is
$$
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
$$

We can generalize to $k$ dimensions by
$$
x_{n+1} = x_n - J^{-1} f(x_n)
$$
where $x$ and $f(x)$ are now vectors, and $J^{-1}$ is the inverse Jacobian matrix. In general, the Jacobian is not a square matrix, and we use the generalized inverse $(J^TJ)^{-1}J^T$ instead, giving
$$
x_{n+1} = x_n - (J^TJ)^{-1}J^T f(x_n)
$$

In multivariate nonlinear estimation problems, we can find the vector of parameters $\beta$ by minimizing the residuals $r(\beta)$,
$$
\beta_{n+1} = \beta_n - (J^TJ)^{-1}J^T r(\beta_n)
$$
where the entries of the Jacobian matrix $J$ are
$$
J_{ij} = \frac{\partial r_i(\beta)}{\partial \beta_j}
$$

## Inverse Quadratic Interpolation

Inverse quadratic interpolation is a type of polynomial interpolation.  Polynomial interpolation simply means we find the polynomial of least degree that fits a set of points.  In quadratic interpolation, we use three points, and find the quadratic polynomial that passes through those three points.

```python

def f(x):
    return (x - 2) * x * (x + 2)**2

x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,-1.0])
y0=f(x0)
f2 = interp1d(x0, y0,kind='quadratic')

#Plot parabola
xs = np.linspace(-3, -1, num=10000, endpoint=True)
plt.plot(xs, f2(xs))

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');

#New x value
xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]

plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='black');

#New triplet
x1=np.append([-3,-2.5],xnew)
y1=f(x1)
f2 = interp1d(x1, y1,kind='quadratic')

#New Parabola
xs = np.linspace(min(x1), max(x1), num=100, endpoint=True)
plt.plot(xs, f2(xs))

xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]
plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='green');

```

So that's the idea behind quadratic interpolation. Use a quadratic approximation, find the zero of interest, use that as a new point for the next quadratic approximation.

Inverse quadratic interpolation means we do quadratic interpolation on the *inverse function*.  So, if we are looking for a root of $f$, we approximate $f^{-1}(x)$ using quadratic interpolation. This just means fitting $x$ as a function of $y$, so that the quadratic is turned on its side and we are guaranteed that it cuts the x-axis somewhere. Note that the secant method can be viewed as a *linear* interpolation on the inverse of $f$. We can write:

$$f^{-1}(y) = \frac{(y-f(x_n))(y-f(x_{n-1}))}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{(y-f(x_n))(y-f(x_{n-2}))}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{(y-f(x_{n-2}))(y-f(x_{n-1}))}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n-1}$$

We use the above formula to find the next guess $x_{n+1}$ for a zero of $f$ (so $y=0$):

$$x_{n+1} = \frac{f(x_n)f(x_{n-1})}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{f(x_n)f(x_{n-2})}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{f(x_{n-2})f(x_{n-1})}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n}$$

We aren't so much interested in deriving this as we are understanding the procedure:

```python
x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Inverse Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,1])
y0=f(x0)
f2 = interp1d(y0, x0,kind='quadratic')

#Plot parabola
xs = np.linspace(min(f(x0)), max(f(x0)), num=10000, endpoint=True)
plt.plot(f2(xs), xs)

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');
```

Convergence rate is approximately $1.8$. The advantage of the inverse method is that we will *always* have a real root (the parabola will always cross the x-axis). A serious disadvantage is that the initial points must be very close to the root or the method may not converge.

That is why it is usually used in conjunction with other methods.

## Brentq Method

Brent's method is a combination of bisection, secant and inverse quadratic interpolation.  Like bisection, it is a 'bracketed' method (starts with points $(a,b)$ such that $f(a)f(b)<0$.

Roughly speaking, the method begins by using the secant method to obtain a third point $c$, then uses inverse quadratic interpolation to generate the next possible root. Without going into too much detail, the algorithm attempts to assess when interpolation will go awry, and if so, performs a bisection step. Also, it has certain criteria to reject an iterate. If that happens, the next step will be linear interpolation (secant method).

To find zeros, use

```python
x = np.arange(-5,5, 0.1);
p1=plt.plot(x, f(x))
plt.xlim(-4, 4)
plt.ylim(-10, 20)
plt.xlabel('x')
plt.axhline(0)
pass
```

```python
scipy.optimize.brentq(f,-1,.5)
```

    -7.864845203343107e-19

```python
scipy.optimize.brentq(f,.5,3)
```

    2.0

## Roots of polynomials

One method for finding roots of polynomials converts the problem into an eigenvalue one by using the **companion matrix** of a polynomial. For a polynomial

$$
p(x) = a_0 + a_1x + a_2 x^2 + \ldots + a_m x^m
$$

the companion matrix is

$$
A = \begin{bmatrix}
-a_{m-1}/a_m & -a_{m-2}/a_m & \ldots & -a_0/a_m \\
1 & 0 & \ldots & 0 \\
0 & 1 & \ldots & 0 \\
\vdots & \vdots & \ldots & \vdots \\
0 & 0 & \ldots & 0
\end{bmatrix}
$$

The characteristic polynomial of the companion matrix is $\lvert \lambda I - A \rvert$ which expands to

$$
a_0 + a_1 \lambda + a_2 \lambda^2 + \ldots + a_m \lambda^m
$$

In other words, the roots we are seeking are the eigenvalues of the companion matrix.

For example, to find the cube roots of unity, we solve $x^3 - 1 = 0$. The `roots` function uses the companion matrix method to find roots of polynomials.

```python
# Coefficients of $x^3, x^2, x^1, x^0$

poly = np.array([1, 0, 0, -1])
```

```python
x = np.roots(poly)
x
```

    array([-0.5+0.8660254j, -0.5-0.8660254j,  1. +0.       j])

```python
plt.scatter([z.real for z in x], [z.imag for z in x])
theta = np.linspace(0, 2*np.pi, 100)
u = np.cos(theta)
v = np.sin(theta)
plt.plot(u, v, ':')
plt.axis('square')
pass
```

## Using `scipy.optimize`

### Finding roots of univariate equations

```python
def f(x):
    return x**3-3*x+1
```

```python
x = np.linspace(-3,3,100)
plt.axhline(0, c='red')
plt.plot(x, f(x))
pass
```

```python
from scipy.optimize import brentq, newton
```

#### `brentq` is the recommended method

```python
brentq(f, -3, 0), brentq(f, 0, 1), brentq(f, 1,3)
```

    (-1.8793852415718166, 0.3472963553337031, 1.532088886237956)

#### Secant method

```python
newton(f, -3), newton(f, 0), newton(f, 3)
```

    (-1.8793852415718169, 0.34729635533385395, 1.5320888862379578)

#### Newton-Raphson method

```python
fprime = lambda x: 3*x**2 - 3
newton(f, -3, fprime), newton(f, 0, fprime), newton(f, 3, fprime)
```

    (-1.8793852415718166, 0.34729635533386066, 1.532088886237956)

### Finding fixed points

Finding the fixed points of a function $g(x) = x$ is the same as finding the roots of $g(x) - x$. However, specialized algorithms also exist - e.g. using `scipy.optimize.fixedpoint`.

```python
from scipy.optimize import fixed_point
```

```python
x = np.linspace(-3,3,100)
plt.plot(x, f(x), color='red')
plt.plot(x, x)
pass
```

```python
fixed_point(f, 0), fixed_point(f, -3), fixed_point(f, 3)
```

    (array(0.25410169), array(-2.11490754), array(1.86080585))

### Mutlivariate roots and fixed points

Use `root` to solve polynomial equations. Use `fsolve` for non-polynomial equations.

```python
from scipy.optimize import root, fsolve
```

Suppose we want to solve a sysetm of $m$ equations with $n$ unknowns

\begin{align}
f(x_0, x_1) &= x_1 - 3x_0(x_0+1)(x_0-1) \\
g(x_0, x_1) &= 0.25 x_0^2 + x_1^2 - 1
\end{align}

Note that the equations are non-linear and there can be multiple solutions. These can be interpreted as fixed points of a system of differential equations.

```python
def f(x):
    return [x[1] - 3*x[0]*(x[0]+1)*(x[0]-1),
            .25*x[0]**2 + x[1]**2 - 1]
```

```python
sol = root(f, (0.5, 0.5))
sol.x
```

    array([1.11694147, 0.82952422])

```python
fsolve(f, (0.5, 0.5))
```

    array([1.11694147, 0.82952422])

```python
r0 = root(f,[1,1])
r1 = root(f,[0,1])
r2 = root(f,[-1,1.1])
r3 = root(f,[-1,-1])
r4 = root(f,[2,-0.5])

roots = np.c_[r0.x, r1.x, r2.x, r3.x, r4.x]
```

```python
Y, X = np.mgrid[-3:3:100j, -3:3:100j]
U = Y - 3*X*(X + 1)*(X-1)
V = .25*X**2 + Y**2 - 1

plt.streamplot(X, Y, U, V, color=U, linewidth=2, cmap=plt.cm.autumn)
plt.scatter(roots[0], roots[1], s=50, c='none', edgecolors='k', linewidth=2)
pass
```

#### We can also give the Jacobian

```python
def jac(x):
    return [[-6*x[0], 1], [0.5*x[0], 2*x[1]]]
```

```python
sol = root(f, (0.5, 0.5), jac=jac)
sol.x, sol.fun
```

    (array([1.11694147, 0.82952422]), array([-4.23383550e-12, -3.31612515e-12]))

#### Check that values found are really roots

```python
np.allclose(f(sol.x), 0)
```

    True

#### Starting from other initial conditions, different roots may be found

```python
sol = root(f, (12,12))
sol.x
```

    array([ 0.77801314, -0.92123498])

```python
np.allclose(f(sol.x), 0)
```

    True

```python

```


```python
from sympy import *
init_printing()
x=symbols("x")
a=Integral(cos(x)*exp(x),x)
a
```

```python
x,y=symbols("x y")

expr=x+2*y
expr
```

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
expr+1
```

```python
expr-x
```

```python
x*expr
```

```python
expanded_expr=expand(_)
expanded_expr
```

```python
factor_expr=factor(_)
factor_expr
```

```python
x,t,z,nu=symbols("x t z nu")
```

```python
diff(cos(x**2)/x,x)
```

```python
print(_)
```

    -2*sin(x**2) - cos(x**2)/x**2

```python
Derivative(cos(x**2)/x,x)
```

```python
integrate(exp(-x**2),(x,-oo,+oo))
```

```python
limit((cos(x+t)-cos(x))/t,t,0)
```

```python
abs(sin(x))/sin(x)
```

```python
limit(_,x,0)
```

```python
limit(abs(sin(x))/sin(x),x,0,"+")
```

```python
limit(abs(sin(x))/sin(x),x,0,"-")
```

```python
Integral(cos(z)*sin(nu),z,nu,t)
```

```python
r,phi,theta,R=symbols("r phi theta R")
```

```python
Integral(r**2*sin(theta),(theta,0,pi),(phi,0,2*pi),(r,0,R))
```

```python
Limit(exp(cos(t**2)),t,5,"+")
```

```python
solve(x**2-2,x)
```

```python
Eq(x**2-2,0)
```

```python
solve(_,x)
```

### Ecuacion diferencial!!!

$y''-y=e^t$

```python
y=Function("y")
y(t)
n=symbols("n")
```

```python
Derivative(y(t),t)
```

```python
Derivative(y(t),(t,n))
```

```python
Eq(Derivative(y(t),(t,2))-y(t),exp(t))
```

```python
dsolve(_,y(t))
```

```python
Matrix([[0,-I],[I,0]])
```

$\displaystyle \left[\begin{matrix}0 & - i\\i & 0\end{matrix}\right]$

```python
_.eigenvals()
```

```python
Eq(4*t*y(t).diff(t,t)+y(t).diff(t)-y(t),0)
```

```python
dsolve(_)
```

```python
print(latex(_))
```

    y{\left(t \right)} = t^{\frac{3}{8}} \left(C_{1} J_{\frac{3}{4}}\left(i \sqrt{t}\right) + C_{2} Y_{\frac{3}{4}}\left(i \sqrt{t}\right)\right)


# Rate of Change
Functions are often visualized as a line on a graph, and this line shows how the value returned by the function changes based on changes in the input value.

## Linear Rate of Change

For example, imagine a function that returns the number of meters travelled by a cyclist based on the number of seconds that the cyclist has been cycling.

Here is such a function:

\begin{equation}q(x) = 2x + 1\end{equation}

We can plot the output for this function for a period of 10 seconds like this:

```python
%matplotlib inline

def q(x):
    return 2*x + 1

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.xticks(range(0,11, 1))
plt.yticks(range(0, 22, 1))
plt.grid()

# Plot x against q(x)
plt.plot(x,q(x), color='green')

plt.show()
```

It's clear from the graph that ***q*** is a *linear* function that describes a slope in which distance increases at a constant rate over time. In other words, the cyclist is travelling at a constant speed.

But what speed?

Speed, or more technically, velocity is a measure of change - it measures how the distance travelled changes over time (which is why we typically express it as a unit of distance per a unit of time, like *miles-per-hour* or *meters-per-second*). So we're looking for a way to measure the change in the line created by the function.

The change in values along the line define its *slope*, which we know from a previous lesson is represented like this:

\begin{equation}m = \frac{\Delta{y}}{\Delta{x}} \end{equation}

We can calculate the slope of our function like this:

\begin{equation}m = \frac{q(x)_{2} - q(x)_{1}}{x_{2} - x_{1}} \end{equation}

So we just need two ordered pairs of ***x*** and ***q(x)*** values from our line to apply this equation.

- After 1 second, ***x*** is 1 and ***q***(1) = **3**.
- After 10 seconds, ***x*** is 10 and ***q***(10) = 21.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{21 - 3}{10 - 1} \end{equation}

This is the same as:

\begin{equation}m = \frac{18}{9} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{2}{1} \end{equation}

So our rate of change is 2/1 or put another way, the cyclist is travelling at 2 meters-per-second.

## Average Rate of Change
OK, let's look at another function that calculates distance travelled for a given number of seconds:

\begin{equation}r(x) = x^{2} + x\end{equation}

Let's take a look at that using Python:

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

plt.show()
```

This time, the function is not linear. It's actually a quadratic function, and the line from 0 seconds to 10 seconds shows an exponential increase; in other words, the cyclist is *accelerating*.

Technically, acceleration itself is a measure of change in velocity over time; and velocity, as we've already discussed, is a measure of change in distance over time. So measuring accelleration is pretty complex, and requires *differential calculus*, which we're going to cover shortly. In fact, even just measuring the velocity at a single point in time requires differential calculus; but we can use algebraic methods to calculate an *average* rate of velocity for a given period shown in the graph.

First, we need to define a *secant* line that joins two points in our exponential arc to create a straight slope. For example, a secant line for the entire 10 second time span would join the following two points:

- 0, ***r***(0)
- 10, ***r***(10)

Run the following Python code to visualize this line:

```python
%matplotlib inline

def r(x):
    return (x)**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([0,10])

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.show()
```

Now, because the secant line is straight, we can apply the slope formula we used for a linear function to calculate the average velocity for the 10 second period:

- At 0 seconds, ***x*** is 0 and ***r***(0) = **0**.
- At 10 seconds, ***x*** is 10 and ***r***(10) = 110.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{110 - 0}{10 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{110}{10} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{11}{1} \end{equation}

So our rate of change is 11/1 or put another way, the cyclist is travelling at an average velocity of 11 meters-per-second over the 10-second period.

Of course, we can measure the average velocity between any two points on the exponential line. Use the following Python code to show the secant line for the period between 2 and 7 seconds, and calculate the average velocity for that period

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([2,7])

# Calculate rate of change
x1 = s[0]
x2 = s[-1]
y1 = r(x1)
y2 = r(x2)
a = (y2 - y1)/(x2 - x1)

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.annotate('Average Velocity =' + str(a) + ' m/s',((x2+x1)/2, (y2+y1)/2))

plt.show()

```

Basics of Calculus by Hiren/02-01-Rate of Change.ipynb


#### Distribución Bernoulli

La distribución Bernoulli es una **distribución de probabilidad discreta**.

Un ensayo Bernoulli se define como un experimento aleatorio con sólo dos resultados posibles: éxito o fracaso.

La probabilidad de éxito es $p$ y la probabilidad de fracaso es $1-p$

Definimos la variable aleatoria X como la función de mapea el resultado éxito al número 1, y fracaso al número 0. Entonces X tiene distribución Bernoulli con parámetro $p \in (0,1) \hspace{0.5cm} X \sim Be(p)$

La función de probabilidad es

\begin{equation}
  f_X(x)=\begin{cases}
    1-p, & \text{si $x = 0$} \\
    p, & \text{si $x = 1$} \\
    0, & \text{en otro caso}
  \end{cases}
\end{equation}

Que también puede escribirse como

\begin{equation}
  f_X(x)=\begin{cases}
    p^{x}(1-p)^{1-x}, & \text{si $x \in \{0, 1\}$} \\
    0, & \text{en otro caso}
  \end{cases}
\end{equation}

**Ejemplos**:

* X: un tratamiento médico es efectivo

* X: al lanzar una moneda sale cara

* X: al lanzar un dado sale 3

---

Vamos a ver ahora cómo generar datos con esta distibución de probabilidad.

Necesitamos un generador de números aleatorios, que expone métodos para generar números aleatorios con alguna distribución de probabilidad especificada. Construimos este generador de este modo `np.random.default_rng()`

https://docs.scipy.org/doc/numpy/reference/random/generator.html

Estas son las distribuciones de probabilidad disponibles:
https://docs.scipy.org/doc/numpy/reference/random/generator.html#distributions

Recordemos que la distribución de Bernoulli es un caso particular de la distribucón Binomial con una única repetición. Por eso vamos a generar datos con distribución empleando el método `binomial` con `n=1` https://docs.scipy.org/doc/numpy/reference/random/generated/numpy.random.Generator.binomial.html#numpy.random.Generator.binomial

```python
import numpy as np
random_generator = np.random.default_rng()
sample_size = 1000
random_bernoulli_data = random_generator.binomial(n=1, p = 0.7, size = sample_size)
```

Usamos la misma función `distribution_plotter` para graficar los datos generados

```python
import seaborn as sns
def distribution_plotter(data, label, bins=None):
    sns.set(rc={"figure.figsize": (10, 7)})
    sns.set_style("white")
    dist = sns.distplot(data, bins= bins, hist_kws={'alpha':0.2}, kde_kws={'linewidth':5})
    dist.set_title('Distribucion de ' + label + '\n', fontsize=16)
```

```python
#print(random_bernoulli_data)
distribution_plotter(random_bernoulli_data, "bernoulli", bins=[0,.1,.9,1])
```

#### Referencias

https://www.statisticshowto.datasciencecentral.com/bernoulli-distribution/

Gráficos: https://en.wikipedia.org/wiki/List_of_probability_distributions


# Google Page Rank Algorithm

In this notebook, we learn and code up a simplified version of Google's Page Rank Algorithm, which is a direct application of Eigenvectors and Eigenvalues we learnt in Linear Algebra.

Reference to the original paper: $\href{http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf}{here}$

Reference to blog:
$\href{https://www.dhruvonmath.com/2019/03/20/pagerank/}{here}$

Google Page Rank Algorithm predicts a rank for each webpage on the internet. This rank depends on the number of ingoing and outgoing links to a webpage.

You can think of rank of a webpage as follows: If a web page gets rank $1$, it means that a random web searcher who clicks links randomly would spend the most amount of time in this web page. If a web page gets rank $2$, it means that a random web searcher who keeps clicking links randomly would spend the second most amount of time in this web page and so on and so forth.

Given the structure of the web, it is very obvious to model the web as a Graph Data structure. The nodes of the graph are the web pages. Edges of the graph represent links between the web pages.

Note that the graph is a directed graph. There maybe a link from webpage 'A' to webpage 'B', it may not always be the case that there exists a link from webpage 'B' to webpage 'A'.

A graph with $N$ nodes can be represented with a $N \times N$ matrix. This matrix is known as Adjacency matrix.

$A_{ij}$ represents the weight of the edge connecting the $j^{th}$ node to the $i^{th}$ node.
We can think of this adjacency matrix as concatenation of $N$ column vectors. The $i^{th}$ column defines the edges from node $i$ to all other nodes.

**Exercise:**
Can you draw the graph corresponding to the following adjacency matrix?

\begin{equation}
A = \begin{pmatrix}
0 & 1/2 & 0 & 0\\
1/3 & 0 & 0 & 1/2\\
1/3 & 0 & 0 & 1/2\\
1/3 & 1/2 & 1 & 0\\
\end{pmatrix}
\end{equation}

**Exercise:**
Can you write down the adjacency matrix for the following graph?

Let us normalize each column of the adjacency matrix so that the entries sum upto $1$. This is because we want to output the rank as a probability value of the amount of time spent on that webpage.

We start with equiprobable ranks for all webpages.

We update the ranks($r$) as follows:
\begin{equation}
r(i) = \underset{j}{\sum} r(j) * A(i, j)
\end{equation}
\begin{equation}
r' = A.r
\end{equation}
The above is a recursive definition.

```python
import numpy as np
```

```python
# Complete the below function to return A after normalizing
# each of it's columns
# Normalizing a column means sum of entries in each column adds to 1.
# PLEASE USE VECTORISED CODE FOR EFFICIENCY
def normalize_columns(A):
    sums = A.sum(axis=0)
    return (A/sums)

```

```python
A = np.array([[0, 0, 1, 1], [1, 0, 0, 0], [1, 1, 0, 1], [1, 1, 0, 0]])
print(normalize_columns(A))
```

    [[0.         0.         1.         0.5       ]
     [0.33333333 0.         0.         0.        ]
     [0.33333333 0.5        0.         0.5       ]
     [0.33333333 0.5        0.         0.        ]]

**Expected Output:**

[[0.         0.         1.         0.5       ]
 [0.33333333 0.         0.         0.        ]
 [0.33333333 0.5        0.         0.5       ]
 [0.33333333 0.5        0.         0.        ]]

```python
# Complete the below function to take a matrix A and a vector r.
# Return the updated rank.
# PLEASE USE VECTORISED CODE WITHOUT LOOPS
def update_rank(A, r):
    r = np.dot(A,r)
    return r
```

```python
# Complete the below function to check if two vectors a and b are equal
# Since we are dealing with real numbers,
# we say two elements(x and y) are equal if abs(x - y) <= epsilon

# PLEASE USE VECTORISED CODE WITHOUT LOOPS
ep = 1e-8
def check_equality(a, b):
    val = np.abs(a - b) <= ep
    if(np.any(val[:]== False)):
        return False
    return True

```

```python
# Complete the below function to compute ranks iteratively until
# ranks stabilise.
# We say ranks become stabilised when the after updation the ranks
# do not change.
# Use the functions defined above.
def compute_iteratively(A, initial_rank):
    curr_rank = initial_rank
    prev_rank=np.zeros(initial_rank.shape[0])
    while(check_equality(curr_rank,prev_rank)!=True):
        prev_rank = curr_rank
        curr_rank = update_rank(A,curr_rank)
    return curr_rank

```

```python
# Complete the below function to compute final ranks at one shot using
# eigen values and eigen vectors
# You may use inbuilt functions to compute eigenvectors and eigenvalues
import scipy.linalg as la
def compute_using_eig(A, initial_rank):
    eigenvals, eigenvecs = la.eig(A)
    egi = eigenvals.astype(int)
    i = np.where(egi==1)
    i = i[0][0]

    return normalize_columns(eigenvecs[:,i])
```

```python
A = np.array([[0, 1, 0, 0], [1, 0, 0, 1], [1, 0, 0, 1], [1, 1, 1, 0]])
A = normalize_columns(A)
r = np.array([0.25, 0.25, 0.25, 0.25])
print("Rank computed iteratively: \n", compute_iteratively(A, r))
print("Rank computed using eigen values and eigen vectors: \n", compute_using_eig(A, r))
```

    Rank computed iteratively:
     [0.12 0.24 0.24 0.4 ]
    Rank computed using eigen values and eigen vectors:
     [0.12 0.24 0.24 0.4 ]

    /usr/lib/python3/dist-packages/ipykernel_launcher.py:7: ComplexWarning: Casting complex values to real discards the imaginary part
      import sys

**Expected Output:**

Rank computed iteratively:
 [0.12       0.24       0.24       0.39999999]

Rank computed using eigen values and eigen vectors:
 [0.12 0.24 0.24 0.4 ]

```python
A = np.array([[0, 0, 1, 1], [1, 0, 0, 0], [1, 1, 0, 1], [1, 1, 0, 0]])
A = normalize_columns(A)
r = np.array([0.25, 0.25, 0.25, 0.25])
print("Rank computed iteratively: \n", compute_iteratively(A, r))
print("Rank computed using eigen values and eigen vectors: \n", compute_using_eig(A, r))
```

    Rank computed iteratively:
     [0.38709677 0.12903226 0.29032258 0.19354839]
    Rank computed using eigen values and eigen vectors:
     [0.38709677+0.j 0.12903226+0.j 0.29032258+0.j 0.19354839+0.j]

    /usr/lib/python3/dist-packages/ipykernel_launcher.py:7: ComplexWarning: Casting complex values to real discards the imaginary part
      import sys

**Expected Output:**

Rank computed iteratively:

[0.38709677 0.12903226 0.29032258 0.19354839]

Rank computed using eigen values and eigen vectors:

[0.38709677 0.12903226 0.29032258 0.19354839]

day8/morning/Google Page Rank Algorithm Assignment/Page Rank Algorithm.ipynb


```python
from sympy import *
```

```python
def build_polynomial(matrix, iter):
    x = Symbol('x')
    if iter == -1:
        return 1
    if iter == 0:
        return poly(matrix[1][1] - x)
    return poly(
        (matrix[iter][iter] - x)*build_polynomial(matrix, iter - 1) -
        matrix[iter - 1][iter]*matrix[iter][iter - 1]*build_polynomial(matrix, iter - 2))
```

```python
def halleys_method(function, initial, precision):
    x = Symbol('x')
    iters = 0

    lfunction = lambdify(x, function, 'numpy')

    dfunction = function.diff(x)
    ldfunction = lambdify(x, dfunction, 'numpy')

    d2function = dfunction.diff(x)
    ld2function = lambdify(x, d2function, 'numpy')

    while abs(lfunction(initial)) > precision:
        t = - lfunction(initial)/ldfunction(initial)
        r = ld2function(initial)*t**2/ldfunction(initial)
        initial = initial + t**2/(t + 0.5*r)
        iters += 1

    residual = lfunction(initial)
    return {'root': initial, 'iterations': iters, 'residual': residual}
```

```python
def eigenvalues(matrix, initial, precision):
    x = Symbol('x')
    eigenvalues = {'value':[], 'residual':[]}
    n = len(matrix)
    polynomial = build_polynomial(matrix, n-1)
    for _ in range(n):
        root = halleys_method(polynomial.as_expr(), initial, precision)
        eigenvalues['value'].append(root['root'])
        eigenvalues['residual'].append(root['residual'])
        polynomial = pquo(polynomial, poly(x - root['root']))
    return eigenvalues
```

```python
matrix = [[1, 2, 0, 0],
          [3, 1, 2, 0],
          [0, 3, 1, 2],
          [0, 0, 3, 2]]
initial = 0
precision = 0.001
```

```python
# результат написанного алгоритма с невязкой для каждого значения
eigenvalues(matrix, initial, precision)
```

    {'value': [-0.1881911215025469,
      2.8998266041121847,
      -2.8518950613854894,
      5.14025957877585],
     'residual': [8.34833535634516e-09,
      1.497466683986204e-08,
      -7.894759500359783e-06,
      0.0]}

```python
import numpy
# результат встроенной функции
numpy.linalg.eigvals(numpy.array(matrix))
```

    array([-2.85189605, -0.18819112,  5.14026057,  2.89982661])


## RIHAD VARIAWA, Data Scientist - Who has fun LEARNING, EXPLORING & GROWING
## Exponentials, Radicals, and Logs
Up to this point, all of our equations have included standard arithmetic operations, such as division, multiplication, addition, and subtraction. Many real-world calculations involve exponential values in which numbers are raised by a specific power.

## Exponentials
A simple case of of using an exponential is squaring a number; in other words, multipying a number by itself. For example, 2 squared is 2 times 2, which is 4. This is written like this:

\begin{equation}2^{2} = 2 \cdot 2 = 4\end{equation}

Similarly, 2 cubed is 2 times 2 times 2 (which is of course 8):

\begin{equation}2^{3} = 2 \cdot 2 \cdot 2 = 8\end{equation}

In Python, you use the ****** operator, like this example in which **x** is assigned the value of 5 raised to the power of 3 (in other words, 5 x 5 x 5, or 5-cubed):

```python
x = 5**3
print(x)
```

Multiplying a number by itself twice or three times to calculate the square or cube of a number is a common operation, but you can raise a number by any exponential power. For example, the following notation shows 4 to the power of 7 (or 4 x 4 x 4 x 4 x 4 x 4 x 4), which has the value:

\begin{equation}4^{7} = 16384 \end{equation}

In mathematical terminology, **4** is the *base*, and **7** is the *power* or *exponent* in this expression.

## Radicals (Roots)
While it's common to need to calculate the solution for a given base and exponential, sometimes you'll need to calculate one or other of the elements themselves. For example, consider the following expression:

\begin{equation}?^{2} = 9 \end{equation}

This expression is asking, given a number (9) and an exponent (2), what's the base? In other words, which number multipled by itself results in 9? This type of operation is referred to as calculating the *root*, and in this particular case it's the *square root* (the base for a specified number given the exponential **2**). In this case, the answer is 3, because 3 x 3 = 9. We show this with a **√** symbol, like this:

\begin{equation}\sqrt{9} = 3 \end{equation}

Other common roots include the *cube root* (the base for a specified number given the exponential **3**). For example, the cube root of 64 is 4 (because 4 x 4 x 4 = 64). To show that this is the cube root, we include the exponent **3** in the **√** symbol, like this:

\begin{equation}\sqrt[3]{64} = 4 \end{equation}

We can calculate any root of any non-negative number, indicating the exponent in the **√** symbol.

The **math** package in Python includes a **sqrt** function that calculates the square root of a number. To calculate other roots, you need to reverse the exponential calculation by raising the given number to the power of 1 divided by the given exponent:

```python
import math

# Calculate square root of 25
x = math.sqrt(25)
print (x)

# Calculate cube root of 64
cr = round(64 ** (1. / 3))
print(cr)
```

The code used in Python to calculate roots other than the square root reveals something about the relationship between roots and exponentials. The exponential root of a number is the same as that number raised to the power of 1 divided by the exponential. For example, consider the following statement:

\begin{equation} 8^{\frac{1}{3}} = \sqrt[3]{8} = 2 \end{equation}

Note that a number to the power of 1/3 is the same as the cube root of that number.

Based on the same arithmetic, a number to the power of 1/2 is the same as the square root of the number:

\begin{equation} 9^{\frac{1}{2}} = \sqrt{9} = 3 \end{equation}

You can see this for yourself with the following Python code:

```python
import math

print (9**0.5)
print (math.sqrt(9))
```

## Logarithms
Another consideration for exponential values is the requirement occassionally to determine the exponent for a given number and base. In other words, how many times do I need to multiply a base number by itself to get the given result. This kind of calculation is known as the *logarithm*.

For example, consider the following expression:

\begin{equation}4^{?} = 16 \end{equation}

In other words, to what power must you raise 4 to produce the result 16?

The answer to this is 2, because 4 x 4 (or 4 to the power of 2) = 16. The notation looks like this:

\begin{equation}log_{4}(16) = 2 \end{equation}

In Python, you can calculate the logarithm of a number using the **log** function in the **math** package, indicating the number and the base:

```python
import math

x = math.log(16, 4)
print(x)
```

The final thing you need to know about exponentials and logarithms is that there are some special logarithms:

The *common* logarithm of a number is its exponential for the base **10**. You'll occassionally see this written using the usual *log* notation with the base omitted:

\begin{equation}log(1000) = 3 \end{equation}

Another special logarithm is something called the *natural log*, which is a exponential of a number for base ***e***, where ***e*** is a constant with the approximate value 2.718. This number occurs naturally in a lot of scenarios, and you'll see it often as you work with data in many analytical contexts. For the time being, just be aware that the natural log is sometimes written as ***ln***:

\begin{equation}log_{e}(64) = ln(64) = 4.1589 \end{equation}

The **math.log** function in Python returns the natural log (base ***e***) when no base is specified. Note that this can be confusing, as the mathematical notation *log* with no base usually refers to the common log (base **10**). To return the common log in Python, use the **math.log10** function:

```python
import math

# Natural log of 29
print (math.log(29))

# Common log of 100
print(math.log10(100))
```

## Solving Equations with Exponentials
OK, so now that you have a basic understanding of exponentials, roots, and logarithms; let's take a look at some equations that involve exponential calculations.

Let's start with what might at first glance look like a complicated example, but don't worry - we'll solve it step-by-step and learn a few tricks along the way:

\begin{equation}2y = 2x^{4} ( \frac{x^{2} + 2x^{2}}{x^{3}} ) \end{equation}

First, let's deal with the fraction on the right side. The numerator of this fraction is x2 + 2x2 - so we're adding two exponential terms. When the terms you're adding (or subtracting) have the same exponential, you can simply add (or subtract) the coefficients. In this case, x2 is the same as 1x2, which when added to 2x2 gives us the result 3x2, so our equation now looks like this:

\begin{equation}2y = 2x^{4} ( \frac{3x^{2}}{x^{3}} ) \end{equation}

Now that we've condolidated the numerator, let's simplify the entire fraction by dividing the numerator by the denominator. When you divide exponential terms with the same variable, you simply divide the coefficients as you usually would and subtract the exponential of the denominator from the exponential of the numerator. In this case, we're dividing 3x2 by 1x3: The coefficient 3 divided by 1 is 3, and the exponential 2 minus 3 is -1, so the result is 3x-1, making our equation:

\begin{equation}2y = 2x^{4} ( 3x^{-1} ) \end{equation}

So now we've got rid of the fraction on the right side, let's deal with the remaining multiplication. We need to multiply 3x-1 by 2x4. Multiplication, is the opposite of division, so this time we'll multipy the coefficients and add the exponentials: 3 multiplied by 2 is 6, and -1 + 4 is 3, so the result is 6x3:

\begin{equation}2y = 6x^{3} \end{equation}

We're in the home stretch now, we just need to isolate y on the left side, and we can do that by dividing both sides by 2. Note that we're not dividing by an exponential, we simply need to divide the whole 6x3 term by two; and half of 6 times x3 is just 3 times x3:

\begin{equation}y = 3x^{3} \end{equation}

Now we have a solution that defines y in terms of x. We can use Python to plot the line created by this equation for a set of arbitrary *x* and *y* values:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 3*df['x']**3

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that the line is curved. This is symptomatic of an exponential equation: as values on one axis increase or decrease, the values on the other axis scale *exponentially* rather than *linearly*.

Let's look at an example in which x is the exponential, not the base:

\begin{equation}y = 2^{x} \end{equation}

We can still plot this as a line:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 2.0**df['x']

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that when the exponential is a negative number, Python reports the result as 0. Actually, it's a very small fractional number, but because the base is positive the exponential number will always positive. Also, note the rate at which y increases as x increases - exponential growth can be be pretty dramatic.

So what's the practical application of this?

Well, let's suppose you deposit $100 in a bank account that earns 5% interest per year. What would the balance of the account be in twenty years, assuming you don't deposit or withdraw any additional funds?

To work this out, you could calculate the balance for each year:

After the first year, the balance will be the initial deposit ($100) plus 5% of that amount:

\begin{equation}y1 = 100 + (100 \cdot 0.05) \end{equation}

Another way of saying this is:

\begin{equation}y1 = 100 \cdot 1.05 \end{equation}

At the end of year two, the balance will be the year one balance plus 5%:

\begin{equation}y2 = 100 \cdot 1.05 \cdot 1.05 \end{equation}

Note that the interest for year two, is the interest for year one multiplied by itself - in other words, squared. So another way of saying this is:

\begin{equation}y2 = 100 \cdot 1.05^{2} \end{equation}

It turns out, if we just use the year as the exponent, we can easily calculate the growth after twenty years like this:

\begin{equation}y20 = 100 \cdot 1.05^{20} \end{equation}

Let's apply this logic in Python to see how the account balance would grow over twenty years:

```python
import pandas as pd

# Create a dataframe with 20 years
df = pd.DataFrame ({'Year': range(1, 21)})

# Calculate the balance for each year based on the exponential growth from interest
df['Balance'] = 100 * (1.05**df['Year'])

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.Year, df.Balance, color="green")
plt.xlabel('Year')
plt.ylabel('Balance')
plt.show()
```

AI Professional/2 - Essential Mathematics For Artificial Intelligence/DAT256x/Module01/01-04-Exponentials Radicals and Logarithms.ipynb


## Ainsley Works on Problem Sets

Ainsley sits down on Sunday night to finish S problem sets, where S is a random variable that is equally likely to be 1, 2, 3, or 4. She learns C concepts from the problem sets and drinks D energy drinks to stay awake, where C and D are random and depend on how many problem sets she does. You know that $p_{C|S}(c|s) = 1/(2s+1)$ for $c \in \{ 0,1,\ldots ,2s\}.$ For each problem set she completes, regardless of concepts learned, she independently decides to have an energy drink with probability $q.$ That is, the number of energy drinks she has is binomial with parameters $q$ and $S:$

$$\begin{eqnarray}
    p_{D\mid S}(d\mid s) &= \begin{cases} {s \choose d}\, q^d\, (1-q)^{s-d} & d \in \{0,\ldots,s\} \\
            0 & \text{otherwise} \end{cases}
\end{eqnarray}$$

where ${n \choose k} = \frac{n!}{k!\, (n-k)!}.$

**Question:** Does the conditional entropy $H(C\mid S=s)$ decrease, stay the same, or increase as $s$ increases from $1$ to $4?$

[$\times    $] It decreases.

[$\times    $] It stays the same

[$\checkmark$] It increases.

**Solution:** Conditioned on $S=s, C$ is uniform from $0$ to $2s.$

$$\begin{align} H(C|S=s)&= \sum _{c=0}^{2s} p_{C|S}(c|s) \log \frac{1}{p_{C|S}(c|s)}\\
&= \sum _{c=0}^{2s} \frac{1}{2s+1} \log \frac{1}{\frac{1}{2s+1}}\\
&=  \log \frac{1}{\frac{1}{2s+1}}\\
&=  \log (2s+1)
\end{align}$$

As $s$ increases, so does $\log (2s+1).$

We can also see this intuitively: as $s$ increases, $c$ is uniform over a broader range of possibilities, which implies a higher entropy.

```python
%matplotlib inline
from numpy import log2, arange
import matplotlib.pyplot as plt

f = lambda x: - x * log2(x)
g = lambda s: (2*s + 1) * f(1/(2*s+1))

s = arange(1, 5, 1)
plt.plot(s, g(s), '-', s, g(s), 'ro')
plt.xlabel("$s$")
plt.ylabel("$H(C\mid S=s)$")
plt.show()
```

**Question:** The next morning, her roommate notices that Ainsley drank $d$ energy drinks. What is the expected number of concepts that she learned?

You should derive a general expression for this although in the answer boxes below we only ask you to evaluate the expression for specific choices of $d$ and $q.$ If you're general expression is correct, your answers to these should also be correct.

(Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

1. When $q=0.2, \mathbb {E}[C | D = 1] =$ {{ans1}}
2. When $q=0.5, \mathbb {E}[C | D = 2] =$ {{ans2}}
3. When $q=0.7, \mathbb {E}[C | D = 3] =$ {{ans3}}

**Solution:** We are interested in $\mathbb {E}[C|D=d].$ Since we are given information about $C$ conditioned on $S,$ we will condition on $S$ and use total expectation. We will also use the fact that $C$ and $D$ are conditionally independent given $S:$

$$\begin{align}
\mathbb {E}[C|D=d] &= \sum _{s=1}^4 \mathbb {E}[C|D=d, S=s] \mathbb {P}(S=s | D=d)\\
\text {(by conditional independence)}	&= \sum _{s=1}^4 \mathbb {E}[C|S=s] p_{S|D}(s|d)\\
\text {(by Bayes' rule)} &= \sum _{s=1}^4 \mathbb {E}[C|S=s] \frac{p_{D|S}(d|s) p_ S(s)}{p_ D(d)}\\
&= \frac{\sum _{s=1}^4 \left(\sum _{c=0}^{2s} c \frac{1}{2s+1}\right) p_{D|S}(d|s) p_ S(s)}{\sum _{s=1}^4 p_{D|S}(d|s) p_ S(s)}	\\
&=\frac{\sum _{s=1}^4 s \cdot p_{D|S}(d|s) }{\sum _{s=1}^4 p_{D|S}(d|s)}\\
\text {(since $p_{D|S}(d|s) = 0$ for $s < d$)}&= \frac{\sum _{s=d}^4 s \cdot p_{D|S}(d|s) }{\sum _{s=d}^4 p_{D|S}(d|s)}	\\
&=\frac{\sum _{s=d}^4 s {s \choose d} q^ d (1-q)^{s-d}}{\sum _{s=d}^4 {s \choose d} q^ d (1-q)^{s-d}}
\end{align}$$
Another solution that works is to compute $p_{C|D}(\cdot \mid d)$ and compute the expectation with respect to this distribution. This leads to a very similar set of steps as above.

```python
from scipy.misc import comb
f = lambda s, d, q : comb(s, d) * (q ** d) * ((1 - q) ** (s - d))
ED = lambda d, q : sum([s * f(s, d, q) for s in range(d, 5)]) / \
                   sum([f(s, d, q) for s in range(d, 5)])

ans1 = "{0:.3f}".format(ED(1, 0.2))
ans2 = "{0:.3f}".format(ED(2, 0.5))
ans3 = "{0:.3f}".format(ED(3, 0.7))
```

**Question:** Is the mutual information $I(C ; D)$ greater than, less than, or equal to zero? You should assume that $q$ lies in the range $0 < q < 1.$

[$\checkmark$] Greater than 0

[$\times    $] Less than 0

[$\times    $] Equal to 0

**Solution:** $\boxed {\text {Greater than zero}}.$

Since the conditional expectation in the previous part depends on $d,$ we can infer that they are not independent. We can also justify this intuitively: for example, knowing that $D=4$ guarantees that $S=4,$ and therefore changes our belief about $C$ (i.e., $C$ is more likely to take on higher values).

## Consecutive Sixes

**Question:** On average, how many times do you have to roll a fair six-sided die before getting two consecutive sixes?

Hint: Use total expectation.

**Solution:** Let $\mu = \mathbb {E}[\# \text { rolls until we get two consecutive 6's}].$

The problem can be broken up into two events (that forms a partition of the sample space):

- Event 1: The very first time we roll a 6, the roll right afterward is also a 6.

    The probability of this first event is $1/6.$ You can think of it as we will, with probability $1,$ roll a 6 in a finite amount of time, and then it's just the next roll that we are looking at the probability for, and rolling a 6 in this next roll happens with probability $1/6.$ (Note that the probability that we never see a 6 is $\lim _{n \rightarrow \infty } (5/6)^ n = 0.$)

    Conditioned on this first event, let's compute the expected number of rolls until we get two consecutive 6's: The expected number of rolls until the first 6 is the expectation of a $\text {Geo}(1/6)$ random variable, which is $1/(1/6) = 6.$ The event we are conditioning on says that the next roll is a 6, so there the conditional expectation here is just $6 + 1 = 7$ rolls.

- Event 2: The very first time we roll a 6, the roll right afterward is not a 6.

    The probability for this second event is $5/6,$ i.e., the roll right after getting the first 6 is not a 6.

    Conditioned on this second event, let's compute the expected number of rolls until we get two consecutive 6's: The expected number of rolls until the first 6 is 6 rolls (again, this is the expectation of a $\text {Geo}(1/6)$ random variable), and then the 7th roll is not a 6. And then we restart the whole process over. So the conditional expectation for this case is $7 + \mu.$

Now using the law of total expectation,

$$\begin{align}
\mu	&= 7 \cdot \frac16 + (7 + \mu ) \frac56 \\
&= \frac76 + \frac{35}6 + \frac{5}{6}\mu\\
&= \frac{42}6 + \frac{5}{6}\mu ,
\end{align}$$

so

$$\frac16 \mu = \frac{42}6,\qquad {\text {i.e.,}}\qquad \mu = \boxed {42}.$$

```python

```