# Symbolic Computation with SymPy
Symbolic computation deals with the computation of mathematical objects symbolically. This means that the mathematical objects are represented exactly, not approximately, and mathematical expressions with unevaluated variables are left in symbolic form.

```python
from sympy import *
```

Here is a brief example, if you wanted to compute a square root in python you would do:

```python
import math
math.sqrt(8)
```

    2.8284271247461903

This is an approximate solution because $\sqrt{8}$ is an irrational number. But if you were to use the sympy version of of the square root function:

```python
sqrt(8)
```

$\displaystyle 2 \sqrt{2}$

Notice that this is an exact solution, where the irrational part is left unevaluated by default.

Let's move to a more interesting examples

---
## The power of symbolic computation

```python
x, y = symbols('x y')
type(x)
```

    sympy.core.symbol.Symbol

```python
y
```

$\displaystyle y$

Now the code understands that `x` and `y` are symbols, not variables. These symbols can be used to build expressions:

```python
expr = x + 2*y
expr
```

$\displaystyle x + 2 y$

```python
expr + 1
```

$\displaystyle x + 2 y + 1$

```python
(x**2)*expr
```

$\displaystyle x^{2} \left(x + 2 y\right)$

Notice that in this last expression, the $x^2$ term was not distributed, but that can be requested.

```python
expand((x**2)*expr)
```

$\displaystyle x^{3} + 2 x^{2} y$

The reverse operation of `expand` is `factor`:

```python
factor(x**2 + x)
```

$\displaystyle x \left(x + 1\right)$

Here are a few more interesting examples:

```python
sp.diff(x*sin(x), x)
```

$\displaystyle x \cos{\left(x \right)} + \sin{\left(x \right)}$

```python
sp.integrate(x*cos(x) + sin(x), x)
```

$\displaystyle x \sin{\left(x \right)}$

```python
sp.integrate(sin(x**2), (x, -oo, oo))
```

$\displaystyle \frac{\sqrt{2} \sqrt{\pi}}{2}$

```python
sp.limit(sin(x)/x, x, 0)
```

$\displaystyle 1$

```python
sp.solve(x**2-4, x)
```

    [-2, 2]

```python
t = symbols('t')
y = Function('y')
ode = dsolve(Eq(y(t).diff(t, t) - y(t), exp(t)), y(t))
ode
```

$\displaystyle y{\left(t \right)} = C_{2} e^{- t} + \left(C_{1} + \frac{t}{2}\right) e^{t}$

```python
latex(ode)
```

    'y{\\left(t \\right)} = C_{2} e^{- t} + \\left(C_{1} + \\frac{t}{2}\\right) e^{t}'

---
## Symbols VS Variables

Let's understand a bit of details of SymPy's implementation.

`symbols` takes a string of variable names separated by spaces or commas, and creates Symbols out of them. We can then assign these to variable names.

```python
x, y, z = symbols('x, y, z')
```

The important thing to notice here is that the name of a Symbol and the name of the variable it is assigned to need not have anything to do with one another. In other words, **the variable and the symbol are not the same thing**.

```python
a, b, c = symbols('x, y, z')
```

```python
a
```

$\displaystyle x$

```python
crazy = symbols('unralated')
crazy + 1
```

$\displaystyle unralated + 1$

Take the following example. Changing `x` to 2 had no effect on `expr`. This is because `x = 2` changes the Python variable `x` to 2, but has no effect on the SymPy Symbol x, which was what we used in creating `expr`.

```python
x = symbols('x')
expr = x + 1
x = 2
display(expr)
```

$\displaystyle x + 1$

---
## Basic Operations
Here we discuss some of the most basic operations needed for expression manipulation in SymPy.

```python
x, y, z = symbols("x y z")
```

One of the most common things you might want to do with a mathematical expression is **substitution**. Substitution replaces all instances of something in an expression with something else. It is done using the `subs` method. For example:

```python
expr = cos(x) + 1
expr.subs(x, y)
```

$\displaystyle \cos{\left(y \right)} + 1$

```python
expr.subs(x, 0)
```

$\displaystyle 2$

```python
expr.subs(x, 2)
```

$\displaystyle \cos{\left(2 \right)} + 1$

Notice that this last example the value of the expression was not evaluated. That is because $\cos\left(2\right)$ is irrational (proof [here](https://math.stackexchange.com/questions/94478/sin-1-circ-is-irrational-but-how-do-i-prove-it-in-a-slick-way-and-tan1)). If we wanted to evaluate the expression, we would use the `evalf` method.

```python
expr.subs(x, 2).evalf()
```

$\displaystyle 0.583853163452858$

Notice that the output has the traditional 15 decimal places. But the `evalf` method lets you choose how many digits you want.

```python
expr.subs(x, 2).evalf(30)
```

$\displaystyle 0.583853163452857613002431770499$

Take a look at the first 100 digits of $\pi$

```python
pi.evalf(100)
```

$\displaystyle 3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068$

To perform multiple substitutions at once, pass a list of `(old, new)` pairs to `subs`.

```python
expr = x**3 + 4*x*y - z
expr.subs([(x, 2), (y, 4), (z, 0)])
```

$\displaystyle 40$

You can use python's list comprehensions to generate substitutions. In the example below we replace all instances of $x$ that have an even power with $y$.

```python
expr = x**4 - 4*x**3 + 4*x**2 - 2*x + 3
replacements = [(x**i, y**i) for i in range(5) if i % 2 == 0]
expr.subs(replacements)
```

$\displaystyle - 4 x^{3} - 2 x + y^{4} + 4 y^{2} + 3$

---
## Printing
As you may have noticed, the output of symbolyc expressions in the jupyter notebook environment automatically prints the rendered $\LaTeX$ expression. But you can print outputs in other formats as well.

```python
expr = Integral(sqrt(1/x), x)
expr
```

$\displaystyle \int \sqrt{\frac{1}{x}}\, dx$

You can ask for the latex source code.

```python
latex(expr)
```

    '\\int \\sqrt{\\frac{1}{x}}\\, dx'

Use the `pprint` to print the expression using Unicode characters.

```python
pprint(expr)
```

    ⌠
    ⎮     ___
    ⎮    ╱ 1
    ⎮   ╱  ─  dx
    ⎮ ╲╱   x
    ⌡

Or non-unicode charcters

```python
pprint(expr, use_unicode=False)
```

      /
     |
     |     ___
     |    / 1
     |   /  -  dx
     | \/   x
     |
    /

---
## Simplification
One of the most useful features of a symbolic manipulation system is the ability to simplify mathematical expressions. SymPy has dozens of functions to perform various kinds of simplification. There is also one general function called `simplify()` that attempts to apply all of these functions in an intelligent way to arrive at the simplest form of an expression.

```python
x, y, z = symbols('x y z')
```

SymPy knows trigonometrical identities

```python
simplify(sin(x)**2 + cos(x)**2)
```

$\displaystyle 1$

Facotoring out commom factors in polynomials

```python
simplify((x**3 + x**2 - x - 1)/(x**2 + 2*x + 1))
```

$\displaystyle x - 1$

And even understands identities from big formulas of statitistical distributions, like the gamma function.

```python
simplify(gamma(x)/gamma(x - 2))
```

$\displaystyle \left(x - 2\right) \left(x - 1\right)$

There is a caveat: "simplify" does not have a well-defined target, so the `simplify()` method might not be what you need.

```python
expand((x + 1)**2)
```

$\displaystyle x^{2} + 2 x + 1$

```python
factor(x**2*z + 4*x*y*z + 4*y**2*z)
```

$\displaystyle z \left(x + 2 y\right)^{2}$

Another powerfull tool is the `rewrite` method. The example below rewrites the tangent functions in terms of the sine function

```python
tan(x).rewrite(sin)
```

$\displaystyle \frac{2 \sin^{2}{\left(x \right)}}{\sin{\left(2 x \right)}}$

And the factorial faction using the gamma function.

```python
factorial(x).rewrite(gamma)
```

$\displaystyle \Gamma\left(x + 1\right)$

There are other types of simplifications as well, for powers, more elaborate trigonometric identities, logarithims, etc. Check out the documentations [here](https://docs.sympy.org/latest/tutorial/simplification.html).

---
## Calculus

```python
x, y, z = symbols('x y z')
```

```python
expr = exp(x**2 * y**2)
expr
```

$\displaystyle e^{x^{2} y^{2}}$

The first steps we usually take in calculus usually envolves limits.

```python
limit(sin(x)/x, x, 0)
```

$\displaystyle 1$

and we learn that limits might be different from the right and from the left.

```python
limit(1/x, x, 0, '+')
```

$\displaystyle \infty$

```python
limit(1/x, x, 0, '-')
```

$\displaystyle -\infty$

You can take analytical derivatives with the `diff` method

```python
diff(expr, x)
```

$\displaystyle 2 x y^{2} e^{x^{2} y^{2}}$

You can also take multiple derivatives by chaining input arguments

```python
diff(expr, x, x, y)
```

$\displaystyle 4 y \left(2 x^{4} y^{4} + 5 x^{2} y^{2} + 1\right) e^{x^{2} y^{2}}$

The same idea can be used for integrals.

```python
integrate(cos(x), x)
```

$\displaystyle \sin{\left(x \right)}$

```python
integrate(cos(x), y)
```

$\displaystyle y \cos{\left(x \right)}$

To evaluate an integral, pass variable and their limits as tuples

```python
integrate(cos(x), (x, 0, pi/2))
```

$\displaystyle 1$

```python
integrate(cos(x), (y, 0, pi/2))
```

$\displaystyle \frac{\pi \cos{\left(x \right)}}{2}$

You can even integrate to infinity

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

$\displaystyle \pi$

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

$\displaystyle \pi$

SymPy can also do taylor expansions aroundo a given point. The example below make a 4th degree approximation to the sine function aroun $x=0$.

```python
sin(x).series(x, 0, 4)
```

$\displaystyle x - \frac{x^{3}}{6} + O\left(x^{4}\right)$

And if you are an economist and do not care the $O(x^n)$ notation, you can remove it.

```python
sin(x).series(x, 0, 4).removeO()
```

$\displaystyle - \frac{x^{3}}{6} + x$

---
## Solvers

Writing equations in SymPy assumes $=0$ at the end, unless you are explicit about the right side.

```python
Eq(x**2 -1)
```

$\displaystyle x^{2} - 1 = 0$

```python
Eq(x**2, 1)
```

$\displaystyle x^{2} = 1$

```python
Eq(x**2, x-1)
```

$\displaystyle x^{2} = x - 1$

We can solve equations

```python
expr = Eq(x**2 - 5*x + 6)
solve(expr, x)
```

    [2, 3]

Sympy can even give elaborate answers to solutions. In this case, use `solveset` instead of `solve`

```python
solveset(x - x, x, domain=S.Reals)
```

$\displaystyle \mathbb{R}$

```python
solveset(x - x, x)
```

$\displaystyle \mathbb{C}$

```python
solveset(sin(x) - 1, x)
```

$\displaystyle \left\{2 n \pi + \frac{\pi}{2}\; |\; n \in \mathbb{Z}\right\}$

```python
solveset(x**2 + 1, x)
```

$\displaystyle \left\{- i, i\right\}$

---
## Solving Differential Equations
The first step is to declare functions

```python
f, g = symbols('f, g', cls=Function)
```

`f` and `g` are now undefined functions. We can call `f(x)`, and it will represent an unknown function.

```python
f(x)
```

$\displaystyle f{\left(x \right)}$

```python
f(x).diff()
```

$\displaystyle \frac{d}{d x} f{\left(x \right)}$

```python
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
diffeq
```

$\displaystyle f{\left(x \right)} - 2 \frac{d}{d x} f{\left(x \right)} + \frac{d^{2}}{d x^{2}} f{\left(x \right)} = \sin{\left(x \right)}$

To solve the equation, use the `dsolve` method.

```python
dsolve(diffeq, f(x))
```

$\displaystyle f{\left(x \right)} = \left(C_{1} + C_{2} x\right) e^{x} + \frac{\cos{\left(x \right)}}{2}$

Here, $C_{i}$ denotes arbitrary constants.

---
## Matrices
Matrix operations in SymPy are the same as in numpy. The only diffenrence is that now we can use symbols.

```python
x, y = symbols('x, y')
M = Matrix([[1, 2, 3], [x, 3, 1], [4, y, 6]])
M
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\x & 3 & 1\\4 & y & 6\end{matrix}\right]$

```python
M.det()
```

$\displaystyle 3 x y - 12 x - y - 10$

Now take an example of a mtrix with incomplete rank

```python
M = Matrix([[1, 1, 2], [2 ,1 , 3], [3 , 1, 4]])
M
```

$\displaystyle \left[\begin{matrix}1 & 1 & 2\\2 & 1 & 3\\3 & 1 & 4\end{matrix}\right]$

```python
M.nullspace()  # or "kernel of the linear transformation"
```

    [Matrix([
     [-1],
     [-1],
     [ 1]])]

```python
M.columnspace()  # or "span of the linear transformation"
```

    [Matrix([
     [1],
     [2],
     [3]]), Matrix([
     [1],
     [1],
     [1]])]

Notice that each output is a sympy object

```python
display(M.diagonalize()[0])
display(M.diagonalize()[1])
```

$\displaystyle \left[\begin{matrix}-1 & - \frac{2 \sqrt{11}}{7} - \frac{3}{7} & - \frac{3}{7} + \frac{2 \sqrt{11}}{7}\\-1 & \frac{2}{7} - \frac{\sqrt{11}}{7} & \frac{2}{7} + \frac{\sqrt{11}}{7}\\1 & 1 & 1\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 3 - \sqrt{11} & 0\\0 & 0 & 3 + \sqrt{11}\end{matrix}\right]$

All of this was to get to the following example, the Jacobian Matrix:

```python
from sympy.abc import rho, phi
X = Matrix([rho*cos(phi), rho*sin(phi), rho**2])
Y = Matrix([rho, phi])
X.jacobian(Y)
```

$\displaystyle \left[\begin{matrix}\cos{\left(\phi \right)} & - \rho \sin{\left(\phi \right)}\\\sin{\left(\phi \right)} & \rho \cos{\left(\phi \right)}\\2 \rho & 0\end{matrix}\right]$

Just to grab a better view of what is happening here, let us build a simpler example

```python
x, y, z = symbols('x, y, z')
eq1 = 2*x**2 + 3*y**3 - 2*z
eq2 = 4*x + y**2 - 2*z**3
eq3 = 3*x + 2*y - 3*z**4
M1 = Matrix([eq1, eq2, eq3])
M2 = Matrix([x, y])
M1.jacobian(M2)
```

$\displaystyle \left[\begin{matrix}4 x & 9 y^{2}\\4 & 2 y\\3 & 2\end{matrix}\right]$

This kind of structure will be perfect for DSGE models

Python Lectures/Section 11 - Symbolic Computation with Sympy.ipynb


# Linear Modelling - Maxiumum Likelihood

One approach to learning parameters is minimizing the loss function, another method is to incorporate a random variable to denote _noise_, which has considerable advantages over are former approach.

## The Gaussian (normal) distribution

A Gaussian distribution is defined over the sample space of all real numbers with the pdf for a random varaible $Y$ as the following:

$$
p(y \mid \mu, \sigma^2) = \frac{1}{\sigma \sqrt{2 \pi}} \exp{\left\{ - \frac{1}{2 \sigma^2} (y - \mu)^2 \right\}}
$$

The common shorthand notation is the following:

$$
p(y \mid \mu, \sigma^2) = \mathcal{N}(\mu, \sigma^2)
$$

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

x_axis = np.linspace(-5, 10, 100)

plt.plot(x_axis, norm.pdf(x_axis,-2,0.1 ** 0.5), 'r', label="$\mu = -2, \sigma^2 = 0.1$")
plt.plot(x_axis, norm.pdf(x_axis,0,0.3 ** 0.5), 'g', label="$\mu = 0, \sigma^2 = 0.3$")
plt.plot(x_axis, norm.pdf(x_axis,5,2 ** 0.5), 'b', label="$\mu = 5, \sigma^2 = 2$")
plt.legend()
plt.show()
```

## Multivariate Gaussian

We can generalize the gaussain distribution to define a density function over vectors. For a vector $\mathbf{x} = [x_1, ... x_D]^T$ the density function is defined as:

$$
p(\mathbf{x}) = \frac{1}{(2 \pi)^{\frac{D}{2}}{\begin{vmatrix}\mathbf{\Sigma}\end{vmatrix}^{\frac{1}{2}}}} \exp \left\{ - \frac{1}{2} (\mathbf{x} - \mathbf{\mu})^T \mathbf{\Sigma}^{-1} (\mathbf{x} - \mathbf{\mu}) \right\}
$$

where $\mathbf{\mu}$ is a vector of mean values, and the variance a $D \times D$  covariance matrix (a matrix whose element in the $i$, $j$ position is the covariance between the $i$ th and $j$ th elements).

```python
from scipy.stats import multivariate_normal
from mpl_toolkits.mplot3d import Axes3D

def plot_m_gauss(mu, variance):
    #Create grid and multivariate normal
    x = np.linspace(-2,5,500)
    y = np.linspace(5,-2,500)
    X, Y = np.meshgrid(x,y)
    pos = np.empty(X.shape + (2,))
    pos[:, :, 0] = X; pos[:, :, 1] = Y
    rv = multivariate_normal(mu, variance)

    #Make a 3D plot
    fig = plt.figure()
    ax = fig.gca(projection='3d')
    ax.plot_surface(X, Y, rv.pdf(pos),cmap='viridis',linewidth=0)
    ax.set_xlabel('x1')
    ax.set_ylabel('x2')
    plt.show()

    plt.contour(X, Y, rv.pdf(pos))
    plt.show()

mu_1 = np.array([2, 1]).T
variance_1 = np.array([[1, 0], [0, 1]])
print("mu = {}, Epsilon = {}".format(mu_1, variance_1))
plot_m_gauss(mu_1, variance_1)

mu_2 = np.array([2, 1]).T
variance_2 = np.array([[1, 0.8], [0.8, 1]])
print("mu = {}, Epsilon = {}".format(mu_2, variance_2))
plot_m_gauss(mu_2, variance_2)
```

A special case of the multivariate Gaussian is where the two variables are independent, hence:

$$
\mathbf{\Sigma} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \mathbf{I}
$$

$$
\begin{align}
p(\mathbf{x}) &= \frac{1}{(2 \pi)^{\frac{D}{2}} \begin{vmatrix}\mathbf{I}\end{vmatrix}^{\frac{1}{2}} } \exp \left\{ - \frac{1}{2} (\mathbf{x} - \mathbf{\mu})^T \mathbf{I}^{-1} (\mathbf{x} - \mathbf{\mu}) \right\} \\
&= \frac{1}{(2 \pi)^{\frac{D}{2}} \begin{vmatrix}\mathbf{I}\end{vmatrix}^{\frac{1}{2}} } \exp \left\{ - \frac{1}{2} (\mathbf{x} - \mathbf{\mu})^T (\mathbf{x} - \mathbf{\mu}) \right\} \\
&= \frac{1}{(2 \pi)^{\frac{D}{2}} \begin{vmatrix}\mathbf{I}\end{vmatrix}^{\frac{1}{2}} } \exp \left\{ - \frac{1}{2} \sum_{d=1}^D (x_d - \mu_d)^2 \right\}
\end{align}
$$

The exponential of a sum is a product of exponentials thus

$$
p(\mathbf{x}) = \frac{1}{(2 \pi)^{\frac{D}{2}} \begin{vmatrix}\mathbf{I}\end{vmatrix}^{\frac{1}{2}} } \prod_{d=1}^D \exp \left\{ - \frac{1}{2} (x_d - \mu_d)^2 \right\}
$$

The determinant of $\mathbf{I}$ is 1, and $(2 \pi)^{\frac{D}{2}}$ can be written as $\prod_{d=1}^D (2 \pi)^{\frac{1}{2}}$ thus we arrive at:

$$
p(\mathbf{x}) = \prod_{d=1}^D \frac{1}{\sqrt{2 \pi} } \exp \left\{ - \frac{1}{2} (x_d - \mu_d)^2 \right\}
$$

Each term in the product in a univariate Gaussian (with mean $\mu_d$ and variance $1$), thus by definition of independence ($p(A \cup B) = p(A)p(B)$ iff $A$ and $B$ are independent), the elements of $\mathbf{x}$ is independent. This will work for any $\mathbf{\Sigma}$ with non-zero elements only in the diagonal positions.

## Thinking generatively

If we think how we could generate mens 100m times that looks like the data we observe we would arive at the following:

$$
t_n = \mathbf{w}^T \mathbf{x}_n + \epsilon_n
$$

where $\epsilon_n$ is a random variable.

Now we need to determine the distribution for $\epsilon_n$. Our model is continous thus $\epsilon_n$ is must be a continous random varible. Their is a random variable for each Olympic year, and its a resonable assumption that these values are independent.

$$
p(\epsilon_1, ..., \epsilon_n) = \prod_{n=1}^N p(\epsilon_n)
$$

Lets assume $p(\epsilon_n)$ follows a Gaussian distribution with a zero mean and variance $\sigma$. Our model can now be described as two components:

1. A _deterministic_ component ($\mathbf{w}^T \mathbf{x}_n$) referred to as a _trend_ or _drift_
2. A random component ($\epsilon_n$) referred to as _noise_

In our case the noise is _additive_ but some applications might call for _mulitiplicative_ noise such as pixel degradation.

## Likelihood

Our model is of the following form:

$$
t_n = f(x_n; \mathbf{w}) + \epsilon_n \quad \epsilon_n \sim \mathcal{N}(0, \sigma^2)
$$

We cant minimize the loss since $t_n$ is no longer a fixed value, its a random variable. Adding a constant ($\mathbf{w}^T \mathbf{x}_n$) to a Gausian distributed random variable is equivalent to a new Gausian random variable with the constant added to the mean. Thus $t_n$ has the following pdf:

$$
p(t_n \mid \mathbf{x}_n, \mathbf{w}, \sigma^2) = \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2)
$$

We can use this to find optimal values for $\mathbf{w}$ and $\sigma^2$, consider the year 1980, using the values for $\mathbf{w}$ we found previously and assuming $\sigma^2 = 0.05$ we can plot:

$$
p\left(t_n \mid \mathbf{x}_n = \begin{bmatrix}1\\1980\end{bmatrix}, \mathbf{w} = \begin{bmatrix}36.416\\-0.0133\end{bmatrix}, \sigma^2 = 0.05 \right)
$$

```python
mu = 36.41645590250286 - 0.013330885710960602 * 1980
sigma2 = 0.05

print("mu = {}, sigma^2 = {}".format(mu, sigma2))

x_axis = np.linspace(9, 11, 50)

plt.plot(x_axis, norm.pdf(x_axis, mu, sigma2 ** 0.5), 'r')
plt.show()
```

According to the graph the most _likely_ winning time for 1980 is $10.02$ seconds. The actuall time was $10.25$, thus we need to tune the parameters $\mathbf{w}$ and $\sigma^2$ to make the density as high as possible at $t = 10.25$.

## Dataset likelihood

We can extend this to the whole dataset by finding the joint conditional density:

$$
p(t_1, ..., t_N \mid \mathbf{x}_1, ..., \mathbf{x}_N, \mathbf{w}, \sigma^2)
$$

By using the vector notation defined previously and the assumption that the noise at each datapoint is independent, we get the following:

$$
L = p(\mathbf{t} \mid \mathbf{X}, \mathbf{w}, \sigma^2) = \prod_{n=1}^N p(t_n \mid \mathbf{x_n}, \mathbf{w}, \sigma^2) = \prod_{n=1}^N \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2)
$$

## Maximum likelihood

To find $\widehat{\mathbf{w}}$ and $\widehat{\sigma^2}$ clearly we need to maximize the value of $L$, to do this we will maximize the log-likelyhood (for analytical reasons)

$$
\begin{align}
L &= \prod_{n=1}^N \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2) \\
\log L &= \log \left(\prod_{n=1}^N \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2) \right) \\
&= \sum_{n=1}^N \log \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2) \\
&= \sum_{n=1}^N \log \left( \frac{1}{\sigma \sqrt{2 \pi}} \exp{\left\{ - \frac{1}{2 \sigma^2} (t_n - \mathbf{w}^T \mathbf{x}_n)^2 \right\}} \right) \\
&= \sum_{n=1}^N \left( -\frac{1}{2} \log(2 \pi) - \log \sigma - \frac{1}{2 \sigma^2} (t_n - \mathbf{w}^T \mathbf{x}_n)^2 \right) \\
&= -\frac{N}{2} \log(2 \pi) - N \log \sigma - \frac{1}{2 \sigma^2} \sum_{n=1}^N (t_n - \mathbf{w}^T \mathbf{x}_n)^2 \\
\end{align}
$$

As previosly we differentiate and set to zero to find the turning point, in this case we want a maximum.

$$
\begin{align}
\frac{\partial \log L}{\partial \mathbf{w}} &= \frac{1}{\sigma^2} \sum^N_{n=1} \mathbf{x}_n(t_n - \mathbf{x}_n^T \mathbf{w}) \\
&= \frac{1}{\sigma^2} \sum^N_{n=1} \mathbf{x}_n t_n - \mathbf{x}_n \mathbf{x}_n^T \mathbf{w} \\
\end{align}
$$

Using the vector/matix notation from earlier, $\sum_{n=1}^N \mathbf{x}_n t_n$ becomes $\mathbf{X}^T \mathbf{t}$ and $\sum_{n=1}^N \mathbf{x}_n \mathbf{x}_n^T \mathbf{w}$ becomes $\mathbf{X}^T \mathbf{Xw}$, thus the derivitive becomes:

$$
\frac{\partial \log L}{\partial \mathbf{w}} = \frac{1}{\sigma^2} (\mathbf{X}^T \mathbf{t} - \mathbf{X}^t \mathbf{Xw})
$$

Setting the derivitive to $\mathbf{0}$ (a vector with all zeros) and solving for $\mathbf{w}$ gives us:

$$
\begin{align}
\frac{1}{\sigma^2} (\mathbf{X}^T \mathbf{t} - \mathbf{X}^t \mathbf{Xw}) &= \mathbf{0} \\
\mathbf{X}^T \mathbf{t} - \mathbf{X}^t \mathbf{Xw} &= \mathbf{0} \\
- \mathbf{X}^t \mathbf{Xw} &= - \mathbf{X}^T \mathbf{t} \\
\mathbf{X}^t \mathbf{Xw} &= \mathbf{X}^T \mathbf{t} \\
\widehat{\mathbf{w}} &= (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t} \\
\end{align}
$$

This is the same result as minimizing the squared loss. Minimising the squared loss is equivalent to the maximum likelihood solution if the noise is assumed to be Gaussian.

Now we repeat the process for $\sigma^2$

$$
\begin{align}
\frac{\partial \log L}{\partial \sigma} &= - \frac{N}{\sigma} + \frac{1}{\sigma^3} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2
\end{align}
$$

$$
\begin{align}
- \frac{N}{\sigma} + \frac{1}{\sigma^3} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2 &= 0 \\
\frac{1}{\sigma^3} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2 &= \frac{N}{\sigma} \\
\sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2 &= N \sigma^2 \\
\widehat{\sigma^2} &= \frac{1}{N} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2 \\
\end{align}
$$

This makes sence, the variance is the average square error. We can use the fact that $\sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2$ is equivalent to $(\mathbf{t} - \mathbf{X}\widehat{\mathbf{w}})^T (\mathbf{t} - \mathbf{X}\widehat{\mathbf{w}})$

$$
\begin{align}
\widehat{\sigma^2} &= \frac{1}{N} (\mathbf{t} - \mathbf{X}\widehat{\mathbf{w}})^T (\mathbf{t} - \mathbf{X}\widehat{\mathbf{w}}) \\
&= \frac{1}{N} (\mathbf{t}^T \mathbf{t} - 2 \mathbf{t}^T \mathbf{X} \widehat{\mathbf{w}} + \widehat{\mathbf{w}}^T \mathbf{X}^T \mathbf{X} \widehat{\mathbf{w}}) \\
\end{align}
$$

Now using $\widehat{\mathbf{w}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t}$ and $\widehat{\mathbf{w}}^T = \mathbf{t}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1}$

$$
\begin{align}
\widehat{\sigma^2} &= \frac{1}{N} (\mathbf{t}^T \mathbf{t} - 2 \mathbf{t}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t} + \mathbf{t}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t}) \\
&= \frac{1}{N} (\mathbf{t}^T \mathbf{t} - 2 \mathbf{t}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t} + \mathbf{t}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t}) \\
&= \frac{1}{N} (\mathbf{t}^T \mathbf{t} - \mathbf{t}^T \mathbf{X} (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{t}) \\
&= \frac{1}{N} (\mathbf{t}^T \mathbf{t} - \mathbf{t}^T \mathbf{X} \widehat{\mathbf{w}}) \\
\end{align}
$$

```python
x_values = [1896, 1900, 1904, 1906, 1908, 1912, 1920, 1924, 1928, 1932, 1936, 1948, 1952, 1956, 1960, 1964,
     1968, 1972, 1976, 1980, 1984, 1988, 1992, 1996, 2000, 2004, 2008]
t_values = [12.00, 11.00, 11.00, 11.20, 10.80, 10.80, 10.80, 10.60, 10.80, 10.30, 10.30, 10.30, 10.40, 10.50,
     10.20, 10.00, 9.95, 10.14, 10.06, 10.25, 9.99, 9.92, 9.96, 9.84, 9.87, 9.85, 9.69]

N = len(x_values)
X = np.matrix([[1,x] for x in x_values])

def get_params(X_mat):
    XT = np.transpose(X_mat)
    tT = np.matrix([t_values])
    t = np.transpose(tT)

    best_w = ((XT * X_mat) ** -1) * XT * t
    best_sigma2 = (1/N) * (tT * t - tT * X_mat * best_w)

    return (best_w, best_sigma2)

print("w = {}\n\nsigma^2 = {}".format(*get_params(X)))
```

    w = [[  3.64164559e+01]
     [ -1.33308857e-02]]

    sigma^2 = [[ 0.05030711]]

## Checking the turning point

Previously we had differentatiated the loss function twice to check the turning point was a minimum, we would like to do the same here to check if the likelihood is maximum.

Since the dirivative is with respect to a vector, we need to form a Hessian matrix, a square matrix with all the second order patrial derivatives of a function, for example for a function $f(\mathbf{x}; \mathbf{w})$ where $\mathbf{w} = [w_1, ..., w_K]^T$

$$
\mathbf{H} = \begin{bmatrix}
\dfrac{\partial^2 f}{\partial w_1^2} & \dfrac{\partial^2 f}{\partial w_1 \partial w_2} & \cdots & \dfrac{\partial^2 f}{\partial w_1 \partial w_K} \\
\dfrac{\partial^2 f}{\partial w_2 \partial w_1} & \dfrac{\partial^2 f}{\partial w_2^2} & \cdots & \dfrac{\partial^2 f}{\partial w_2 \partial w_K} \\
\vdots & \vdots & \ddots & \vdots \\
\dfrac{\partial^2 f}{\partial w_K \partial w_1} & \dfrac{\partial^2 f}{\partial w_K \partial w_2} & \cdots & \dfrac{\partial^2 f}{\partial w_K^2} \\
\end{bmatrix}
$$

The turning point is maximum if the matix is negative definite. A real-valued matrix is negative definite if $\mathbf{x}^T \mathbf{H} \mathbf{x} < 0$ for all real values of $\mathbf{x}$

The first order derivative was

$$
\frac{\partial \log L}{\partial \mathbf{w}} = \frac{1}{\sigma^2} (\mathbf{X}^T \mathbf{t} - \mathbf{X}^t \mathbf{Xw})
$$

Integrating with repect to $\mathbf{w}^T$ gives us the Hessian matrix:

$$
\frac{\partial \log L}{\partial \mathbf{w} \partial \mathbf{w}^T} = - \frac{1}{\sigma^2} \mathbf{X}^T \mathbf{X}
$$

Now to check the matrix is negative definite we must show

$$
- \frac{1}{\sigma^2} \mathbf{z}^T \mathbf{X}^T \mathbf{X} \mathbf{z} < 0
$$

for any vector $\mathbf{z}$ or equivalently (since $\sigma^2$ must be positive)

$$
\mathbf{z}^T \mathbf{X}^T \mathbf{X} \mathbf{z} > 0
$$

So that we can explicitly multiply out the various terms, we will rescrict $\mathbf{X}$ to

$$
\mathbf{X}
=
\begin{bmatrix}
\mathbf{x}^T_1 \\
\mathbf{x}^T_2 \\
\vdots \\
\mathbf{x}^T_N
\end{bmatrix}
=
\begin{bmatrix}
x_{11} & x_{12} \\
x_{21} & x_{22} \\
\vdots & \vdots \\
x_{N1} & x_{N2}
\end{bmatrix}
$$

Thus $\mathbf{X}^T \mathbf{X}$ becomes

$$
\mathbf{X}^T \mathbf{X}
=
\begin{bmatrix}
\sum_{i=1}^N{x^2_{i1}} & \sum_{i=1}^N{x_{i1} x_{i2}} \\
\sum_{i=1}^N{x_{i2} x_{i1}} & \sum_{i=1}^N{x^2_{i2}}
\end{bmatrix}
$$

Pre and post multiplying with the arbitary vector $\mathbf{z} = \begin{bmatrix} z_1 \\ z_2 \end{bmatrix}$ gives us:

$$
\begin{align}
\mathbf{z}^T \mathbf{X}^T \mathbf{X} \mathbf{z} &= \mathbf{z}^T
\begin{bmatrix}
\sum_{i=1}^N{x^2_{i1}} & \sum_{i=1}^N{x_{i1} x_{i2}} \\
\sum_{i=1}^N{x_{i2} x_{i1}} & \sum_{i=1}^N{x^2_{i2}}
\end{bmatrix}
\mathbf{z} \\
&=
\begin{bmatrix}
z_1 \sum_{i=1}^N{x^2_{i1}} + z_2 \sum_{i=1}^N{x_{i2} x_{i1}} &
z_1 \sum_{i=1}^N{x_{i1} x_{i2}} + z_2 \sum_{i=1}^N{x^2_{i2}}
\end{bmatrix}
\mathbf{z} \\
&= z_1^2 \sum_{i=1}^N{x^2_{i1}} + 2 z_1 z_2 \sum_{i=1}^N{x_{i1} x_{i2}} + z^2_2 \sum_{i=1}^N{x^2_{i2}}
\end{align}
$$

The terms $z_1^2 \sum_{i=1}^N{x^2_{i1}}$ and $z^2_2 \sum_{i=1}^N{x^2_{i2}}$ are always positive thus proving $\mathbf{z}^T \mathbf{X}^T \mathbf{X} \mathbf{z}$ is positive is equivalent to

$$
z_1^2 \sum_{i=1}^N{x^2_{i1}} + z^2_2 \sum_{i=1}^N{x^2_{i2}} > 2 z_1 z_2 \sum_{i=1}^N{x_{i1} x_{i2}}
$$

The sum of the positive terms must be greater that the other term so that the whole term is greater than zero. Now let $y_{i1} = z_1 x_{i1}$ and $y_{i2} = z_2 x_{i2}$.

$$
\begin{align}
z_1^2 \sum_{i=1}^N{x^2_{i1}} + z^2_2 \sum_{i=1}^N{x^2_{i2}} &> 2 z_1 z_2 \sum_{i=1}^N{x_{i1} x_{i2}}\\
\sum_{i=1}^N{y^2_{i1}} + \sum_{i=1}^N{y^2_{i2}} &> 2 \sum_{i=1}^N{y_{i1} y_{i2}}\\
\sum_{i=1}^N{\left(y^2_{i1} + y^2_{i2} \right)} &> 2 \sum_{i=1}^N{y_{i1} y_{i2}}
\end{align}
$$

Now consider an arbitary $i$

$$
\begin{align}
y^2_{i1} + y^2_{i2} &> 2 y_{i1} y_{i2} \\
y^2_{i1} - 2 y_{i1} y_{i2} + y^2_{i2} &> 0 \\
(y_{i1} - y_{i2})^2 &> 0
\end{align}
$$

Thus the only case where this is not true is when $y_{i1}^2 = y_{i2}^2$ and thus $x_{i1} = x_{i2}$, somthing that is unlikely to happen in practive. Thus for an  arbitary $i$, $y^2_{i1} + y^2_{i2} > 2 y_{i1} y_{i2}$ holds, and thus the summation of the terms holds. Hence, $\mathbf{z}^T \mathbf{X}^T \mathbf{X} \mathbf{z}$ is always positive, thus $\mathbf{H}$, our Hessian matrix is negative definite, thus the solution is a maximum.

Likewise, to check $\widehat{\sigma^2}$ corrsponds to the maximum, we diffentiate

$$
\frac{\partial \log L}{\partial \sigma} = - \frac{N}{\sigma} + \frac{1}{\sigma^3} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2
$$

Again with respect to $\sigma$, giving us

$$
\frac{\partial \log L}{\partial \sigma^2} = - \frac{N}{\sigma^2} + \frac{3}{\sigma^4} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2
$$

Substituting $\widehat{\sigma^2} = \frac{1}{N} \sum_{n=1}^N (t_n - \mathbf{x}^T \widehat{\mathbf{w}})^2$

$$
\begin{align}
\frac{\partial \log L}{\partial \sigma^2} &= - \frac{N}{\widehat{\sigma^2}} + \frac{3}{\left(\widehat{\sigma^2}\right)^2} N \widehat{\sigma^2} \\
&= - \frac{2N}{\widehat{\sigma^2}}
\end{align}
$$

Thus $\widehat{\sigma^2}$ corresponds to a maximum.

## Maximum likelihood favours complexity

By substituting $\widehat{\sigma^2}$ into $\log L$ gives the value of the log-likelihood at the maximum

$$
\begin{align}
\log L &= -\frac{N}{2} \log(2 \pi) - N \log \sigma - \frac{1}{2 \sigma^2} \sum_{n=1}^N (t_n - \mathbf{w}^T \mathbf{x}_n)^2 \\
&= -\frac{N}{2} \log(2 \pi) - N \log \sqrt{\widehat{\sigma^2}} - \frac{1}{2 \widehat{\sigma^2}} N\widehat{\sigma^2} \\
&= -\frac{N}{2} \log(2 \pi) - \frac{N}{2} \log \widehat{\sigma^2} - \frac{N}{2}\\
&= -\frac{N}{2} (1 + \log 2 \pi) - \frac{N}{2} \log \widehat{\sigma^2}\\
\end{align}
$$

Thus by decreasing $\widehat{\sigma^2}$ we increase the log-likeliness. One way to decrease $\widehat{\sigma^2}$ is to modify $f(\mathbf{x};\mathbf{w})$ so that it can capture more of the noise. The same tradeoff between overfitting and generalization as we saw last time occors. Before we used regularization to peanalize complex models, _prior distributions_ on parameter values can acieve the same thing with probablistic models.

```python
# Normalize x, for numerical stability
stable_x = np.array([float(x) for x in x_values]) - x_values[0]
stable_x *= 0.4

orders = list(range(2, 9))
log_Ls = []
for order in orders:
    X = np.matrix([[x**o for o in range(0, order)] for x in stable_x])
    (_, ss) = get_params(X)
    log_L = -(N/2)*(1+np.log(2 * np.pi)) - (N/2) * np.log(ss)
    log_Ls.append(log_L.item(0))

plt.plot(orders, log_Ls, 'r')
plt.xlabel('Polynomial order')
plt.ylabel('log L')
plt.show()
```

## Effect of noise on estimates

It would be useful to determine how much confidence we have in our parameters. Firstly is our estimator $\widehat{w}$ _unbiased_.

Our current model takes the form:

$$
t_n = \mathbf{w}^T \mathbf{x}_n + \epsilon_n
$$

Since we defined $\epsilon_n$ to be normally distributed, the _generating_ distribution (or likelihood) is a product of normal densities:

$$
p(\mathbf{t} \mid \mathbf{X}, \mathbf{w}, \sigma^2) = \prod^N_{n=1} p(t_n \mid \mathbf{x}_n \mathbf{w}) = \prod^N_{n=1} \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2)
$$

We have shown that a product of univariant Gaussians can be rewritten as a multivariant Gaussian with a diangonal covariance, thus

$$
p(\mathbf{t} \mid \mathbf{X}, \mathbf{w}, \sigma^2) = \prod^N_{n=1} \mathcal{N}(\mathbf{w}^T \mathbf{x}_n, \sigma^2) = \mathcal{N}(\mathbf{Xw}, \sigma^2 \mathbf{I})
$$

Linear Modelling - Maximum Likelihood.ipynb


# 13 Linear Algebra: Singular Value Decomposition (Students)

One can always decompose a matrix $\mathsf{A}$

\begin{gather}
\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}\\
\mathsf{U}^T \mathsf{U} = \mathsf{U} \mathsf{U}^T = 1\\
\mathsf{V}^T \mathsf{V} = \mathsf{V} \mathsf{V}^T = 1
\end{gather}

where $\mathsf{U}$ and $\mathsf{V}$ are orthogonal matrices and the $w_j$ are the _singular values_ that are assembled into a diagonal matrix $\mathsf{W}$.

$$
\mathsf{W} = \text{diag}(w_j)
$$

The inverse (if it exists) can be directly calculated from the SVD:

$$
\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T
$$

## Solving ill-conditioned coupled linear equations

```python
import numpy as np
```

### Non-singular matrix
Solve the linear system of equations

$$
\mathsf{A}\mathbf{x} = \mathbf{b}
$$

Using the standard linear solver in numpy:

```python
A = np.array([
        [1, 2, 3],
        [3, 2, 1],
        [-1, -2, -6],
    ])
b = np.array([0, 1, -1])
```

```python

```

Using the inverse from SVD:

$$
\mathbf{x} = \mathsf{A}^{-1} \mathbf{b}
$$

```python

```

First check that the SVD really factors $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python

```

Now calculate the matrix inverse $\mathsf{A}^{-1} = \mathsf{V} \text{diag}(1/w_j) \mathsf{U}^T$:

```python

```

Check that this is the same that we get from `numpy.linalg.inv()`:

```python

```

Now, *finally* solve (and check against `numpy.linalg.solve()`):

```python

```

### Singular matrix

If the matrix $\mathsf{A}$ is *singular* (i.e., its rank (linearly independent rows or columns) is less than its dimension and hence the linear system of equation does not have a unique solution):

For example, the following matrix has the same row twice:

```python
C = np.array([
     [ 0.87119148,  0.9330127,  -0.9330127],
     [ 1.1160254,   0.04736717, -0.04736717],
     [ 1.1160254,   0.04736717, -0.04736717],
    ])
b1 = np.array([ 2.3674474,  -0.24813392, -0.24813392])
b2 = np.array([0, 1, 1])
```

NOTE: failure is not always that obvious: numerically, a matrix can be *almost* singular: Try solving the linear system of equations

$$
\mathsf{D}\mathbf{x} = \mathbf{b}_1
$$
with matrix $\mathsf{D}$ below:

```python
D = C.copy()
D[2, :] = C[0] - 3*C[1]
D
```

Solve:

```python

```

Note that some of the values are huge, and suspiciously like the inverse of machine precision? Sign of a nearly singular matrix.

Now back to the example with $\mathsf{C}$:

#### SVD for singular matrices
If a matrix is *singular* or *near singular* then one can *still* apply SVD.

One can then compute the *pseudo inverse*

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

i.e., any singular $w_j = 0$ is being "augmented" by setting

$$
\frac{1}{w_j} \rightarrow 0 \quad\text{if}\quad w_j = 0
$$

in $\text{diag}(1/w_j)$.

Perform the SVD for the singular matrix $\mathsf{C}$:

```python

```

Note the third value $w_2 \approx 0$: sign of a singular matrix.

Test that the SVD really decomposes $\mathsf{A} = \mathsf{U}\,\text{diag}(w_j)\,\mathsf{V}^{T}$:

```python

```

There are the **singular values** (let's say, $|w_i| < 10^{-12}$):

```python

```

#### Pseudo-inverse

Calculate the **pseudo-inverse** from the SVD

\begin{align}
\mathsf{A}^{-1} &= \mathsf{V} \text{diag}(\alpha_j) \mathsf{U}^T \\
\alpha_j &= \begin{cases}
 \frac{1}{w_j}, &\quad\text{if}\ w_j \neq 0\\
 0, &\quad\text{if}\ w_j = 0
 \end{cases}
\end{align}

Augment:

```python

```

Now solve the linear problem with SVD:

```python

```

Thus, using the pseudo-inverse $\mathsf{C}^{-1}$ we can obtain solutions to the equation

$$
\mathsf{C} \mathbf{x}_1 = \mathbf{b}_1
$$

However, $\mathbf{x}_1$ is not the only solution: there's a whole line of solutions that are formed by the special solution and a combination of the basis vectors in the *null space* of the matrix:

The (right) *kernel* or *null space* contains all vectors $\mathbf{x^0}$ for which

$$
\mathsf{C} \mathbf{x^0} = 0
$$

(The dimension of the null space corresponds to the number of singular values.) You can find a basis that spans the null space. Any linear combination of null space basis vectors will also end up in the null space when $\mathbf{A}$ is applied to it.

Specifically, if $\mathbf{x}_1$ is a special solution and $\lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots$ is a vector in the null space then

$$
\mathbf{x} = \mathbf{x}_1 + ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots )
$$

is **also a solution** because

$$
\mathsf{C} \mathbf{x} = \mathsf{C} \mathbf{x_1} + \mathsf{C} ( \lambda_1 \mathbf{x}^0_1 + \lambda_2 \mathbf{x}^0_2 + \dots ) = \mathsf{C} \mathbf{x_1} + 0 = \mathbf{b}_1 + 0 = \mathbf{b}_1
$$

The $\lambda_i$ are arbitrary real numbers and hence there is an infinite number of solutions.

In SVD:

* The columns $U_{\cdot, i}$ of $\mathsf{U}$ (i.e. `U.T[i]` or `U[:, i]`) corresponding to non-zero $w_i$, i.e. $\{i : w_i \neq 0\}$, form the basis for the _range_ of the matrix $\mathsf{A}$.
* The columns $V_{\cdot, i}$ of $\mathsf{V}$ (i.e. `V.T[i]` or `V[:, i]`) corresponding to zero $w_i$, i.e. $\{i : w_i = 0\}$, form the basis for the _null space_ of the matrix $\mathsf{A}$.

Note that `x1` can be written as a linear combination of `U.T[0]` and `U.T[1]`:

$$
\mathbf{x}_1 = (\mathbf{x}_1\cdot \mathsf{U}^T_{0,.}) \mathsf{U}^T_{0,.} + (\mathbf{x}_1\cdot \mathsf{U}^T_{1,.}) \mathsf{U}^T_{1,.}
$$

```python

```

Thus, **all** solutions are
```
x1 + lambda * VT[2]
```

The solution vector $x_2$ is in the null space:

```python

```

(For more details see the solution notebook.)

## SVD for fewer equations than unknowns
$M$ equations for $N$ unknowns with $M < N$:

* no unique solutions (underdetermined)
* $N-M$ dimensional family of solutions
* SVD: at least $N-M$ zero or negligible $w_j$: columns of $\mathsf{V}$ corresponding to singular $w_j$ span the solution space when added to a particular solution.

Same as the above **Solving ill-conditioned coupled linear equations**.

## SVD for more equations than unknowns
$M$ equations for $N$ unknowns with $M > N$:

* no exact solutions in general (overdetermined)
* but: SVD can provide best solution in the least-square sense
  $$
  \mathbf{x} = \mathsf{V}\, \text{diag}(1/w_j)\, \mathsf{U}^{T}\, \mathbf{b}
  $$
  where

  * $\mathbf{x}$ is a $N$-dimensional vector of the unknowns,
  * $\mathsf{V}$ is a $N \times M$ matrix
  * the $w_j$ form a square $M \times M$ matrix,
  * $\mathsf{U}$ is a $N \times M$ matrix (and $\mathsf{U}^T$ is a $M \times N$ matrix), and
  * $\mathbf{b}$ is the $M$-dimensional vector of the given values

It provides the $\mathbf{x}$ that minimizes the residual

$$
\mathbf{r} := |\mathsf{A}\mathbf{x} - \mathbf{b}|.
$$

### Linear least-squares fitting

This is the *linear least-squares fitting problem*: Given data points $(x_i, y_i)$, fit to a linear model $y(x)$, which can be any linear combination of functions of $x$.

For example:

$$
y(x) = a_1 + a_2 x + a_3 x^2 + \dots + a_M x^{M-1}
$$

or in general
$$
y(x) = \sum_{k=1}^M a_k X_k(x)
$$

The goal is to determine the coefficients $a_k$.

Define the **merit function**
$$
\chi^2 = \sum_{i=1}^N \left[ \frac{y_i - \sum_{k=1}^M a_k X_k(x_i)}{\sigma_i}\right]^2
$$
(sum of squared deviations, weighted with standard deviations $\sigma_i$ on the $y_i$).

Best parameters $a_k$ are the ones that *minimize $\chi^2$*.

*Design matrix* $\mathsf{A}$ ($N \times M$, $N \geq M$), vector of measurements $\mathbf{b}$ ($N$-dim) and parameter vector $\mathbf{a}$ ($M$-dim):

\begin{align}
A_{ij} &= \frac{X_j(x_i)}{\sigma_i}\\
b_i &= \frac{y_i}{\sigma_i}\\
\mathbf{a} &= (a_1, a_2, \dots, a_M)
\end{align}

Minimum occurs when the derivative vanishes:
$$
0 = \frac{\partial\chi^2}{\partial a_k} = \sum_{i=1}^N {\sigma_i}^{-2} \left[ y_i - \sum_{k=1}^M a_k X_k(x_i) \right] X_k(x_i), \quad 1 \leq k \leq M
$$
($M$ coupled equations)
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \mathsf{\beta}
\end{align}
with the $M \times M$ matrix
\begin{align}
\alpha_{kj} &= \sum_{i=1}^N \frac{X_j(x_i) X_k(x_i)}{\sigma_i^2}\\
\mathsf{\alpha} &= \mathsf{A}^T \mathsf{A}
\end{align}
and the vector of length $M$
\begin{align}
\beta_{k} &= \sum_{i=1}^N \frac{y_i X_k(x_i)}{\sigma_i^2}\\
\mathsf{\beta} &= \mathsf{A}^T \mathbf{b}
\end{align}

The inverse of $\mathsf{\alpha}$ is related to the uncertainties in the parameters:
$$
\mathsf{C} := \mathsf{\alpha}^{-1}
$$
in particular
$$
\sigma(a_i) = C_{ii}
$$
(and the $C_{ij}$ are the co-variances).

#### Solution of the linear least-squares fitting problem with SVD
We need to solve the overdetermined system of $M$ coupled equations
\begin{align}
\sum_{j=1}^{M} \alpha_{kj} a_j &= \beta_k\\
\mathsf{\alpha}\mathbf{a} = \mathsf{\beta}
\end{align}

SVD finds $\mathbf{a}$ that minimizes
$$
\chi^2 = |\mathsf{A}\mathbf{a} - \mathbf{b}|
$$

(proof in _Numerical Recipes_ Ch 2.)

The errors are
$$
\sigma^2(a_j) = \sum_{i=1}^{M} \left(\frac{V_{ji}}{w_i}\right)^2
$$
(see _Numerical Recipes_ Ch. 15)

#### Example
Synthetic data

$$
y(x) = 3\sin x - 2\sin 3x + \sin 4x
$$

with noise $r$ added (uniform in range $-5 < r < 5$).

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import numpy as np
```

```python
def signal(x, noise=0):
    r = np.random.uniform(-noise, noise, len(x))
    return 3*np.sin(x) - 2*np.sin(3*x) + np.sin(4*x) + r
```

```python
X = np.linspace(-10, 10, 500)
Y = signal(X, noise=5)
```

```python
plt.plot(X, Y, 'r-', X, signal(X, noise=0), 'k--')
```

```python
def fitfunc(x, a):
    return a[0]*np.cos(x) + a[1]*np.sin(x) + \
           a[2]*np.cos(2*x) + a[3]*np.sin(2*x) + \
           a[4]*np.cos(3*x) + a[5]*np.sin(3*x) + \
           a[6]*np.cos(4*x) + a[7]*np.sin(4*x)

def basisfuncs(x):
    return np.array([np.cos(x), np.sin(x),
                     np.cos(2*x), np.sin(2*x),
                     np.cos(3*x), np.sin(3*x),
                     np.cos(4*x), np.sin(4*x)])
```

```python
M = 8
sigma = 1.
alpha = np.zeros((M, M))
beta = np.zeros(M)
for x in X:
    Xk = basisfuncs(x)
    for k in range(M):
        for j in range(M):
            alpha[k, j] += Xk[k]*Xk[j]
for x, y in zip(X, Y):
    beta += y * basisfuncs(x)/sigma
```

Now use SVD!

```python

```

In this case, the singular values do not immediately show if any basis functions are superfluous (this would be the case for values close to 0).

```python
w
```

... nevertheless, remember to routinely mask any singular values or close to singular values:

```python

```

Compare the fitted values to the original parameters $a_j = 0, +3, 0, 0, 0, -2, 0, +1$.

```python

```

```python
plt.plot(X, fitfunc(X, a_values), 'b-', label="fit")
plt.plot(X, signal(X, noise=0), 'k--', label="signal")
plt.legend(loc="best", fontsize="small")
```

```python

```


```python
import sympy as sym
from scipy import integrate
import numpy as np
import matplotlib.pyplot as plt
```

```python
S, I, R = sym.Function("S"), sym.Function("I"), sym.Function("V")
N, mu, alpha, beta, t = sym.symbols("N, mu, alpha, beta, t")
```

```python
eq1 = sym.Derivative(S(t), t) - (- alpha * S(t) * I(t) - mu * R(t))
eq2 = sym.Derivative(I(t), t) - (alpha * I(t) * S(t) / N - beta * I(t))
eq3 = sym.Derivative(R(t), t) - (beta * I(t) + mu * R(t))
```

```python
sym.dsolve((eq1, eq2, eq3))
```

Further investigation shows that an exact solution to this system of differential equations is difficult. Let us do this numerically:

```python
def dx(x, t, alpha, beta, mu):

    return (- alpha * x[1] * x[0] - mu * x[0],
            alpha * x[1] * x[0]  - beta * x[1],
            beta * x[1] + mu * x[0])
```

```python
alpha = 1 / 1000  # Every 1000 interactions leads to infection
beta = 1 / 5  # take 5 time units to recover
N = 10 ** 4  # Population of 10 thousand people
mu = 0  # 0 vaccination percentage

ts = np.linspace(0, 10, 5000)
xs = integrate.odeint(func=dx, y0=np.array([N - 1, 1, 0]), t=ts, args=(alpha, beta, mu))
S, I, R = xs.T
plt.figure()
plt.plot(ts, S, label="Susceptibles")
plt.plot(ts, I, label="Infected")
plt.plot(ts, R, label="Recovered")
plt.legend()
plt.title(f"$\max(I)={round(max(I))}$ ($\\alpha={alpha}$, $\\beta={beta}$, $\mu={mu}$)")
plt.savefig("base_scenario.pdf");
```

```python
mu = 1 / 2  # Vaccinate half the population
ts = np.linspace(0, 10, 5000)
xs = integrate.odeint(func=dx, y0=np.array([N - 1, 1, 0]), t=ts, args=(alpha, beta, mu))
S, I, R = xs.T
plt.figure()
plt.plot(ts, S, label="Susceptibles")
plt.plot(ts, I, label="Infected")
plt.plot(ts, R, label="Recovered")
plt.legend()
plt.title(f"$\max(I)={round(max(I))}$ ($\\alpha={alpha}$, $\\beta={beta}$, $\mu={mu}$)")
plt.savefig("moderate_vaccination_rate.pdf");
```

```python
mu = 99 / 100  # Vaccinate 99% of the population
ts = np.linspace(0, 10, 5000)
xs = integrate.odeint(func=dx, y0=np.array([N - 1, 1, 0]), t=ts, args=(alpha, beta, mu))
S, I, R = xs.T
plt.figure()
plt.plot(ts, S, label="Susceptibles")
plt.plot(ts, I, label="Infected")
plt.plot(ts, R, label="Recovered")
plt.legend()
plt.title(f"$\max(I)={round(max(I))}$ ($\\alpha={alpha}$, $\\beta={beta}$, $\mu={mu}$)")
plt.savefig("high_vaccination_rate.pdf");
```

```python
vaccination_rates = np.linspace(0, 1, 500)
max_percent_of_infected = []
for mu in vaccination_rates:
    xs = integrate.odeint(func=dx, y0=np.array([N - 1, 1, 0]), t=ts, args=(alpha, beta, mu))
    S, I, R = xs.T
    max_percent_of_infected.append(max(I) / N)
plt.figure()
plt.plot(vaccination_rates, max_percent_of_infected)
plt.xlabel("$\mu$")
plt.ylabel("% of population infected")
plt.savefig("effect_of_vaccination_rate.pdf");
```


# Gradient descent

## Simple linear regression

In a previous notebook, we solved the problem of simple linear regression - finding a straight line that best fits a data set with two variables.
In that case, we were able to find the exact solution.
In this notebook, we'll use a common technique to approximate that solution.

Why would we want to approximate a solution when we can easily find an exact solution?
We don't - it's just that the technique we discuss here can also be used in situations where we can't find an exact solution or don't want to try, for whatever reason.
The approximation technique is called gradient descent.

### Preliminaries

```python
# numpy efficiently deals with numerical multi-dimensional arrays.
import numpy as np

# matplotlib is a plotting library, and pyplot is its easy-to-use module.
import matplotlib.pyplot as pl

# This just sets the default plot size to be bigger.
pl.rcParams['figure.figsize'] = (16.0, 8.0)
```

### Simple linear regression model
In simple linear regression, we have some data points $(x_i, y_i)$, and we decide that they belong to a straight line with a little bit of error involved.
Straight lines in two dimensions are of the form $y = mx + c$, and to fit a line to our data points we must find appropriate values for $m$ and $c$.
Numpy has a function called `polyfit` that finds such values for us.

```python
w = np.arange(1.0, 16.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)

m, c = np.polyfit(w, d, 1)
print("Best fit is m = %f and c = %f" % (m, c))

# Plot the best fit line.
pl.plot(w, d, 'k.', label='Original data')
pl.plot(w, m * w + c, 'b-', label='Best fit: $%0.1f x + %0.1f$' % (m,c))
pl.legend()
pl.show()
```

### Gradient descent
In gradient descent, we select a random guess of a parameter and iteratively improve that guess.
For instance, we might pick $1.0$ as our initial guess for $m$ and then create a `for` loop to iteratively improve the value of $m$.
The way we improve $m$ is to first take the partial derivative of our cost function with respect to $m$.

### Cost function
Recall that our cost function for simple linear regression is:

$$
Cost(m, c) = \sum_i (y_i - mx_i - c)^2
$$

### Calculate the partial derivatives
We calculate the partial derivative of $Cost$ with respect to $m$ while treating $c$ as a constant.
Note that the $x_i$ and $y_i$ values are all just constants.
We'll also calculate the partial derivative with respect to $c$ here.

$$
\begin{align}
Cost(m, c)                       &= \sum_i (y_i - mx_i - c)^2      \\[1cm]
\frac{\partial Cost}{\partial m} &= \sum 2(y_i - m x_i -c)(-x_i) \\
                                 &= -2 \sum x_i (y_i - m x_i -c) \\[0.5cm]
\frac{\partial Cost}{\partial c} & = \sum 2(y_i - m x_i -c)(-1) \\
                                 & = -2 \sum (y_i - m x_i -c)   \\
\end{align}
$$

### Code the partial derivatives
Once we've calculated the partial derivatives, we'll code them up in python.
Here we create two functions, each taking four parameters.
The first two parameters are arrays with our $x_i$ and $y_i$ data set values.
The second two are our current guesses for $m$ and $c$.

```python
def grad_m(x, y, m, c):
  return -2.0 * np.sum(x * (y - m * x - c))
```

```python
def grad_c(x, y, m , c):
  return -2.0 * np.sum(y - m * x - c)
```

### Iterate
Now we can run our gradient descent algorithm.
For $m$, we keep replacing its value with $m - \eta grad\_m(x, y, m, c)$ until it doesn't change.
For $c$, we keep replacing its value with $c - \eta grad\_c(x, y, m, c)$ until it doesn't change.

What is $\eta$? It is called the learning rate and we set it to a small value relative to the data points.

You can see on each iteration, $m$ and $c$ are getting closer to their true values.

```python
eta = 0.0001
m, c = 1.0, 1.0
delta = 0.0000001

mold, cold = m - 1.0, c - 1.0
i = 0
while abs(mold - m) > delta and abs(cold - c) > delta:
  mold, cold = m, c
  m = mold - eta * grad_m(w, d, mold, cold)
  c = cold - eta * grad_c(w, d, mold, cold)

  i = i + 1
  if i % 1000 == 0:
    print("m: %20.16f  c: %20.16f" % (m, c))
```

    m:   5.7149506777702488  c:   4.5518695120877757
    m:   5.5629397868829846  c:   6.1183980049248472
    m:   5.4852588608786546  c:   6.9189286986499470
    m:   5.4455621902673004  c:   7.3280175818245992
    m:   5.4252763139437015  c:   7.5370710455442040
    m:   5.4149097826482908  c:   7.6439019874021357
    m:   5.4096122559659570  c:   7.6984949605832087
    m:   5.4069051027099357  c:   7.7263931768575951
    m:   5.4055216875445105  c:   7.7406497821666767
    m:   5.4048147318094859  c:   7.7479352226744274
    m:   5.4044534617990276  c:   7.7516582438453998
    m:   5.4042688448364773  c:   7.7535607899748742

## Newton's method for square roots
Newton's method for square roots is a method for approximating the square root of a number $x$.
We begin with an initial guess $z_0$ of the square root - it doesn't have to be particularly good.
We then apply the following calculation repeatedly, to calculate $z_1$, then $z_2$, and so on:

$$
z_{i+1} = z_i - \frac{z_i^2 - x}{2z_i}
$$

### Coding the calculation
We can create a function that calculates the next value of $z$ based on the current value of $z$ as follows.

```python
def next_z(x, z):
  return z - (z**2 - x) / (2 * z)
```

### Calculating the square root of $x$
Suppose we want to calculate the square root of $x$.
We start with a random guess for the square root, $z_0$.
We then apply the `next_z` function repeatedly until the value of $z$ stops changing.
Let's create a function to do this.
We'll include the next_z function inside the `newtsqrt` function to make it one all-inclusive package.

```python
def newtsqrt(x):
  next_z = lambda x, z: z - (z**2 - x) / (2 * z)
  z = 2.0
  n = next_z(x, z)

  while z != n:
    z, n = n, next_z(x, n)
    print(z)

  return z

newtsqrt(11)
```

    3.75
    3.341666666666667
    3.316718620116376
    3.3166247916826186
    3.3166247903554

    3.3166247903554

### Comparison with the standard library
We can compare our square root method return value to the value calculated by Python's `math` standard library package. It has a `sqrt` function.

```python
import math
math.sqrt(11)
```

    3.3166247903554

### Being careful
Due to the complexities of floating point numbers, the `nextsqrt` function could get into an infinite loop.
For instance, calculating the square root of 10 gives an infinite loop.

```python
# Uncommenting will result in infinite loop.
# newtsqrt(10)
```

To counteract this problem, the condition of the loop is better written as:
```python
abs(z - n) > 0.001
```

## Gradient descent for square roots
Newton's method for square roots is efficient, but we can also use gradient descent to approximate the square root of a real number $x$.
Here, we use the following cost function.

$$
Cost(z \mid x) = (x - z^2)^2
$$

### Example value
Let's use it to calculate the square root of 20, i.e. $x = 20$.
Then the cost function is:

$$
Cost(z \mid x=20) = (20 - z^2)^2
$$

Our goal is to find the $z$ that minimises this.

### Plotting the cost function
Let's plot the cost function.
Given that we know the best $z$ will be between $4$ and $5$, we'll let $z$ range over 0 to 10.

```python
i = np.linspace(0.0, 10.0, 1000)
j = (20.0 - i**2)**2

pl.plot(i, j, 'k-', label='$(20-z^2)^2$')
pl.legend()
pl.show()
```

### The derivative
Looks like there's a low point at about $4.5$.
Let's take the derivative of the cost function with respect to $z$.

$$
\begin{align}
Cost(z) &= ( 20.0 - z^2 )^2 \\
\Rightarrow \frac{\partial Cost}{\partial z} &= 2(20.0 - z^2)(-2z) \\
                                              &= 4z(z^2 - 20) \\
                                              &= 4z^3 - 80z
\end{align}
$$

The derivative tells us what the slope of the tangent to the curve is at any point on the cost function.
What does that mean?
It means that if we pick a value of $z$, e.g. $8.0$, that the derivative tells us that a line going through the point $(8.0, (20.0 - (8.0)^2)^2)$ with the slope $4(8.0)^3 - 80(8.0)$ perfectly touches the graph above.
Let's plot that.

When you simply, the point $(8.0, (20.0 - (8.0)^2)^2)$ becomes $(8,1936)$.
The slope is $4(8.0)^3 - 80(8.0)$ which when simplified becomes $1408$.
So, the claim is that the line with slope $1408$ going through the point $(8,1936)$ touches the graph.
To calculate the equation of the line, we'll use $(y - y_1) = m(x - x_1)$:

$$
y - 1936 = 1408(x - 8) \\
\Rightarrow y = 1408x - 11264 + 1936 \\
\Rightarrow y = 1408x - 9328
$$

Let's plot that line and the cost function together.

```python
i = np.linspace(0.0, 10.0, 1000)
j = (20.0 - i**2)**2
k = 1408 * i - 9328

pl.plot(i, j, 'k-', label='$(20-z^2)^2$')
pl.plot(i, k, 'b-', label='$1408z - 9328$')
pl.legend()
pl.show()
```

### Why do we care about the slope?
It's a bit hard to see, but the blue line is perfectly touching the curve.
We care about this because the slope of the blue line tells us in which way to change $z$ in order to make the cost less.
If we increase $z$ the cost goes up.
If we decrease it the cost goes down.

### Gradient descent
Let's use the gradient descent algorithm to calculate the best $z$.
We'll start with the guess $z=8$, and then use the derivative to move $z$ ever so slightly in the direction that decreases the cost.
By ever so slightly, we mean $0.001$ times the slope:

$$
\begin{align}
z_{i+1} &= z_i - \eta \frac{\partial Cost}{\partial z} \\
        &= z_i - (0.001) (4 z_i^3 - 80 z_i)
\end{align}
$$

So, for our initial guess $z_0 = 8.0$ we get:

$$
\begin{align}
z_1 &= 8.0 - (0.001) (4 (8.0)^3 - 80 (8.0)) \\
    &= 8.0 - 1.408 = 6.592
\end{align}
$$

Let's code it up.

```python
def next_z(z, x, eta=0.001):
  return z - eta * (4.0 * z**3 - 80 * z)

def sqrt_grad_desc(x, z, verbose=False):
  while abs(z - next_z(z, x)) > 0.001:
    if verbose:
      print("Current: %14.8f\tNext: %14.8f" % (z, next_z(z, x)))
    z = next_z(z, x)
  return z

ans =sqrt_grad_desc(20.0, 8.0, True)
print("Square root:", ans, "\tSquared:", ans**2)
```

    Current:     8.00000000	Next:     6.59200000
    Current:     6.59200000	Next:     5.97355269
    Current:     5.97355269	Next:     5.59881186
    Current:     5.59881186	Next:     5.34469983
    Current:     5.34469983	Next:     5.16157297
    Current:     5.16157297	Next:     5.02444369
    Current:     5.02444369	Next:     4.91903017
    Current:     4.91903017	Next:     4.83645229
    Current:     4.83645229	Next:     4.77084541
    Current:     4.77084541	Next:     4.71815685
    Current:     4.71815685	Next:     4.67548576
    Current:     4.67548576	Next:     4.64069702
    Current:     4.64069702	Next:     4.61218330
    Current:     4.61218330	Next:     4.58871218
    Current:     4.58871218	Next:     4.56932433
    Current:     4.56932433	Next:     4.55326362
    Current:     4.55326362	Next:     4.53992784
    Current:     4.53992784	Next:     4.52883326
    Current:     4.52883326	Next:     4.51958845
    Current:     4.51958845	Next:     4.51187478
    Current:     4.51187478	Next:     4.50543157
    Current:     4.50543157	Next:     4.50004463
    Current:     4.50004463	Next:     4.49553736
    Current:     4.49553736	Next:     4.49176369
    Current:     4.49176369	Next:     4.48860255
    Current:     4.48860255	Next:     4.48595333
    Current:     4.48595333	Next:     4.48373229
    Current:     4.48373229	Next:     4.48186965
    Current:     4.48186965	Next:     4.48030717
    Current:     4.48030717	Next:     4.47899619
    Current:     4.47899619	Next:     4.47789603
    Square root: 4.477896027970839 	Squared: 20.05155283731702

### A question
Let's try some other initial guesses: 4.0, 1.0 and -1.0.
Can you explain the square root returned with -1.0?

```python
print("With initial guess %6.2f: %10.6f" % (4.0, sqrt_grad_desc(20.0, 4.0, False)))
print("With initial guess %6.2f: %10.6f" % (1.0, sqrt_grad_desc(20.0, 1.0, False)))
print("With initial guess %6.2f: %10.6f" % (-1.0, sqrt_grad_desc(20.0, -1.0, False)))
```

    With initial guess   4.00:   4.465951
    With initial guess   1.00:   4.465963
    With initial guess  -1.00:  -4.465963

### End


## Task 01:
Use the Multi Segment Simpsons 1/3 rule for finding Integral of the following function within the closed interval of $[0,1]$ and Plot the Relative Absolute Error for the different number of segments. The the integrad is as follows:
$$ f(x) = 15\sqrt{x^{3}} + 8\sqrt[\leftroot{-1}\uproot{2}\scriptstyle 3]{x^{2}} + x $$

```
from sympy import sympify, symbols, integrate
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

x = symbols('x')
# original simpsons 1/3 method
def simpsons_one_third(f= None, a = 0, b = 0):
    #Fixing the values of X
    X = np.array([[a,(a+b)/2,b]]).T
    #Fixing the values of Y
    Y = np.vectorize(lambda t: f.subs(x,t))(X)
    #Taking the powers of X
    X = np.concatenate((X**0,X**1,X**2),axis = 1)
    #Solving for A
    A = np.linalg.inv(X)@Y
    #Substituted values of X after integration stored in b_a
    b_a = np.array([(b-a),(b**2-a**2)/2,(b**3-a**3)/3])
    result = b_a@A
    return float(result)

def simpsons_three_eight(f= None, a = 0, b = 0):
    #setting the value of h
    h = (b-a)/3
    #Fixing the values of X
    X = np.array([[a+i*h for i in range(3+1)]]).T
    #Fixing the values of Y
    Y = np.vectorize(lambda t: f.subs(x,t))(X)
    #Taking the powers of X
    X = np.concatenate(tuple([X**i for i in range(4)]),axis = 1)
    #Solving for A
    A = np.linalg.inv(X)@Y
    #Substituted values of X after integration stored in b_a
    b_a = np.array([(b**i-a**i)/i for i in range(1,5)])
    result = b_a@A
    return float(result)

# multisegment simpsons 1/3 method
def multi_segment_simpsons_one_third(f, a=0, b=0, n=2, verbose = False):
    n = n//2
    h = (b-a)/n
    X = [a+i*h for i in range(n+1)]
    result = 0
    for i in range(n):
        result += simpsons_one_third(f,X[i],X[i+1])
    return result

# multisegment simpsons 3/8 method
def multi_segment_simpsons_three_eight(f, a=0, b=0, n=3, verbose = False):
    n = n//3
    h = (b-a)/n
    X = [a+i*h for i in range(n+1)]
    result = 0
    for i in range(n):
        result += simpsons_three_eight(f,X[i],X[i+1])
    return result

def combined_simpsons(f, a=0, b=0, n=5, verbose = False):
    n = n//5
    h = (b-a)/n
    X = [a+i*h for i in range(n+1)]
    result = 0
    for i in range(n):
        s = (X[i+1]-X[i])/5*2
        result += simpsons_one_third(f,X[i],X[i]+s)
        result += simpsons_three_eight(f,X[i]+s,X[i+1])
    return result
```

```
def simpsons_13(f= None, a = 0, b = 0):
    #Fixing the values of X
    X = np.array([[a,(a+b)/2,b]]).T
    #Fixing the values of Y
    Y = np.vectorize(lambda t: f.subs(x,t))(X)
    h = (b-a)/2
    result = h/3 *(Y[0]+4*Y[1]+Y[2])
    return float(result)

def simpsons_38(f= None, a = 0, b = 0):
    #setting the value of h
    h = (b-a)/3
    #Fixing the values of X
    X = np.array([[a+i*h for i in range(3+1)]]).T
    #Fixing the values of Y
    Y = np.vectorize(lambda t: f.subs(x,t))(X)
    result = h*3/8 * (Y[0]+3*Y[1]+3*Y[2]+Y[3])
    return float(result)

def trap(f= None, a = 0, b = 0):
    fa = f.subs(x,a)
    fb = f.subs(x,b)
    result = (b-a)/2*(fa+fb)
    return float(result)
```

```
f = '-9.1688*10**-6*x**3+2.7961*10**-3*x**2-2.8487*10**-1*x+9.6778'
[a,b] = [0,100]
f = sympify(f)
print(f'Numerical Prediction with simpsons 1/3 rule: {simpsons_13(f, a= a, b = b)}')
```

    Numerical Prediction with simpsons 1/3 rule: 246.24333333333337

# Input Parameters

```
# Testing Stub
f = '15*(x^(1.5)) + 8*(x ^(0.6666667)) + x'
[a,b] = [0,1]
f = sympify(f)
print(f'Given Integrad: {f}')
integral_f = integrate(f, x)
print(f'Original Integration: {integral_f}')
original_value = integral_f.subs(x, 1)- integral_f.subs(x, 0)
print('Original Result:', original_value)
print(f'Numerical Prediction with simpsons 1/3 rule: {simpsons_one_third(f, a= 0, b = 1)}')
print(f'Numerical Prediction with multiple segment simpsons 1/3 rule: {multi_segment_simpsons_one_third(f, a= 0, b = 1, n= 16, verbose = False)}')
print(f'Numerical Prediction with simpsons 3/8 rule: {simpsons_three_eight(f, a= 0, b = 1)}')
print(f'Numerical Prediction with multiple segment simpsons 3/8 rule: {multi_segment_simpsons_three_eight(f, a= 0, b = 1, n= 15, verbose = False)}')
print(f'Numerical Prediction with combined simpsons rule: {combined_simpsons(f, a= 0, b = 1, n= 15, verbose = False)}')

```

# Error Calculation:

```
errors_one_third = []
errors_three_eight = []
errors_combined = []
n_value_one_third = []
n_value_three_eight = []
n_value_combined = []
for i in range(2, 61, 2):
    errors_one_third.append((original_value - multi_segment_simpsons_one_third(f, a= 0, b = 1, n= i, verbose = False))/original_value*100)
    n_value_one_third.append(i)

for i in range(3, 61, 3):
    errors_three_eight.append((original_value - multi_segment_simpsons_three_eight(f, a= 0, b = 1, n= i, verbose = False))/original_value*100)
    n_value_three_eight.append(i)

for i in range(5, 61, 5):
    errors_combined.append((original_value - combined_simpsons(f, a= 0, b = 1, n= i, verbose = False))/original_value*100)
    n_value_combined.append(i)

```

# Error Plotting

```
plt.Figure(facecolor='black', linewidth=2)
plt.rcParams["figure.figsize"] = (20,15)
# plt.plot(n_value, errors, 'r-*', markercolor = 'b')
#plt.plot(n_value, errors, color='red', linestyle='dashed', marker='*', markerfacecolor='blue', markersize=10)
plt.plot(n_value_one_third, errors_one_third, "k--", label="multiple segment simpsons 1/3 rule",marker='*',color='red')
plt.plot(n_value_three_eight, errors_three_eight, "k:", label="multiple segment simpsons 3/8 rule",marker='.',color='green')
plt.plot(n_value_combined, errors_combined, "k", label="combined simpsons rule",marker='+',color='blue')
plt.grid(axis = 'both')
plt.xlabel('Intercepts: (n)')
plt.ylabel('Relative Approximate Error')
plt.title('Error Analysis of Simpsons Methods')
plt.legend(loc="upper right", shadow=True, fontsize="large")
plt.show()
```

```
f = '15*(x^(1.5)) + 8*(x ^(0.6666667)) + x'

```


# Excercises : Finite-difference basics

```python
%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

## Create a numpy array representing a one-dimensional mesh

```python
# Create a uniform mesh for the domain [-1,1]
# x =
```

## Discretize a function on the mesh

Let's start with a very simple function

\begin{equation}
f(x) = x
\end{equation}

We know the function, $f(x)$, is continuous over the entire domain, $(-\infty, \infty)$. Using a numpy array, how do we define this function on the discrete mesh we created in the previous step?

```python
#f =
```

```python
# Let's plot the function
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(9, 6))

line = ax.plot(f, x, lw=2, color='b', label=r'$f(x) = x')

ax.set_xlim((0.0, 1.0))

ax.set_title(r'Plot of the function $f(x)$', fontsize=15)
ax.legend(loc=4, fontsize=15)
ax.set_xlabel(r'$x$', fontsize=15.0)
ax.set_ylabel(r'$f(x)$', fontsize=15.0)
```

## Change the number of points in the mesh

Often we will want to study the behavior of a numerical method we different levels of refinement. For a uniform mesh, refinement simply means increasing the total number of discrete points (nodes) used to define the mesh of the computational domain. Good practice is to use three levels of successive refinement. We will define a base number of mesh points, $N$, and the a **coarse** mesh with half the number of points, and a **fine** mesh with double the number of points.

To simplify the notation, let's refer to these as $N_0$, $N_1$, and $N_2$

\begin{equation}
N_\textrm{coarse} = N_0 = \frac{N}{2}
\end{equation}

\begin{equation}
N_\textrm{base} = N_1 = N
\end{equation}

\begin{equation}
N_\textrm{fine} = N_2 = 2 N
\end{equation}

```python
# Create three uniform meshs for the domain [-1,1]
#   - x_coarse = x_0
#   - x_base = x_1
#   - x_fine = x_2
```

## Discretize a function on each mesh

Let's use a slightly more complex function,

\begin{equation}
f(x) = x^2
\end{equation}

Recall we are interested in how the derivatives of a function are computed on a discrete domain (mesh) using a finite-difference approximation. Since the derivative of $f(x) = x^2$, is easy to compute analytically, $f^{\prime}(x) = x$, we can use it to confirm wether or not our finite-difference approximation is qualitiatively correct.

```python
# f_0 =
# f_1 =
# f_2 =
```

## Compute the finite-difference approximation of the first derivative

Compute $f^{\prime}(x)$ at each mesh point using the following finite-difference approximation for the first derivative:

\begin{equation}
\frac{\textrm{d} f}{\textrm{d} x} \approx \frac{ f(x_i + \Delta x) - f(x_i) }{\Delta x} = \frac{f_{i+1} - f_i }{\Delta x}
\end{equation}

To analyze how our approximations might change for different number of mesh points, compute the first derivative for each of the different size meshes.

```python
# dfdx_1 =
# dfdx_2 =
# dfdx_3 =
```

## How do we know our finite-difference approximations are accurate?

```python

```

## Convert the finite-difference approximation into a matrix operator

As discussed during lecture, we can write the finite-difference approximations at each point in the mesh as a matrix operator, $\mathbf{A}$. Consider the 1-D, advection equation,

\begin{equation}
\frac{\partial u}{\partial t} + a \frac{\partial u}{\partial x} = 0
\end{equation}

Using the method of lines, we can write a numerical approximation to the 1-D, advection equation like

\begin{equation}
\frac{\textrm{d} \mathbf{u} }{\textrm{d} t} = \mathbf{A} \mathbf{u}
\end{equation}

where the vector $\mathbf{u} = [u(x_0,t), u(x_1,t), \dots, u(x_N,t)]$ where the matrix $\mathbf{A}$ is

\begin{equation}
\mathbf{A} = -a \mathcal{D}
\end{equation}

where $\mathcal{D}$ is the difference-operator, sometimes it is written as $\delta_x$ or as $\mathcal{D}_x$, where the subscript means the difference-operator is an approximation of the first-derivative with respect to $x$, $\delta_{xx}$ would be an approximationof the second-derivative with respect to $x$.

Using the approximation of the first derivative,

\begin{equation}
\frac{\textrm{d} u}{\textrm{d} x} \approx \frac{ u(x_i + \Delta x) - u(x_i) }{\Delta x} = \frac{u_{i+1} - u_i }{\Delta x}
\end{equation}

what is the matrix operator for the one-dimensional, advection equation. Assume the initial condition for $u(x,t)$ is given as

\begin{equation}
u(x,0) = f(x)
\end{equation}

where $f(x) = x^2$

```python
# what is Dx?
```

## Compute the RHS using the matrix operator

Compute $\mathbf{A}\mathbf{u}$. There are a few ways to do this, and which way is best really depends on the scale of the problem we are trying to solve. Recall that $\mathbf{A}$ is a sparse matrix with most of the elements in the matrix being zero. Is there some pattern to the non-zero entries of $\mathbf{A}$ we can use to design a more compact way to represesnting the matrix? What about the fact that for periodic boundaries, the matrix $\mathbf{A}$ is circulant?

The answer to all of those questions is yes. There are ways to effeciently represent the sparse matrix $\mathbf{A}$, and yes we should do this (if only because it real-world applications you must, and because it is good practice to write memory efficient programs).

**But**, let's start with a naive implementation. Use $N=4$.

How do you know it is correct?

```python
# A = -a D
# u = [u0, u1, u2, ..., uN]

# Au = -a * dudx
```

## Can A be represented as a banded-matrix?

A banded matrix is a matrix whose only non-zero elements are along the diagonals of the matrix. For non-periodic boundary conditions, the matrix $\mathbf{A}$ is banded. (We can modify how we treat the periodicity of the domain if we want to use a periodic matrix, we just can not encode all that information into the matrix $\mathbf{A}$.)

```python

```


# Solution {-}

The stationary process $X(t)$ has an autocorrelation function of the form:

\begin{equation}
  R_X(t) = \sigma^2 e^{-\beta|\tau|}
\end{equation}

Another process $Y(t)$ is related to $X(t)$ by the deterministic equation:

\begin{equation}
  Y(t) = aX(t) + b
\end{equation}

$X(t)$ is stationary, so Y(t) is also stationary.

\begin{align}
  R_Y(t) =& E[Y(t) \cdot Y (t + \tau)] \\
         =& E[(aX(t) + b)(aX(t + \tau) + b)] \\
         =& E[(a^2 X(t)X(t + \tau)] + E[abX(t)] + E[baX(t)] + E[b^2] \\
         =& a^2 R_X(t) + b^2 \\
         =& a^2 \sigma^2 e^{-\beta|\tau|} + b^2 \\
\end{align}


```python
import numpy as np
import sympy as sp
import math

import yx_ODE as p1
import unstable_ODE as p2

import matplotlib.pyplot as plt

# Needed only in Jupyter to render properly in-notebook
%matplotlib inline
```

# Chinmai Raman

## Homework 6

## C.3 Solving an ODE Numerically and Symbolically

$\frac{dy}{dx} = \frac{1}{2(y-1)},   y(0) = 1 + \sqrt{\epsilon},   x \subset [0,4],   \epsilon = 10^{-3}$

The approximation of y for x = 4 using Euler's method given a step size of 1:

```python
print p1.euler(1, 1 + np.sqrt(1e-3))[1][-1]
```

    16.9375021945

The approximation of y for x = 4 using Euler's method given a step size of 0.25:

```python
print p1.euler(0.25, 1 + np.sqrt(1e-3))[1][-1]
```

    5.43162981571

The approximation of y for x = 4 using Euler's method given a step size of 0.01:

```python
print p1.euler(0.01, 1 + np.sqrt(1e-3))[1][-1]
```

    3.01196145341

The exact solution to the differential equation can be calculated symbolically. The value of C1 depends on the epsilon value.

```python
p1.sym_solve()
```

    f(x) == sqrt(C1 + x) + 1

The graph below shows Euler's method implemented for a step size of 1 (red), 0.25 (green), and 0.01 (blue). It also shows the exact solution (yellow). The problem is hard to solve numerically because it requires an extremely large number of points to come even close to estimating the exact solution.

```python
p1.graph_euler()
```

## C.4 Demonstrating instability of an ODE

$u^` = \alpha u, u(0) = u_0$

The graphs below show that $u_k = (1 + \alpha \Delta t)^k u_0$ and that the numerical solution of the above ODE problem will oscillate if $\Delta t > \frac{-1}{\alpha}$

$\alpha = -1, \Delta t = 1.1$

```python
p2.graph(-1, 1, 1.1, 0, 20)
```

$\alpha = -1, \Delta t = 1.5$

```python
p2.graph(-1, 1, 1.5, 0, 20)
```

$\alpha = -1, \Delta t = 1.9$

```python
p2.graph(-1, 1, 1.9, 0, 120)
```

$\alpha = -1, \Delta t = 2.1$

```python
p2.graph(-1, 1, 2.1, 0, 120)
```

$\alpha = -1, \Delta t = 1$

```python
p2.graph(-1, 1, 1, 0, 120)
```

These graphs show that as $ \Delta t$ increases (when $\frac{-1}{\alpha} < \Delta t < \frac{-2}{\alpha}$), the value of u as $k -> \infty$ converges to 0, but at a decreasing rate and with oscillations that fall lower into the fourth quadrant. This is because the term $(1 + \alpha \Delta t)^k$ will converge to 0 when the value inside the parenthesis is a fraction. It follows that when the value inside the parenthesis is $>1$, the sequence will diverge, as is the case when $\Delta t > \frac{-2}{\alpha}$


# Computing gradients in tensorflow

## Partial derivatives using pure python

Let's say we have a derivable function:

\begin{equation}
f(x) = 2 x_1^2 + 3 x_1 x_2
\end{equation}

```python
def f(x1, x2):
    return 2 * x1 ** 2 + 3 * x1 * x2
```

It's easy to find analytically the derivative of this function:

\begin{align}
\frac{\partial{f}}{\partial x_1} & = 4x_1 + 3x_2\\
\frac{\partial{f}}{\partial x_2} & = 3x_1
\end{align}

So, for the point $x=(2,1)$, the result will be $(11,6)$. To check that everything goes as expected, we can compute the partial derivatives with regard to both variables using the definition:

\begin{equation}
\frac{\partial}{{\partial x}}f \left( x \right) = \mathop {\lim }\limits_{\epsilon \to 0} \frac{{f\left( {x + \epsilon } \right) - f\left( x \right)}}{\epsilon }
\end{equation}

```python
x1, x2 = 2, 1
eps = 1e-04
```

```python
(f(x1 + eps, x2) - f(x1, x2)) / (eps)
```

    11.000200000026439

```python
(f(x1, x2 + eps) - f(x1, x2)) / (eps)
```

    6.00000000000378

Great!

## Partial derivatives using tensorflow

We will do the same, but this time we will use tensorlfow to calculate the results. It may not be as interesting, but it will certainly be more efficient.

```python
import tensorflow as tf
```

```python
x1, x2 = tf.Variable(2.), tf.Variable(1.)
```

```python
with tf.GradientTape() as tape:
    y = f(x1, x2)
gradients = tape.gradient(y, [x1, x2])
```

```python
[g.numpy() for g in gradients]
```

    [11.0, 6.0]

Within the `tf.GradientTape` context, tensorflow will track each operation applied to any variable. But be careful! To save memory, tensorflow will remove the tape contents after calling the `.gradient()` method. To avoid this, you can explicitly indicate that you do not want them to disappear (with the `persistent=True` parameter of the `GradientTape`), but try not to do so if there is no good reason, or even remove it from memory once you've done with it.

By default, the tape will record all the operations involving variables (because de default value for the tape's `watch_accessed_variables` parameter is `True`). We can track the operations that involves a constant adding `tape.watch(my_constant)` at the begining of the context, or setting `watch_accessed_variables=False` and select the variables we want to track through the `watch` method. This is useful if we want to add information about the variation of the inputs in our loss function.

## Higher order derivatives

We can even compute second (or higher) order derivatives by nesting tapes. For example:

```python
def f(x):
    return 5*x**3
```

In this case,

\begin{equation}
\frac{\partial f}{\partial x} = 15 x ^2 \\
\frac{\partial^2 f}{\partial x^2} = 30 x
\end{equation}

```python
x = tf.Variable(0.1)
with tf.GradientTape() as tape1:
    with tf.GradientTape() as tape2:
        y = f(x)
    dy_dx  = tape2.gradient(y, x)
d2y_dx2 = tape1.gradient(dy_dx, x)

print(f"dy/dx at x={x.numpy():.2f}: {dy_dx.numpy():.2f}")
print(f"d2y/dx2 at x={x.numpy():.2f}: {d2y_dx2.numpy():.2f}")
```

    dy/dx at x=0.10: 0.15
    d2y/dx2 at x=0.10: 3.00

## Derivatives of different variables

If we try to calculate the gradient of several variables separately, tensorlow will calculate the sum of the gradients.

**Tip:** Until now we have only used the `.gradient()` method with variables or lists of variables for its two main parameters. However, it also accepts dictionaries.

```python
x = tf.Variable(2.0)
with tf.GradientTape(persistent=True) as tape:
  y0 = x**2
  y1 = -4 * x

print(tape.gradient({'y0': y0, 'y1': y1}, x).numpy())
print(tape.gradient(y0, x).numpy(), tape.gradient(y1, x).numpy())
```

    0.0
    4.0 -4.0

However, if we compute the gradients of a single variable, contaning several components (all of them affected by the same calculations), we will get the gradients of each component.

```python
x = tf.linspace(-1.0, 1.0, 3)

with tf.GradientTape() as tape:
  tape.watch(x) # x is a constant
  y = tf.nn.sigmoid(x)

dy_dx = tape.gradient(y, x)
dy_dx.numpy()
```

    array([0.19661194, 0.25      , 0.19661193], dtype=float32)

## Jacobian

Now we know how to compute the derivatives of a single value regarding a set of variables. Let's see now how to compute the derivatives of a vector (two-dimensional tensor).

If you want to compute, for example, gradients for an array of losses, tensorflow will compute the gradients of the sum of all of them. To compute all the derivatives one step before, we will need to use the tape's `jacobian()` method.

```python
def f(x):
    x1 = 2 * x[0] ** 2
    x2 = x[1] ** 3
    x3 = x[2] + x[1]
    return tf.stack([x1, x2, x3])
```

\begin{equation}
\mathcal{J}_u(x_1, x_2, x_3) =
\begin{bmatrix}
  \frac{\partial u_1}{\partial x_1} &
    \frac{\partial u_1}{\partial x_2} &
    \frac{\partial u_1}{\partial x_3} \\[1ex] % <-- 1ex more space between rows of matrix
  \frac{\partial u_2}{\partial x_1} &
    \frac{\partial u_2}{\partial x_2} &
    \frac{\partial u_2}{\partial x_3} \\[1ex]
  \frac{\partial u_3}{\partial x_1} &
    \frac{\partial u_3}{\partial x_2} &
    \frac{\partial u_3}{\partial x_3}
\end{bmatrix}
\end{equation}

```python
x = tf.Variable([1.0, 1.0, 1.0])
with tf.GradientTape() as tape:
    y = f(x)
tape.jacobian(y, x)
```

    <tf.Tensor: shape=(3, 3), dtype=float32, numpy=
    array([[4., 0., 0.],
           [0., 3., 0.],
           [0., 1., 1.]], dtype=float32)>

## Derivatives involving matrix operations

We can also calculate the derivatives of any variable involved in matrix operations.

```python
W = tf.Variable(tf.random.normal((3, 1)), name='W')
b = tf.Variable(tf.zeros(1, dtype=tf.float32), name='b')
X = tf.constant([[1., 2., 3.], [4., 5., 6.]])
y_true = tf.constant([[5.], [16.]])

with tf.GradientTape(persistent=True) as tape:
  y = X @ W + b
  loss = tf.reduce_mean((y - y_true)**2)
```

```python
dloss_dW, dloss_db = tape.gradient(loss, [W, b])
print(dloss_dW.numpy(), dloss_db.numpy())
```

    [[ -79.16642]
     [-104.15854]
     [-129.15067]] [-24.992123]

This is especially useful when working with deep learning models, and of course, we can do exacly the same with the variable inside a keras layer/model:

```python
layer = tf.keras.layers.Dense(1, activation='relu')

with tf.GradientTape() as tape:
  # Forward pass
  y = layer(X)
  loss = tf.reduce_mean((y - y_true)**2)

# Calculate gradients with respect to every trainable variable
grad = tape.gradient(loss, layer.trainable_variables)
```

```python
[g.numpy() for g in grad]
```

    [array([[-41.165585],
            [-52.70707 ],
            [-64.24856 ]], dtype=float32),
     array([-11.541485], dtype=float32)]