## Computing Sums

Suppose we want to compute the sum of a sequence of numbers $x_0$, $x_1$, $x_2$, $x_3$, $\dots$, $x_n$. There are at least two approaches:

1. Compute the entire sequence, store it as a list $[x_0,x_1,x_2,\dots,x_n]$ and then use the built-in function `sum`.
2. Initialize a variable with value 0 (and name it `result` for example), create and add each element in the sequence to `result` one at a time.

The advantage of the second approach is that we don't need to store all the values at once. For example, here are two ways to write a function which computes the sum of squares.

For the first approach, use a list comprehension:

```python
def sum_of_squares_1(N):
    "Compute the sum of squares 1**2 + 2**2 + ... + N**2."
    return sum([n**2 for n in range(1,N + 1)])
```

```python
sum_of_squares_1(4)
```

    30

For the second approach, use a `for` loop with the initialize-and-update construction:

```python
def sum_of_squares_2(N):
    "Compute the sum of squares 1**2 + 2**2 + ... + N**2."
    # Initialize the output value to 0
    result = 0
    for n in range(1,N + 1):
        # Update the result by adding the next term
        result = result + n**2
    return result
```

```python
sum_of_squares_2(4)
```

    30

Again, both methods yield the same result however the second uses less memory!

## Computing Products

There is no built-in function to compute products of sequences therefore we'll use an initialize-and-update construction similar to the example above for computing sums.

Write a function called `factorial` which takes a positive integer $N$ and return the factorial $N!$.

```python
def factorial(N):
    "Compute N! = N(N-1) ... (2)(1) for N >= 1."
    # Initialize the output variable to 1
    product = 1
    for n in range(2,N + 1):
        # Update the output variable
        product = product * n
    return product
```

Let's test our function for input values for which we know the result:

```python
factorial(2)
```

    2

```python
factorial(5)
```

    120

We can use our function to approximate $e$ using the Taylor series for $e^x$:

$$
e^x = \sum_{k=0}^{\infty} \frac{x^k}{k!}
$$

For example, let's compute the 100th partial sum of the series with $x=1$:

```python
sum([1/factorial(k) for k in range(0,101)])
```

    2.7182818284590455

## Searching for Solutions

We can use `for` loops to search for integer solutions of equations. For example, suppose we would like to find all representations of a positive integer $N$ as a [sum of two squares](https://en.wikipedia.org/wiki/Sum_of_two_squares_theorem). In other words, we want to find all integer solutions $(x,y)$ of the equation:

$$
x^2 + y^2 = N
$$

Write a function called `reps_sum_squares` which takes an integer $N$ and finds all representations of $N$ as a sum of squares $x^2 + y^2 = N$ for $0 \leq x \leq y$. The function returns the representations as a list of tuples. For example, if $N = 50$ then $1^2 + 7^2 = 50$ and $5^2 + 5^2 = 50$ and the function returns the list `[(1, 7),(5, 5)]`.

Let's outline our approach before we write any code:

1. Given $x \leq y$, the largest possible value for $x$ is $\sqrt{\frac{N}{2}}$
2. For $x \leq \sqrt{\frac{N}{2}}$, the pair $(x,y)$ is a solution if $N - x^2$ is a square
3. Define a helper function called `is_square` to test if an integer is square

```python
def is_square(n):
    "Determine if the integer n is a square."
    if round(n**0.5)**2 == n:
        return True
    else:
        return False

def reps_sum_squares(N):
    '''Find all representations of N as a sum of squares x**2 + y**2 = N.

    Parameters
    ----------
    N : integer

    Returns
    -------
    reps : list of tuples of integers
        List of tuples (x,y) of positive integers such that x**2 + y**2 = N.

    Examples
    --------
    >>> reps_sum_squares(1105)
    [(4, 33), (9, 32), (12, 31), (23, 24)]
    '''
    reps = []
    if is_square(N/2):
        # If N/2 is a square, search up to x = (N/2)**0.5
        max_x = round((N/2)**0.5)
    else:
        # If N/2 is not a square, search up to x = floor((N/2)**0.5)
        max_x = int((N/2)**0.5)
    for x in range(0,max_x + 1):
        y_squared = N - x**2
        if is_square(y_squared):
            y = round(y_squared**0.5)
            # Append solution (x,y) to list of solutions
            reps.append((x,y))
    return reps
```

```python
reps_sum_squares(1105)
```

    [(4, 33), (9, 32), (12, 31), (23, 24)]

What is the smallest integer which can be expressed as the sum of squares in 5 different ways?

```python
N = 1105
num_reps = 4
while num_reps < 5:
    N = N + 1
    reps = reps_sum_squares(N)
    num_reps = len(reps)
print(N,':',reps_sum_squares(N))
```

    4225 : [(0, 65), (16, 63), (25, 60), (33, 56), (39, 52)]

## Examples

### Prime Numbers

A positive integer is [prime](https://en.wikipedia.org/wiki/Prime_number) if it is divisible only by 1 and itself. Write a function called `is_prime` which takes an input parameter `n` and returns `True` or `False` depending on whether `n` is prime or not.

Let's outline our approach before we write any code:

1. An integer $d$ divides $n$ if there is no remainder of $n$ divided by $d$.
2. Use the modulus operator `%` to compute the remainder.
3. If $d$ divides $n$ then $n = d q$ for some integer $q$ and either $d \leq \sqrt{n}$ or $q \leq \sqrt{n}$ (and not both), therefore we need only test if $d$ divides $n$ for integers $d \leq \sqrt{n}$

```python
def is_prime(n):
    "Determine whether or not n is a prime number."
    if n <= 1:
        return False
    # Test if d divides n for d <= n**0.5
    for d in range(2,round(n**0.5) + 1):
        if n % d == 0:
            # n is divisible by d and so n is not prime
            return False
    # If we exit the for loop, then n is not divisible by any d
    # and therefore n is prime
    return True
```

Let's test our function on the first 30 numbers:

```python
for n in range(0,31):
    if is_prime(n):
        print(n,'is prime!')
```

    2 is prime!
    3 is prime!
    5 is prime!
    7 is prime!
    11 is prime!
    13 is prime!
    17 is prime!
    19 is prime!
    23 is prime!
    29 is prime!

Our function works! Let's find all the primes between 20,000 and 20,100.

```python
for n in range(20000,20100):
    if is_prime(n):
        print(n,'is prime!')
```

    20011 is prime!
    20021 is prime!
    20023 is prime!
    20029 is prime!
    20047 is prime!
    20051 is prime!
    20063 is prime!
    20071 is prime!
    20089 is prime!

### Divisors

Let's write a function called `divisors` which takes a positive integer $N$ and returns the list of positive integers which divide $N$.

```python
def divisors(N):
    "Return the list of divisors of N."
    # Initialize the list of divisors (which always includes 1)
    divisor_list = [1]
    # Check division by d for d <= N/2
    for d in range(2,N // 2 + 1):
        if N % d == 0:
            divisor_list.append(d)
    # N divides itself and so we append N to the list of divisors
    divisor_list.append(N)
    return divisor_list
```

Let's test our function:

```python
divisors(10)
```

    [1, 2, 5, 10]

```python
divisors(100)
```

    [1, 2, 4, 5, 10, 20, 25, 50, 100]

```python
divisors(59)
```

    [1, 59]

### Collatz Conjecture

Let $a$ be a positive integer and consider the recursive sequence where $x_0 = a$ and

$$
x_{n+1} = \left\\{ \begin{array}{cl} x_n/2 & \text{if } x_n \text{ is even} \\\\ 3x_n+1 & \text{if } x_n \text{ is odd}  \end{array} \\right.
$$

The [Collatz conjecture](https://en.wikipedia.org/wiki/Collatz_conjecture) states that this sequence will *always* reach 1. For example, if $a = 10$ then $x_0 = 10$, $x_1 = 5$, $x_2 = 16$, $x_3 = 8$, $x_4 = 4$, $x_5 = 2$ and $x_6 = 1$.

Write a function called `collatz` which takes one input parameter `a` and returns the sequence of integers defined above and ending with the first occurrence $x_n=1$.

```python
def collatz(a):
    "Compute the Collatz sequence starting at a and ending at 1."
    # Initialize list with first value a
    sequence = [a]
    # Compute values until we reach 1
    while sequence[-1] > 1:
        # Check if the last element in the list is even
        if sequence[-1] % 2 == 0:
            # Compute and append the new value
            sequence.append(sequence[-1] // 2)
        else:
            # Compute and append the new value
            sequence.append(3*sequence[-1] + 1)
    return sequence
```

Let's test our function:

```python
print(collatz(10))
```

    [10, 5, 16, 8, 4, 2, 1]

```python
collatz(22)
```

    [22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5, 16, 8, 4, 2, 1]

The Collatz conjecture is quite amazing. No matter where we start, the sequence always terminates at 1!

```python
a = 123456789
seq = collatz(a)
print("Collatz sequence for a =",a)
print("begins with",seq[:5])
print("ends with",seq[-5:])
print("and has",len(seq),"terms.")
```

    Collatz sequence for a = 123456789
    begins with [123456789, 370370368, 185185184, 92592592, 46296296]
    ends with [16, 8, 4, 2, 1]
    and has 178 terms.

Which $a < 1000$ produces the longest sequence?

```python
max_length = 1
a_max = 1
for a in range(1,1001):
    seq_length = len(collatz(a))
    if seq_length > max_length:
        max_length = seq_length
        a_max = a
print('Longest sequence begins with a =',a_max,'and has length',max_length)
```

    Longest sequence begins with a = 871 and has length 179

## Exercises

1. [Fermat's theorem on the sum of two squares](https://en.wikipedia.org/wiki/Fermat%27s_theorem_on_sums_of_two_squares) states that every prime number $p$ of the form $4k+1$ can be expressed as the sum of two squares. For example, $5 = 2^2 + 1^2$ and $13 = 3^2 + 2^2$. Find the smallest prime greater than $2019$ of the form $4k+1$ and write it as a sum of squares. (Hint: Use the functions `is_prime` and `reps_sum_squares` from this section.)

2. What is the smallest prime number which can be represented as a sum of squares in 2 different ways?

3. What is the smallest integer which can be represented as a sum of squares in 3 different ways?

4. Write a function called `primes_between` which takes two integer inputs $a$ and $b$ and returns the list of primes in the closed interval $[a,b]$.

5. Write a function called `primes_d_mod_N` which takes four integer inputs $a$, $b$, $d$ and $N$ and returns the list of primes in the closed interval $[a,b]$ which are congruent to $d$ mod $N$ (this means that the prime has remainder $d$ after division by $N$). This kind of list is called [primes in an arithmetic progression](https://en.wikipedia.org/wiki/Dirichlet%27s_theorem_on_arithmetic_progressions).

6. Write a function called `reciprocal_recursion` which takes three positive integers $x_0$, $x_1$ and $N$ and returns the sequence $[x_0,x_1,x_2,\dots,x_N]$ where

    $$
    x_n = \frac{1}{x_{n-1}} + \frac{1}{x_{n-2}}
    $$

7. Write a function called `root_sequence` which takes input parameters $a$ and $N$, both positive integers, and returns the $N$th term $x_N$ in the sequence:

    $$
    \begin{align}
    x_0 &= a \\\
    x_n &= 1 + \sqrt{x_{n-1}}
    \end{align}
    $$

    Does the sequence converge to different values for different starting values $a$?

8. Write a function called `fib_less_than` which takes one input $N$ and returns the list of Fibonacci numbers less than $N$.

9. Write a function called `fibonacci_primes` which takes an input parameter $N$ and returns the list of Fibonacci numbers less than $N$ which are also prime numbers.

10. Let $w(N)$ be the number of ways $N$ can be expressed as a sum of two squares $x^2 + y^2 = N$ with $1 \leq x \leq y$. Then

    $$
    \lim_{N \to \infty} \frac{1}{N} \sum_{n=1}^{N} w(n) = \frac{\pi}{8}
    $$

    Compute the left side of the formula for $N=100$ and compare the result to $\pi / 8$.

11. A list of positive integers $[a,b,c]$ (with $1 \leq a < b$) are a [Pythagorean triple](https://en.wikipedia.org/wiki/Pythagorean_triple) if $a^2 + b^2 = c^2$. Write a function called `py_triples` which takes an input parameter $N$ and returns the list of Pythagorean triples `[a,b,c]` with $c \leq N$.

Python/1. Python Basics/Notebooks/1. The Basics.- Startups/1.0 Fundamentals Continued/1.4b Using Functions to Solve Math Problems.ipynb


# Finding Roots of Equations

## Calculus review

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy as scipy
from scipy.interpolate import interp1d
```

Let's review the theory of optimization for multivariate functions. Recall that in the single-variable case, extreme values (local extrema) occur at points where the first derivative is zero, however, the vanishing of the first derivative is not a sufficient condition for a local max or min.  Generally, we apply the second derivative test to determine whether a candidate point is a max or min (sometimes it fails - if the second derivative either does not exist or is zero).  In the multivariate case, the first and second derivatives are *matrices*.  In the case of a scalar-valued function on $\mathbb{R}^n$, the first derivative is an $n\times 1$ vector called the *gradient* (denoted $\nabla f$). The second derivative is an $n\times n$ matrix called the *Hessian* (denoted $H$)

Just to remind you, the gradient and Hessian are given by:

$$\nabla f(x) = \left(\begin{matrix}\frac{\partial f}{\partial x_1}\\ \vdots \\\frac{\partial f}{\partial x_n}\end{matrix}\right)$$

$$H = \left(\begin{matrix}
  \dfrac{\partial^2 f}{\partial x_1^2} & \dfrac{\partial^2 f}{\partial x_1\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_1\,\partial x_n} \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_2\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_2^2} & \cdots & \dfrac{\partial^2 f}{\partial x_2\,\partial x_n} \\[2.2ex]
  \vdots & \vdots & \ddots & \vdots \\[2.2ex]
  \dfrac{\partial^2 f}{\partial x_n\,\partial x_1} & \dfrac{\partial^2 f}{\partial x_n\,\partial x_2} & \cdots & \dfrac{\partial^2 f}{\partial x_n^2}
\end{matrix}\right)$$

One of the first things to note about the Hessian - it's symmetric. This structure leads to some useful properties in terms of interpreting critical points.

The multivariate analog of the test for a local max or min turns out to be a statement about the gradient and the Hessian matrix.  Specifically, a function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ has a critical point at $x$ if $\nabla f(x) = 0$ (where zero is the zero vector!).  Furthermore, the second derivative test at a critical point is as follows:

* If $H(x)$ is positive-definite ($\iff$ it has all positive eigenvalues), $f$ has a local minimum at $x$
* If $H(x)$ is negative-definite ($\iff$ it has all negative eigenvalues), $f$ has a local maximum at $x$
* If $H(x)$ has both positive and negative eigenvalues, $f$ has a saddle point at $x$.

If you have $m$ equations with $n$ variables, then the $m \times n$ matrix of first partial derivatives is known as the Jacobian $J(x)$. For example, for two equations $f(x, y)$ and $g(x, y)$, we have

$$
J(x) = \begin{bmatrix}
\frac{\delta f}{\delta x} & \frac{\delta f}{\delta y} \\
\frac{\delta g}{\delta x} & \frac{\delta g}{\delta y}
\end{bmatrix}
$$

We can now express the multivariate form of Taylor polynomials in a familiar format.

$$
f(x + \delta x) = f(x) + \delta x \cdot J(x) + \frac{1}{2} \delta x^T H(x) \delta x + \mathcal{O}(\delta x^3)
$$

## Main Issues in Root Finding in One Dimension

* Separating close roots
* Numerical Stability
* Rate of Convergence
* Continuity and Differentiability

## Bisection Method

The bisection method is one of the simplest methods for finding zeros of a non-linear function.  It is guaranteed to find a root - but it can be slow. The main idea comes from the intermediate value theorem:  If $f(a)$ and $f(b)$ have different signs and $f$ is continuous, then $f$ must have a zero between $a$ and $b$.  We evaluate the function at the midpoint, $c = \frac12(a+b)$. $f(c)$ is either zero, has the same sign as $f(a)$ or the same sign as $f(b)$.  Suppose $f(c)$ has the same sign as $f(a)$ (as pictured below).  We then repeat the process on the interval $[c,b]$.

```python
def f(x):
    return x**3 + 4*x**2 -3

x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

a = -3.0
b = -0.5
c = 0.5*(a+b)

plt.text(a,-1,"a")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([a,b,c], [f(a), f(b),f(c)], s=50, facecolors='none')
plt.scatter([a,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

```python
x = np.linspace(-3.1, 0, 100)
plt.plot(x, x**3 + 4*x**2 -3)

d = 0.5*(b+c)

plt.text(d,-1,"d")
plt.text(b,-1,"b")
plt.text(c,-1,"c")

plt.scatter([d,b,c], [f(d), f(b),f(c)], s=50, facecolors='none')
plt.scatter([d,b,c], [0,0,0], s=50, c='red')

xaxis = plt.axhline(0)
pass
```

We can terminate the process whenever the function evaluated at the new midpoint is 'close enough' to zero. This method is an example of what are known as 'bracketed methods'. This means the root is 'bracketed' by the end-points (it is somewhere in between). Another class of methods are 'open methods' - the root need not be somewhere in between the end-points (but it usually needs to be close!)

## Secant Method

The secant method also begins with two initial points, but without the constraint that the function values are of opposite signs.  We use the secant line to extrapolate the next candidate point.

```python
def f(x):
    return (x**3-2*x+7)/(x**4+2)

x = np.arange(-3,5, 0.1);
y = f(x)

p1=plt.plot(x, y)
plt.xlim(-3, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
t = np.arange(-10, 5., 0.1)

x0=-1.2
x1=-0.5
xvals = []
xvals.append(x0)
xvals.append(x1)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--']
while (notconverge==1 and count <  3):
    slope=(f(xvals[count+1])-f(xvals[count]))/(xvals[count+1]-xvals[count])
    intercept=-slope*xvals[count+1]+f(xvals[count+1])
    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(f(nextval)) < 0.001:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

plt.show()
```

The secant method has the advantage of fast convergence.  While the bisection method has a linear convergence rate (i.e. error goes to zero at the rate that $h(x) = x$ goes to zero, the secant method has a convergence rate that is faster than linear, but not quite quadratic (i.e. $\sim x^\alpha$, where $\alpha = \frac{1+\sqrt{5}}2 \approx 1.6$) however, the trade-off is that the secant method is not guaranteed to find a root in the brackets.

A variant of the secant method is known as the **method of false positions**. Conceptually it is identical to the secant method, except that instead of always using the last two values of $x$ for linear interpolation, it chooses the two most recent values that maintain the bracket property (i.e $f(a) f(b) < 0$). It is slower than the secant, but like the bisection, is safe.

## Newton-Raphson Method

We want to find the value $\theta$ so that some (differentiable) function $g(\theta)=0$.
Idea: start with a guess, $\theta_0$.  Let $\tilde{\theta}$ denote the value of $\theta$ for which $g(\theta) = 0$ and define $h = \tilde{\theta} - \theta_0$.  Then:

$$
\begin{eqnarray*}
g(\tilde{\theta}) &=& 0 \\\\
&=&g(\theta_0 + h) \\\\
&\approx& g(\theta_0) + hg'(\theta_0)
\end{eqnarray*}
$$

This implies that

$$ h\approx \frac{g(\theta_0)}{g'(\theta_0)}$$

So that

$$\tilde{\theta}\approx \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

Thus, we set our next approximation:

$$\theta_1 = \theta_0 - \frac{g(\theta_0)}{g'(\theta_0)}$$

and we have developed an iterative procedure with:

$$\theta_n = \theta_{n-1} - \frac{g(\theta_{n-1})}{g'(\theta_{n-1})}$$

#### Example

Let $$g(x) = \frac{x^3-2x+7}{x^4+2}$$

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Example Function')
plt.show()
```

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Good Guess')
t = np.arange(-5, 5., 0.1)

x0=-1.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge=0
    else:
        xvals.append(nextval)
    count = count+1

```

From the graph, we see the zero is near -2.  We make an initial guess of $$x=-1.5$$

We have made an excellent choice for our first guess, and we can see rapid convergence!

```python
funval
```

    0.007591996330867034

In fact, the Newton-Raphson method converges quadratically.  However, NR (and the secant method) have a fatal flaw:

```python
x = np.arange(-5,5, 0.1);
y = (x**3-2*x+7)/(x**4+2)

p1=plt.plot(x, y)
plt.xlim(-4, 4)
plt.ylim(-.5, 4)
plt.xlabel('x')
plt.axhline(0)
plt.title('Bad Guess')
t = np.arange(-5, 5., 0.1)

x0=-0.5
xvals = []
xvals.append(x0)
notconverge = 1
count = 0
cols=['r--','b--','g--','y--','c--','m--','k--','w--']
while (notconverge==1 and count <  6):
    funval=(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)
    slope=-((4*xvals[count]**3 *(7 - 2 *xvals[count] + xvals[count]**3))/(2 + xvals[count]**4)**2) + (-2 + 3 *xvals[count]**2)/(2 + xvals[count]**4)

    intercept=-slope*xvals[count]+(xvals[count]**3-2*xvals[count]+7)/(xvals[count]**4+2)

    plt.plot(t, slope*t + intercept, cols[count])
    nextval = -intercept/slope
    if abs(funval) < 0.01:
        notconverge = 0
    else:
        xvals.append(nextval)
    count = count+1
```

We have stumbled on the horizontal asymptote.  The algorithm fails to converge.

### Convergence Rate

The following is a derivation of the convergence rate of the NR method:

Suppose $x_k \; \rightarrow \; x^*$ and $g'(x^*) \neq 0$.  Then we may write:

$$x_k = x^* + \epsilon_k$$.

Now expand $g$ at $x^*$:

$$g(x_k) = g(x^*) + g'(x^*)\epsilon_k + \frac12 g''(x^*)\epsilon_k^2 + ...$$
$$g'(x_k)=g'(x^*) + g''(x^*)\epsilon_k$$

We have that

\begin{eqnarray}
\epsilon_{k+1} &=& \epsilon_k + \left(x_{k-1}-x_k\right)\\
&=& \epsilon_k -\frac{g(x_k)}{g'(x_k)}\\
&\approx & \frac{g'(x^*)\epsilon_k + \frac12g''(x^*)\epsilon_k^2}{g'(x^*)+g''(x^*)\epsilon_k}\\
&\approx & \frac{g''(x^*)}{2g'(x^*)}\epsilon_k^2
\end{eqnarray}

## Gauss-Newton

For 1D, the Newton method is
$$
x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
$$

We can generalize to $k$ dimensions by
$$
x_{n+1} = x_n - J^{-1} f(x_n)
$$
where $x$ and $f(x)$ are now vectors, and $J^{-1}$ is the inverse Jacobian matrix. In general, the Jacobian is not a square matrix, and we use the generalized inverse $(J^TJ)^{-1}J^T$ instead, giving
$$
x_{n+1} = x_n - (J^TJ)^{-1}J^T f(x_n)
$$

In multivariate nonlinear estimation problems, we can find the vector of parameters $\beta$ by minimizing the residuals $r(\beta)$,
$$
\beta_{n+1} = \beta_n - (J^TJ)^{-1}J^T r(\beta_n)
$$
where the entries of the Jacobian matrix $J$ are
$$
J_{ij} = \frac{\partial r_i(\beta)}{\partial \beta_j}
$$

## Inverse Quadratic Interpolation

Inverse quadratic interpolation is a type of polynomial interpolation.  Polynomial interpolation simply means we find the polynomial of least degree that fits a set of points.  In quadratic interpolation, we use three points, and find the quadratic polynomial that passes through those three points.

```python

def f(x):
    return (x - 2) * x * (x + 2)**2

x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,-1.0])
y0=f(x0)
f2 = interp1d(x0, y0,kind='quadratic')

#Plot parabola
xs = np.linspace(-3, -1, num=10000, endpoint=True)
plt.plot(xs, f2(xs))

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');

#New x value
xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]

plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='black');

#New triplet
x1=np.append([-3,-2.5],xnew)
y1=f(x1)
f2 = interp1d(x1, y1,kind='quadratic')

#New Parabola
xs = np.linspace(min(x1), max(x1), num=100, endpoint=True)
plt.plot(xs, f2(xs))

xnew=xs[np.where(abs(f2(xs))==min(abs(f2(xs))))]
plt.scatter(np.append(xnew,xnew), np.append(0,f(xnew)), c='green');

```

So that's the idea behind quadratic interpolation. Use a quadratic approximation, find the zero of interest, use that as a new point for the next quadratic approximation.

Inverse quadratic interpolation means we do quadratic interpolation on the *inverse function*.  So, if we are looking for a root of $f$, we approximate $f^{-1}(x)$ using quadratic interpolation. This just means fitting $x$ as a function of $y$, so that the quadratic is turned on its side and we are guaranteed that it cuts the x-axis somewhere. Note that the secant method can be viewed as a *linear* interpolation on the inverse of $f$. We can write:

$$f^{-1}(y) = \frac{(y-f(x_n))(y-f(x_{n-1}))}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{(y-f(x_n))(y-f(x_{n-2}))}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{(y-f(x_{n-2}))(y-f(x_{n-1}))}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n-1}$$

We use the above formula to find the next guess $x_{n+1}$ for a zero of $f$ (so $y=0$):

$$x_{n+1} = \frac{f(x_n)f(x_{n-1})}{(f(x_{n-2})-f(x_{n-1}))(f(x_{n-2})-f(x_{n}))}x_{n-2} + \frac{f(x_n)f(x_{n-2})}{(f(x_{n-1})-f(x_{n-2}))(f(x_{n-1})-f(x_{n}))}x_{n-1} + \frac{f(x_{n-2})f(x_{n-1})}{(f(x_{n})-f(x_{n-2}))(f(x_{n})-f(x_{n-1}))}x_{n}$$

We aren't so much interested in deriving this as we are understanding the procedure:

```python
x = np.arange(-5,5, 0.1);
plt.plot(x, f(x))
plt.xlim(-3.5, 0.5)
plt.ylim(-5, 16)
plt.xlabel('x')
plt.axhline(0)
plt.title("Inverse Quadratic Interpolation")

#First Interpolation
x0=np.array([-3,-2.5,1])
y0=f(x0)
f2 = interp1d(y0, x0,kind='quadratic')

#Plot parabola
xs = np.linspace(min(f(x0)), max(f(x0)), num=10000, endpoint=True)
plt.plot(f2(xs), xs)

#Plot first triplet
plt.plot(x0, f(x0),'ro');
plt.scatter(x0, f(x0), s=50, c='yellow');
```

Convergence rate is approximately $1.8$. The advantage of the inverse method is that we will *always* have a real root (the parabola will always cross the x-axis). A serious disadvantage is that the initial points must be very close to the root or the method may not converge.

That is why it is usually used in conjunction with other methods.

## Brentq Method

Brent's method is a combination of bisection, secant and inverse quadratic interpolation.  Like bisection, it is a 'bracketed' method (starts with points $(a,b)$ such that $f(a)f(b)<0$.

Roughly speaking, the method begins by using the secant method to obtain a third point $c$, then uses inverse quadratic interpolation to generate the next possible root. Without going into too much detail, the algorithm attempts to assess when interpolation will go awry, and if so, performs a bisection step. Also, it has certain criteria to reject an iterate. If that happens, the next step will be linear interpolation (secant method).

To find zeros, use

```python
x = np.arange(-5,5, 0.1);
p1=plt.plot(x, f(x))
plt.xlim(-4, 4)
plt.ylim(-10, 20)
plt.xlabel('x')
plt.axhline(0)
pass
```

```python
from scipy import optimize
```

```python
scipy.optimize.brentq(f,-1,.5)
```

    -7.864845203343107e-19

```python
scipy.optimize.brentq(f,.5,3)
```

    2.0

## Roots of polynomials

One method for finding roots of polynomials converts the problem into an eigenvalue one by using the **companion matrix** of a polynomial. For a polynomial

$$
p(x) = a_0 + a_1x + a_2 x^2 + \ldots + a_m x^m
$$

the companion matrix is

$$
A = \begin{bmatrix}
-a_{m-1}/a_m & -a_{m-2}/a_m & \ldots & -a_0/a_m \\
1 & 0 & \ldots & 0 \\
0 & 1 & \ldots & 0 \\
\vdots & \vdots & \ldots & \vdots \\
0 & 0 & \ldots & 0
\end{bmatrix}
$$

The characteristic polynomial of the companion matrix is $\lvert \lambda I - A \rvert$ which expands to

$$
a_0 + a_1 \lambda + a_2 \lambda^2 + \ldots + a_m \lambda^m
$$

In other words, the roots we are seeking are the eigenvalues of the companion matrix.

For example, to find the cube roots of unity, we solve $x^3 - 1 = 0$. The `roots` function uses the companion matrix method to find roots of polynomials.

```python
# Coefficients of $x^3, x^2, x^1, x^0$

poly = np.array([1, 0, 0, -1])
```

Manual construction

```python
A = np.array([
    [0,0,1],
    [1,0,0],
    [0,1,0]
])
```

```python
scipy.linalg.eigvals(A)
```

    array([-0.5+0.8660254j, -0.5-0.8660254j,  1. +0.j       ])

Using built-in function

```python
x = np.roots(poly)
x
```

    array([-0.5+0.8660254j, -0.5-0.8660254j,  1. +0.j       ])

```python
plt.scatter([z.real for z in x], [z.imag for z in x])
theta = np.linspace(0, 2*np.pi, 100)
u = np.cos(theta)
v = np.sin(theta)
plt.plot(u, v, ':')
plt.axis('square')
pass
```

## Using `scipy.optimize`

### Finding roots of univariate equations

```python
def f(x):
    return x**3-3*x+1
```

```python
x = np.linspace(-3,3,100)
plt.axhline(0, c='red')
plt.plot(x, f(x))
pass
```

```python
from scipy.optimize import brentq, newton
```

#### `brentq` is the recommended method

```python
brentq(f, -3, 0), brentq(f, 0, 1), brentq(f, 1,3)
```

    (-1.8793852415718166, 0.3472963553337031, 1.532088886237956)

#### Secant method

```python
newton(f, -3), newton(f, 0), newton(f, 3)
```

    (-1.8793852415718166, 0.34729635533385395, 1.5320888862379578)

#### Newton-Raphson method

```python
fprime = lambda x: 3*x**2 - 3
newton(f, -3, fprime), newton(f, 0, fprime), newton(f, 3, fprime)
```

    (-1.8793852415718166, 0.34729635533386066, 1.532088886237956)

### Finding fixed points

Finding the fixed points of a function $g(x) = x$ is the same as finding the roots of $g(x) - x$. However, specialized algorithms also exist - e.g. using `scipy.optimize.fixedpoint`.

```python
from scipy.optimize import fixed_point
```

```python
x = np.linspace(-3,3,100)
plt.plot(x, f(x), color='red')
plt.plot(x, x)
pass
```

```python
fixed_point(f, 0), fixed_point(f, -3), fixed_point(f, 3)
```

    (array(0.25410169), array(-2.11490754), array(1.86080585))

### Mutlivariate roots and fixed points

Use `root` to solve polynomial equations. Use `fsolve` for non-polynomial equations.

```python
from scipy.optimize import root, fsolve
```

Suppose we want to solve a sysetm of $m$ equations with $n$ unknowns

\begin{align}
f(x_0, x_1) &= x_1 - 3x_0(x_0+1)(x_0-1) \\
g(x_0, x_1) &= 0.25 x_0^2 + x_1^2 - 1
\end{align}

Note that the equations are non-linear and there can be multiple solutions. These can be interpreted as fixed points of a system of differential equations.

```python
def f(x):
    return [x[1] - 3*x[0]*(x[0]+1)*(x[0]-1),
            .25*x[0]**2 + x[1]**2 - 1]
```

```python
sol = root(f, (0.5, 0.5))
sol.x
```

    array([1.11694147, 0.82952422])

```python
fsolve(f, (0.5, 0.5))
```

    array([1.11694147, 0.82952422])

```python
r0 = root(f,[1,1])
r1 = root(f,[0,1])
r2 = root(f,[-1,1.1])
r3 = root(f,[-1,-1])
r4 = root(f,[2,-0.5])

roots = np.c_[r0.x, r1.x, r2.x, r3.x, r4.x]
```

```python
Y, X = np.mgrid[-3:3:100j, -3:3:100j]
U = Y - 3*X*(X + 1)*(X-1)
V = .25*X**2 + Y**2 - 1

plt.streamplot(X, Y, U, V, color=U, linewidth=2, cmap=plt.cm.autumn)
plt.scatter(roots[0], roots[1], s=50, c='none', edgecolors='k', linewidth=2)
pass
```

#### We can also give the Jacobian

```python
def jac(x):
    return [[-6*x[0], 1], [0.5*x[0], 2*x[1]]]
```

```python
sol = root(f, (0.5, 0.5), jac=jac)
sol.x, sol.fun
```

    (array([1.11694147, 0.82952422]), array([-4.23383550e-12, -3.31612515e-12]))

#### Check that values found are really roots

```python
np.allclose(f(sol.x), 0)
```

    True

#### Starting from other initial conditions, different roots may be found

```python
sol = root(f, (12,12))
sol.x
```

    array([ 0.77801314, -0.92123498])

```python
np.allclose(f(sol.x), 0)
```

    True

```python

```


## The usual imports

```python
%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
# And a new import - scipy's basic ODE integration function
from scipy.integrate import odeint
```

## Intro - The Euler Method

A first-order ordinary differential equation can be written as
$$\frac{d y}{dt} = g(y, t)$$
The time derivative can depend on the value of $y(t)$ as well as on $t$.

To solve a differential equation on a computer, we use a discrete approximation to the derivative
$$\frac{d y}{dt} = \frac{y_{i+1} - y_{i}}{t_{i+1} - t_{i}}$$
where we're evaluating $y$ at discrete *time steps* $t_{i}$.

We'll work with equal time steps given by $t_{i+1} - t_{i} = \tau$. The Euler method makes a simple approximation:
$$y_{i+1} = y_{i} + g_{i} \tau + O(\tau^2)$$
The last term just means that we've ignored any terms of order $\tau^2$ or higher. The bigger $\tau$ is, the bigger the error will be.

We'll see how this works by solving the equation
$$\frac{d y}{dt} = - y ~,$$
which has the solution $y(t) = A~e^{-t}$.

```python
tau = 1.
yi = 1.
result1 = [(0., yi)]
for t in np.arange(tau, 6.+tau, tau):
    yi = yi + (-yi) * tau
    result1.append((t, yi))
result1 = np.array(result1)
```

```python
times = np.linspace(0.,6.,1000)
plt.plot(times, np.exp(-times))
plt.scatter(result1[:,0], result1[:,1], c='g')
```

**Exercise 0.1**: Make the time step above $\tau = 1.5$. What happens?

**Exercise 0.2**: Now make two new sets of results with smaller values of $\tau$ and add them to the plot. Does the solution get better?

## Ex 1 - The Decaying Exponential

Now let's start to use scipy's odeint to solve some differential equations with nice sophisticated integrators. We'll continue to solve
$$\frac{d y}{dt} = - y$$
We start by defining a function for the right-hand side.

```python
# Defining the function is simple.
def f1(y, t):
    return -y
```

```python
# We'll solve from t = 0 to t = 4, at 100 equally-spaced points.
times = np.linspace(0.,4.,100)
```

```python
y_init1 = 1. # Starting value for y, at t = 0
yarr1 = odeint(f1, y_init1, times)
```

```python
plt.figure(figsize=(8,6))
plt.plot(times, np.exp(-times), c='g', label='True solution')
plt.scatter(times, yarr1, s=2, c='k', label='ODE solver')
plt.legend()
```

**Exercise 1.1**: Can you make this decay twice as fast?

```python

```

**Exercise 1.2**: Can you change the amplitude, i.e. make the solution $2 e^{-t}$ rather than $e^{-t}$?

```python

```

## Ex 2 - The Sinusoid

Now let's try to get a sinusoid. We can't do that with a single derivative -- we need a second-order equation. It looks like
$$ \frac{d^2 y}{d t^2} = - \omega^2 y$$
But odeint doesn't say anything about second derivatives. How are we supposed to solve this?

The trick is, we're allowed to have many variables. So let's define a new variable
$$v_y = \frac{d y}{d t}$$

If we take another derivative, we get:
$$\frac{d v_y}{d t} = \frac{d^2 y}{d t}$$

Now we can re-write the above equation as two pieces:
$$\begin{align}
\frac{d v_y}{d t} &= -\omega^2 y \\
\frac{d y}{d t} &= v_y
\end{align}$$

We've gone from one second-order equation to two first-order equations. Our new variable is the velocity.

```python
# The function now takes a vector as input.
omega = np.pi
def f2(vec, t):
    y, vy = vec
    return (vy, -omega*omega*y)
```

```python
y_init2 = (1., 0.) # Starting value for y, vy at t = 0
yarr2 = odeint(f2, y_init2, times)
```

```python
yarr2.shape
```

    (100, 2)

The result now has two variables, each one is its own column. Each time is a row. We can use numpy's multi-dimensional slicing to access just the first variable by doing

yarr2\[:, 0\]

In the first slot, we put the slice operator ':' all by itself. This just means take all the data in that dimension. Then we specify that we want column zero, which is the first variable since Python counts from zero.

```python
plt.figure(figsize=(8,6))
plt.plot(times, np.cos(omega*times), 'g')
plt.plot(times, yarr2[:,0], 'r:')
```

**Exercise 2.1**: How do you double the amplitude?

```python

```

**Exercise 2.2**: Can you change the result in y from a cosine to a sine? Hint: A sine starts at zero, but it has some initial slope.

```python

```

**Exercise 2.3**: Can you change the frequency in the same way? If not, how do you change it?

```python

```

**Exercise 2.4**: Copying the example from above, can you modify the equation to add damping? The result should be a decaying sinusoid.

```python

```


## What is symbolic computation?

"Symbolic computation deals with the computation of mathematical objects symbolically. This means that the mathematical objects are represented exactly, not approximately, and mathematical expressions with unevaluated variables are left in symbolic form."

- [SymPy Documentation](https://docs.sympy.org/latest/tutorial/intro.html)

### A More Interesting Example

```python
from sympy import *
init_printing()
```

```python
#Definimos los símbolos (SymPy) 'x' e 'y' y los definimos en las
#variables (Python) 'x' e 'y'
x,y=symbols('x y')

expr=x+2*y
expr
```

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
expr+1
```

```python
expr-x
```

```python
#No expande automáticamente
x*expr
```

```python
#Podemos expandir una expresión o factorizarla de la siguiente forma+
from sympy import expand, factor
expanded_expr=expand(x*expr)
expanded_expr
```

```python
factor_expr=factor(expanded_expr)
factor_expr
```

### The Power of Symbolic Computation

```python
x,t,z,nu=symbols('x t z nu')
```

```python
#Tomamos la derivsada de cos(x**2)/x
diff(cos(x**2)/x,x)
```

```python
#Podemos ver la versión simbólica de output anterior usando
#la función print()
print(_)
```

    -2*sin(x**2) - cos(x**2)/x**2

```python
#Calculemos la integral de exp(x)*sin(x)+exp(x)*cos(x)
integrate(exp(x)*sin(x)+exp(x)*cos(x),x)
```

```python
#Calculemos la integral de exp(-x**2) desde -oo hatsta +oo
integrate(exp(-x**2),(x,-oo,+oo))
```

```python
#Calcular el límite de (sin(x+t)-sin(x))/t cuando t va a 0
limit((sin(x+t)-sin(x))/t,t,0)
```

```python
#De igual forma
limit((cos(x+t)-cos(x))/t,t,0)
```

```python
#Por defecto el límite se toma por la derecha.
#Por ejemplo, calculemos el límite cuando 'x' va a cero,
#luego por la derecha e izquierda de la siguiente expresión

abs(sin(x))/sin(x)
```

```python
limit(_,x,0)
```

```python
limit(abs(sin(x))/sin(x),x,0,'+')
```

```python
limit(abs(sin(x))/sin(x),x,0,'-')
```

```python
#También podemos imprimir las expresiones que vamos a computar
Integral(cos(z)*sin(nu),z,nu)
```

```python
r,phi,theta,R=symbols('r phi theta R')
```

```python
Integral(r**2*sin(theta),(theta,0,pi),(phi,0,2*pi),(r,0,R))
```

```python
integrate(r**2*sin(theta),(theta,0,pi),(phi,0,2*pi),(r,0,R))
```

```python
Limit(exp(cos(t**2)),t,5,'+')
```

```python
#Podemos resolver ecuaciones introduciendo en el código la expresión
#que es igual a cero. Por ejemplo, resolvamos la siguiente expresión
```

$x^2-2=0\quad\quad\quad\quad(1)$

```python
solve(x**2-2,x)
```

```python
#Otra forma de igualar la parte izqueirda y derecha de la ecuación (1)
#es mediante la instrucción 'Eq()'
```

```python
Eq(x**2-2,0)
```

```python
solve(_,x)
```

```python
#Resolvamos la siguiente ecuación diferencial
```

$y''-y=e^t\quad\quad\quad\quad(2)$

```python
#Primero definimos 'y' como una función y a continuación establecemos
#su argumento
y=Function('y')

#Ahora aplicamos la primera derivada
y(t).diff(t)
```

```python
#Ahora aplicamos la segunda derivada
y(t).diff(t,t)
```

```python
#Veamos el código simbólico
print(_)
```

    Derivative(y(t), (t, 2))

```python
#Ahora sí, a resolver la ecuación diferencial usando 'Eq()'
dsolve(Eq(y(t).diff(t,t)-y(t),exp(t)),y(t))
```

```python
Eq(4*t*y(t).diff(t,t)+y(t).diff(t)-y(t),0)
```

```python
dsolve(_)
```

```python
print(latex(_))
```

    y{\left(t \right)} = t^{\frac{3}{8}} \left(C_{1} J_{\frac{3}{4}}\left(i \sqrt{t}\right) + C_{2} Y_{\frac{3}{4}}\left(i \sqrt{t}\right)\right)

```python
#Calculemos los autovalores de la siguiente matriz
```

$\begin{pmatrix}
0 & -i\\
i & 0
\end{pmatrix}
$

```python
Matrix([[0,-I],[I,0]]).eigenvals()
#En el resultado tenemos un diccionario donde el 'key' es el autovalor
#y el 'value' es la multiplicidad
```

```python
#Por úñtimo podemos imprimir en código LaTeX para poder copiar
#directamente a nuestro documento. Para ello usamos 'print()' y 'latex()'
Integral(cos(x)**2, (x, 0, pi))

```

```python
print(latex(_))
```

    \int\limits_{0}^{\pi} \cos^{2}{\left(x \right)}\, dx

```python
print(Integral(cos(x)**2, (x, 0, pi)))
```

    Integral(cos(x)**2, (x, 0, pi))


# Bond Pricing

* maturity (T)
    - date
* coupon
    - type
    - rate
    - frequency
* bid/ask prices
    - clean vs. dirty
    - accrued interest
* yield (r)

bid/ask prices usually are quoted as percentage of par value at a clean price.

\begin{equation}
\text{Bond value}=\sum_{t=1}^{T}\frac{\text{Coupon}}{(1+r)^t}+\frac{\text{Par Value}}{(1+r)^T}
\end{equation}

If the coupons and r are fixed:

\begin{equation}
\text{Bond value}=\frac{\text{Coupon}}{r}\times(1-(1+r)^{-T})+\text{Par Value}\times(1+r)^{-T}
\end{equation}

## Accrued Interest and Quoted Prices

If you buy a bond x days after its issuance you must pay the accrued interest to the seller as follow:

$$\text{Dirty Price}=\text{Clean Price}+\text{Accrued interest}$$

where:

$$\text{Accrued Interest}=\frac{\text{Annual coupon payment}}{\text{coupon pmt p.a.}}\times\frac{\text{days since last pmt}}{\text{days between pmts}}$$

If you don't know the clean price:

1. $T = \text{maturity date}-\text{last coupon date}$;
2.  Calculate the FV from the last coupon date to the date you are interest in.

*this price will be the dirty price*

## Yield to Maturity (YtM)
YtM is the IRR of the bond - the interest rate that makes the PV of a bond's payments equal to its price.
It's the average rate of return that would be earned on a bond held until maturity, if all coupons are reinvested at the same interest rate.

**The relationship between Yield and Bond Price is negative as follows**

$$\uparrow\text{YtM}\Longleftrightarrow\downarrow\text{Bond price}$$

```python
def quoted_bond_price(maturity, yield_, coupon):
    """Prices a bond as percentage of par value."""
    annuity_factor = (1/yield_)*(1-(1+yield_)**(-maturity))
    pv_factor = (1+yield_)**(-maturity)
    return coupon*annuity_factor + pv_factor
```

```python
def bond_prices_vs_yield(maturity, coupon, yield_i, yield_f):
    """Plots the relationship between bond price and yields."""
    import numpy as np
    import pandas as pd
    import seaborn as sns
    yields = np.arange(yield_i, yield_f, step=0.01)
    prices = [quoted_bond_price(maturity, x, coupon) for x in yields]
    bond_df = pd.DataFrame({'Yield': yields, 'Price': prices})
    return sns.lineplot(x=bond_df.Yield, y=bond_df.Price)
```

```python
bond_prices_vs_yield(30, 0.05, 0.01, 0.2)

```

**The yield-to-Bond price relationship derived above is sensitive to the maturity. Long-term bond's price vary more than short-term bond's price as follows**

```python
import numpy as np
for i in np.arange(1, 30):
    bond_prices_vs_yield(i, 0.05, 0.01, 0.2)
```

# Interest Rate Sensivity
* **Yield curve**: is a plot of YtM as function of maturities

1. *Bond prices* and *yields* are *inversely related*: as yields increase, bond prices fall; as yields fall, bond prices rise
2. An *increase in a bond’s yield to maturity* results in a *smaller price change* than a *decrease in yield* of equal magnitude
3. *Prices of long-term bonds* tend to be *more sensitive* to *interest rate changes* than prices of short-term bonds
4. The *sensitivity of bond prices to changes in yields increases at a decreasing rate as maturity increases*. In other words, interest rate risk is less than proportional to bond maturity.
5. *Interest rate risk is inversely related to the bond’s coupon rate*. Prices of low-coupon bonds are more sensitive to changes in interest rates than prices of high-coupon bonds.
6. The *sensitivity of a bond’s price to a change in its yield is inversely related to the yield to maturity at which the bond currently is selling*

## Macaulay duration
Measures the "real" maturity of a bond. It serves as a guide to the sensitivity of a bond to interest rate.

$$D=\frac{\sum_{t=1}^{T}t\times\frac{CF_t}{(1+YtM)^t}}{\text{Bond Price}}$$

### Modified duration
\begin{array}
\text{MD}&=\frac{D}{(1+YtM)}\\
\frac{\Delta P}{P}&=-MD\times\Delta YtM\\
\end{array}

* The duration of a zero-coupon bond is equal its maturity
* Duration is lower when the coupon is higher
* Duration is higher when YtM is lower
* Practitioners use BPV (or PV01): price change in \% or \$ for one base point.

$$PV01=MD\times Price\times 0.01\%$$

## wrapping up
- $\uparrow\text{YtM}\Longleftrightarrow\downarrow\text{Bond price}$
- $\uparrow\text{MD}\Longleftrightarrow\downarrow\text{YtM}$
- $\uparrow\text{MD}\Longleftrightarrow\downarrow\text{coupon}$
- $\uparrow\text{MD}\Longleftrightarrow\uparrow\text{maturity}$
- $\downarrow\Delta\text{MD}\Longleftrightarrow\uparrow\text{maturity}$

```python
def bond_cash_flows(maturity, coupon_rate, coupons_per_year=1):
    """Returns a series of cash flows generated by a bond."""
    import numpy as np
    import pandas as pd
    n_coupons = round(maturity*coupons_per_year)
    coupon_pmt = (coupon_rate/coupons_per_year)
    coupon_index = np.arange(1, n_coupons + 1)
    cash_flows = pd.Series(data = coupon_pmt, index = coupon_index)
    cash_flows.iloc[-1] += 1
    return pd.Series(cash_flows)

def discount(t, r):
    """Returns a discounted rates series."""
    import pandas as pd
    discounts = pd.Series([(r+1)**-i for i in t])
    discounts.index = t
    return discounts

def pv(flows, r):
    """Computes the pv of cash flows indexed by time."""
    dates = flows.index
    discounts = discount(dates, r)
    return discounts.multiply(flows, axis='rows').sum()

def bond_price(maturity, coupon_rate, coupons_per_year, discount_rate):
    """Prices a bond based on its cash flows."""
    import pandas as pd
    if maturity <= 0:
        return (1 + coupon_rate/coupons_per_year)
    else:
        cash_flows = bond_cash_flows(maturity,
                                     coupon_rate, coupons_per_year)
        return pv(cash_flows, discount_rate/coupons_per_year)

def macauly_duration(cash_flows, discount_rate):
    """Computes the Macauly duration of a bond."""
    import numpy as np
    import pandas as pd
    discounted_flows = discount(cash_flows.index, discount_rate)*cash_flows
    weights = discounted_flows/discounted_flows.sum()
    return np.average(pd.Series(cash_flows.index), weights=weights)

def modified_duration(macauly_duration, ytm, maturity):
    """Computes the modified duration of a bond."""
    return macauly_duration/(1 + ytm/maturity)
```

```python
def md_vs_maturity(maturity_i, maturity_f, coupon_rate, ytm):
    import numpy as np
    import pandas as pd
    import seaborn as sns
    maturities = np.arange(maturity_i, maturity_f, step=1)
    mds = []
    for maturity in maturities:
        cf = bond_cash_flows(maturity, coupon_rate=coupon_rate, coupons_per_year=1)
        mds.append(modified_duration(macauly_duration(cf, ytm), ytm, maturity))
    bond_df = pd.DataFrame({'MD': mds, 'Maturity': maturities})
    return sns.lineplot(x=bond_df.Maturity, y=bond_df.MD)
```

**MD (sensibility) increases as Maturity increases**

```python
md_vs_maturity(1, 30, 0.08, 0.05)
```

**MD (sensibility) increases as YtM decreases**

```python
import numpy as np
for i in np.arange(0.01, 0.15, step=0.01):
    md_vs_maturity(1, 30, 0.08, i)
```

## Convexity
As the relationship between bond prices and yields are not linear, the duration rule is good approximation just for small changes in bond yield. Convexity allows improvement on duration aproximation for bond price changes.
\begin{array}
\text{Convexity}&=\frac{1}{P\times(1+YtM)^2}\times\sum_{t=1}^{T}\big(\frac{CF_t}{(1+YtM)^t}\times(t^2+t)\big)\\
\frac{\Delta P}{P}&=-MD\times\Delta YtM+\frac{\text{Convexity}\times\Delta YtM^2}{2}\\
\end{array}

```python
# A bond with 10y maturity, sensitivity 8, yield 4% is priced at 98% what is the coupon?
# if coupon = yield -> price = 1 -> deltaytm = delta p/MD -> coupon - ytm = delta p/md
md = 8
delta_p = 0.98 - 1
ytm = 0.04
coupon = ytm + delta_p/md
coupon
```

    0.0375

```python
# A bond is quoted at 110%, its md is 5. supposing the yield goes down by 10 basis points. What would be its price?
md = 5
p = 1.10
delta_y = -0.001
-(delta_y*md - p)
```

    1.105

```python
# A 10y bond, coupon 2% and md equal to 8.978 and yield 1.95%
# Price?
# md = delta price/delta yield -> md*(0.02 - 0.0195)=(p - 1)
md = 8.978
delta_y = 0.02 - 0.0195
p = md*delta_y + 1
p
```

    1.004489

```python
# 210 days later the bond trades with a yield of 2.5%. What are the clean and dirty price?
dp = quoted_bond_price(10, 0.025, 0.02)*(1+0.025)**(210/365)
ai = 0.02*(210/365)
cp = dp - ai
print('clean Price: ', cp, '; dirty Price: ', dp)
```

    clean Price:  0.9584148073395676 ; dirty Price:  0.9699216566546361

```python
# P&L (10M)
(dp - p)*10
```

    -0.3456734334536382

# Hedging with duration
$$\text{Hedging ratio}=\frac{P_1\times MD_1}{P_2\times MD_2}$$
or:
$$HR=-BPV_1/BPV_2$$

1. 10M 3%10y, YtM 2,5% - quoted 104.376%, MD=8,6
2. x? 2.5%7y, YtM 1.75% - quoted 104.901%, MD=6.41

To hedge 10M of 1 you need to sell 13.349M of the 2.

```python
hr = (1.04376*8.6)/(1.04901*6.41)
hr*10
```

    13.349390669080591

# Floating Rate Notes (FRN)
- variable coupon
- Discount Margin (cristalization)
    1. last known reference
    2. yield for a given price
    3. Discount Margin = yield - Reference

5y bond paying Euribor + 1% coupons
    - last euribor = 2%
    - price 102%
    - MD = 4.61

       cristalized coupon = 2% + 1%

```python
# delta p/p = -md * deltaY
coupon = 0.03
md = 4.61
p =1.02
delta_p = 1 - p
delta_y = -(delta_p/(p*md))
y = 0.03 - delta_y
y - 0.02
```

    0.005746671770660537


# Part I - Linear Equations

## 1. Views of linear equations

It can be reasonable to state that the the linear equations are the most simple equation system and the one that we know the best. Thus, it constitute the basic part of our toolkit and the study of linear algebra.

For the below linear equations system,

$$
\begin{equation}
2x - y = 0 \\
-x + 2y = 3 \\
\end{equation}
$$

It can be organized the system into a matrix representation as below.And that will lead to two kinds of views of the system.

$$
\begin{equation}
\left[
\begin{matrix}
 2 & -1 \\
-1 &  2 \\
\end{matrix}
\right]
\left[
\begin{matrix}
x \\
y \\
\end{matrix}
\right]
=
\left[
\begin{matrix}
0 \\
3 \\
\end{matrix}
\right]
\end{equation}
$$

And the representation can further be summarized in the general form:

$$\begin{equation}
\rm{A}
\textbf{x}
=
\textbf{b}
\end{equation}$$

where
$
\begin{equation}
\rm{A}
=
\left[
\begin{matrix}
 2 & -1 \\
-1 &  2 \\
\end{matrix}
\right]
\end{equation}
$
is called the $\textit{coefficient matrix}$,
$
\begin{equation}
\textbf{x}
=
\left[
\begin{matrix}
x \\
y \\
\end{matrix}
\right]
\end{equation}
$
is the $\textit{vector of unknowns}$,
and
$
\begin{equation}
\textbf{b}
=
\left[
\begin{matrix}
0 \\
3 \\
\end{matrix}
\right]
\end{equation}
$
is the vector coming from the right hand side of the equations.

### View of Row (View of Separate Equations)

By seeing the system as separate equations, we get the view of row.

$$
\begin{equation}
\begin{cases}
2x - y = 0, & Eq1\\
-x + 2y = 3, & Eq2\\
\end{cases}
\end{equation}
$$

Under the view of row, solutions are interpreted as the set of points that satisfy all of the equations. Therefore, in geometry, they are the **intersection points of all the geometric objects (lines/planes/hyperplanes)** these equation represent.

### View of Column (View of Linear Combination)

An alternative view can be: Since all the equations share the same vector of unknowns (if some unknowns are missing, it can be represented as unknowns with 0 coefficient), we can **"factor out"** those unknowns and reshape the system like this.

$$
\begin{equation}
x
\left[
\begin{matrix}
 2 \\
-1 \\
\end{matrix}
\right]
+
y
\left[
\begin{matrix}
-1 \\
2 \\
\end{matrix}
\right]
=
\left[
\begin{matrix}
0 \\
3 \\
\end{matrix}
\right]
\end{equation}
$$

In this view, the different rows of the coefficient vectors are treated as the independent "coordinates". For example, the $2$ and $-1$ in
$
\left[
\begin{matrix}
 2 \\
-1 \\
\end{matrix}
\right]
$
are treated as the x-coordinate and the y-coordinate.
And the solution to the euqations are interpreted as the **correct multiples** of these coefficient vectors such that the sum of them equals the the right hand side $\textbf{b}$.

Following this view, since the coefficient vectors constitute the columns of the coefficient matrix, we call them the $\textit{column vectors}$. And solving the linear equations is equivalent to finding the correct multiples of the column vectors. Or to say, finding the correct $\textit{linear combination}$ of the column vectors.

By linear combination, we mean adding/subtracting and multipling vectors by **scalars (pure numbers)**.

## 2. Gaussian Elimination

### Prerequisite: Matrix Multiplication

The most elementary and mechanical view of matrix multiplication is the **element view**. This view focuses on every single element of the resulting matrix, and is more often used in simple calculations. In particular, for a $m\times{n}$ matrix $\rm{A}$ and a $n\times{p}$ matrix $\rm{B}$, the resulting matrix $\rm{C}=\rm{AB}$ is a $m\times{p}$ matrix. And a general element in row $i$ and column $j$ of matrix $\rm{C}$ is:

$$
\begin{equation}
c_{ij} = \sum_{k=1}^{n} a_{ik}b_{kj}
\end{equation}
$$

From Section 1 we have seen that when a matrix is multiplied by a vector on the right, it is equivalent to taking the $\textit{linear combination}$ of the columns of that matrix. And that helps to introduce the **vector view** of matrix multiplication.

Notice that from the **element view** formula, all of the elements in column $j$ (denoted as $\rm{C_{j}}$) of the resulting matrix $\rm{C}$ are only influenced by the repective column $j$ (denoted as $\rm{B_{j}}$) in matrix $\rm{B}$, rather then the other columns from matrix $\rm{B}$. And the elements of $\rm{C_{j}}$ can be seen as the linear combination of the columns of $\rm{A}$. In other words, $\rm{A}B_{j}=C_{j}$.

For detailed demonstration, think about the case that a $3\times{2}$ matrix $\rm{A}$ multiplies a $2\times{3}$ matrix $\rm{B}$, where the resulting matrix is a $3\times{3}$ matrix $\rm{C}$. The first column of $\rm{C}$ comes from the first columns of $\rm{B}$ with elements computed as follows.

$$
\begin{equation}
\begin{aligned}
c_{11} &= a_{11}b_{11} + a_{12}b_{21} + a_{13}b_{31} \\
c_{21} &= a_{21}b_{11} + a_{22}b_{21} + a_{23}b_{31} \\
c_{31} &= a_{31}b_{11} + a_{32}b_{21} + a_{33}b_{31} \\
\end{aligned}
\end{equation}
$$

Going through the trick we have seen in Section 1 we may have,

$$
\begin{equation}
C_{1}
=
\left[
\begin{matrix}
c_{11} \\
c_{21} \\
c_{31} \\
\end{matrix}
\right]
=
b_{11}
\left[
\begin{matrix}
a_{11} \\
a_{21} \\
a_{31} \\
\end{matrix}
\right]
+
b_{21}
\left[
\begin{matrix}
a_{12} \\
a_{22} \\
a_{32} \\
\end{matrix}
\right]
+
b_{31}
\left[
\begin{matrix}
a_{13} \\
a_{23} \\
a_{33} \\
\end{matrix}
\right]
=
\left[
\begin{matrix}
a_{11} & a_{12} & a_{13}\\
a_{21} & a_{22} & a_{23}\\
a_{31} & a_{32} & a_{33}\\
\end{matrix}
\right]
\left[
\begin{matrix}
b_{11} \\
b_{21} \\
b_{31} \\
\end{matrix}
\right]
=
\rm{AB_{1}}
\end{equation}
$$

Therefore, in **verctor view**, we have the following important conclusions:

For the product of matrix multiplication $\rm{C}=\rm{AB}$,
1. The column of $\rm{C}$ is a linear combination of the columns of of $\rm{A}$.
2. The row of $\rm{C}$ is a linear combination of the rows of of $\rm{B}$.

The second conclusion can be proved like the way we do above. Since matrix multiplication is not commutative, the $\rm{A}$ and $\rm{B}$ cannot be interchanged under most circumstances.

### Warm-up from High Schools

In high school, students learned how to solve simple linear equations through eliminating variables and simplifying the system. For general linear equation systems, since they can be represented by matrices, the idea of finding the solutions is alike but extended to the operation of matrices.

For example, concerning the linear equations below, we may cancel the $x$ variable in $Eq2$ through multiplying the $Eq1$ by 3 and subtract that from $Eq2$, which will result in a new equation $Eq2^{'}$ with no $x$ variable. And then further eliminating $y$ in $Eq3$ by subtracting 2 times of $Eq2^{'}$ from it. Then the equations can be easily solved by $\textit{backsubstitution}$: solving z by $Eq3^{'}$, then using z to solve y by $Eq2^{'}$...

$$
\begin{equation}
\begin{aligned}
x + 2y + z &= 2  &Eq1\\
3x + 8y + z &= 12 &Eq2\\
    4y + z &= 2  &Eq3\\
\end{aligned}
\ \rightarrow\
\begin{aligned}
x + 2y + z &= 2  &Eq1\\
   2y - 2z &= 6 &Eq2^{'}\\
    4y + z &= 2 &Eq3\\
\end{aligned}
\ \rightarrow\
\begin{aligned}
x + 2y + z &= 2  &Eq1\\
   2y - 2z &= 6 &Eq2^{'}\\
      5z &= -10 &Eq3^{'}\\
\end{aligned}
\end{equation}
$$

### Matrix Representation of the Elimination Process

Since the linear equations can be represented by matrices, the elimination process above can also be stated in the language of matrices. First of all, the original equations can be represented by the following $\textit{augmented matrix}$. By $\textit{augmented}$, we mean adding an extra column to the coefficient matrix $\rm{A}$ to represent the right hand side vector $\textbf{b}$.

Remember that in order to have the same solution, when multiplying or subtracting the equations in the system should do the same manipulation to the both sides of the equations.

$$
\begin{equation}
\left[
\begin {array}{c|c}
\rm{A}&
\textbf{b}
\end {array}
\right]
=
\left[{
\begin {array}{c|c}
\begin{matrix}
1 & 2 & 1 \\
3 & 8 & 1 \\
0 & 4 & 1 \\
\end{matrix}&
\begin{matrix}
2 \\
12 \\
2\\
\end{matrix}
\end{array}}
\right]
\end{equation}
$$

Given this the elimination above can be restated in matrix language, which again, can be solved in ease.

$$
\begin{equation}
\left[
{\begin{array}{c|c}
\begin{matrix}
1 & 2 & 1 \\
3 & 8 & 1 \\
0 & 4 & 1 \\
\end{matrix}&
\begin{matrix}
2 \\
12 \\
2\\
\end{matrix}
\end{array}}
\right]
\ \rightarrow \
\left[
{\begin{array}{c|c}
\begin{matrix}
1 & 2 & 1 \\
0 & 2 & -2 \\
0 & 4 & 1 \\
\end{matrix}&
\begin{matrix}
2 \\
6 \\
2\\
\end{matrix}
\end{array}}
\right]
\ \rightarrow \
\left[
{\begin{array}{c|c}
\begin{matrix}
1 & 2 & 1 \\
0 & 2 & -2 \\
0 & 0 & 5 \\
\end{matrix}&
\begin{matrix}
2 \\
6 \\
-10\\
\end{matrix}
\end{array}}
\right]
=
\left[
\begin {array}{c|c}
\rm{U}&
\textbf{c}
\end {array}
\right]
\end{equation}
$$

### Mathematical View of Matrix Operation

The next thing we are going to do is to express the operations in the elimination process through mathematical language.

In the elimination process, what we have done is taking multiples of the equations and performing addition or subtraction operations among them.

In matrix language, we are manipulating the rows of the coefficient matrix or the augmented matrix. Recall from matrix multiplication, this can be expressed by multiplying a matrix on the left side, since taking multiples of the rows and performing addition or subtraction operations among them are just taking the $\textit{linear combinations}$.

Basically, the procedures taken in the elimination process can be **decomposed** into fundamental ones represented by the so-called $\textit{elimination matrices}$. In particular, the matrix representing the procedures needed to eliminate the entry $\textit a_{ij}$ is denoted as the elimination matrix $\textit E_{ij}$.

For example, for eliminating the 3 in row 2 and column 1 in the above matrix $\rm{A}$, we need to multiply the first row by 3 and subtract it from the second row of $\rm{A}$. In the language of matrix operation,
$
\begin{equation}
\textit E_{21}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
-3 & 1 & 0 \\
0 & 0 & 1 \\
\end{matrix}
\right]
\end{equation}
$.
And
$\begin{equation}
\textit E_{21}\rm{A}
=
\left[
\begin{matrix}
1 & 2 & 1 \\
0 & 2 & -2 \\
0 & 4 & 1 \\
\end{matrix}
\right]
\end{equation}
$ will eliminate the element $\textit a_{21}=3$.

Then the process can continue to eliminate $\textit a_{32}$ by multiplying
$
\begin{equation}
\textit E_{32}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & -2 & 1 \\
\end{matrix}
\right]
\end{equation}
$
and get the final result $\rm{U}$.
Since matrix multiplication is **associative**, all the elimination matrix can be grouped together and computed first, denoted as $\textit{E}$.

$$
\begin{equation}
\textit E_{32}\ (E_{21}\rm{A})
=
(\textit E_{32}\ \textit E_{21})\rm{A}
=
\left[
\begin{matrix}
1 & 2 & 1 \\
0 & 2 & -2 \\
0 & 0 & 5 \\
\end{matrix}
\right]
=
\rm{U}
\end{equation}
$$
$$
\textit E = \textit E_{32}\ \textit E_{21}
$$

And for the equations to have the same solutions as the original ones, the same multiplication should also be applied to the right hand side vector $\textbf{b}$, such that $\textit E\textbf{b} = \textbf{c}$.

Thus, the original linear equations system
$
\begin{equation}
\rm{A}
\textbf{x}
=
\textbf{b}
\end{equation}
$
has been transformed into a new system
$
\begin{equation}
(\textit E\rm{A})
\textbf{x}
=
\rm{U}
\textbf{x}
=
\textbf{c}
\end{equation}
$, which can be easily solved by the $\textit{backsubstitution}$ procedure.
And most importantly, the solutions stay the same.

### Additional comments

When conductiong the elimination process, the final elements we put on the diagonal are often called as $\textit{pivot}$, including the elements $\textit u_{11}=1$, $\textit u_{22}=2$, $\textit u_{33}=5$ in the resulting matrix $\rm{U}$. Those are the elements we usually used to eliminate other elements below them.

What worths noticing is that a $\textit{pivot}$ cannot be zero. In case of seeing a zero in the diagonal position, we will try to do **row exchange** with a row below and make the pivot non-zero. This can be done by multiplying a group of matrices called the $\textit{permutation matrices}$.
However, if no non-zero element is available, then the matrix will have some problems or some additional properties. We called this kind of matrix $\textit{not invertible}$.

For example, if
$\begin{equation}
\rm{A}
=
\left[
\begin{matrix}
1 & 2 & 1 \\
3 & 6 & 1 \\
0 & 4 & 1 \\
\end{matrix}
\right]
\end{equation}
$,
then after eliminating the $a_{21}=3$,
$\textit E_{21}\rm{A}
=
\left[
\begin{matrix}
1 & 2 & 1 \\
0 & 0 & -2 \\
0 & 4 & 1 \\
\end{matrix}
\right]
$
would have $a_{22}=0$.

At the time, we need to multiply an additional permutation matrix which will perform a swap between row 2 and row 3 of
$\textit E_{21}\rm{A}$. And the permutation matrix
$\begin{equation}
P_{23}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{matrix}
\right]
\end{equation}
$
would finish the job. (Using the **vector view** of matrix multiplication, multiplying a matrix from the left is taking linear combination of the rows. Therefore, the second row of the product is the third row of $\textit E_{21}\rm{A}$, and the third row is the second row of $\textit E_{21}\rm{A}$.Thus,
$\begin{equation}
P_{23}\textit E_{21}\rm{A}
=
\left[
\begin{matrix}
1 & 2 & 1 \\
0 & 4 & 1 \\
0 & 0 & -2 \\
\end{matrix}
\right]
\end{equation}
$.)

## 3. Inverse

For square matrices, there is one special family of members that deserves further attention. Each of them, say $\rm{A}$, is paired with another matrix called its $\textit{inverse}$ $\rm A^{-1}$ such that $\rm AA^{-1} = A^{-1}A = I$.

Not all of the square matrices have the corresponding inverse matrices. Those do have their corresponding inverse matrices are called $\textit{invertible}$ or $\textit{non-singular}$. And those cannot find the corresponding inverse matrices are called $\textit{singluar matrices}$.

The most simple example of inverse matrices are those corresponding to the $\textit{elimination matrices}$ or the $\textit{permutation matrices}$, which just undo the elimination or permutation steps.

For example, if
$
\begin{equation}
\textit E_{21}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
-3 & 1 & 0 \\
0 & 0 & 1 \\
\end{matrix}
\right]
\end{equation}
$,
then the corresponding inverse
$
\begin{equation}
\textit E_{21}^{-1}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
3 & 1 & 0 \\
0 & 0 & 1 \\
\end{matrix}
\right]
\end{equation}
$.

In words, what $\textit E_{21}^{-1}$ is doing is just adding 3 times of the row 1 to row 2 of matrix $\rm{A}$, which is just the opposite of what $\textit E_{21}$ is doing. Therefore, the result of the two operations is just the matrix $\rm{A}$ itself, represented by $\textit E_{21}^{-1}\textit E_{21}=\rm{I}$, ie multiplying an identity matrix.

### Relation with linear equations

The idea of inverse matrices is closely related to the solution of linear equations. If the coefficient matrix $\rm{A}$ has an inverse, then solution of the linear equation system
$
\begin{equation}
\rm{A}
\textbf{x}
=
\textbf{b}
\end{equation}
$
can be easily found by multiplying the inverse $\rm{A^{-1}}$ on both sides, such that

$$
\begin{equation}
\rm{A^{-1}A}
\textbf{x}
=
\rm{A^{-1}}\textbf{b} \\
\textbf{x}
=
\rm{A^{-1}}\textbf{b}
\end{equation}
$$.

Moreover, if $\rm{A}$ has an inverse, the corresponding **homogeneous equation system** $\rm{A}𝐱 = 0$ has the unique solution $𝐱=0$, which means that the column vectors in $\rm{A}$ are $\textit{linear independent}$. The concept of linear independence is the key to finding overall properties of the solutions to a linear equation system
$
\begin{equation}
\rm{A}
\textbf{x}
=
\textbf{b}
\end{equation}
$
and will be introduced in later parts.

### Gauss-Jordan Elimination

The next issue becomes, how to find an inverse of a square matrix, if it exists. In fact, finding the inverse can also be accomplished through **row operations**, given that $\rm A^{-1}A = I$.

Assume that we are doing a series of row operations and get the inverse of $\rm{A}$, and the product of those row operation matrices is $E=\rm A^{-1}$. Now suppose we perform these series of operations with two matrices $\rm{A}$ and $\rm{I}$ simutaneously, and record them through two blocks of a matrix. Then the inverse will automatically turn out in the second block.

$$
\begin{equation}
E
\begin{matrix}
\left[
\begin {array}{c|c}
\rm{A}&
\rm{I}
\end {array}
\right]
\end{matrix}
=
\begin{matrix}
\left[
\begin {array}{c|c}
\rm{I}&
E
\end {array}
\right]
\end{matrix}
\end{equation}
$$

Therefore, by **augmenting** an extra block of a identity matrix, we can keep track of the row operations which turn $\rm{A}$ into $\rm{I}$. Specifically, the row operations are done in the manner of Gaussian elimination, but continue to **eliminate the upper right entries** above the diagonal, and normalize the diagonal entries into 1.

For example, to find the inverse of the matrix
$
\left[
\begin{matrix}
1 & 3  \\
2 & 7  \\
\end{matrix}
\right]
$,
we augment an identity matrix to the right, and start to perform the elimination.

$$
\left[
\begin{array}{c|c}
\begin{matrix}
1 & 3  \\
2 & 7  \\
\end{matrix}&
\begin{matrix}
1 & 0  \\
0 & 1  \\
\end{matrix}
\end{array}
\right]
\ \rightarrow \
\left[
\begin{array}{c|c}
\begin{matrix}
1 & 3  \\
0 & 1  \\
\end{matrix}&
\begin{matrix}
1 & 0  \\
-2 & 1  \\
\end{matrix}
\end{array}
\right]
\ \rightarrow \
\left[
\begin{array}{c|c}
\begin{matrix}
1 & 0  \\
0 & 1  \\
\end{matrix}&
\begin{matrix}
7 & -3  \\
-2 & 1  \\
\end{matrix}
\end{array}
\right]
$$

## 4. First Factorization: A=LU

The elimination process produces $\textit E\rm{A} = \rm{U}$ and $\rm{U}\textbf{x} = \text{c}$, where $\textit {E}$ is the product of a series of elimination matrices. Assume that we do not need to perform any permutation, or to say multiply any permutation matrices during the elimination process.

We know that the $\textit{elimination matrices}$ always have their corresponding inverses, since what they do is taking $\textit{linear combinations}$ of the rows in $\rm{A}$. For example,
$
\begin{equation}
\textit E_{21}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
-3 & 1 & 0 \\
0 & 0 & 1 \\
\end{matrix}
\right]
\end{equation}
$
above is just taking 3 times of the first row and subtracting it from the second row of matrix $\rm{A}$. Therefore, adding 3 times of the first row to the second row of the **product**, namely multiplying
$
\begin{equation}
\textit E_{21}^{-1}
=
\left[
\begin{matrix}
1 & 0 & 0 \\
3 & 1 & 0 \\
0 & 0 & 1 \\
\end{matrix}
\right]
\end{equation}
$
on the left will give the identity matrix $\rm{I}$ back.

Back to the product of elimination matrices $\textit {E}$, $\textit {E}$ must have an inverse. If we multiply the inverse $\textit E^{-1}$ on both sides of $\textit E\rm{A} = \rm{U}$. We will get,

$$
\rm{A} = \textit E^{-1}\rm{U}
$$

Notice there are two important properties for the two matrices on the right,
1. $\textit {E}$ and $\textit E^{-1}$ are both lower triangular matrices.
2. $\rm {U}$ is a upper triangular matrix.

The fact that $\rm{U}$ is upper triangular is obvious, since we knock off all the entries below the diagonal in the elimination process. And every elimination matrix $\textit E_{ij}$ is lower triangular, since we always subtract a multiple of one **upper** row from row $i$ that contains the element $a_{ij}$. Similarly, the inverse of $\textit E_{ij}$, which **undo** the row operation in $\textit E_{ij}$, is also a lower triangular matrix. Lastly, the **product** of two lower triangular matrices is once agin a lower triangular matrix, which will give us property 1.

Because of these two properties, we often denote the $\textit E^{-1}$ on the right hand side as $\rm{L}$ to signify its lower triangular nature. Thus, we arrived at the first **factorization** of a matrix.

$$
\rm{A} = \rm{L} \rm{U}
$$

This means the information contains in $\rm{A}$ is now **decomposed** into two parts, storing in $\rm{L}$ and $\rm{U}$ respectively.

### Additional comments

In order to decompose $\rm{A}$ into two triangular matrices $\rm{L}$ and $\rm{U}$, we have assumed that no permutation will need to perform. That is surely not the case all the time. If we do need to perform any permutation, represented by a permutation matrix $\textit{P}$, we will accomplish it **before** the elimination or decomposition process, such that the result of permutation $\textit{P}\rm{A}$, will become a matrix that does not require further permutations.

This additional procedure gives us back to the case discussed above, with a generalized form of

$$
\textit{P} \rm{A} = \rm{L} \rm{U}
$$


# Homework - 1

#####Vectors and Matrices

Consider the matrix X and the vectors y and z below:

$$
\mathbf{X} =
\begin{bmatrix}
2&4 \\
1&3
\end{bmatrix}
$$

$$\mathbf{y} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}$$

$$\mathbf{z} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$$

**1. What is the inner product of the vectors y and z? (this is also sometimes called the dot product, and is sometimes written $y^Tz$)**

Inner Product of y and z ($y^T z$) = $(1 * 2) + ( 3 * 3) = 11$

```
# Inner Product of two vectors using Numpy
import numpy as np
y =  np.array([[1], [3]])
z = np.array([[2], [3]])

print np.vdot(y,z)
```

    11

**2. What is the product Xy?**

Xy = $\begin{bmatrix} (2 * 1) + ( 1 * 3) \\(4 * 1) + (3 * 3) \end{bmatrix}$

Xy = $\begin{bmatrix} 5 \\ 13 \end{bmatrix}$

```
# Matrix Vector multiplication using Numpy
X = np.array([[2,1],[4,3]])
y =  np.array([[1], [3]])

print X.dot(y)
```

    [[ 5]
     [13]]

**3. Is X invertible? If so, give the inverse, and if no, explain why not.**

A n x n square matrix A is said to be invertible or nonsingular if there exists any n x n square matrix B such that $$ AB = BA = I_n $$ , where $I_n$ is a n x n Identity matrix.

The inverse matrix of $X$, denoted by $X^{-1}$ is $$\mathbf{X^{-1}} = \begin{bmatrix} 1.5&-0.5 \\ -2&1 \end{bmatrix}$$

```
# Inverse of X in Numpy
from numpy.linalg import inv
X_inv = inv(X)
print 'Inverse of X is:'
print X_inv

print 'X * X_inv is an Identity Matrix'
print X.dot(X_inv)
```

    Inverse of X is:
    [[ 1.5 -0.5]
     [-2.   1. ]]
    X * X_inv is an Identity Matrix
    [[ 1.  0.]
     [ 0.  1.]]

**4. What is the rank of X?**

The rank of X is 2, since the column rank is 2.

##### Calculus

**1. If $y = x^3 + x − 5$ then what is the derivative of y with respect to x?**

Derivative of y w.r.t x is: $ \frac{dy}{dx} = 3x^2 + 1$

**2. If $y = x\:sin(z)\:e^{−x}$ then what is the partial derivative of y with respect to x?**

Using Product Rule and the fact that derivative of $e^{-x} = -e^{-x}$, $x = 1$, we have

$\frac{\partial y}{\partial x} = sin(z)\:e^{-x} - x\:sin(z)\:e^{-x}$

##### Probability and Statistics

Consider a sample of data S = {1, 1, 0, 1, 0} created by flipping a coin x five times, where 0 denotes that the coin turned up heads and 1 denotes that it turned up tails.

**1. What is the sample mean for this data?**

Sample Mean is $\frac{1 + 1 + 0 + 1 + 0}{5} = \frac{3}{5}$

**2. What is the sample variance for this data?**

Sample Variance is $\frac{6}{25}$

** 3. What is the probability of observing this data, assuming it was generated by flipping a coin with an equal probability of heads and tails (i.e. the probability distribution is p(x = 1) = 0.5, p(x = 0) = 0.5).**

Since it is a sequence of five independent tosses with P(H) = P(T) = 0.5,
P(S) = $(\frac{1}{2})^5 = \frac{1}{32}$

** 4. Note that the probability of this data sample would be greater if the value of p(x = 1) was not 0.5, but instead some other value. What is the value that maximizes the probability of the sample S. Please justify your answer.**

Answer:
Let p(x=1) = p. Therefore, we have p(x=0) = 1-p. If n is the number of tosses, we have the total probability as $\prod\limits_{i=1}^n p^{x_i} \: (1-p)^{x_i} = p^{\sum_{i=1}^n x_i} \: (1-p)^{n - \sum_{i=1}^n x_i}$

Let y = $ {\sum_{i=1}^n x_i} $.

The above equation can be now written as $ p^{y} \: (1-p)^{n - y}$

Since we need to find the value of p that maximizes the probability of sample S, we can take the log of the above equation, take its derivative w.r.t p, set it to zero and solve for p.

Taking the log, we have $y\>log(p) + (n-y)\>log(1-p)$, since $log(a^p) = p\>log(a)$ and $log(a * b) = log(a) + log(b)$.

Taking the derivative w.r.t p and setting it to 0 and solving for p, we have:

$$\begin{equation}
\begin{split}
\frac{dl(p)}{dp} & = \frac{y}{p} + \frac{n-y}{(1-p)} = 0 \\
 \implies 0 & = \frac{(1-p)\>y + p\>(n - y)}{p\>(1-p)}  \\
 \implies 0 & = \frac{y - py + py - pn}{p\>(1-p)} \\
 \implies 0 & = \frac{y - pn}{p\>(1-p)} \\
 pn & = y \\
 p & = \frac{1}{n} y \\
 p & = \frac{1}{n} {\sum_{i=1}^n x_i} \\
\end{split}
\end{equation}$$

Plugging in the values for $x_i$ and n=5, we get $\boxed{p = \frac{3}{5}}$

```
# Custom Styling - Please ignore
# Custom CSS Styling
from IPython.core.display import HTML

def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()

```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


# Diving into symbolic computing with SymPy

```
from sympy import *
init_printing()
```

```
var('x y')
```

```
x, y = symbols('x y')
```

```
expr1 = (x + 1) ** 2
expr2 = x**2 + 2 * x + 1
```

```
expr1 == expr2
```

    False

```
simplify(expr1 - expr2)
```

```
expr1.subs(x, expr1)
```

```
expr1.subs(x, pi)
```

```
expr1.subs(x, S(1) / 2)
```

```
_.evalf()
```

```
f = lambdify(x, expr1)
```

```
import numpy as np
f(np.linspace(-2., 2., 5))
```

    array([ 1.,  0.,  1.,  4.,  9.])


```python
from scipy import optimize
import cvxopt

import matplotlib.pyplot as plt

import numpy as np

import sympy
sympy.init_printing()
```

```python
# Minimize the area of a cylinder with unit volume
r, h = sympy.symbols("r, h")

Area = 2 * sympy.pi * r**2 + 2 * sympy.pi * r * h

Volume = sympy.pi * r**2 * h

h_r = sympy.solve(Volume - 1)[0]

Area_r = Area.subs(h_r)

rsol = sympy.solve(Area_r.diff(r))[0]
rsol
```

```python
_.evalf()
```

```python
Area_r.diff(r, 2).subs(r, rsol)
```

```python
Area_r.subs(r, rsol)
```

```python
_.evalf()
```

```python
def f(r):
    return 2 * np.pi * r**2 + 2 / r

r_min = optimize.brent(f, brack=(0.1, 4))
r_min
```

```python
f(r_min)
```

```python
optimal = optimize.minimize_scalar(f, bracket=(0.1, 4))
```

```python
r = np.linspace(0.065, 2, 1000)

fig, ax = plt.subplots()

ax.plot(r, f(r), color="blue", label="$f(r)=2{\pi}r^2 + 2 / r$")
ax.plot(optimal.x, optimal.fun, "*", color="red", markersize=10)
ax.legend()
```

### Multivariable optimization example

```python
x1, x2 = sympy.symbols("x_1, x_2")

f_sym = (x1-1)**4 + 5 * (x2-1)**2 - 2*x1*x2

fprime_sym = [f_sym.diff(x_) for x_ in (x1, x2)]
fprime_sym

```

```python
sympy.Matrix(fprime_sym)
```

$\displaystyle \left[\begin{matrix}- 2 x_{2} + 4 \left(x_{1} - 1\right)^{3}\\- 2 x_{1} + 10 x_{2} - 10\end{matrix}\right]$

```python
fhess_sym = [
    [f_sym.diff(x1_, x2_) for x1_ in (x1, x2)]
    for x2_ in (x1, x2)
]
fhess_sym
```

```python
sympy.Matrix(fhess_sym)
```

$\displaystyle \left[\begin{matrix}12 \left(x_{1} - 1\right)^{2} & -2\\-2 & 10\end{matrix}\right]$

```python
f_lmbda = sympy.lambdify((x1, x2), f_sym, 'numpy')
fprime_lmbda = sympy.lambdify((x1, x2), fprime_sym, 'numpy')
fhess_lmbda = sympy.lambdify((x1, x2), fhess_sym, 'numpy')
```

```python
def func_XY_to_X_Y(f):
    """
    Wrapper for f(X) -> f(X[0], X[1])
    """
    return lambda X: np.array(f(X[0], X[1]))

f = func_XY_to_X_Y(f_lmbda)
fprime = func_XY_to_X_Y(fprime_lmbda)
fhess = func_XY_to_X_Y(fhess_lmbda)
```

```python
x_opt = optimize.fmin_ncg(f, (0, 0), fprime=fprime, fhess=fhess)
```

    Optimization terminated successfully.
             Current function value: -3.867223
             Iterations: 8
             Function evaluations: 10
             Gradient evaluations: 10
             Hessian evaluations: 8

```python
x_opt
```

    array([1.88292613, 1.37658523])

### Contour plot example

```python
fig, ax = plt.subplots(figsize=(6, 4))
x_ = y_ = np.linspace(-1, 4, 100)

X, Y = np.meshgrid(x_, y_)

c = ax.contour(X, Y, f_lmbda(X, Y), 50)
ax.plot(x_opt[0], x_opt[1], 'r*', markersize=15)

ax.set_xlabel(r"$x_1$", fontsize=18)
ax.set_ylabel(r"$x_2$", fontsize=18)

plt.colorbar(c, ax=ax)
```

```python
# Broyden-Fletcher-Goldfarb-Shanno (BFGS)

x_opt = optimize.fmin_bfgs(f, (0, 0), fprime=fprime)
```

    Optimization terminated successfully.
             Current function value: -3.867223
             Iterations: 9
             Function evaluations: 13
             Gradient evaluations: 13

```python
x_opt
```

    array([1.88292645, 1.37658596])

```python
# conjugate gradient methods

x_opt = optimize.fmin_cg(f, (0, 0), fprime=fprime)
```

    Optimization terminated successfully.
             Current function value: -3.867223
             Iterations: 8
             Function evaluations: 18
             Gradient evaluations: 18

```python
x_opt
```

    array([1.88292612, 1.37658523])

```python
# without providing a function for the gradient as well

x_opt = optimize.fmin_bfgs(f, (0, 0))
```

    Optimization terminated successfully.
             Current function value: -3.867223
             Iterations: 9
             Function evaluations: 39
             Gradient evaluations: 13

```python
x_opt
```

    array([1.88292644, 1.37658595])

```python
def f(X):
    x, y = X
    return (4 * np.sin(np.pi * x) + 6 * np.sin(np.pi * y)) + (x - 1)**2 + (y - 1)**2
```

```python
x_start = optimize.brute(f, (slice(-3, 5, 0.5), slice(-3, 5, 0.5)), finish=None)
x_start
```

    array([1.5, 1.5])

```python
f(x_start)
```

```python
x_opt = optimize.fmin_bfgs(f, x_start)
```

    Optimization terminated successfully.
             Current function value: -9.520229
             Iterations: 4
             Function evaluations: 21
             Gradient evaluations: 7

```python
x_opt
```

    array([1.47586906, 1.48365787])

```python
f(x_opt)
```

```python
def func_X_Y_to_XY(f, X, Y):
    """
    Wrapper for f(X, Y) -> f([X, Y])
    """
    s = np.shape(X)
    return f(np.vstack([X.ravel(), Y.ravel()])).reshape(*s)

fig, ax = plt.subplots(figsize=(6, 4))

x_ = y_ = np.linspace(-3, 5, 100)
X, Y = np.meshgrid(x_, y_)

c = ax.contour(X, Y, func_X_Y_to_XY(f, X, Y), 25)

ax.plot(x_opt[0], x_opt[1], 'r*', markersize=15)

ax.set_xlabel(r"$x_1$", fontsize=18)
ax.set_ylabel(r"$x_2$", fontsize=18)

plt.colorbar(c, ax=ax)
```

```python
optimize.minimize(f, x_start, method= 'BFGS')
```

          fun: -9.520229273055016
     hess_inv: array([[2.41596001e-02, 4.61008275e-06],
           [4.61008275e-06, 1.63490348e-02]])
          jac: array([-7.15255737e-07, -7.15255737e-07])
      message: 'Optimization terminated successfully.'
         nfev: 21
          nit: 4
         njev: 7
       status: 0
      success: True
            x: array([1.47586906, 1.48365787])


# Matrix

## Create a matrix $$\begin{bmatrix} 1 & 2 \\ -2 & 3 \\ 3 & 4 \end{bmatrix}$$

```python
# Create a matrix
from sympy import *

init_printing(use_unicode=True)

Matrix([[1, 2], [-2, 3], [3, 4]])
```

$\displaystyle \left[\begin{matrix}1 & 2\\-2 & 3\\3 & 4\end{matrix}\right]$

## Create some matrix
$$M = \begin{bmatrix} 1 & 2 \\ -2 & 1 \end{bmatrix}$$
$$N = \begin{bmatrix} -1 \\ 1 \end{bmatrix}$$

```python
# Manipulate the matrix
M = Matrix([[1, 2], [-2, 1]])
N = Matrix([-1, 1])
N
```

$\displaystyle \left[\begin{matrix}-1\\1\end{matrix}\right]$

## Manipulate the matrix
$$M \times N$$

```python
M*N
```

$\displaystyle \left[\begin{matrix}1\\3\end{matrix}\right]$

```python
# Basic Operations
# Get the shape of the matrix M
shape(M)
```

```python
# Accessing rows
M.row(0)
```

$\displaystyle \left[\begin{matrix}1 & 2\end{matrix}\right]$

```python
# Accessing columns
M.col(-1)
```

$\displaystyle \left[\begin{matrix}2\\1\end{matrix}\right]$

## Addition of M and N $$\begin{bmatrix} 1 & 2 \\ -2 & 1 \end{bmatrix} + \begin{bmatrix} -1 & 1 \\ 1 & 2 \end{bmatrix} = \begin{bmatrix} 0 & 3 \\ -1 & 3 \end{bmatrix}$$

```python
# Addition of matrices
M = Matrix([[1, 2], [-2, 1]])
N = Matrix([[-1, 1], [1, 2]])
M + N
```

$\displaystyle \left[\begin{matrix}0 & 3\\-1 & 3\end{matrix}\right]$

## Multiplication of M and N $$\begin{bmatrix} 1 & 2 \\ -2 & 1 \end{bmatrix} \times \begin{bmatrix} -1 & 1 \\ 1 & 2 \end{bmatrix} = \begin{bmatrix} 1 & 5 \\ 3 & 0 \end{bmatrix}$$

```python
# Multiplication of matrices
M*N
```

$\displaystyle \left[\begin{matrix}1 & 5\\3 & 0\end{matrix}\right]$

## Power of matrix M $$\begin{bmatrix} 1 & 2 \\ -2 & 1 \end{bmatrix}^2 = \begin{bmatrix} -3 & 4 \\ -4 & -3 \end{bmatrix}$$

```python
# Power of a matrix
M**2
```

$\displaystyle \left[\begin{matrix}-3 & 4\\-4 & -3\end{matrix}\right]$

## Inverse of matrix M $$\begin{bmatrix} 1 & 2 \\ -2 & 1 \end{bmatrix}^{-1} = \begin{bmatrix} \frac{1}{5} & -\frac{2}{5} \\ \frac{2}{5} & \frac{1}{5} \end{bmatrix}$$

```python
# Inverse of a matrix
M**-1
```

$\displaystyle \left[\begin{matrix}\frac{1}{5} & - \frac{2}{5}\\\frac{2}{5} & \frac{1}{5}\end{matrix}\right]$

## Transpose of matrix M $$MT = \begin{bmatrix} 1 & -2 \\ 2 & 1 \end{bmatrix}$$

```python
# Transpose of a matrix
M.T
```

$\displaystyle \left[\begin{matrix}1 & -2\\2 & 1\end{matrix}\right]$

## Create a matrix identity $$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 1 \end{bmatrix}$$

```python
# Matrix Constructor
# Identity Matrix
eye(3)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

## Create all zeros matrix $$\begin{bmatrix} 0 & 0 \\ 0 & 0 \end{bmatrix}$$

```python
# All zeros matrix
zeros(2, 2)
```

$\displaystyle \left[\begin{matrix}0 & 0\\0 & 0\end{matrix}\right]$

## Create all ones matrix $$\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$$

```python
# All ones matrix
ones(2, 2)
```

$\displaystyle \left[\begin{matrix}1 & 1\\1 & 1\end{matrix}\right]$

## Create a diagonal matrix $$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \end{bmatrix}$$

```python
# Diagonal Matrix
diag(1, 2, 3)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 2 & 0\\0 & 0 & 3\end{matrix}\right]$

## Determinant of matrix M $$a \times d - b \times c$$ $$=1 \times 1 - 2 \times (-2) = 5$$

```python
# Determinant of a matrix
M.det()
```


# Exercises 2 - Answers

## Exercise 1

The profit that you make for each kwH is $2-0.01x^2-(1-0.01x)$. Thus, the amount of profit that you make is
$$
(1+0.01x-0.01x^2)x=x+0.01x^2-0.01x^3.
$$
Thus, the optimization problem is
$$
\begin{align}
\max \qquad & x+0.01x^2-0.01x^3\\
\text{s.t.} \qquad & 0\leq x \leq 50.
\end{align}
$$

## Exercise 2

```python
def bisection_line_search(a,b,f,L,epsilon):
    x = a
    y = b
    while y-x>2*L:
        if f((x+y)/2+epsilon)>f((x+y)/2-epsilon):
            y=(x+y)/2+epsilon
        else:
            x = (x+y)/2-epsilon
    return (x+y)/2
```

```python
def f_ex2(x):
    return (1-x)**2+x
```

```python
print f_ex2(0.5), f_ex2(0), f_ex2(1)
```

    0.75 1 1

```python
bisection_line_search(0,2,f_ex2,0.0001,1e-6)
```

    0.49993946490478514

## Exercise 3

```python
import math
def golden_section_line_search(a,b,f,L):
    x = a
    y = b
    f_left = f(y-(math.sqrt(5.0)-1)/2.0*(y-x)) #funtion eval
    f_right = f(x+(math.sqrt(5.0)-1)/2.0*(y-x)) #function eval
    while y-x>2*L:
        if f_left > f_right:
            x = y-(math.sqrt(5.0)-1)/2.0*(y-x)
            f_left = f_right #no function eval
            f_right = f(x+(math.sqrt(5.0)-1)/2.0*(y-x)) #function eval
        else:
            y = x+(math.sqrt(5.0)-1)/2.0*(y-x)
            f_right = f_left #no function eval
            f_left = f(y-(math.sqrt(5.0)-1)/2.0*(y-x)) #function eval
    return (x+y)/2
```

```python
golden_section_line_search(0,2,f_ex2,0.0001)
```

    0.4999795718254958

## Exercise 4

Now, $f(x) = (1-x)^2+x$. Thus,
$$
f'(x)=2(1-x)(-1)+1= -1 +2x.
$$

If $f'(x) = 0$, then $2x=1$ and $x=\frac 12$.
This is a local minimum since,
$$
f''(x) = 2,
$$
which is greater than $0$.

This means that the algorithms work!


sergazy.nurbavliyev@gmail.com © 2021

## Detecting Unfair Coin

Question: Assume we have a fair coin (one side heads and the other side tails) also we have an unfair coin (both sides heads). You pick randomly one coin, and flip it 5 times. You get 5 heads in a row. What is the probability that you are indeed flipping the unfair coin?

### Intuition

Assume you flip a fair coin 5 times, what is the probability of getting 5 heads in a row. That is $\frac{1}{2^5}$.
If this is confusing, think this way, take 5 fair coin and flip once, and now the same question what is the probability of getting 5 heads? The same answer that is $\frac{1}{2^5}$. Which is around 0.03125. If you dont have a calculator, then rougly you can guess $3/96\approx 3/100=0.03$. That tells us probability of flipping an unfair coin is around 0.97.

If you are flipping an unfair coin well then probability of getting 5 head is 1. Now correct answer should be close to these numbers.

```python
1/32
```

    0.03125

```python

```

### Theoritical result

As you can guess we will use Bayes theorem here.
We will denote with $U$ letter if we are flipping the unfair coin and $F$ letter if we are flipping a fair
Since we are picking randomly one coin, $\mathbb{P}(U)=\mathbb{P}(F)=1/2$.

Let 5H represent the case where we get 5 heads in a row.  Then  we want to find the probability that we are flipping the unfair coin, given that we saw 5 heads in a row.
i.e. $\mathbb{P}(U|5H)$.
If we are given an unfair coin and then the probability of getting 5 heads in a row would be 1. i.e. $\mathbb{P}(5H|U)=1$. With the same logic if we have fair coin and then the probability of getting 5 heads in a row would be 1/32. i.e. $\mathbb{P}(5H|F)=\frac{1}{2^5}.$ We actually collected all the information we want. Now using Bayes rule we get
\begin{equation}
\mathbb{P}(U|5H)=\dfrac{\mathbb{P}(5H|U)\mathbb{P}(U)}{\mathbb{P}(5H|U)\mathbb{P}(U)+\mathbb{P}(5H|F)\mathbb{P}(F)}=\frac{\frac{1}{2}*1}{\frac{1}{2}*1+\frac{1}{2}*\frac{1}{2^5}}=\frac{32}{33}= 0.9696969696969697
\end{equation}

```python
32/33
```

    0.9696969696969697

## Python code for our intuition

```python
import scipy.stats as stats
def fair(n_trials):
    return stats.bernoulli.rvs(0.5, size=n_trials) # this returns an array of 0s and 1s
def unfair(n_trials):
    return stats.bernoulli.rvs(1, size=n_trials) ## this returns an array of 1s
```

```python
import numpy as np
count_f=0
count_unf=0
for j in range(100):
    data = fair(5)
    if np.sum(data)==5:
        count_f+=1
for j in range(100):
    data = unfair(5)
    if np.sum(data)==5:
        count_unf+=1

```

```python
count_f/100,count_unf/10000
```

    (0.06, 0.01)

```python

```