# Solving Linear Equations

```python
import numpy as np
import scipy.linalg as la
```

## Linear Equations

Consider a set of  $m$ linear equations in $n$ unknowns:

\begin{align*}
a_{11} x_1 + &a_{12} x_2& +& ... + &a_{1n} x_n &=& b_1\\
\vdots  && &&\vdots &= &\vdots\\
a_{m1} x_1 + &a_{m2} x_2& +& ... + &a_{mn} x_n &=&b_m
\end{align*}

We can let

\begin{align*}
    A=\left[\begin{matrix}a_{11}&\cdots&a_{1n}\\
               \vdots & &\vdots\\
               a_{m1}&\cdots&a_{mn}\end{matrix}\right]
\end{align*}

\begin{align*}
x = \left[\begin{matrix}x_1\\
               \vdots\\
               x_n\end{matrix}\right] & \;\;\;\;\textrm{   and } &
b =  \left[\begin{matrix}b_1\\
               \vdots\\
               b_m\end{matrix}\right]
\end{align*}

and re-write the system

$$ Ax = b$$

### Linear independence and existence of solutions

* If $A$ is an $m\times n$ matrix and $m>n$, if all $m$ rows are linearly independent, then the system is *overdetermined* and *inconsistent*.  The system cannot be solved exactly.  This is the usual case in data analysis, and why least squares is so important. For example, we may be finding the parameters of a linear model, where there are $m$ data points and $n$ parameters.

* If $A$ is an $m\times n$ matrix and $m<n$, if all $m$ rows are linearly independent, then the system is *underdetermined* and there are *infinite* solutions.

* If $A$ is an $m\times n$ matrix and some of its rows are linearly dependent, then the system is *reducible*.  We can get rid of some equations. In other words, there are equations in the system that do not give us any new information.

* If $A$ is a square matrix and its rows are linearly independent, the system has a unique solution. ($A$ is invertible.)

For a linear system, we can only get a unique solution, no solution, or infinite solutions.

### Using `solve` to find unique solutions

\begin{align}
x + 2y &= 3 \\
3x + 4y &= 17
\end{align}

```python
A = np.array([[1,2],[3,4]])
A
```

    array([[1, 2],
           [3, 4]])

```python
b = np.array([3,17]).reshape((-1,1))
b
```

    array([[ 3],
           [17]])

```python
x = la.solve(A, b)
x
```

    array([[11.],
           [-4.]])

```python
np.allclose(A @ x, b)
```

    True

#### Under the hood of `solve`

The `solve` function uses the `dgesv` fortran function in `lapack` (Linear Algebra Package) to do the actual work. In turn, `lapack` calls even lower level routines in `blas` (Basic Linear Algebra Subprograms). In particular, `solve` uses the LU matrix decomposition that we will show in detail shortly.

There is rarely any reason to use `blas` or `lapack` functions directly becuase the `linalg` package provides more convenient functions that also perfrom error checking, but you can use Python to experiment with `lapack` or `blas` before using them in a language like C or Fortran.

- [How to interpret lapack function names](http://www.netlib.org/lapack/lug/node24.html)
- [Summary of BLAS functions](http://cvxopt.org/userguide/blas.html)
- [Sumary of Lapack functions](http://cvxopt.org/userguide/lapack.html)

```python
import scipy.linalg.lapack as lapack
```

```python
lu, piv, x, info = lapack.dgesv(A, b)
x
```

    array([[11.],
           [-4.]])

## Gaussian elimination

Let's review how Gaussian elimination (ge) works.  We will deal with a $3\times 3$ system of equations for conciseness, but everything here generalizes to the $n\times n$ case. Consider the following equation:

$$\left(\begin{matrix}a_{11}&a_{12} & a_{13}\\a_{21}&a_{22}&a_{23}\\a_{31}&a_{32}&a_{33}\end{matrix}\right)\left(\begin{matrix}x_1\\x_2\\x_3\end{matrix}\right) = \left(\begin{matrix}b_1\\b_2\\b_3\end{matrix}\right)$$

For simplicity, let us assume that the leftmost matrix $A$ is non-singular.  To solve the system using ge, we start with the 'augmented matrix':

$$\left(\begin{array}{ccc|c}a_{11}&a_{12} & a_{13}& b_1 \\a_{21}&a_{22}&a_{23}&b_2\\a_{31}&a_{32}&a_{33}&b_3\end{array}\right)$$

We begin at the first entry, $a_{11}$.  If $a_{11} \neq 0$, then we divide the first row by $a_{11}$ and then subtract the appropriate multiple of the first row from each of the other rows, zeroing out the first entry of all rows. (If $a_{11}$ is zero, we need to permute rows.  We will not go into detail of that here.)  The result is as follows:

$$\left(\begin{array}{ccc|c}
1 & \frac{a_{12}}{a_{11}} & \frac{a_{13}}{a_{11}} & \frac{b_1}{a_{11}} \\
0 & a_{22} - a_{21}\frac{a_{12}}{a_{11}} & a_{23} - a_{21}\frac{a_{13}}{a_{11}}  & b_2 - a_{21}\frac{b_1}{a_{11}}\\
0&a_{32}-a_{31}\frac{a_{12}}{a_{11}} & a_{33} - a_{31}\frac{a_{13}}{a_{11}}  &b_3- a_{31}\frac{b_1}{a_{11}}\end{array}\right)$$

We repeat the procedure for the second row, first dividing by the leading entry, then subtracting the appropriate multiple of the resulting row from each of the third and first rows, so that the second entry in row 1 and in row 3 are zero. Gaussian elimination stops at *row echelon* form (upper triangular, with ones on the diagonal), and then uses *back substitution* to obtain the final answer.

Note that in some cases, it is necessary to permute rows to obtain row echelon form (when the pivot would otherwise be zero). This is called *partial pivoting*.

### Example

We perform Gaussian elimination on the the following augmented matrix

$$
\left(
\begin{array}{ccc|c}
1 & 3 & 4 & 1\\
2 & 1 & 3 & 2 \\
4 & 7 & 2 & 3
\end{array}
\right)
$$

We need to multiply row $1$ by $2$ and subtract from row $2$ to eliminate the first entry in row $2$, and then multiply row $1$ by $4$ and subtract from row $3$.

$$
\left(
\begin{array}{ccc|c}
1 & 3 & 4 & 1 \\
0 & -5 & -5 & 0\\
0 & -5 &-14 & -1
\end{array}
\right)
$$

And then we eliminate the second entry in the third row:

$$
\left(
\begin{matrix}
1 & 3 & 4 & 1\\
0 & -5 & -5 & 0 \\
0 & 0 & -9 & -1
\end{matrix}
\right)
$$

We can now do back-substitution to solve

$$
9x_3 = 1 \\
x_2 = -x_3 = 0 \\
x_1 = -3x_2 -4x_3 + 1
$$

to get

$$
x_1 = 8/9 \\
x_2 = -1/9 \\
x_3 = 1/9
$$

### Check

```python
A = np.array([
    [1,3,4],
    [2,1,3],
    [4,7,2]
])
b = np.array([1,2,3]).reshape((-1,1))

la.solve(A, b)
```

    array([[ 0.88888889],
           [-0.11111111],
           [ 0.11111111]])

## Gauss-Jordan elimination

With Gauss-Jordan elimination, we make all the pivots equal to 1, and set all entries above and below a pivot 0.

$$
\left(
\begin{matrix}
1 & 0 & 0 & 8/9 \\
0 & 1 & 0 & -1/9 \\
0 & 0 & 1 & 1/9
\end{matrix}
\right)
$$

Then we can just read off the values of $x_1$, $x_2$ and $x_3$ directly. This is known as the *reduced row echelon form*.

## Gaussian-Jordan elimination and the number of solutions

Consider 3 matrices $m > n$, $m = n$ and $m < n$ and provide some intuition as to the existence of a no, unique or infinite solutions.

## LU Decomposition

LU stands for 'Lower Upper', and so an LU decomposition of a matrix $A$ is a decomposition so that
$$A= LU$$
where $L$ is lower triangular and $U$ is upper triangular.

Now, LU decomposition is essentially gaussian elimination, but we work only with the matrix $A$ (as opposed to the augmented matrix).

Gaussian elimination is all fine when we are solving a system one time, for one outcome $b$.  Many applications involve solutions to multiple problems, where the left-hand-side of our matrix equation does not change, but there are many outcome vectors $b$.  In this case, it is more efficient to *decompose* $A$.

First, we start just as in ge, but we 'keep track' of the various multiples required to eliminate entries.  For example, consider the matrix

$$A = \left(\begin{matrix} 1 & 3 & 4 \\
                           2 & 1 & 3\\
                           4 & 7 & 2
                           \end{matrix}\right)$$

We need to multiply row $1$ by $2$ and subtract from row $2$ to eliminate the first entry in row $2$, and then multiply row $1$ by $4$ and subtract from row $3$. Instead of entering zeroes into the first entries of rows $2$ and $3$, we record the multiples required for their elimination, as so:

$$\left(\begin{matrix} 1 & 3 & 4 \\
                           (2)& -5 & -5\\
                           (4)&-5 &-14
                           \end{matrix}\right)$$

And then we eliminate the second entry in the third row:

$$\left(\begin{matrix} 1 & 3 & 4 \\
                           (2)& -5 & -5\\
                           (4)& (1)&-9
                           \end{matrix}\right)$$

And now we have the decomposition:
$$L= \left(\begin{matrix} 1 & 0 & 0 \\
                           2& 1 & 0\\
                           4& 1 &1
                           \end{matrix}\right) \,
                          U = \left(\begin{matrix} 1 & 3 & 4 \\
                           0& -5 & -5\\
                           0&0&-9
                           \end{matrix}\right)$$

### Elementary matrices

Why does the algorithm for LU work? Gaussian elimination consists of 3 elementary operations

- Op1: swapping two rows
- Op2: replace a row by the sum of that row and a multiple of another
- Op3: multiplying a row by a non-zero scalar

These can be recast as matrix operations - in particular, pre-multiplication with corresponding elementary matrices.

- Op1: swapping two rows uses a permutation matrix

$$
\begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}\begin{bmatrix}
3 & 4 \\
1 & 2
\end{bmatrix} = \begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix}
$$

- Op2: replace a row by the sum of that row and a multiple of another uses an lower triangular matrix

$$
\begin{bmatrix}
1 & 0 \\
-3 & 1
\end{bmatrix}\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix} = \begin{bmatrix}
1 & 2 \\
0 & -2
\end{bmatrix}
$$

Note: The inverse operation just substitutes the negative of the multiple, and is also lower triangular.

- Op3: multiplying a row by a non-zero scalar uses an lower triangular matrix

$$
\begin{bmatrix}
1 & 0 \\
0 & -0.5
\end{bmatrix}\begin{bmatrix}
1 & 2 \\
0 & -2
\end{bmatrix} = \begin{bmatrix}
1 & 2 \\
0 & 1
\end{bmatrix}
$$

Note: The inverse operation just substitutes the inverse of the scalar, and is also lower triangular.

Multiplying an upper triangular matrix by another lower triangular matrix gives an lower triangular matrix. Hence if we put the permutations aside (i.e. keep a separate permutation matrix), Gaussian elimination can be expressed as a product of lower triangular matrices, which is just another lower triangular matrix $L$, with the original matrix $A$ to give an upper triangular matrix $U$. The lower triangular matrix $L$ is then the product of the inverse operations.

```python
A = np.array([
    [1,3,4],
    [2,1,3],
    [4,7,2]
])
```

```python
A
```

    array([[1, 3, 4],
           [2, 1, 3],
           [4, 7, 2]])

#### Construct U

```python
b1 = np.array([
    [1, 0, 0],
    [-2, 1, 0],
    [0, 0, 1]
])
```

```python
b2 = np.array([
    [1, 0, 0],
    [0, 1, 0],
    [-4, 0, 1]
])
```

```python
b3 = np.array([
    [1, 0, 0],
    [0, 1, 0],
    [0, -1, 1]
])
```

```python
b1 @ A
```

    array([[ 1,  3,  4],
           [ 0, -5, -5],
           [ 4,  7,  2]])

```python
b2 @ b1 @ A
```

    array([[  1,   3,   4],
           [  0,  -5,  -5],
           [  0,  -5, -14]])

```python
b3 @ b2 @ b1 @ A
```

    array([[ 1,  3,  4],
           [ 0, -5, -5],
           [ 0,  0, -9]])

```python
U = b3 @ b2 @ b1 @ A
U
```

    array([[ 1,  3,  4],
           [ 0, -5, -5],
           [ 0,  0, -9]])

#### Construct L

```python
ib1 = np.array([
    [1, 0, 0],
    [2, 1, 0],
    [0, 0, 1]
])
```

```python
ib2 = np.array([
    [1, 0, 0],
    [0, 1, 0],
    [4, 0, 1]
])
```

```python
ib3 = np.array([
    [1, 0, 0],
    [0, 1, 0],
    [0, 1, 1]
])
```

```python
L = ib1 @ ib2 @ ib3
L
```

    array([[1, 0, 0],
           [2, 1, 0],
           [4, 1, 1]])

#### A is factorized into LU

```python
L @ U
```

    array([[1, 3, 4],
           [2, 1, 3],
           [4, 7, 2]])

```python
A
```

    array([[1, 3, 4],
           [2, 1, 3],
           [4, 7, 2]])

We can now use the LU decomposition to solve for *any* $b$ without having to perform Gaussian elimination again.

- First solve $Ly = b$
- Then solve $Ux = y$

Since $L$ and $U$ are triangular, they can be cheaply solved by substitution.

```python
b
```

    array([[1],
           [2],
           [3]])

```python
y = la.solve_triangular(L, b, lower=True)
y
```

    array([[ 1.],
           [ 0.],
           [-1.]])

```python
x = la.solve_triangular(U, y)
x
```

    array([[ 0.88888889],
           [-0.11111111],
           [ 0.11111111]])

### LDU Decomposition

Note that $L$ is a unit triangular matrix while $U$ is not. It is sometimes instructive to factorize $A = LDU$ where both $L$ and $U$ are unit triangular, and $D$ is a diagonal matrix.

```python
U
```

    array([[ 1,  3,  4],
           [ 0, -5, -5],
           [ 0,  0, -9]])

```python
D = np.diag(np.diag(U))
D
```

    array([[ 1,  0,  0],
           [ 0, -5,  0],
           [ 0,  0, -9]])

The next step is just element-wise division.

```python
U1 = U/np.diag(U)[:, None]
U1
```

    array([[ 1.,  3.,  4.],
           [-0.,  1.,  1.],
           [-0., -0.,  1.]])

```python
D @ U1
```

    array([[ 1.,  3.,  4.],
           [ 0., -5., -5.],
           [ 0.,  0., -9.]])

```python
np.allclose(A, L @ D @ U1)
```

    True

### LU Decomposition in practice

```python
P, L, U = la.lu(A)
```

```python
L
```

    array([[ 1.  ,  0.  ,  0.  ],
           [ 0.5 ,  1.  ,  0.  ],
           [ 0.25, -0.5 ,  1.  ]])

```python
U
```

    array([[ 4. ,  7. ,  2. ],
           [ 0. , -2.5,  2. ],
           [ 0. ,  0. ,  4.5]])

P is a permutation matrix.

```python
P
```

    array([[0., 0., 1.],
           [0., 1., 0.],
           [1., 0., 0.]])

In practice, we can store both L and U in a single matrix LU.

```python
LU, P = la.lu_factor(A)
```

```python
LU
```

```python
la.lu_solve((LU, P), b)
```

## Cholesky Decomposition

Recall that a square matrix $A$ is positive definite if

$$u^TA u > 0$$

for any non-zero n-dimensional vector $u$,

and a symmetric, positive-definite matrix $A$ is a positive-definite matrix such that

$$A = A^T$$

For a positive definite square matrix, all the pivots (diagonal elements of $U$) are positive. If the matrix is also symmetric, then we must have an $LDU$ decomposition of the form $LDL^T$, where all the diagonal elements of $D$ are positive. Given this, $D^{1/2}$ is well-defined, and we have

$$
A = LDL^T = LD^{1/2}D^{1/2}L^T = LD^{1/2}(LD^{1/2})^T = CC^{T}
$$

where $C$ is lower-triangular with positive diagonal elements and $C^T$ is its transpose.  This decomposition is known as the Cholesky decomposition, and $C$ may be interpreted as the 'square root' of the matrix $A$.

### Algorithm

Let $A$ be an $n\times n$ matrix.  We find the matrix $L$ using the following iterative procedure:

$$A = \left(\begin{matrix}a_{11}&A_{12}\\A_{12}&A_{22}\end{matrix}\right) =
\left(\begin{matrix}\ell_{11}&0\\
L_{12}&L_{22}\end{matrix}\right)
\left(\begin{matrix}\ell_{11}&L_{12}\\0&L_{22}\end{matrix}\right)
$$

1.) Let $\ell_{11} = \sqrt{a_{11}}$

2.) $L_{12} = \frac{1}{\ell_{11}}A_{12}$

3.) Solve $A_{22} - L_{12}L_{12}^T = L_{22}L_{22}^T$ for $L_{22}$

### Example

$$A = \left(\begin{matrix}1&3&5\\3&13&23\\5&23&42\end{matrix}\right)$$

$$\ell_{11} = \sqrt{a_{11}} = 1$$

$$L_{12} = \frac{1}{\ell_{11}} A_{12} = A_{12}$$

$\begin{eqnarray*}
A_{22} - L_{12}L_{12}^T &=& \left(\begin{matrix}13&23\\23&42\end{matrix}\right) - \left(\begin{matrix}9&15\\15&25\end{matrix}\right)\\
&=& \left(\begin{matrix}4&8\\8&17\end{matrix}\right)
\end{eqnarray*}$

This is also symmetric and positive definite, and can be solved by another iteration

$\begin{eqnarray*}
&=& \left(\begin{matrix}2&0\\4&\ell_{33}\end{matrix}\right) \left(\begin{matrix}2&4\\0&\ell_{33}\end{matrix}\right)\\
&=& \left(\begin{matrix}4&8\\8&16+\ell_{33}^2\end{matrix}\right)
\end{eqnarray*}$

And so we conclude that $\ell_{33}=1$.

This yields the decomposition:

$$\left(\begin{matrix}1&3&5\\3&13&23\\5&23&42\end{matrix}\right) =
\left(\begin{matrix}1&0&0\\3&2&0\\5&4&1\end{matrix}\right)\left(\begin{matrix}1&3&5\\0&2&4\\0&0&1\end{matrix}\right)$$

```python
A = np.array([
    [1,3,5],
    [3,13,23],
    [5,23,42]
])
```

```python
C = la.cholesky(A)
C
```

    array([[1., 3., 5.],
           [0., 2., 4.],
           [0., 0., 1.]])

```python
C1 = la.cho_factor(A)
C1
```

    (array([[ 1.,  3.,  5.],
            [ 3.,  2.,  4.],
            [ 5., 23.,  1.]]), False)

```python
C1 = la.cho_factor(A)
la.cho_solve(C1, b)
```


### Algebraic definition

A set G equipped with operation $\bullet$

> Closure: $\forall a,b \in G, a \bullet b \in G$
>Associativity: $(a \bullet b) \bullet c = a \bullet (b \bullet c)$
>Identity element: $\exists e \in G, such \ that \ \forall a \in G \ a \bullet e = a$
>Inverse element: $\forall a \in G, \exists a^{-1}, such \ that \ a \bullet a^{-1} = e$

Can be shown, that $e \bullet a = a$ and $a^{-1} \bullet a = e$ and inverse and e are unique

**Abelian group**: $\forall a,b \in G, a \bullet b = b \bullet a$

If a group is abelian, usually group operation is denoted by + and the identity element by 0

As usual in algebra are difined Homomorphism, Isomorphism, Endomorphism, Automorphism. In Algebra isomorph groups are deemed the same.

group homomorphism $f:G \to H$ such that $f(g_1 \bullet g_2) = f(g_1) * f(g_2)$

**kernel** of a homomorphism is defined as the set of elements that get mapped to the identity element in the image.

Group operation may be viewed as bijection from G to G.

**Subgroup**: if H subset of G, which is closed under group operation and inverse operation. $H \leq G$, proper subgroup: $H \lt G$

Finite groups can be described by **Cayley table**:

$
\begin{bmatrix}
& Elmnts & e & a & b & c  \\
& e & e & a  & b & c  \\
& a & a & a \bullet a & a \bullet b & a \bullet c  \\
& b & b & b \bullet a & b \bullet b & b \bullet c  \\
& c & c & c \bullet a & c \bullet b & c \bullet c  \\
\end{bmatrix}
$

Group can be described by **Presentation of a group**:

$\langle S \vert R  \rangle$ where S is set of generators and R is a set of rules

free group $F_S$ over a given set $S$ consists of all expressions (a.k.a. words, or terms) that can be built from members of S: $\langle S \vert \emptyset  \rangle$

### Normal Subgroup

**Coset left** (right): if $H \lt G$, set defined by an element $gH = \{g \bullet h : h \in H\}$ ($Hg = \{h \bullet g : h \in H\}$)

cosets form a partitioning of group, any element of a coset is a represantative of the coset. the number of distinct cosets is called index of subgroup.

if $H \lt G$ then $ index= \frac {\vert G  \vert} {\vert H  \vert} $ . A finite groups  of prime size can not have subgroups.

if $\forall g \in G : gH = Hg$, then $H$ is said to be a normal subgroup. Subgroups of  abelian group are normal.

**Conjugacy class**: $Cl(a)=\{ b \in G : \exists g \in G \text{ such that } b = gag^-1\}$

Conjugacy class is an equivalence relation and thus partitions the group. For abelian group $Cl(a) = \{a\}$

group of **Inner automorphisms**; $Inn(G)$
>$\varphi_g: G \rightarrow  G$ such that $\varphi_g(x)=gxg^-1$
> $Inn(AbelianGroup) \simeq \{e\}$
>Normal subgoup is preserved by inner automorphisms.
>kernel of a homomorphism is equal to a normal subgroup

**Quotient group (factor group)**: we can define a group operation on cosets as the following $g_1H*g_2H=(g_1 \bullet g_2)H$.

If $H \vartriangleleft G$ then  $G/H$ is a group. ( $G/\{e\}=G$ , $G/G=\{e\}$ )

**Direct product**: $G \times H$ is the ordered pairs $(g,h)$ where $g \in G$ and $h \in H$ such that $(g_1,h_1) \bullet (g_2,h_2) == (g_1 * g_2, h_1*'h_2)$

Let a group $P$ has 2 subgroups $G,H$.

$
P=G \times H
\Leftrightarrow
\begin{cases}
 & G \cap H == \{e\} \\
 & \text{ every element can be expressed in } g \bullet h \\
 & G \text{ comutes with } H \text{( or stronger:} G \text{ and } H \text{ are normal subgroups} \\
\end{cases}
$

### Geometric definition

**Transformation group (Group action)** : $\varphi$ is a function of G on set X

$\varphi : G \times X \rightarrow X:(g,x) \mapsto \varphi(g,x) $ where $\varphi(e,x)=x$, and $\varphi$ is bijective map.

**symmetric group**: defined over any set is the group whose elements are all the bijections from the set to itself. For finite set of size n, it is $S_n$: all permutations of finite set of size n.

**Cayley's theorem**: every group can be realized as a subgroup of a symmetric group over some set

**Orbit**: Suppose $G$ is a group acting on a set $S$. ${G_{.s}}$ the orbit of $s \in S$ is defined as $\{t \in S \vert \exists g \in G, g.t=s \}$

The set of orbits of (points x in) X under the action of G form a partition of X.

A group action is termed **transitive** if it has exactly one orbit: $\forall x,y \in S, \exists g \in G , g.x = y$

**Stabilizer (isotropy group)**: denoted $Stab_G(s)$, is defined as: $\{ h \in G \mid \ h.s = s \} $.

Actions of groups on vector spaces are called **representations** of the group

Alternativly a **representation** of a group G on a vector space V is a group homomorphism $\varphi: G \rightarrow GL(V,F)$

**General linear group** : $GL(V,F)$ are all matrices over a field $F$, with 0 determinant (invertible).
> $SL(n, F)$: determinant is 1. In $R^n$  volume and orientation preserving linear transformations
> $O(n)$: orthogonal matrcies $O^TO=I$. Preserves the dot product of vectors.
>$SU(n)$: n×n unitary matrices with determinant 1 on $\mathbb{C}$

A **faithful representation** is one in which the homomorphism G → GL(V) is injective; in other words, one whose kernel is the trivial subgroup {e} consisting only of the group's identity element.

### examples

small groups of order n:
> n=1, the only  group {e} == **trivial group**

> n=2, only one group $Z_2 \simeq S_2$
>> $
\begin{bmatrix}
&    & e & a \\
& e & e & a \\
& a & a & e \\
\end{bmatrix}
$

> n=3, only one group $Z_3$
>>$\begin{bmatrix}
&    & e & a & b \\
& e & e & a & b \\
& a & a & b & e\\
& b & b & e & a\\
\end{bmatrix}$
>> that is $b = a \bullet a == a^2$

> n=4, two non isomorphic groups
>> $Z_4$
>>>$\begin{bmatrix}
&    & e & g & g^2 & g^3 \\
& e & e & g & g^2 & g^3 \\
& g & g & g^2 & g^3 & e\\
& g^2 & g^2 & g^3 & e & g\\
& g^3 & g^3 & e & g & g^2\\
\end{bmatrix}$
>> this can be viewed as a group of multiplications of $\{1, i, -1, -i\}$
>> one proper subgroup $Z_2 \lt Z_4 : \{ e, g^2 \}$
>> this subgroup partitions the group to the following cosets: $\{e, g^2\}$, $\{g, g^3\}$

>> Klein four-group $K_4$:
>>>$
\begin{bmatrix}
&  & e & a & b & c  \\
& e & e & a  & b & c  \\
& a & a & e & c & b  \\
& b & b & c & e & a  \\
& c & c & b & a & e  \\
\end{bmatrix}
$
>> 3 proper subgroups : $\{ e, a \},\{ e, b \},\{ e, c \}$
>> $K_4 = Z_2 \times Z_2 $ and can be presented as $ \langle a, b \vert a^2,b^2, ab=ba  \rangle$

**symmetric group**:  $S_3$

> order of the group is $3!=6$
> cycle examples:
>>$(1,2) == \begin{pmatrix}
1 & 2 & 3\\
2 & 1 & 3
\end{pmatrix}$
>>$(1,2,3) == \begin{pmatrix}
1 & 2 & 3\\
2 & 3 & 1
\end{pmatrix}$
> elements (in cycles):  { e, (1,2), (1,3), (2,3), (1,2,3), (1,3,2) }
> first non abelian group:  (1,2)(2,3) = (1,2,3) and (2,3)(1,2)=(1,3,2)
>presentations: $\langle a, b, c \vert a^2,b^2, c^3, abc  \rangle$ or $\langle s_1, s_2 \vert s_1^2,s_2^2, (s_1s_2)^3  \rangle$ or $\langle s, t \vert s^3,t^2, (st)=ts^2  \rangle$

> proper subgroups:
>>$Z_2$: {e,(1,2)}, {e,(1,3)}, {e,(2,3)}
>> $Z_3$: {e,(1,2,3), (1,3,2)}
> conjugacy class partitioning
>> {e}
>>{ (1,2), (1,3), (2,3) }
>>{ (1,2,3), (1,3,2) } this is the only normal subgroup

>$Inn(S_3) \simeq S_3$

**Cyclic group** of order n:  $Z_n$ : $\langle a \vert a^n  \rangle$, that is one generator and $a^n = e$
> $Z_n$ is abelian group

> it may be looked as
>> modulo n arithmetics
>> rotations by $2\pi \frac {m} {n}$

**Free group** of order 1:  $F_1: \langle a \vert \emptyset  \rangle$
> consists of all possible strings of $a$ and $a^{-1}$ reducted by the group identity $a \bullet a^{-1} = e$
> for example $"aaaaaa"$ or $"a^{-1}a^{-1}a^{-1}"$
>  $F_1 \simeq Z$ where $Z$ with + is treated as Cyclic goup

## example  Dihedral transformation group
$D_n$ is the group of symmetries of a regular polygon (includes rotations and reflections).

Orbit of a point in $R^2$ is the set of vertexes of a polygon.

$D_n = \langle r,s \vert r^n, s^2, (sr)^2  \rangle$, order of the group = 2n,  $D_3 \simeq S_3$

$(sr)^2=e \Leftrightarrow srs=r^{-1}$

$r_i==r^i; s_i==r_is \Rightarrow r_ir_j=r_{i+j}, r_is_j=s_{i+j},s_jr_i=s_{i-j},s_is_j=r_{i-j}$

natural represantation in $R^2$:
$
r_i=\begin{pmatrix}
cos \frac{2 \pi i}{n} & -sin \frac{2 \pi i}{n}\\
sin \frac{2 \pi i}{n} & cos \frac{2 \pi i}{n}
\end{pmatrix}
,
s_i=\begin{pmatrix}
cos \frac{2 \pi i}{n} & sin \frac{2 \pi i}{n}\\
sin \frac{2 \pi i}{n} & -cos \frac{2 \pi i}{n}
\end{pmatrix}
$

$D_4 \lt O(2)$, $D_4 \lt SO(3)$

$D_4$: group of symmetries of a square

> conjucacy classes:
>> $\{e\}$
>> $\{r^2\}$ rotation by $\pi$
>> $\{s,s_2\}$ reflections horisontal and vertical
>> $\{s_1,s_3\}$ diagonal reflections
>> $\{r,r^3\}$ rotations by $\pi/2$ and $3\pi/2$

> proper subgroups:
>> $\{e,r^2\}$ Normal subgroup
>> $\{e,s\}$,$\{e,s_1\}$,$\{e,s_2\}$,$\{e,s_3\}$
>> $\{e,s,r^2,r^2s\}$, $\{e,rs,r^2,r^3s\}$
>> $\{e,r,r^2,r^3\}$  Normal subgroup

> $Inn(D_4) \simeq K_4$

```python
# some sympy Permutation examples

# this is needed for google.colab
#from IPython.display import HTML
#display(HTML(""))

from sympy import init_printing
init_printing(use_latex='mathjax')

import sympy.combinatorics.permutations as P

p1 = P.Permutation(4)
display(p1.array_form)
display(p1)  #p1.cyclic_form
print('='*50)

p1 = P.Permutation(3,1)(0,2,4)
display(p1.array_form)
display(p1)
print('='*50)

p2=p1*p1
display(p2.array_form)
display(p2)
print('='*50)

```

$$\left [ 0, \quad 1, \quad 2, \quad 3, \quad 4\right ]$$

$$\left( 4\right)$$

    ==================================================

$$\left [ 2, \quad 3, \quad 4, \quad 1, \quad 0\right ]$$

$$\left( 0\; 2\; 4\right)\left( 1\; 3\right)$$

    ==================================================

$$\left [ 4, \quad 1, \quad 0, \quad 3, \quad 2\right ]$$

$$\left( 0\; 4\; 2\right)$$

    ==================================================

```python
p1('abcde')
```

    ['c', 'd', 'e', 'b', 'a']

```python
import sympy.combinatorics.generators as G
d = G.dihedral(4)
list(d)
```

$$\left [ \left( 3\right), \quad \left( 0\; 3\right)\left( 1\; 2\right), \quad \left( 0\; 1\; 2\; 3\right), \quad \left( 1\; 3\right), \quad \left( 0\; 2\right)\left( 1\; 3\right), \quad \left( 0\; 1\right)\left( 2\; 3\right), \quad \left( 0\; 3\; 2\; 1\right), \quad \left( 0\; 2\right)\left( 3\right)\right ]$$

```python
#S3
from sympy.combinatorics.free_groups import free_group, FreeGroup
import sympy.combinatorics.fp_groups
F, a, b = free_group("a, b")
G = FpGroup(F, [a**2, b**3, (a*b)**4])
print("Order is:",G.order())
#l = sympy.combinatorics.fp_groups.low_index_subgroups(G,2)
#for t in l:
#    print(t.table)
```

    Order is: 24

## Links

[The Group Properties Wiki](https://groupprops.subwiki.org/wiki/Main_Page)

[Magma Computational Algebra System](http://magma.maths.usyd.edu.au/magma/)

[GAP - Groups, Algorithms, Programming -a System for Computational Discrete Algebra](https://www.gap-system.org/)

http://www.sagemath.org/

[Handbook of Computational Group Theory. Derek F. Holt, Bettina Eick, Eamonn A. O'Brien](https://books.google.am/books?id=rnTLBQAAQBAJ&source=gbs_book_other_versions)

https://docs.sympy.org/latest/modules/combinatorics/perm_groups.html

https://people.maths.bris.ac.uk/~matyd/GroupNames/index.html

```python

```


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
alpha, beta = symbols('alpha beta')
```

```python
eq3 = Eq(diff(f(t), t), alpha*f(t) + beta*f(t)**2)
```

```python
solution_eq = dsolve(eq3)
```

```python
general = solution_eq.rhs
```

```python
at_0 = general.subs(t, 0)
```

```python
solutions = solve(Eq(at_0, p_0), C1)
value_of_C1 = solutions[0]
```

```python
particular = general.subs(C1, value_of_C1)
particular.simplify()
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.interpolate import interp1d
from typing import List, Tuple
```

```python
data = np.array([0.5, -0.32, -0.55, -0.76, -0.07, 0.44, -0.48])
theta = np.linspace(-10, 10, 10000)
```

(a) Let $x_1,...,x_n \sim DExp(\theta)$. The contribution of $x_i$ to the likelihood is:
$$L_i(\theta) = \frac{1}{2} e^{|x_i - \theta|} $$

and the total log-likelihood is:
\begin{align}
\log L(\theta) & = \sum_{i=1}^n \log L_i(\theta) \\
& = -n \log 2 + \sum_{i=1}^n |x_i - \theta|
\end{align}

```python
def log_likelihood_double_exp(theta: List[float], data: List[float]) -> List[float]:
    log_like = []
    for th in theta:
        ll = 0
        for dt in data:
            ll += np.abs(dt - th)
        ll *= -np.log(2)
        log_like.append(ll)
    return log_like
```

```python
def plot_log_likelihood(theta: List[float], data: List[float]):
    log_like = log_likelihood_double_exp(theta, data)
    plt.plot(theta, log_like)
    plt.title('Double exponential likelihood')
    plt.xlabel(r'$\theta$')
    plt.ylabel('Log likelihood');
```

```python
plot_log_likelihood(theta, data)
```

(b) Then $\hat{\theta}$ the MLE of $\theta$ is given by the solution to the score equation:

\begin{align}
S(\theta) = 0 & = \frac{\partial}{\partial \theta} \log L(\theta) \\
& = \sum_{i=1}^n sgn |x_i - \theta| \\
& = median(x)
\end{align}

where the last statement is due to the following argument: https://math.stackexchange.com/questions/1678740/mle-of-double-exponential.

```python
print(f"MLE = {np.median(data)}")
```

or alternatively we can use the log-likelihood directly:

```python
log_like = log_like = log_likelihood_double_exp(theta, data)
print(f"MLE = {np.round(theta[np.argmax(log_like)], 2)}")
```

The likelihood based interval:

```python
def likelihood_interval(theta: List[float],
                        likelihood: List[float],
                        cutoff: float) -> Tuple[float, float]:
    # intersection points occur below and above the maximum likelihood estimate
    mle_index = np.argmax(likelihood)
    interp_below_max = interp1d(likelihood[:mle_index], theta[:mle_index])
    interp_above_max = interp1d(likelihood[mle_index:], theta[mle_index :])
    lower_int = np.round(interp_below_max(cutoff).flatten()[0], 2)
    upper_int = np.round(interp_above_max(cutoff).flatten()[0], 2)
    return (lower_int, upper_int)
```

at a cut-off of:

```python
c = 0.15 # 95% confidence interval
print(f'Likelihood interval for c = {c} is {likelihood_interval(theta, np.exp(log_like), c)}')
```

(c) If the largest value in the data is now 2.5 instead of 0.5:

```python
data[0] = 2.5
data
```

then the log-loikelihood function is now:

```python
plot_log_likelihood(theta, data)
```

Despite the change in the largest value, the inference of the MLE $\hat{\theta}$ does not change. This is expected as it is the median of the data which remains unchanged. The median is known to be robust to outliers.

```python
print(f"MLE = {np.median(data)}")
```


# Linear Regression

This notebook implements linear regression in numpy using the least squares method. Some fitness metrics are also discussed.

Some imports:

```python
import matplotlib.pyplot as plt
import numpy as np
import sys

from mpl_toolkits.mplot3d import Axes3D

sys.path.append("..")
from models.linear_regression import LinearRegression
```

```python
%matplotlib inline
```

## Simple linear regression: One variable

Linear regression is a method that allows to predict the value of a quantitative respone Y on the basis of some predictor variables $\{X_1, X_2, ..., X_n\}$. This approach assumes a linear relationship between the response and the predictors. This takes the form:

$$
\begin{equation}
Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + ... + \beta_nX_n,
\end{equation}
$$

where $\beta_0, \beta_1, ..., \beta_n$ are unknown parameters. First, we implement a simple version where there is only one variable $X$. This leave us with 2 paremeters, the intercept($\beta_0$) and the coefficient($\beta_1$) of the function:
$$
\begin{equation}
Y = \beta_0 + \beta_1X
\end{equation}
$$

Based on a training set we produce the estimates $\hat{\beta_0}$ and $\hat{\beta_1}$ that fit the training data, we can later use them to predict future values of the response $Y$ for new values of $X$ as:
$$
\begin{equation}
\hat{y} = \hat{\beta_0} + \hat{\beta_1}x_i
\end{equation}
$$

In this notation $\hat{y}$ is the predicted value of $Y$ for $X = x_i$

### Generate the data

We are gonna try to interpolate the function $y = 5x + 2$

We take randomly 20 points in the interval $[0, 5)$, this correspond to our set of training samples $X = \{x_1, x_2, ..., x_{20}\}$

```python
sample_size = 20

# Parameters to estimate
b0 = 2
b1 = 5

# Standard deviation of the simulated irreducible error
e = 2

x = np.random.random_sample(sample_size)*5
```

We compute the dependent variable, $y_i$, mapping with the function and adding some random noise to the data.

We are using zero mean and two standar deviation normal distributed noise.

```python
y = (b1*x + b0) + e*np.random.randn(sample_size)
```

Let's check the data we are going to fit

```python
plt.plot(x, y, 'o')
plt.show()
```

## Least squares method

Now fit the data using the least squares technique, we are going to estimate parameters $\hat{\beta_0}$ and $\hat{\beta_1}$ corresponing to the parameters of the line $y = \beta_0 + \beta_1x$ that generated the data. The least square method computes the parameters that minimize the residual sum of squares (RSS) defined as:
$$
\text{RSS} = \sum_{i=1}^n (y_i - \hat{y}_i)^2
$$

The analytic solution of the values of $\hat{\beta_0}$ and $\hat{\beta_1}$ that minimize the RSS is:
$$
\hat{\beta_1} = \frac{\sum_{i=1}^n (x_i - \overline{x})(y_i - \overline{y})}{\sum_{i=1}^n (x_i - \overline{x})^2}, \\
\hat{\beta_0} = \overline{y} - \hat{\beta_1}\overline{x}
$$

```python
simple_model = LinearRegression()
b_hat = simple_model.fit(x.reshape(-1, 1), y)

print("Estimated parameters of the linear regression model:\n"
      + "Intercept = "+str(b_hat[0])+" \nCoefficient = " + str(b_hat[1]))
```

    Estimated parameters of the linear regression model:
    Intercept = 3.390914159450319
    Coefficient = 4.504515480115419

```python
# Plottin rergession line over data
x_axis = np.linspace(0, 5, 100)
plt.plot(x, y, 'o')
plt.plot(x_axis, b1*x_axis + b0, label='True line', color='green')
plt.plot(x_axis, b_hat[1]*x_axis + b_hat[0], label='Regressed line')
plt.legend()
plt.show()
```

Using the LinearRegression class:

### Making new predictions

We can now predict the response for a new value $x_i$ as:
$$
\hat{y_i} = \hat{\beta_0} + \hat{\beta_1}x_i
$$

```python
x_test = 2

pred_test = b_hat[0] + x_test*b_hat[1]

result = "For a test value of x = %.2f, the predicted response is y_hat = %.2f, and the true value is y = %.2f" % (x_test, pred_test, (b1*x_test + b0))
print(result)
```

    For a test value of x = 2.00, the predicted response is y_hat = 12.40, and the true value is y = 12.00

We may also use the predict method:

```python
pred_test = simple_model.predict(x_test)
print("Predicted value: %.2f" % pred_test)
```

    Predicted value: 12.40

## Fitness metrics

### Residual Standard Error of the estimation

The residual standard error (RSE) is an estimate of the standard deviation of the random error term $\epsilon$. Typically we dont know $\epsilon$ and it is considered the irreducible error. However in this case we have generated the data and simulated noise, so $\epsilon$ can be identified with the added noise. This measure gives an estimation of how much the prediction deviates from the true line in average measured in the units of the respond. The metric is defined as follows
$$
\text{RSE} = \sqrt{\frac{1}{n-2}\sum_{i=1}^n(y_i - \hat{y}_i)^2},
$$
where $n$ is the number of samples.

```python
print("Estimated standard error of the model: %.2f" % simple_model.std_err)
```

    Estimated standard error of the model: 1.52

Notice that the output is in units of $y$, so the number on its own does not give much information, it should be considered along with the magnitud of $y$. It is not the same to deviate $2$ units from the true value when this is $y_i = 10000$ than when it is $y_i = 1$. For this reason the RSE has to be interpreted along with the scale of $Y$. To overcome this limitation we can use the $R^2$ Statistic.

### $R^2$ Statistic

The R squared is used to verify how well the model describes the data. This metric is the proportion of variability explained, because it's a proportion it takes values in the range $[0, 1]$ and avoids the problem of being subject to the scale of $Y$. But how can we measure the explained variability? To do this we can exploit a couple concepts.

The total variance of the response $Y$ can be measured using the total sum of squares (TSS) defined as
$$
\text{TSS} = \sum_{i=1}^n(y_i - \overline{y})^2
$$

Besides, the residual sum of squares (RSS) represents the amount of variability that is left after performing the regression. The R squared statistic is defined as:
$$
R^2 = \frac{\text{TSS} - \text{RSS}}{\text{TSS}} = 1 - \frac{\text{RSS}}{\text{TSS}} = \frac{\sum_{i=1}^n (y_i - \hat{y_i})^2}{\sum_{i=1}^n(y_i - \overline{y})^2}
$$

Looking into our particular case:

```python
print("R squared score of the model: %.5f" % simple_model.r2)
```

    R squared score of the model: 0.94928

It is actually very good, it might be because we have small noise.
For noisier data it should perform worse.

```python
# Use a bigger standard deviation for the simulated error, this is more noise
noisier_e = 5

y_noisy = (b1*x + b0) + noisier_e*np.random.randn(sample_size)

# Estimate parameters
[b0_hat, b1_hat] = simple_model.fit(x.reshape(-1, 1), y_noisy)

# Fitness metrics
print("Estimated standard error: %.2f" % simple_model.std_err)
print("R squared score: %.2f" % simple_model.r2)
```

    Estimated standard error: 4.40
    R squared score: 0.78

We can see that the estimate of the standard error has grown (keeping track of the real error) and the r2 score is worse too.

## Comparison to the sklearn implementation

To the aim of validating our results let's compare them against the linear regression implementation of sklearn.

```python
from sklearn import linear_model
```

```python
linear = linear_model.LinearRegression()
linear.fit(x.reshape(-1, 1), y.reshape(-1, 1))

print("Sklearn beta0: %.5f" % np.squeeze(linear.intercept_))
print("Sklearn beta1: %.5f" % np.squeeze(linear.coef_))
print("Sklearn R squared: %.5f" % linear.score(x.reshape(-1, 1), y.reshape(-1, 1)))
```

    Sklearn beta0: 3.39091
    Sklearn beta1: 4.50452
    Sklearn R squared: 0.94928

The same! Not surprise really.. This is beacuase the algorithm is deterministic. In more complex algorithms such as in a MLP we could different results due to, for example, random initialization.

## General linear regression: multiple independant variables

The model can be generalized as:
$$
Y = X\beta
$$
where $\beta$ is the vector $[\beta_0, \beta_1, ..., \beta_n]$ and X is the collection of examples(one per row). The first entry in each training example is $1$ as it is the coefficient of the intercept. Typically the linear model can be written as:
$$
Y = \beta_0 + X\beta_{1:n},
$$
but adding a first column on $1$s to $X$ we can rewritte the equation as the more compact one of the begining.

In this case instead of fitting a line we fit an n-dimensional hyperplane. Let's see this in 2D

```python
sample_size = 100

# Parameters to estimate
beta = np.array([1, 2, -2])

# Standard deviation of the simulated irreducible error
e = 2

# Generate random points
x = np.random.rand(sample_size, 2)*5

# Simulate data with noise
y = (np.matmul(x, beta[1:]) + beta[0]) + e*np.random.randn(sample_size)
```

### Visualize the data

```python
fig = plt.figure()

ax = plt.axes(projection='3d')
ax.scatter3D(x[:, 0], x[:, 1], y);
plt.show()
```

### Estimator for $\beta$

Using the least squares method the resulting estimator is:
$$
\hat{\beta} = (X^TX)^{-1}X^TY
$$
This assumes $X$ has full column rank. If $M^+$ is the Moore-Penrose pseudoinverse of any matrix $M$, then $\beta$ can be estimated as:

$$
\hat{\beta} = X^+Y,
$$

which can be expressed as:
$$
\hat{\beta} = (X^TX)^+X^TY,
$$

obtaining a correct estimator for $\beta$ even when $X$ has less than full rank.

```python
linear_model = LinearRegression()

# The model expects the data without the column of ones and with shape (n_samples, n_features)
beta_hat = linear_model.fit(x, y)
print("Estimated parameters:\n B0={}\n B1={}\n B2={}".format(beta_hat[0], beta_hat[1], beta_hat[2]))
```

    Estimated parameters:
     B0=2.364948307515827
     B1=1.6136037083970096
     B2=-2.1559887289541617

```python
xx, yy = np.meshgrid(np.linspace(0, 5, 100), np.linspace(0, 5, 100))
z = beta[0] + beta[1]*xx + beta[2]*yy
z_hat = beta_hat[0] + beta_hat[1]*xx + beta_hat[2]*yy

fig = plt.figure()

ax = plt.axes(projection='3d')
ax.scatter3D(x[:, 0], x[:, 1], y);
ax.plot_surface(xx, yy, z, alpha=0.5, color='green', label='True plane')
ax.plot_surface(xx, yy, z_hat, alpha=0.5, color='orange', label='Regressed plane')
plt.show()
```

### Fitness metrics

```python
print("Estimated standard error: %.2f" % linear_model.std_err)
print("R squared score: %.2f" % linear_model.r2)
```

    Estimated standard error: 2.00
    R squared score: 0.80

## Bibliography

* [Wikipedia: Linear Regression](https://en.wikipedia.org/wiki/Linear_regression)
* [Proofs involving ordinary least squares](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Least_squares_estimator_for_.CE.B2)
* [Regression via pseudoinverse](https://spartanideas.msu.edu/2015/10/21/regression-via-pseudoinverse/)


# RSA Cryptography Algorithm

## About RSA Algorithm
RSA (Rivest–Shamir–Adleman) is one of the first public-key cryptosystems and is widely used for secure data transmission. In such a cryptosystem, the encryption key is public and distinct from the decryption key which is kept secret (private).

## Algorithm

1. Select 2 Prime Numbers - **p & q**
2. Calculate **n = p x q**
3. Calculate Euler's Totient Function of n,  **φ(n) = (p-1) x (q-1)**
4. Select PUBLIC KEY, **e** such that **e & φ(n) are Co-primes**  i.e, **gcd(e , φ(n))=1**
5. Calculate PRIVATE KEY, **d** such that **(d x e) mod φ(n) = 1**

## Public and Private Keys

1. The Public  key is { e , n }, which is known to all in the network.
2. The Private key is { d , n }, which is known ONLY to the User to whom message is to be sent.

## Encryption & Decryption

#### Encryption Algorithm

The Cipher Text, C is generated from the plaintext, M using the public key, e as:

**C = Me mod n**

#### Decryption Algorithm

The Plain Text, M is generated from the ciphertext, C using the private key, d as:

**M = Cd mod n**

## Implementation of RSA using Python

```python
from sympy import *
import math

#Generate p and q
p = randprime(1, 10)
q = randprime(11, 20)

#Generate n and l(n)
n = p*q
l = (p-1)*(q-1)

#Function to test Co-Primality for generation of list of Public Keys
def isCoPrime(x):
    if math.gcd(l,x)==1:
        return True
    else:
        return False

#Function to find mod Inverese of e withl(n) to generate d
def modInverse(e, l) :
    e = e % l;
    for x in range(1, l) :
        if ((e * x) % l == 1) :
            return x
    return 1

#List for Co-Primes
listOfCP = []
for i in range(1, l):
    if isCoPrime(i) == True:
        listOfCP.append(i)

#Print values of P, Q, N, L
print("Value of P = ", p)
print("Value of Q = ", q)
print("Value of N = ", n)
print("Value of L = ", l)

print(" ")

#Print List of Co-Primes for e
print("List of Available Public Keys")
print(listOfCP)

print(" ")

#select a Public Key from list of Co-Primes
e = int(input("Select Public Key from the Above List ONLY: "))

#Value of d
d = modInverse(e, l)

print(" ")

#Print Public and Private Keys
print("PUBLIC KEY  : { e , n } = {", e ,",", n , "}")
print("PRIVATE KEY : { d , n } = {", d ,",", n , "}")

print(" ")

#Encryption Algorithm
def encrypt(plainText):
    return (plainText**e)%n

#Decryption Algorithm
def decrypt(cipherText):
    pvtKey = int(input("Enter your Private Key: "))
    return (cipherText**pvtKey)%n

#Driver Code

#Message Input
pt = int(input('Enter the Plain Text: '))
print("CipherText: ", encrypt(pt))

print(" ")

#CipherText Input
ct = int(input('Enter the Cipher Text: '))
print("PlainText: ", decrypt(ct))
```

    Value of P =  7
    Value of Q =  19
    Value of N =  133
    Value of L =  108

    List of Available Public Keys
    [1, 5, 7, 11, 13, 17, 19, 23, 25, 29, 31, 35, 37, 41, 43, 47, 49, 53, 55, 59, 61, 65, 67, 71, 73, 77, 79, 83, 85, 89, 91, 95, 97, 101, 103, 107]

    Select Public Key from the Above List ONLY: 47

    PUBLIC KEY  : { e , n } = { 47 , 133 }
    PRIVATE KEY : { d , n } = { 23 , 133 }

    Enter the Plain Text: 51
    CipherText:  116

    Enter the Cipher Text: 116
    Enter your Private Key: 23
    PlainText:  51

## Encryption & Decryption Mechanism

The Encryption and Decryption mechanishm block diagram is shown below

## Explanation

The explanation for the above example is shown below

#### Tanmoy Sen Gupta
[tanmoysg.com](http://tanmoysg.com) | +91 9864809029 | tanmoysps@gmail.com


## Plotting

```python
from sympy import init_session
init_session(quiet=True)
```

```python
expr = exp(sin(x**2+pi))
expr.series(x)
```

```python
expr.series(x, 1, n=7)
```

Lets compare the original function with its series expansion visually

SymPy includes plotting functions, which are (by default) based on matplotlib (and also a legacy pyglet-based plotting...)

```python
%matplotlib inline
```

```python
plot(expr);
```

```python
p1 = plot(expr, expr.series(x, 1, 7).removeO(), (x, -4, 7),
    ylim = (-3,3),
    show = False,
    legend = True
)
p1[1].line_color="r"
p1[1].label="series(6)"
p1.show()
```

```python
p0 = plot(expr, (x, -2.5, 5),
    ylim = (-0,3),
    show = False,
    legend = True
)
for n in range(1, 6):
    p1 = plot(expr.series(x, 1, n=n).removeO(), (x, -2.5, 5), show = False)
    p1[0].line_color = "C%d"%n
    p1[0].label = "n=%d"%(n,)
    p0.append(p1[0])
p0.show()
```

For more complicated plots I personally prefer to use matplotlib

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
numexpr = lambdify(x, expr)
xpts = np.linspace(-4, 4, 500)
plt.plot(xpts, numexpr(xpts), label="$%s$"%latex(expr), lw=2)
for n in range(1, 5):
    plt.plot(xpts, lambdify(x, expr.series(x, 1, n=n).removeO())(xpts)*np.ones_like(xpts),
             c="C%d"%n, label="$n=%d$"%n, lw=1)
plt.ylim(0.0, 3)
plt.xlim(-4, 4)
plt.legend(loc="lower left", fontsize=12);
```

We have used the lambdify function, which turns a sympy expression into a numerical function (using numpy by default)

```python
fun = lambdify(x, expr)
```

It is a numpy function...

```python
import inspect
inspect.getsource(fun)
```

    'def _lambdifygenerated(x):\n    return (exp(-sin(x**2)))\n'

3D plots are also supported

```python
from sympy.plotting import plot3d, plot3d_parametric_surface
plot3d(sin(x)*cos(y), (x, -5, 5), (y, -5, 5));
```

```python
u, v = symbols('u v')
plot3d_parametric_surface(cos(u + v), sin(u - v), u - v, (u, -5, 5), (v, -5, 5));
```


# Series (Lecture 6)

```python
# This cell just imports relevant modules

import numpy
from sympy import sin, cos, exp, ln, Function, Symbol, diff, integrate, limit, oo, series, factorial
from math import pi
import mpmath
import matplotlib.pyplot as plt
```

## Sequences

**Slide 10**

```python
# Store elements of the sequence in a list
finite_sequence = [2*k for k in range(1, 5)]

print(finite_sequence)
```

    [2, 4, 6, 8]

Remember: `range(A,B)` generates integers from `A` up to `B-1`, so we need to use `B=5` here.

### Convergence of sequences

**Slides 11, 13, 14**

```python
k = Symbol('k')
print("As k->infinity, the sequence {k} tends to:", limit(k, k, oo))
# The 'oo' here is SymPy's notation for infinity

print("As k->infinity, the sequence {1/k} tends to:", limit(1.0/k, k, oo))

print("As k->infinity, the sequence {exp(1/k)} tends to:", limit(exp(1.0/k), k, oo))

print("As k->infinity, the sequence {(k**3 + 2*k - 4)/(k**3 + 1)} tends to:",
      limit((k**3 + 2*k - 4)/(k**3 + 1), k, oo))
```

    As k->infinity, the sequence {k} tends to: oo
    As k->infinity, the sequence {1/k} tends to: 0
    As k->infinity, the sequence {exp(1/k)} tends to: 1
    As k->infinity, the sequence {(k**3 + 2*k - 4)/(k**3 + 1)} tends to: 1

## Series

**Slide 15**

```python
# Using list comprehension:
print("The sum of 3*k + 1 (from k=0 to k=4) is:", sum([3*k + 1 for k in range(0,5)]))
# Note: we could also use the nsum function (part of the module mpmath):
# print(mpmath.nsum(lambda k: 3*k + 1, [0, 4]))
```

    The sum of 3*k + 1 (from k=0 to k=4) is: 35

```python
x = 1
print(f"The sum of (x**k)/(k!) from k=0 to k=4, with x = {x}, is:",
      sum([x**k/factorial(k) for k in range(1,5)]))
```

    The sum of (x**k)/(k!) from k=0 to k=4, with x = 1, is: 41/24

### Arithmetic progression

**Slide 18**

```python
print("The sum of 5 + 4*k up to the 11th term (i.e. up to k=10) is:",
      sum([5 + 4*k for k in range(0,11)]))
```

    The sum of 5 + 4*k up to the 11th term (i.e. up to k=10) is: 275

### Geometric progression

**Slide 21**

```python
print("The sum of 3**k up to the 7th term (i.e. up to k=6) is:",
      sum([3**k for k in range(0,7)]))
```

    The sum of 3**k up to the 7th term (i.e. up to k=6) is: 1093

### Infinite series

**Slides 23, 24, 25**

```python
print("The sum of the infinite series sum(1/(2**k)) is:",
      mpmath.nsum(lambda k: 1/(2**k), [1, mpmath.inf]))
print("The sum of the infinite alternating series sum(((-1)**(k+1))/k) is:",
      mpmath.nsum(lambda k: ((-1)**(k+1))/k, [1, mpmath.inf]))
```

    The sum of the infinite series sum(1/(2**k)) is: 1.0
    The sum of the infinite alternating series sum(((-1)**(k+1))/k) is: 0.693147180559945

### Ratio test

**Slide 27**

A diverging example:

```python
k = Symbol('k')
f = (2**k)/(3*k)
f1 = (2**(k+1))/(3*(k+1))
ratio = f1/f

lim = limit(ratio, k, oo)
print("As k -> infinity, the ratio tends to:", lim)
if(lim < 1.0):
    print("The series converges")
elif(lim > 1.0):
    print("The series diverges")
else:
    print("The series either converges or diverges")
```

    As k -> infinity, the ratio tends to: 2
    The series diverges

A converging example:

```python
f = (2**k)/(5**k)
f1 = (2**(k+1))/(5**(k+1))
ratio = f1/f

lim = limit(ratio, k, oo)
print("As k -> infinity, the ratio tends to:", lim)
if(lim < 1.0):
    print("The series converges")
elif(lim > 1.0):
    print("The series diverges")
else:
    print("The series either converges or diverges")
```

    As k -> infinity, the ratio tends to: 2/5
    The series converges

### Power series

**Slide 30**

```python
k = Symbol('k')
x = Symbol('x')

a = 1/k
f = a*(x**k)

a1 = 1/(k+1)
f1 = a1*(x**(k+1))

ratio = abs(a/a1)
R = limit(ratio, k, oo)
print("The radius of convergence (denoted R) is:", R)

x = 0.5
if(abs(x) < 1):
    print(f"The series converges for |x| = {abs(x)} (< R)")
elif(abs(x) > 1):
    print(f"The series diverges for |x| = {abs(x)} (> R)")
else:
    print(f"The series either converges or diverges for |x| = {abs(x)} (== R)\n")
```

    The radius of convergence (denoted R) is: 1
    The series converges for |x| = 0.5 (< R)

## Useful series

**Slide 34**

```python
x = Symbol('x')
r = Symbol('r')

# Note: the optional argument 'n' allows us to truncate the series
# after a certain order of x has been reached.
print("1/(1+x) = ", series(1.0/(1.0+x), x, n=4))
print("1/(1-x) = ", series(1.0/(1.0-x), x, n=4))
print("ln(1+x) = ", series(ln(1.0+x), x, n=4))
print("exp(x) = ", series(exp(x), x, n=4))
print("cos(x) = ", series(cos(x), x, n=7))
print("sin(x) = ", series(sin(x), x, n=8))
```

    1/(1+x) =  1.0 - 1.0*x + 1.0*x**2 - 1.0*x**3 + O(x**4)
    1/(1-x) =  1.0 + 1.0*x + 1.0*x**2 + 1.0*x**3 + O(x**4)
    ln(1+x) =  1.0*x - 0.5*x**2 + 0.333333333333333*x**3 + O(x**4)
    exp(x) =  1 + x + x**2/2 + x**3/6 + O(x**4)
    cos(x) =  1 - x**2/2 + x**4/24 - x**6/720 + O(x**7)
    sin(x) =  x - x**3/6 + x**5/120 - x**7/5040 + O(x**8)

### Taylor series

The more terms there are in the series the more accurate it is as it better approximates the function. The error of the Taylor series decreases as \\(x\\) approaches \\(0\\).

#### \\( \ln(1+x) \\) with different number of terms \\(n\\):

```python
def ln_taylor(x, n):
    y = numpy.zeros(x.shape)
    for i in range(1, n):
        y = y + (-1)**(i+1) * (x**i)/(factorial(i))  # taylor series of ln(1+x)
    return y

x = numpy.linspace(-0.9, 0.9, 181)  # [-0.9, -0.89, ..., 0.88, 0.89, 0.9]

ln = numpy.log(1+x)  # actual function

n = [i for i in range(1, 6)]  # list [1, 2, 3, 4, 5]
colour = ['y', 'b', 'c', 'g', 'r']

fig, ax = plt.subplots(1, 2, figsize=(15, 10))

for i in range(len(n)):
    y_ln = ln_taylor(x, n[i])  # values from taylor series
    ln_error = abs(y_ln - ln)  # difference between taylor series and function

    ax[0].plot(x, y_ln, colour[i], label=f'n={n[i]}')
    ax[1].plot(x, ln_error, colour[i], label=f'n={n[i]}')

ax[0].plot(x, ln, 'k', label='y=ln(1+x)')

ax[0].set_ylabel('y')
ax[0].set_title('ln(1+x) vs taylor series', fontsize=14)
ax[1].set_ylabel('error')
ax[1].set_title('Error plot of taylor series for ln(1+x)', fontsize=14)
for i in range(len(ax)):
    ax[i].set_xlabel('x')
    ax[i].legend(loc='best', fontsize=14)
    ax[i].grid(True)

fig.tight_layout()
plt.show()
```

#### \\( \exp(x) \\) with different number of terms \\(n\\)

```python
def exp_taylor(x, n):
    y = numpy.zeros(x.shape)
    for i in range(n):
        y = y + x**i / factorial(i)  # taylor series for exp(x)
    return y

x = numpy.linspace(0, 5, 501)  # x between 0 and 5

expx = numpy.exp(x)  # actual function

n = [1, 2, 3, 4, 5]  # number of terms between 1 and 5
colour = ['y', 'b', 'c', 'g', 'r']

fig, ax = plt.subplots(1, 2, figsize=(15, 10))

for i in range(len(n)):
    y_exp = exp_taylor(x, n[i])  # values from taylor series
    exp_error = abs(y_exp - expx)  # difference between taylor series and function

    ax[0].plot(x, y_exp, colour[i], label=f'n={n[i]}')
    ax[1].plot(x, exp_error, colour[i], label=f'n={n[i]}')

ax[0].plot(x, expx, 'k', label='y=exp(x)')

ax[0].set_ylabel('y')
ax[0].set_title('exp(x) vs taylor series', fontsize=14)
ax[1].set_ylabel('error')
ax[1].set_title('Error plot of taylor series for exp(x)', fontsize=14)
for i in range(len(ax)):
    ax[i].set_xlabel('x')
    ax[i].legend(loc='best', fontsize=14)
    ax[i].grid(True)

fig.tight_layout()
plt.show()
```

#### \\( \sin(x) \\) with different number of terms \\(n\\)

```python
def sin_taylor(x, n):
    y = numpy.zeros(x.shape)
    for i in range(n):
        y = y + (-1)**(i)*(x**(2*i+1) / factorial(2*i+1)) # taylor series for sin(x)
    return y

x = numpy.linspace(-5, 5, 1001)  # x between -5 and 5

sinx = numpy.sin(x) # actual function

n = [1, 2, 3, 4, 5]  # number of terms between 1 and 5
colour = ['y', 'b', 'c', 'g', 'r']

fig, ax = plt.subplots(2, 1, figsize=(10, 15))

for i in range(len(n)):
    y_sin = sin_taylor(x, n[i])  # values from taylor series
    sin_error = abs(y_sin - sinx)  # difference between taylor series and function

    ax[0].plot(x, y_sin, colour[i], label=f'n={n[i]}')
    ax[1].plot(x, sin_error, colour[i], label=f'n={n[i]}')

ax[0].plot(x, sinx, 'k', label='y=sin(x)')

ax[0].set_ylabel('y')
ax[0].set_title('sin(x) vs taylor series', fontsize=14)
ax[1].set_ylabel('error')
ax[1].set_title('Error plot of taylor series for sin(x)', fontsize=14)
for i in range(len(ax)):
    ax[i].set_xlabel('x')
    ax[i].legend(loc='best', fontsize=14)
    ax[i].grid(True)

fig.tight_layout()
plt.show()
```

```python

```


# Hamiltonian Monte Carlo (HMC)

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from functools import partial
```

## Hamiltonian Monte Carlo (HMC)

HMC uses an auxiliary variable corresponding to the momentum of particles in a potential energy well to generate proposal distributions that can make use of gradient information in the posterior distribution. For reversibility to be maintained, the total energy of the particle has to be conserved - hence we are interested in Hamiltonian systems. The main attraction of HMC is that it works much better than other methods when variables of interest are highly correlated. Because we have to solve problems involving momentum, we need to understand how to numerically solve differential equations in a way that is both accurate (i.e. second order) and preserves total energy (necessary for a Hamiltonian system).

Example adapted from [MCMC: Hamiltonian Monte Carlo (a.k.a. Hybrid Monte Carlo)](https://theclevermachine.wordpress.com/2012/11/18/mcmc-hamiltonian-monte-carlo-a-k-a-hybrid-monte-carlo/)

### Hamiltonian systems

In a Hamiltonian system, we consider particles with position $x$ and momentum (or velocity if we assume unit mass) $v$. The total energy of the system $H(x, v) = K(v) + U(x)$, where $K$ is the kinetic energy and $U$ is the potential energy, is conserved. Such a system satisfies the following Hamiltonian equations

$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta H}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta H}{dx}
\end{align}
$$

Since $K$ depends only on $v$ and $U$ depends only on $x$, we have
$$
\begin{align}
\frac{dx}{dt} &= & \frac{\delta K}{dv} \\
\frac{dv}{dt} &= & -\frac{\delta U}{dx}
\end{align}
$$

#### Harmonic oscillator

We will consider solving a classical Hamiltonian system - that of a undamped spring governed by the second order differential equation

$$
x'' + x = 0
$$

We convert this to two first order ODEs by using a dummy variable $x' = v$ to get

$$
\begin{align}
x' &= v \\
v' &= -x
\end{align}
$$

From the Hamiltonian equations above, this is equivalent to a system with kinetic energy $K(v) = \frac{1}{2}v^2$ and potential energy $U(x) = \frac{1}{2}x^2$.

Writing in matrix form,

$$
A = \pmatrix{ x' \\ v' } = \pmatrix{0 & 1 \\ -1 & 0} \pmatrix{x \\ v}
$$

and in general, for the state vector $x$,

$$
x' = Ax
$$

We note that $A$ is anti- or skew-symmetric ($A^T = -A$), and hence has purely imaginary eigenvalues. Solving $|A - \lambda I = 0$, we see that the eigenvalues and eigenvectors are $i, \pmatrix{1\\i}$ and $-i, \pmatrix{1\\-i}$. Since the eigenvalues are pure imaginary, we see that the solution for the initial conditions $(x,v) = (1, 0)$ is $x(t) = e^{it}$ and the orbit just goes around a circle with a period of $2\pi$, neither growing nor decaying. Another weay of seeing this is that the Hamiltonian $H(u, v)$ or sum of potential ($U(x)) = \frac{1}{2}x^2$) and kinetic energy ($K(v) = \frac{1}{2}v^2$) is constant, i.e. in vector form, $(x^T x) = \text{constant}$.

### Finite difference methods

We want to find a finite difference approximation to $u' = Au$ that is **accurate** and **preserves total energy**. If total energy is not preserved, the orbit will either spiral in towards zero or outwards away from the unit circle. If the accuracy is poor, the orbit will not be close to its starting value after $t = 2\pi$. This gives us an easy way to visualize how good our numerical scheme is. We can also compare the numerical scheme to the Taylor series to evaluate its accuracy.

#### Forward Euler

The simplest finite difference scheme for integrating ODEs is the forward Euler

$$
\frac{u_{n+1} - u_n}{\Delta t} = A u_n
$$

Rearranging terms, we get

$$
u_{n+1} = u_n + \Delta t A u_n = \left( I + \Delta t A \right) u_n
$$

Since the eigenvalues of $A$ are $\pm i$, we see that the eigenvalues of the forward Euler matrix are $1 \pm i$. Since the absolute value of the eigenvalues is greater than 1, we expect **growing** solutions - i.e. the solution will spiral away from the unit circle.

```python
import scipy.linalg as la
```

```python
def f_euler(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u = u + dt * A @ u
        orbit[i] = u
    return orbit
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
orbit = f_euler(A, u, N)
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.3600318484671192

##### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

##### Accuracy and conservation of energy

We can see that forward Euler is not very accurate and also does not preserve energy since the orbit spirals away from the unit circle.

#### The trapezoidal method

The trapezoidal method uses the following scheme

$$
\frac{u_{n+1} - u_n}{\Delta t} = \frac{1}{2}  ( A u_{n+1} + A u_{n})
$$

This is an implicit scheme (because $u_{n+1}$ appears on the RHS) whose solution is

$$
u_{n+1} = \left(I - \frac{\Delta t}{2} A \right)^{-1} \left(I + \frac{\Delta t}{2} A \right) u_{n} = B u_n
$$

By inspection, we see that the eigenvalues are the complex conjugates of

$$
\frac{1 + \frac{\Delta t}{2} i}{1 - \frac{\Delta t}{2} i}
$$

whose absolute value is 1 - hence, energy is conserved. If we expand the matrix $B$ using the geometric series and compare with the Taylor expansion, we see that the trapezoidal method has local truncation error $O(h^3)$ and hence accuracy $O(h^2)$, where $h$ is the time step.

```python
def trapezoidal(A, u, N):
    p = len(u)
    orbit = np.zeros((N,p))

    dt = 2*np.pi/N
    for i in range(N):
        u = la.inv(np.eye(p) - dt/2 * A) @ (np.eye(p) + dt/2 * A) @ u
        orbit[i] = u
    return orbit
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
orbit = trapezoidal(A, u, N)
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.005039305635733781

##### Conservation of energy

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

#### The leapfrog method

The leapfrog method uses a second order difference to update $u_n$. The algorithm simplifies to the following explicit scheme:

- First take one half-step for v
- Then take a full step for u
- Then take one final half step for v

It performs almost as well as the trapezoidal method, with the advantage of being an explicit scheme and cheaper to calculate, so the leapfrog method is used in HMC.

```python
def leapfrog(A, u, N):
    orbit = np.zeros((N,2))

    dt = 2*np.pi/N
    for i in range(N):
        u[1] = u[1] + dt/2 * A[1] @ u
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt/2 * A[1] @ u
        orbit[i] = u
    return orbit
```

##### If we don't care about the intermediate steps, it is more efficient to just take 1/2 steps at the beginning and end

```python
def leapfrog2(A, u, N):
    dt = 2*np.pi/N

    u[1] = u[1] + dt/2 * A[1] @ u
    for i in range(N-1):
        u[0] = u[0] + dt * A[0] @ u
        u[1] = u[1] + dt * A[1] @ u

    u[0] = u[0] + dt * A[0] @ u
    u[1] = u[1] + dt/2 * A[1] @ u
    return u
```

```python
A = np.array([[0,1],[-1,0]])
u = np.array([1.0,0.0])
N = 64
```

```python
orbit = leapfrog(A, u, N)
```

##### Accuracy

```python
la.norm(np.array([1.0,0.0]) - orbit[-1])
```

    0.0025229913808033464

##### Conservation of energy

Note that unlike the trapezoidal scheme, energy is not perfectly conserved.

```python
plt.plot([p @ p for p in orbit])
pass
```

```python
ax = plt.subplot(111)
plt.plot(orbit[:, 0], orbit[:,1], 'o')
ax.axis('square')
plt.axis([-1.5, 1.5, -1.5, 1.5])
pass
```

### From Hamiltonians to probability distributions

The physical analogy considers the negative log likelihood of the target distribution $p(x)$ to correspond to a potential energy well, with a collection of particles moving on the surface of the well. The state of each particle is given only by its position and momentum (or velocity if we assume unit mass for each particle). In a Hamiltonian system, the total energy $H(x, v) = U(x) + K(v)$ is conserved. From statistical mechanics, the probability of each state is related to the total energy of the system

$$
\begin{align}
p(x, v) & \propto e^{-H(x, v)} \\
&= e^{-U(x) - K(v)} \\
&= e^{-P(x)}e^{-K(v)} \\
& \propto p(x) \, p(v)
\end{align}
$$

Since the joint distribution factorizes $p(x, v) = p(x)\, p(v)$, we can select an initial random $v$ for a particle, numerically integrate using a finite difference method such as the leapfrog and then use the updated $x^*$ as the new proposal. The acceptance ratio for the new $x^*$ is

$$
\frac{ e^{ -U(x^*)-K(v^*) }} { e^{-U(x)-K(v)} } = e^{U(x)-U(x^*)+K(x)-K(x^*)}
$$

If our finite difference scheme was exact, the acceptance ration would be 1 since energy is conserved with Hamiltonian dynamics. However, as we have seen, the leapfrog method does not conserve energy perfectly and an accept/reject step is still needed.

#### Example of HMC

We will explore how HMC works when the target distribution is bivariate normal centered at zero

$$
x \sim N(0, \Sigma)
$$

In practice of course, the target distribution will be the posterior distribution and depend on both data and distributional parameters.

The potential energy or negative log likelihood is proportional to
$$
U(x) = \frac{x^T\Sigma^{-1} x}{2}
$$

The kinetic energy is given by
$$
K(v) = \frac{v^T v}{2}
$$

where the initial $v_0$ is chosen at random from the unit normal at each step.

To find the time updates, we use the Hamiltonian equations and find the first derivatives of total energy with respect to $x$ and $v$

$$
\begin{align}
x' &= \frac{\delta K}{\delta v} &= v \\
v' &= -\frac{\delta U}{\delta x} &= -\Sigma^{-1} x \\
\end{align}
$$

giving us the block matrix

$$
A = \pmatrix{0 & 1 \\ -\Sigma^{-1} & 0}
$$

By using the first derivatives, we are making use of the gradient information on the log posterior to guide the proposal distribution.

##### This is what the target distribution should look like

```python
sigma = np.array([[1,0.8],[0.8,1]])
mu = np.zeros(2)
ys = np.random.multivariate_normal(mu, sigma, 1000)
sns.kdeplot(ys[:,0], ys[:,1])
plt.axis([-3.5,3.5,-3.5,3.5])
pass
```

##### This is the HMC posterior

```python
def E(A, u0, v0, u, v):
    """Total energy."""
    return (u0 @ tau @ u0 + v0 @ v0) - (u @ tau@u + v @ v)
```

```python
def leapfrog(A, u, v, h, N):
    """Leapfrog finite difference scheme."""
    v = v - h/2 * A @ u
    for i in range(N-1):
        u = u + h * v
        v = v - h * A @ u

    u = u + h * v
    v = v - h/2 * A @ u

    return u, v
```

```python
niter = 100
h = 0.01
N = 100

tau = la.inv(sigma)

orbit = np.zeros((niter+1, 2))
u = np.array([-3,3])
orbit[0] = u
for k in range(niter):
    v0 = np.random.normal(0,1,2)
    u, v = leapfrog(tau, u, v0, h, N)

    # accept-reject
    u0 = orbit[k]
    a = np.exp(E(A, u0, v0, u, v))
    r = np.random.rand()

    if r < a:
        orbit[k+1] = u
    else:
        orbit[k+1] = u0
```

```python
sns.kdeplot(orbit[:, 0], orbit[:, 1])
plt.plot(orbit[:,0], orbit[:,1], alpha=0.2)
plt.scatter(orbit[:1,0], orbit[:1,1],  c='red', s=30)
plt.scatter(orbit[1:,0], orbit[1:,1],  c=np.arange(niter)[::-1], cmap='Reds')
plt.axis([-3.5,3.5,-3.5,3.5])
pass
```

```python

```


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
dfdt = diff(f(t),t)
```

```python
alpha,beta = symbols('alpha beta')

eq1 = Eq(dfdt,(alpha * f(t)) + (beta * f(t)**2))
```

```python
solution_eqn = dsolve(eq1)
```

```python
general = solution_eqn.rhs
```

```python
at_0 = general.subs(t, 0)
```

```python
solutions = solve(Eq(at_0, p_0), C1)
value_of_C1 = solutions[0]
```

```python
particular = general.subs(C1, value_of_C1)
```

```python
particular = simplify(particular)
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python
general = solution_eqn.rhs
```

```python
particular.subs(t,0)
```