```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

import tensorflow as tf
```

Part of the training process requires calculating derivatives that involve tensors. So let's learn about TensorFlow's built-in [automatic differentiation](https://www.tensorflow.org/guide/autodiff) engine, using a very simple example. Let's consider the following two tensors:

$$
\begin{align}
  U =
  \begin{bmatrix}
    1 & 2
  \end{bmatrix}
  &&
  V =
  \begin{bmatrix}
    3 & 4 \\
    5 & 6
  \end{bmatrix}
\end{align}
$$

Now let's suppose that we want to multiply $U$ by $V$, and then sum all the values in the resulting tensor, such that the result is a scalar. In math notation, we might represent this as the following scalar function $f$:

$$
f(U, V) = \mathrm{sum} (U \, V) = \sum_j \sum_i u_i \, v_{ij}
$$

Our goal is to calculate the derivative of $f$ with respect to each of its inputs: $\frac{\partial f}{\partial U}$ and $\frac{\partial f}{\partial V}$. We start by creating the two tensors $U$ and $V$. We then create a [tf.GradientTape](https://www.tensorflow.org/guide/autodiff#gradient_tapes), and tell TensorFlow to watch for mathematical operations involving $U$ and $V$, recording those operations onto our "tape." The tape then enables us to calculate the derivatives of the function $f$ with respect to $U$ and $V$.

```python
# Decimal points in tensor values ensure they are floats, which automatic differentiation requires.
U = tf.constant([[1., 2.]])
V = tf.constant([[3., 4.], [5., 6.]])

with tf.GradientTape(persistent=True) as tape:
  tape.watch(U)
  tape.watch(V)
  W = tf.matmul(U, V)
  f = tf.math.reduce_sum(W)

print(tape.gradient(f, U)) # df/dU
print(tape.gradient(f, V)) # df/dV
```

    tf.Tensor([[ 7. 11.]], shape=(1, 2), dtype=float32)
    tf.Tensor(
    [[1. 1.]
     [2. 2.]], shape=(2, 2), dtype=float32)

TensorFlow automatically watches tensors that are defined as `Variable` instances. So let's turn `U` and `V` into variables, and remove the `watch` calls:

```python
# Decimal points in tensor values ensure they are floats, which automatic differentiation requires.
U = tf.Variable(tf.constant([[1., 2.]]))
V = tf.Variable(tf.constant([[3., 4.], [5., 6.]]))

with tf.GradientTape(persistent=True) as tape:
  W = tf.matmul(U, V)
  f = tf.math.reduce_sum(W)

print(tape.gradient(f, U)) # df/dU
print(tape.gradient(f, V)) # df/dV
```

    tf.Tensor([[ 7. 11.]], shape=(1, 2), dtype=float32)
    tf.Tensor(
    [[1. 1.]
     [2. 2.]], shape=(2, 2), dtype=float32)

As you will see later, in deep learning, we will need to calculate the derivatives of the loss function with respect to the model parameters. Those parameters are variables because they change during training. Therefore, the fact that variables are automatically watched is handy in our scenario.

## Optional explanation of the math

Let's take a look at the math used to compute the derivatives. You only need to understand matrix multiplication and partial derivatives to follow along, but if the math isn't as interesting to you, feel free to skip to the next notebook.

We'll start by thinking of $U$ and $V$ as generic 1 × 2 and 2 × 2 matrices:

$$
\begin{align}
  U =
  \begin{bmatrix}
    u_1 & u_2
  \end{bmatrix}
  &&
  V =
  \begin{bmatrix}
    v_{11} & v_{12} \\
    v_{21} & v_{22}
  \end{bmatrix}
\end{align}
$$

Then the scalar function $f$ can be written as:

$$
\begin{align}
  f(U, V)
  &= \mathrm{sum}(U \, V) \\
  &= \mathrm{sum}
    \left(
      \begin{bmatrix}
        u_1 & u_2
      \end{bmatrix}
      \begin{bmatrix}
        v_{11} & v_{12} \\
        v_{21} & v_{22}
      \end{bmatrix}
    \right) \\
  &= \mathrm{sum}
    \left(
      \begin{bmatrix}
        u_1 v_{11} + u_2 v_{21} & u_1 v_{12} + u_2 v_{22}
      \end{bmatrix}
    \right) \\
  &= u_1 v_{11} + u_2 v_{21} + u_1 v_{12} + u_2 v_{22}
\end{align}
$$

We can now calculate the derivatives of $f$ with respect to each of its inputs:

$$
\frac{\partial f}{\partial U} =
  \begin{bmatrix}
    \frac{\partial f}{\partial u_1} & \frac{\partial f}{\partial u_2}
  \end{bmatrix} =
  \begin{bmatrix}
    v_{11} + v_{12} & v_{21} + v_{22}
  \end{bmatrix} =
  \begin{bmatrix}
    7 & 11
  \end{bmatrix}
$$

$$
\frac{\partial f}{\partial V} =
  \begin{bmatrix}
    \frac{\partial f}{\partial v_{11}} & \frac{\partial f}{\partial v_{12}} \\
    \frac{\partial f}{\partial v_{21}} & \frac{\partial f}{\partial v_{22}}
  \end{bmatrix} =
  \begin{bmatrix}
    u_1 & u_1 \\
    u_2 & u_2
  \end{bmatrix} =
  \begin{bmatrix}
    1 & 1 \\
    2 & 2
  \end{bmatrix}
$$

As you can see, when we plug in the numerical values of $U$ and $V$, we get the same result as TensorFlow's automatic differentiation.


# Lecture 2: Different types of point processes and transformations

## 1. Binomial Point Process

- Fixed number of points
- Independent and identically distributed points over a given region
- How to simulate a BPP?

```python
#Numpy library
import numpy as np
#Number of points: N
x_realization_BPP = 50

#Square side: L
L = 10

#Random points in axis x
position_x_realization_BPP = np.random.uniform(0,L,x_realization_BPP)

#Random points in axis y
position_y_realization_BPP = np.random.uniform(0,L,x_realization_BPP)

#Manipulating arrays
position_x_realization_BPP_t=np.transpose(position_x_realization_BPP)
position_y_realization_BPP_t=np.transpose(position_y_realization_BPP)
#position_final_BPP = []
#position_final_BPP = [[position_x_realization_BPP_t[ix], position_y_realization_BPP_t[ix]] for ix in range(0, x_realization_BPP)]
```

```python
#Ploting libraries
import matplotlib.pyplot as plt
#Plot commands
plt.figure(figsize=(8,8), dpi=1200)
plt.plot(position_x_realization_BPP, position_y_realization_BPP, marker='.', color='g',linestyle = '')
plt.xlabel('x')
plt.ylabel('y')
plt.title('BPP')
plt.show()
```

What can you say about this process?

## 2. Poisson Point Process

- Random number of points: Poisson distribution
- Independent and identically distributed point over a given region
- How to simulate a PPP?

```python
#Square side: L
L = 10

#Average number of points (Poisson)
x_average_PPP = 50

#Number of points N that is a Poisson random variable
x_realization_PPP = np.random.poisson(x_average_PPP,1)

#Random points in axis x
position_x_realization_PPP = np.random.uniform(0,L,x_realization_PPP)

#Random points in axis y
position_y_realization_PPP = np.random.uniform(0,L,x_realization_PPP)

#Manipulating arrays
position_x_realization_PPP_t=np.transpose(position_x_realization_PPP)
position_y_realization_PPP_t=np.transpose(position_y_realization_PPP)
#position_final = []
#position_final = [[position_x_realization_PPP_t[ix], position_y_realization_PPP_t[ix]] for ix in range(0, x_realization_PPP)]
```

```python
#Number of points in this realization
print 'Number of points:', x_realization_PPP

#Ploting libraries
import numpy as np
import matplotlib.pyplot as plt
#Plot commands
plt.figure(figsize=(8,8), dpi=1200)
plt.plot(position_x_realization_PPP, position_y_realization_PPP, marker='.', color='r',linestyle = '')
plt.xlabel('x')
plt.ylabel('y')
plt.title('PPP')
plt.axis([0, L, 0, L])
plt.show()
```

How about the statistics?
- Let's compare the average number of points based on the simulation and the analytic formula of Poisson distribution.

```python
#Sympy library
from sympy import Symbol, symbols, init_printing, oo
from sympy import exp, gamma

# Setting things up
init_printing()

#Symbols
x,k = symbols('x k', real = True, positive = True)
#Defining Poisson distribution
f_Poisson = ((x)**k/gamma(k)) * exp(-x)
f_Poisson
```

```python
#Simulating Poisson distribution
s = np.random.poisson(x_realization_PPP, 100000)

#Analytic distribution
from scipy.stats import poisson
#
plt.figure(figsize=(10,8), dpi=1200)
plt.hist(s,normed=True, color="#6495ED")
k = np.arange(100)
plt.plot(k, poisson.pmf(k, x_realization_PPP), '-o')
plt.xlabel('$k$')
plt.ylabel('Probability that $n = k$')
plt.title('Poisson distribution')
plt.show()
```

## 3. Superposition transformation

What is a superposition transformation?

```python
import matplotlib.pyplot as plt
#Plot commands
plt.figure(figsize=(8,8), dpi=100)
plt.plot(position_x_realization_BPP, position_y_realization_BPP, marker='.', color='b',linestyle = '')
plt.plot(position_x_realization_PPP, position_y_realization_PPP, marker='.', color='b',linestyle = '')
plt.xlabel('x')
plt.ylabel('y')
plt.title('Superposition: BPP and PPP')
plt.show()
```

By looking this plot, can you say which point process is which?

Marking the process

```python
import matplotlib.pyplot as plt
#Plot commands
plt.figure(figsize=(8,8), dpi=1200)
plt.plot(position_x_realization_BPP, position_y_realization_BPP, marker='x', color='g',linestyle = '')
plt.plot(position_x_realization_PPP, position_y_realization_PPP, marker='o', color='r',linestyle = '')
plt.xlabel('x')
plt.ylabel('y')
plt.title('Superposition: BPP and PPP')
plt.show()
```

## 4. Mapping transformation

What is a  mapping transformation?

```python
#Square side: L
L = 10

#Average number of points (Poisson)
x_average_PPP = 50

#Number of points N that is a Poisson random variable
x_realization_PPP = np.random.poisson(x_average_PPP,1)

#Scaling transformation
s = 2

#Random points in axis x
position_x_realization_PPP = s * np.random.uniform(0,L,x_realization_PPP)

#Random points in axis y
position_y_realization_PPP = s * np.random.uniform(0,L,x_realization_PPP)
```

```python
#Number of points in this realization
print 'Number of points:', x_realization_PPP

#Ploting libraries
import numpy as np
import matplotlib.pyplot as plt
#Plot commands
plt.figure(figsize=(8,8), dpi=1200)
plt.plot(position_x_realization_PPP, position_y_realization_PPP, marker='.', color='g',linestyle = '')
plt.xlabel('x')
plt.ylabel('y')
plt.title('PPP with scaling')
plt.show()
```

## 5. Thinning transformation

What is a thinning transformation?

Delete points

- Independent: e.g. erase with a given probability
- Dependent: e.g. erase the points within a given range

You can try to implentent this at home

## 6. Cluster formation

How to form clusters?

- Think about if every point in the original point process can be the parent of other point process around it
- Try also to implement this at home

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

```python

```


# Euler Equations

The Euler equations in one-dimension are given by the following set of partial differential equations representing the conservation of mass, momentum, and energy.

$$
\begin{align}
& \frac{\partial \rho}{\partial t} + \frac{\partial \rho u}{\partial x} = 0 \\
& \frac{\partial \rho u}{\partial t} + \frac{\partial (\rho u^2 + p)}{\partial x} = 0 \\
& \frac{\partial \rho e}{\partial t} + \frac{\partial}{\partial x}\Bigg[ \left(e + p\right)u \Bigg] = 0
\end{align}
$$

The Euler equations can be written in vector form as

$$
\frac{\partial \mathbf{U} }{\partial t} + \frac{\partial \mathbf{F}}{\partial x} = 0
$$

where

$$
\mathbf{U} = \begin{bmatrix}
\rho \\
\rho u \\
e
\end{bmatrix}
$$

and

$$
\mathbf{F} = \begin{bmatrix}
\rho u\\
\rho u^2 + p \\
 u (e + p)
\end{bmatrix}
$$

with

$$
p = (\gamma - 1)\left(e - \rho \frac{u^2}{2}\right)
$$

where the vector $\mathbf{U}$ is a vector of the conservative variables, and the vector $\mathbf{F}$ is the net flux of those conservative variables in and out of the control volume.

We want to be able to relate the vector form of the Euler equations to our model hyperbolic PDE; however, for this PDE, we have the partial derivative of $\mathbf{U}$ with time, but instead of $\mathbf{U}$, the partial derivative with respect to $x$ is the function, $\mathbf{F}$.

## Homogeneous of Degree p

Consider $\mathbf{F} = \mathbf{F}(\mathbf{U})$, where $\mathbf{F}$ is a function of the elements of $\mathbf{U}$. Let $\mathbf{U}^{\prime} = \alpha \mathbf{U}$, then the following statement is true,

$$
\mathbf{F}(\mathbf{U}^{\prime}) = \alpha^p \mathbf{F}(\mathbf{U}),
$$

if and only if $\mathbf{F}$ is homogenous of degree $p$ with respect to $\mathbf{U}$.

## Euler Equations are Homogeneous with p = 1

Using this theorem, we can state that if $\mathbf{F}$ in the Euler equations,

$$
\frac{\partial \mathbf{U} }{\partial t} + \frac{\partial \mathbf{F}}{\partial x} = 0
$$

is homogenous of degree $p=1$ with respect to $\mathbf{U}$, then

$$
\frac{\partial \mathbf{F}}{\partial \mathbf{U}}\mathbf{U} = \overline{\mathbf{A}} \mathbf{U} = \mathbf{F}
$$

where the matrix $\overline{\mathbf{A}}$ is called the Jacobian is defined as

$$
\overline{\mathbf{A}} = \frac{\partial \mathbf{F}}{\partial \mathbf{U}}
$$

and

$$
\frac{\partial \mathbf{U} }{\partial t} + \overline{\mathbf{A}} \frac{\partial \mathbf{U}}{\partial x} = 0
$$

### Example

Considering the following system of equations

$$
\frac{\partial \mathbf{U} }{\partial t} + \frac{\partial \mathbf{F}}{\partial x} = 0
$$

where

$$
\mathbf{F} =
\begin{bmatrix}
\rho \\
\rho u^2
\end{bmatrix}
$$

and

$$
\mathbf{U} =
\begin{bmatrix}
\rho \\
\rho u
\end{bmatrix}
$$

compute the Jacobian matrix

$$
\overline{\mathbf{A}} = \frac{\partial \mathbf{F}}{\partial \mathbf{U}}
$$

and demonstrate that $\mathbf{F} = \overline{\mathbf{A}} \mathbf{U}$.

### Solution

Writing this in component form,

$$
\overline{\mathbf{A}} = \begin{bmatrix}
\frac{\partial f_1}{\partial u_1} \Big\rvert_{u_2} & \frac{\partial f_1}{\partial u_2} \Big\rvert_{u_1}\\
\frac{\partial f_2}{\partial u_1} \Big\rvert_{u_2} & \frac{\partial f_2}{\partial u_2} \Big\rvert_{u_1}
\end{bmatrix}
$$

where

$$
\overline{\mathbf{A}} = \begin{bmatrix}
\frac{\partial \rho}{\partial \rho}  \Big\rvert_{\rho u} & \frac{\partial \rho}{\partial (\rho u)} \Big\rvert_{\rho} \\
\frac{\partial \rho u^2}{\partial \rho} \Big\rvert_{\rho u} & \frac{\partial \rho u^2}{\partial (\rho u)} \Big\rvert_{\rho}
\end{bmatrix}
=
\begin{bmatrix}
1 & 0 \\
\frac{\partial }{\partial \rho} \left[ \frac{1}{\rho} (\rho u)^2 \right]_{\rho u} & \frac{\partial }{\partial (\rho u)} \left[ \frac{1}{\rho} (\rho u)^2 \right]_{\rho}
\end{bmatrix}
$$

Simplifying this results in

$$
\overline{\mathbf{A}} =
\begin{bmatrix}
1 & 0 \\
-\frac{1}{\rho^2} (\rho u)^2 & \frac{1}{\rho} \left(2 \rho u \right)
\end{bmatrix}
=
\begin{bmatrix}
1 & 0 \\
-u^2 & 2 u
\end{bmatrix}
$$

and we can show that

$$
 \begin{bmatrix}
\frac{\partial \rho }{\partial t} \\
\frac{\partial \rho u }{\partial t}
\end{bmatrix}
+ \begin{bmatrix}
1 & 0 \\
-\frac{1}{\rho^2} (\rho u)^2 & \frac{1}{\rho} \left(2 \rho u \right)
\end{bmatrix} \begin{bmatrix}
\frac{\partial \rho }{\partial x} \\
\frac{\partial \rho u }{\partial x}
\end{bmatrix}= 0
$$

# Euler Equations

To write the Euler equations in this form would require determining the Jacobian

$$
\overline{\mathbf{A}} = \begin{bmatrix}
\frac{\partial \rho u}{\partial \rho}  \Big\rvert_{\rho u,e} &
\frac{\partial \rho u}{\partial (\rho u)} \Big\rvert_{\rho,e} &
\frac{\partial \rho u}{\partial e} \Big\rvert_{\rho,\rho u} \\
\frac{\partial \rho u^2 + p}{\partial \rho}  \Big\rvert_{\rho u,e} &
\frac{\partial \rho u^2 + p}{\partial (\rho u)} \Big\rvert_{\rho,e} &
\frac{\partial \rho u^2 + p}{\partial e} \Big\rvert_{\rho,\rho u} \\
\frac{\partial (e + p)u}{\partial \rho}  \Big\rvert_{\rho u,e} &
\frac{\partial (e + p)u}{\partial (\rho u)} \Big\rvert_{\rho,e} &
\frac{\partial (e + p)u}{\partial e} \Big\rvert_{\rho,\rho u}
\end{bmatrix}
$$

Since evaluating this matrix is not straight-forward, let us apply a transformation of variables and write the Euler equations in primitve form using the vector of primitive variables

$$
\mathbf{V} =
\begin{bmatrix}
\rho \\
u \\
p
\end{bmatrix}
\qquad \textrm{and} \qquad
\mathbf{U} =
\begin{bmatrix}
\rho u\\
\rho u^2 \\
e
\end{bmatrix}
$$

Let the matrix $\overline{\mathbf{T}}$ be the transformation Jacobian, which is defined as

$$
\overline{\mathbf{T}} = \frac{\partial \mathbf{U}}{\partial \mathbf{V}}
$$

Using the transformation Jacobian we can write

$$
\frac{\partial \mathbf{U}}{\partial \mathbf{V}} \frac{\partial \mathbf{V}}{\partial t} + \overline{\mathbf{A}} \left( \frac{\partial \mathbf{U}}{\partial \mathbf{V}} \frac{\partial \mathbf{V}}{\partial x} \right) = 0
$$

Multiply by the inverse of $\partial \mathbf{U} / \partial \mathbf{V}$,

$$
\left( \frac{\partial \mathbf{U}}{\partial \mathbf{V}}\right)^{-1} \frac{\partial \mathbf{U}}{\partial \mathbf{V}} \frac{\partial \mathbf{V}}{\partial t} + \left( \frac{\partial \mathbf{U}}{\partial \mathbf{V}}\right)^{-1} \overline{\mathbf{A}} \left( \frac{\partial \mathbf{U}}{\partial \mathbf{V}} \frac{\partial \mathbf{V}}{\partial x} \right) = 0
$$

simplifying,

$$
\frac{\partial \mathbf{V}}{\partial t} + \left[ \left( \frac{\partial \mathbf{U}}{\partial \mathbf{V}}\right)^{-1} \overline{\mathbf{A}} \frac{\partial \mathbf{U}}{\partial \mathbf{V}} \right] \frac{\partial \mathbf{V}}{\partial x} = 0
$$

which gives

$$
\frac{\partial \mathbf{V}}{\partial t} + \overline{\mathbf{A}^{\prime}} \frac{\partial \mathbf{V}}{\partial x} = 0
$$

where

$$
\overline{\mathbf{A}^{\prime}} = \overline{\mathbf{T}}^{-1} \, \overline{\mathbf{A}} \, \overline{\mathbf{T}}
$$

By induction, we can then define

$$
\overline{\mathbf{A}^{\prime}} =
\begin{bmatrix}
u & \rho & 0 \\
0 & u & \frac{1}{\rho} \\
0 & \gamma p & u \\
\end{bmatrix}
$$

where

$$
\begin{bmatrix}
\frac{\partial \rho }{\partial t} \\
\frac{\partial u }{\partial t} \\
\frac{\partial p}{\partial t}
\end{bmatrix}
+
\begin{bmatrix}
u & \rho & 0 \\
0 & u & \frac{1}{\rho} \\
0 & \gamma p & u \\
\end{bmatrix}
\begin{bmatrix}
\frac{\partial \rho }{\partial x} \\
\frac{\partial u }{\partial x} \\
\frac{\partial p}{\partial x}
\end{bmatrix}
 = 0
$$

### Proof

$$
\frac{\partial e }{\partial t} = \frac{\partial}{\partial t}\left[ \frac{p}{\gamma - 1} + \rho \frac{ u^2 }{2} \right]
$$

$$
\frac{\partial (e + p)u }{\partial x} = \frac{\partial}{\partial x}\left[ u \frac{p}{\gamma - 1} + \rho u \frac{ u^2 }{2} + p u\right] = \frac{\partial}{\partial x}\left[ p u \left( 1 + \frac{1}{\gamma - 1}\right) + \rho u \frac{ u^2 }{2} \right] = \frac{\partial}{\partial x}\left[ p u \frac{\gamma}{\gamma - 1} + \rho u \frac{ u^2 }{2} \right]
$$

Combining the terms

$$
\frac{\partial e }{\partial t} + \frac{\partial (e + p)u }{\partial x} = \frac{\partial}{\partial t}\left[ \frac{p}{\gamma - 1} + \rho \frac{ u^2 }{2} \right] + \frac{\partial}{\partial x}\left[ p u \frac{\gamma}{\gamma - 1} + \rho u \frac{ u^2 }{2} \right] = 0
$$

$$
\frac{\partial e }{\partial t} + \frac{\partial (e + p)u }{\partial x} = \frac{\partial}{\partial t}\left[ \frac{p}{\gamma - 1} \right]  + \frac{\partial}{\partial x}\left[ p u \frac{\gamma}{\gamma - 1} \right] + \frac{\partial}{\partial t}\left[ \rho \frac{ u^2 }{2} \right] + \frac{\partial}{\partial x}\left[\rho u \frac{ u^2 }{2} \right] = 0
$$

Using the chain rule, we know

$$
\frac{\partial f(x^2)}{\partial x} = 2 x \frac{\partial f(x)}{\partial x}
$$

which we can use to simplify the equation as follows

$$
\frac{\partial e }{\partial t} + \frac{\partial (e + p)u }{\partial x} = \frac{1}{\gamma - 1} \left( \frac{\partial p}{\partial t} +  \frac{\partial \gamma p u }{\partial x} \right) + u \frac{\partial (\rho u)}{\partial t} + u \frac{\partial \rho u^2 }{\partial x} = 0
$$

and factoring

$$
\frac{\partial e }{\partial t} + \frac{\partial (e + p)u }{\partial x} = \frac{1}{\gamma - 1} \left( \frac{\partial p}{\partial t} + \frac{\partial \gamma p u }{\partial x} \right) + u \left( \frac{\partial (\rho u)}{\partial t} + \frac{\partial \rho u^2 }{\partial x} \right) = 0
$$

then using the fact that

$$
\frac{\partial (\rho u)}{\partial t} + \frac{\partial \rho u^2  + p}{\partial x} = 0
$$

we can rewrite the equation about adding and subtracting $\frac{\partial p }{\partial x}$ term,

$$
\frac{\partial e }{\partial t} + \frac{\partial (e + p)u }{\partial x} = \frac{1}{\gamma - 1} \left( \frac{\partial p}{\partial t} +  \frac{\partial \gamma p u }{\partial x} \right) + u \left( \frac{\partial (\rho u)}{\partial t} + \frac{\partial (\rho u^2 + p)}{\partial x}  - \frac{\partial p}{\partial x} \right) = 0
$$

we get the conservation of energy in the primitive form as

$$
\frac{\partial p}{\partial t} + \frac{\partial ( \gamma  p u) }{\partial x} - u (\gamma - 1) \frac{\partial p }{\partial x} = 0
$$

expaning results in

$$
\frac{\partial p}{\partial t} + u  \gamma \frac{\partial p }{\partial x} + \gamma p \frac{\partial u }{\partial x} - u \gamma \frac{\partial p }{\partial x} + u \frac{\partial p }{\partial x} = 0
$$

Finally,

$$
\frac{\partial p}{\partial t} + \gamma p \frac{\partial u }{\partial x} + u \frac{\partial p }{\partial x}= 0
$$

## One-dimensional Euler Equations

In conservative form, the one-dimensional Euler equations are

$$
\frac{\partial \mathbf{U}}{\partial t} + \frac{\partial \mathbf{F}}{\partial x} = 0
$$

where

$$
\mathbf{U} = \begin{bmatrix}
\rho \\
\rho u \\
e
\end{bmatrix}
$$

and

$$
\mathbf{F} = \begin{bmatrix}
\rho u\\
\rho u^2 + p \\
 u (e + p)
\end{bmatrix}
$$

with

$$
p = (\gamma - 1)\left(e - \rho \frac{u^2}{2}\right)
$$

Since the Euler equations are homogenous to degeree 1, we can write the same set of equations in terms of the Jacobian matrix $\mathbf{A}$,

$$
\frac{\partial \mathbf{U}}{\partial t} + \overline{\mathbf{A}} \frac{\partial \mathbf{U}}{\partial x} = 0
$$

It is often more convenient to work with the primitive set of variables $(\rho, u, p)$ instead of the conservative variables. Using this set of variables, the Euler equations in terms of the primitive variables is written

$$
\frac{\partial \mathbf{V}}{\partial t} + \overline{\mathbf{A}^{\prime}} \frac{\partial \mathbf{V}}{\partial x} = 0
$$

Note that this form of the equation is also referred to as the non-conservative form of the Euler equations though sometimes the entropy is used as the primitive variable instead of the pressure. The transformed Jacobian matrix is defined as

$$
\overline{\mathbf{A}^{\prime}} = \overline{\mathbf{T}}^{-1} \, \overline{\mathbf{A}} \, \overline{\mathbf{T}}
$$

where

$$
\overline{\mathbf{A}^{\prime}} =
\begin{bmatrix}
u & \rho & 0 \\
0 & u & \frac{1}{\rho} \\
0 & \gamma p & u \\
\end{bmatrix}
$$

## Hyperbolic Requirement

The above equation is hyperbolic if the eigenvalues of the matrix $\overline{\mathbf{A}}$ or $\overline{\mathbf{A}^{\prime}}$ are real and have a complete set of eigenvectors (the matrix is diagonalizable). We can determine the eigenvalues by diagonalizing the matrix

$$
\Lambda = \overline{\mathbf{P}} \, \overline{\mathbf{A}^{\prime}} \, \overline{\mathbf{P}}^{-1} =
\begin{bmatrix}
u & 0 & 0 \\
0 & u + c& 0 \\
0 & 0& u -c \\
\end{bmatrix}
$$

where

$$
\overline{\mathbf{P}} =
\begin{bmatrix}
1 & 0 & -\frac{1}{c^2} \\
0 & \rho c & 1 \\
0 & -\rho c& 1 \\
\end{bmatrix}
$$

and

$$
\overline{\mathbf{P}}^{-1} =
\begin{bmatrix}
1 & \frac{1}{c^2} & \frac{1}{c^2} \\
0 & \frac{1}{2\rho c} & -\frac{1}{2\rho c} \\
0 & \frac{1}{2} & \frac{1}{2} \\
\end{bmatrix}
$$


# Regression

## 1.Linear models

***

* 1.1 Formulation

***

Assume instance $ \mathbf{x} = (x_1, x_2, ... , x_d)^T  $, here $T$ means the matrix transpose. Linear model use linear combination of all attributes to do prediction.

\begin{equation}
f(\mathbf{x}) = w_1x_1 + w_2x_2 + ... + w_dx_d + b, \tag{1}
\end{equation}

or

\begin{equation}
    f(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + b. \tag{2}
\end{equation}

***

* 1.2 Linear regression

***

Use the linear model to obtain the relationship between the dependent variable (test results, $y_i$) and independent variables (selected features, $x_i$). If only one explanatory variable is considered, it is called simple linear regression, for more than one, it is called multiple linear regression. If multiple dependent variables are considered, it is called multivariate linear regression. Let's start with the simple linear regression.

Given $x_i$ and $y_i$, how to find w and b so that $f(x_i) = wx_i + b \rightarrow y_i$?

We may use the least square approach:

$(w^* , b^*) = \underset{(w , b)}{\arg\min} \sum\limits_{i=1}^m (f(x_i) - y_i)^2 = \underset{(w , b)}{\arg\min} \sum\limits_{i=1}^m (y_i - wx_i - b)^2$, here $m$ means we have $m$ instances.

Let's set
$E_{(w, b)} = \sum\limits_{i=1}^m (y_i - wx_i - b)^2$, here $E_{(w, b)}$ is the cost function.

Perform parameter estimation of least square approach

\begin{equation}
    \frac{\partial E}{\partial w} = 2 \left( w \sum\limits_{i=1}^m x_i^2 - \sum\limits_{i=1}^m (y_i - b)x_i \right) = 0, \tag{3}
\end{equation}

\begin{equation}
    \frac{\partial E}{\partial b} = 2 \left( mb - \sum\limits_{i=1}^m (y_i - wx_i) \right) = 0. \tag{4}
\end{equation}

We then have

\begin{equation}
    w = \frac{ \sum\limits_{i=1}^m y_i (x_i - \frac{1}{m}\sum\limits_{i=1}^m x_i )}{\sum\limits_{i=1}^m x_i^2 - \frac{1}{m} \left( \sum\limits_{i=1}^m x_i \right)^2}, \tag{5}
\end{equation}

\begin{equation}
    b = \frac{1}{m}\sum\limits_{i=1}^m (y_i - w x_i). \tag{6}
\end{equation}

Similarly, the cost function for multiple variables (let's say d varaibles) is: $E_{(\mathbf{w}, b)} = E_{(w_1, w_2, ..., w_d, b)} = \sum\limits_{i=1}^m (y_i - \mathbf{w}^T\mathbf{x} - b)^2$. Combine coeficients $\mathbf{w}$ and $b$, we obtain a new vector $\hat{\mathbf{w}} = (\mathbf{w} ; b)$. The dataset $\mathbf{XX}$ with the size $m \times (d+1)$ can be read as

\begin{equation}
\mathbf{XX} = \begin{pmatrix}
x_{11} & x_{12} & ... & x_{1d}\\
x_{21} & x_{22} & ... & x_{2d}\\
\vdots & \vdots & \ddots & \vdots \\
x_{m1} & x_{m2} & ... & x_{md}
\end{pmatrix} =  \begin{pmatrix}
\mathbf{x_{1}^T} & 1 \\
\mathbf{x_{2}^T} & 1 \\
\vdots & \vdots \\
\mathbf{x_{m}^T} & 1
\end{pmatrix}.       \tag{7}
\end{equation}

The label vector is $\mathbf{y} = (y_1; y_2; ...; y_m)$. Based on least square method, we need obtain the following

\begin{equation}
\mathbf{\hat{w}}^* = \underset{\mathbf{\hat{w}}}{\arg\min} (\mathbf{y} - \mathbf{XX\hat{w}})^T  (\mathbf{y} - \mathbf{XX\hat{w}}).  \tag{8}
\end{equation}

Let $E_{\mathbf{\hat{w}}} = (\mathbf{y} - \mathbf{XX\hat{w}})^T  (\mathbf{y} - \mathbf{XX\hat{w}})$, then the derivative w.r.t. $\mathbf{\hat{w}}$ is

\begin{equation}
\frac{\partial E_{\mathbf{\hat{w}}}}{\partial \mathbf{\hat{w}}} = 2\mathbf{XX}^T (\mathbf{XX}\mathbf{\hat{w}} - \mathbf{y}) .  \tag{9}
\end{equation}

We then use Gradient Descent method to iterativly update the unknown coefficients

\begin{equation}
\mathbf{\hat{w}}^{(n+1)} = \mathbf{\hat{w}}^{(n)} - \left ( \alpha \frac{\partial E_{\mathbf{\hat{w}}}}{\partial \mathbf{\hat{w}}} \right)^n.  \tag{10}
\end{equation}

***

* 1.3 Hand-on example

***

```python
import numpy as np
import pandas as pd
import math as m
import matplotlib.pyplot as plt

def train_test_split(X, Y, train_size, shuffle):
    ''' Perform tran/test datasets splitting '''
    if shuffle:
        randomize = np.arange(len(X))
        np.random.shuffle(randomize)
        X = X[randomize]
        Y = Y[randomize]
    s_id = int(len(Y) * train_size)
    X_train, X_test = X[:s_id], X[s_id:]
    Y_train, Y_test = Y[:s_id], Y[s_id:]

    return X_train, X_test, Y_train, Y_test

def metric_mse(Y_label, Y_pred):
    ''' Evaluate mean squared error (MSE) '''
    return np.mean(np.power(Y_label - Y_pred, 2))

def metric_rmse(Y_label, Y_pred):
    ''' Evaluate root mean squared error (RMSE) '''
    return m.sqrt(np.mean(np.power(Y_label - Y_pred, 2)))

def readin_data(path):
    ''' Evaluate root mean squared error (RMSE) '''
    df = pd.read_csv(path)
    X = df.iloc[:,:-1].values
    Y = df.iloc[:,1].values
    return X, Y

def generate_dataset_simple(beta, n, std_dev):
    ''' Generate dataset '''
    X = np.random.rand(n)
    e = np.random.randn(n) * std_dev
    Y = X * beta + e
    X = X.reshape((n,1))
    return X, Y

class LinearRegression() :
    ''' Linear Regression model.
        Used to obtain the relationship between dependent variable and independent variables.'''
    def __init__(self, iterations, learning_rate):
        self.lr = learning_rate
        self.it = iterations

    def fit(self, X, Y):
        # m instances, d atrributes
        self.m, self.d = X.shape
        # weight initialization
        self.W = np.zeros(self.d+1)
        self.X = X
        self.XX = np.ones((self.m, self.d+1))
        self.XX[:,:-1] = self.X
        self.Y = Y
        for i in range(self.it):
            self.update_weights()
        return self

    def update_weights(self):
        Y_pred = self.predict(self.XX)
        # calculate gradients
        dW = (self.XX.T).dot(Y_pred - self.Y)/self.m
        # update weights
        self.W = self.W - self.lr * dW
        return self

    def predict(self, X):
        return X.dot(self.W)

def main():
    # Import data
    X, Y = generate_dataset_simple(10, 200, 0.5)
    # Splitting dataset into train and test set
    X_train, X_test, Y_train, Y_test = train_test_split(X, Y, train_size=.5, shuffle=False)
    # Model Learning
    model = LinearRegression(learning_rate = 0.01, iterations = 15000)
    model.fit(X_train, Y_train)
    # Model Working
    M, D = X_test.shape
    TEST = np.ones((M, D+1))
    TEST[:,:-1] = X_test
    Y_pred = model.predict(TEST)
    # Statistics
    mse = metric_mse(Y_test, Y_pred)
    rmse = metric_rmse(Y_test, Y_pred)
    print('Coefficients: ', 'W = ', model.W[:-1], ', b = ', model.W[-1])
    print('MSE = ', mse)
    print('RMSE = ', rmse)
    # Visualization
    plt.scatter( X_test, Y_test, color = 'black', s=8)
    plt.plot( X_test, Y_pred, color = 'red', linewidth=3)
    plt.title( 'X_test v.s. Y_test')
    plt.xlabel( 'X_test')
    plt.ylabel( 'Y_test')
    X_actual = np.array([0, 1])
    Y_actual = X_actual*10
    plt.plot(X_actual, Y_actual, 'c--', linewidth=3)
    plt.legend(('Regression Line', 'Actual Line'),loc='upper left', prop={'size': 15})
    plt.show()

if __name__ == '__main__':
    main()
```

```python
import numpy as np
import pandas as pd
import math as m
import matplotlib.pyplot as plt

def train_test_split_po(X, Y, train_size, shuffle):
    ''' Perform tran/test datasets splitting '''
    if shuffle:
        randomize = np.arange(len(X))
        np.random.shuffle(randomize)
        X = X[randomize]
        Y = Y[randomize]
    s_id = int(len(Y) * train_size)
    X_train, X_test = X[:s_id], X[s_id:]
    Y_train, Y_test = Y[:s_id], Y[s_id:]
    Y_train = Y_train.reshape((-1, 1))
    X_train1 = np.append(X_train, Y_train, axis = 1)
    X_train1 = X_train1[np.argsort(X_train1[:, 0])]
    Y_test = Y_test.reshape((-1, 1))
    X_test1 = np.append(X_test, Y_test, axis = 1)
    X_test1 = X_test1[np.argsort(X_test1[:, 0])]
    X_train, X_test = X_train1[:,:-1], X_test1[:,:-1]
    Y_train, Y_test = X_train1[:,-1], X_test1[:,-1]
    Y_train=np.squeeze(Y_train)
    Y_test=np.squeeze(Y_test)
    return X_train, X_test, Y_train, Y_test

def metric_mse(Y_label, Y_pred):
    ''' Evaluate mean squared error (MSE) '''
    return np.mean(np.power(Y_label - Y_pred, 2))

def metric_rmse(Y_label, Y_pred):
    ''' Evaluate root mean squared error (RMSE) '''
    return m.sqrt(np.mean(np.power(Y_label - Y_pred, 2)))

def readin_data(path):
    ''' Evaluate root mean squared error (RMSE) '''
    df = pd.read_csv(path)
    X = df.iloc[:,:-1].values
    Y = df.iloc[:,1].values
    return X, Y

def generate_dataset_simple(beta, n, std_dev):
    ''' Generate dataset '''
    X = np.random.rand(n)
    e = np.random.randn(n) * std_dev
    Y = X * beta + e
    X = X.reshape((n,1))
    return X, Y

def generate_dataset_polynomial(beta, n, std_dev):
    ''' Generate polynomial dataset '''

    e = np.random.randn(n) * std_dev/n
    X = np.random.random_sample(n)
    X = np.sort(X)
    Y = 1- 6*X +36*X**2 - 53*X**3 + 22*X**5 + e
    X = X.reshape((n,1))
    return X, Y

def standardization(X,degree):
    """ A scaling technique where the values
    are centered around the mean with
    a unit standard deviation.
    This means that the mean of the attribute
    becomes zero and the resultant distribution
    has a unit standard deviation.
    ----------------------------------------
    degree: polynomial regression degree
    """
    X[:, :(degree)] = (X[:, :(degree)] - np.mean(X[:, :(degree)], axis = 0))/ \
    np.std(X[:, :(degree)], axis = 0)
    return X

def normalization(X,degree):
    """ A scaling technique in which values
    are shifted and rescaled so that they
    end up ranging between 0 and 1.
    It is also known as Min-Max scaling
    ----------------------------------------
    degree: polynomial regression degree
    """
    X[:, :(degree)] = (X[:, :(degree)] - np.amin(X[:, :(degree)], axis = 0))/ \
    (np.amax(X[:, :(degree)], axis = 0) - np.amin(X[:, :(degree)], axis = 0))
    return X

def transformation(m, X, degree):
    tmp = np.zeros([m, 1])
    for j in range(degree + 1):
        if j != 0:
            x_pow = np.power(X, j)
            tmp = np.append(tmp, x_pow.reshape(-1, 1), axis = 1)
    tmp = np.append(tmp, np.ones((m, 1)), axis = 1)
    Xt=tmp[:,1:]
    return Xt

def transformation_predict(m, X, degree):
    tmp = np.zeros([m, 1])
    for j in range(degree + 1):
        if j != 0:
            x_pow = np.power(X, j)
            tmp = np.append(tmp, x_pow.reshape(-1, 1), axis = 1)
    tmp = np.append(tmp, np.ones((m, 1)), axis = 1)
    Xt=tmp[:,1:-1]
    return Xt

class PolynomialRegression():
    ''' Univariate Polynomial Regression model.
        Used to obtain the relationship between dependent variable and independent variables.
        -------------------------------------------------------------------------------------
        degree shall be great equal 2
        '''

    def __init__(self, iterations, learning_rate, degree):
        self.lr = learning_rate
        self.it = iterations
        self.de = degree

    def fit(self, X, Y):
        # m instances, d atrributes
        self.m, self.d = X.shape
        # weight initialization
        self.W = np.zeros(self.de+1)
        self.X = X
        self.Xt = transformation(self.m, self.X, self.de)
        self.Xs = standardization(self.Xt, self.de)
        self.Y = Y
        for i in range(self.it):
            self.update_weights()
        return self

    def update_weights(self):
        Y_pred = self.predict(self.X)
        # calculate gradients
        dW = (self.Xs.T).dot(Y_pred - self.Y)/self.m
        # update weights
        self.W = self.W - self.lr * dW
        return self

    def predict(self, X):
        self.Xt = transformation(X.shape[0], X, self.de)
        self.Xs = standardization(self.Xt, self.de)
        return self.Xs.dot(self.W)

def main():
    # Import data
    X, Y = generate_dataset_polynomial(10, 200, 20)
    # Splitting dataset into train and test set
    X_train, X_test, Y_train, Y_test = train_test_split_po(X, Y, train_size=.5, shuffle=True)
    # Model Learning
    model = PolynomialRegression(learning_rate = 0.2, iterations = 20000, degree=5)
    model.fit(X_train, Y_train)
    print(X_test.shape)
    Y_pred = model.predict(X_test)
    print(Y_test.shape)
    print(Y_pred.shape)
    # Statistics
    mse = metric_mse(Y_test, Y_pred)
    rmse = metric_rmse(Y_test, Y_pred)
    print('Coefficients: ', 'W = ', model.W[-1], model.W[:-1])
    print('MSE = ', mse)
    print('RMSE = ', rmse)
    # Visualization
    l1=plt.scatter(X_train, Y_train, color = 'black', s=8)
    l2=plt.scatter(X_test, Y_test, color = 'green', s=8)
    l3=plt.plot( X_test, Y_pred, color = 'red', linewidth=3)
    plt.title( 'X v.s. Y')
    plt.xlabel( 'X')
    plt.ylabel( 'Y')
    plt.legend((l1, l2),('Train Points', 'Test Points'),loc='lower left', prop={'size': 20})
    plt.show()

if __name__ == '__main__':
    main()
```

***
* 1.4 Additional notes about linear regression
***

1. When performing **Gradient Descent** approach, all features/attributes must have similar scale, or **feature scaling** is required to increase Gradient Descent convergence. We may import `from sklearn.preprocessing import StandardScaler`, then use `StandardScaler()`. (refer [feature scaling](https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/))

2. To avoid local minimum, and to quickly find the global minimum, make sure the cost function is a **convex function** ($\displaystyle i.e., f(\frac{a+b}{2}) \leq \frac{f(a)+f(b)}{2} $).

3. Linear regression assumptions:
    - Exogeneity weak. Independent variable X is fixed variable, it is not random variable;
    - **Linearity**. $f$ is a linear combination of the parameters/coefficients and the independent variables X. Note that, linearity is a restriction on the parameters, not the independent variables X, e.g., polynomial regression can also be linear regression;
    - **Constant variable(Homoscedasticity)**. The variance of residual is the same for any value of independent variables;
    - **Independence**. Observations are independent of each other. The errors are uncorrelated with each other;
    - **Normality**. For any fixed value of X, Y is normally distributed.


# Zero-based indexing

In math we use 1-based indexing, but in many programming languages including Python, we use 0-based indexing.

For example, a three dimensional vector $\vec{v} \in \mathbb{R}^3$ are denoted as $(v_1,v_2,v_3)$ in components form.

In Python a vector `vec` can be denoted as a list (or a Matrix), and its three components will be `vec[0]`, `vec[1]`, and `vec[2]`. The variable we to index the components is assumed to start at zero, not one.

```python
from sympy import Matrix, symbols
a, b, c = symbols('a b c')

vec = Matrix([a,b,c])
vec
```

$\displaystyle \left[\begin{matrix}a\\b\\c\end{matrix}\right]$

```python
vec[0]
```

$\displaystyle a$

```python
vec[1]
```

$\displaystyle b$

```python
vec[2]
```

$\displaystyle c$

```python
# trying to access vec[3] leads to an error,
# since a 3 in 0-based indexing is accessing a fourth component which doesn't exist
vec[3]
```

## Ranges

```python
# a list of five numbers
list(range(0,5))
# Note the second argument of the function range specifies
#      the first number not to be included in the range...
```

    [0, 1, 2, 3, 4]

```python

```

## SymPy shorthand for ranges

```python
# consider a 2x3 matrix
A = Matrix([
        [1,2,3],
        [4,5,6]])
A
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\end{matrix}\right]$

```python
# can access individual entry on row i, col j using A[i-1,j-1]
print('top left =',    A[0,0], '    top right =', A[0,2])
print('bottom left =', A[1,0], ' bottom right =', A[1,2])
```

    top left = 1     top right = 3
    bottom left = 4  bottom right = 6

```python
# can access entire colum j using A[:,j-1]
A[:,0]
```

$\displaystyle \left[\begin{matrix}1\\4\end{matrix}\right]$

```python
# can access entire row i using A[i-1,:]
A[0,:]
```

$\displaystyle \left[\begin{matrix}1 & 2 & 3\end{matrix}\right]$

```python
# can access left 2x2 submatrix using i:j shorhand range notation
A[0:2,0:2]
```

$\displaystyle \left[\begin{matrix}1 & 2\\4 & 5\end{matrix}\right]$

```python
# the above notation is MATLAB-like syntax sugar that translates to
A[range(0,2),range(0,2)]
```

$\displaystyle \left[\begin{matrix}1 & 2\\4 & 5\end{matrix}\right]$

```python

```

```python

```


## OSY

Osyczka and Kundu used the following six-variable
test problem:

**Definition**

\begin{equation}
\newcommand{\boldx}{\mathbf{x}}
\begin{array}
\mbox{Minimize} & f_1(\boldx) = -\left[25(x_1-2)^2+(x_2-2)^2 + (x_3-1)^2+(x_4-4)^2  + (x_5-1)^2\right], \\
\mbox{Minimize} & f_2(\boldx) = x_1^2 + x_2^2 + x_3^2 + x_4^2 + x_5^2 + x_6^2,
\end{array}
\end{equation}

\begin{equation}
\begin{array}
\mbox{\text{subject to}} & C_1(\boldx) \equiv x_1 + x_2 - 2 \geq 0, \\
& C_2(\boldx) \equiv 6 - x_1 - x_2 \geq 0, \\
& C_3(\boldx) \equiv 2 - x_2 + x_1 \geq 0, \\
& C_4(\boldx) \equiv 2 - x_1 + 3x_2 \geq 0, \\
& C_5(\boldx) \equiv 4 - (x_3-3)^2 - x_4 \geq 0, \\
& C_6(\boldx) \equiv (x_5-3)^2 + x_6 - 4 \geq 0, \\[2mm]
& 0 \leq x_1,x_2,x_6 \leq 10,\quad 1 \leq x_3,x_5 \leq 5,\quad 0\leq x_4 \leq 6.
\end{array}
\end{equation}

**Optimum**

The Pareto-optimal region is a concatenation of
five regions. Every region lies on some of the constraints. However, for the
entire Pareto-optimal region, $x_4^{\ast} = x_6^{\ast} = 0$.
In table below shows the other variable values in each of the five
regions and the constraints that are active in each region.

<div style="display: block;margin-left: auto;margin-right: auto;width: 40%;">

</div>

**Plot**

```python
from pymoo.factory import get_problem
from pymoo.util.plotting import plot

problem = get_problem("osy")
plot(problem.pareto_front(), no_fill=True)
```


# Generating Partitions

Python code for generating all integer partitions of a given integer $n > 0$
using different algorithms. Let's define some terminology.

**Integer partition:** An integer partition is a way to write a nonnegative integer $n$
as sum of positive integers (note that the order of the summand doesn't matter). Therefore,
3+2 and 2+3 are the same partitions of 5. The number of partition of $n$ is denoted by
$p(n)$. The function $p(\cdot)$ is called the _partition function_.

Example: There are 7 partitions of 5. Therefore, $p(5) = 7$.

\begin{align}
5 &= 1 + 1 + 1 + 1 + 1 \\
  &= 2 + 1 + 1 + 1 \\
  &= 2 + 2 + 1 \\
  &= 3 + 1 + 1 \\
  &= 3 + 2 \\
  &= 4 + 1 \\
  &= 5.
\end{align}

**Ascending/Descending composition:** Partitions are represented as an ordered list of
summands. Depending on whether they are ordered as weakly increasing or decreasing, it's
called either an _ascending_ or _descending composition_.

For example, `[2, 2, 1]` is a descending composition, whereas, `[1, 4]` is an ascending
composition.

**Ordering of the output:** An algorithm to generate all integer partitions may output the partition of the given input $n > 0$ in different order. Most commonly, they output the partitions as _lexicographically increasing_ or _decreasing_ order as list of summands.
This is independent of whether partitions are represented as ascending or descending
compositions.

```python
import partition
```

```python
dir(partition)
```

    ['__builtins__',
     '__doc__',
     '__file__',
     '__name__',
     '__package__',
     'accel_asc',
     'accel_desc',
     'merca1',
     'merca2',
     'merca3',
     'rule_asc',
     'rule_desc',
     'zs1',
     'zs2']

## ZS Algorithms (1998)

See [Zoghbi-Stojmenovic-1998] for reference. There are two algorithms, `zs1` and `zs2`.

```python
def print_partitions(algo_name, n):
    """Print all integer partions of n using the given algorithm.

    Inputs:
        n            A nonnegative integer
        algo_name    The algorithm name (a string)
    """
    method = getattr(partition, algo_name)
    for p in method(n):
        print p

def count_partitions(algo_name, n):
    """Count the number of all integer partitions of n using the given algorithm.

    Inputs:
        n            A nonnegative integer
        algo_name    The algorithm name (a string)

    Returns:
        The count, p(n) (integer or long).
    """
    method = getattr(partition, algo_name)
    count = 0
    for p in method(n):
        count += 1
    return count

# We will print partitions of a small number
n = 5
```

```python
print_partitions('zs1', n)
```

    [5]
    [4, 1]
    [3, 2]
    [3, 1, 1]
    [2, 2, 1]
    [2, 1, 1, 1]
    [1, 1, 1, 1, 1]

As we can see, `zs1` outputs partitions as descending compositions in lexicographically decreasing order.

```python
print_partitions('zs2', n)
```

    [1, 1, 1, 1, 1]
    [2, 1, 1, 1]
    [2, 2, 1]
    [3, 1, 1]
    [3, 2]
    [4, 1]
    [5]

`zs2` generates partitions as descending compositions, but in lexicographically increasing order.

Now let's compare the time to generate partiton of a large number (say N = 50). We won't print out the partitions, only count them.

Notice that the pure python implementation is much slower than the C implementation.

**TODO:** Later I will wrap the C program as a python module. Or, try other optimizations.

```python
# Define ans in the global scope:
ans = 0

# Moderately large number to test
N = 50

# It may take a long time
%timeit global ans; ans = count_partitions('zs1', N)
print ans
```

    10 loops, best of 3: 124 ms per loop
    204226

```python
%timeit global ans; ans = count_partitions('zs2', N)
print ans
```

    10 loops, best of 3: 161 ms per loop
    204226

## Kelleher's Algorithms (2006)

See [Kelleher-2006] for reference.  There are four algorithms: `rule_desc`, `rule_asc`, `accel_desc` and `accel_desc`.

```python
print_partitions('rule_desc', n)
```

    [5]
    [4, 1]
    [3, 2]
    [3, 1, 1]
    [2, 2, 1]
    [2, 1, 1, 1]
    [1, 1, 1, 1, 1]

The `rule_desc` algorithm generates partitions as descending compositions, in lexicographically decreasing order.

```python
print_partitions('rule_asc', n)
```

    [1, 1, 1, 1, 1]
    [1, 1, 1, 2]
    [1, 1, 3]
    [1, 2, 2]
    [1, 4]
    [2, 3]
    [5]

The `rule_asc` algorithm generates partitions as ascending compositions, in lexicographically increasing order.

```python
print_partitions('accel_desc', n)
```

    [5]
    [4, 1]
    [3, 2]
    [3, 1, 1]
    [2, 2, 1]
    [2, 1, 1, 1]
    [1, 1, 1, 1, 1]

```python
print_partitions('accel_asc', n)
```

    [1, 1, 1, 1, 1]
    [1, 1, 1, 2]
    [1, 1, 3]
    [1, 2, 2]
    [1, 4]
    [2, 3]
    [5]

The algorithms `accel_desc` and `accel_asc` are faster versions of `rule_desc` and `rule_asc` respectively.

Now, let's compare the speed.

```python
%timeit global ans; ans = count_partitions('rule_desc', N)
print ans
```

    1 loop, best of 3: 374 ms per loop
    204226

```python
%timeit global ans; ans = count_partitions('rule_asc', N)
print ans
```

    10 loops, best of 3: 159 ms per loop
    204226

```python
%timeit global ans; ans = count_partitions('accel_desc', N)
print ans
```

    10 loops, best of 3: 126 ms per loop
    204226

```python
%timeit global ans; ans = count_partitions('accel_asc', N)
print ans
```

    10 loops, best of 3: 124 ms per loop
    204226

## Merca's Algorithms

See [Merca-2012] for reference. There are three algorithms in this family: `merca1`, `merca2` and `merca3` -- all generates partitions as ascending compositions and in lexicographically increasing order.

```python
print_partitions('merca1', n)
```

    [1, 1, 1, 1, 1]
    [1, 1, 1, 2]
    [1, 1, 3]
    [1, 2, 2]
    [1, 4]
    [2, 3]
    [5]

```python
print_partitions('merca2', n)
```

    [1, 1, 1, 1, 1]
    [1, 1, 1, 2]
    [1, 1, 3]
    [1, 2, 2]
    [1, 4]
    [2, 3]
    [5]

```python
print_partitions('merca3', n)
```

    [1, 1, 1, 1, 1]
    [1, 1, 1, 2]
    [1, 1, 3]
    [1, 2, 2]
    [1, 4]
    [2, 3]
    [5]

```python
%timeit global ans; ans = count_partitions('merca1', N)
print ans
```

    1 loop, best of 3: 141 ms per loop
    204226

```python
%timeit global ans; ans = count_partitions('merca2', N)
print ans
```

    10 loops, best of 3: 122 ms per loop
    204226

```python
%timeit global ans; ans = count_partitions('merca3', N)
print ans
```

    10 loops, best of 3: 112 ms per loop
    204226

```python

```


# Sympy

## What is Symbolic Computation?

Symbolic computation deals with the computation of mathematical objects
symbolically.  This means that the mathematical objects are represented
exactly, not approximately, and mathematical expressions with unevaluated
variables are left in symbolic form.

Let’s take an example. Say we wanted to use the built-in Python functions to
compute square roots. We might do something like this

```
import math
math.sqrt(9)
```

    3.0

9 is a perfect square, so we got the exact answer, 3. But suppose we computed
the square root of a number that isn’t a perfect square

```
math.sqrt(8)
```

    2.8284271247461903

Here we got an approximate result. 2.82842712475 is not the exact square root
of 8 (indeed, the actual square root of 8 cannot be represented by a finite
decimal, since it is an irrational number).  If all we cared about was the
decimal form of the square root of 8, we would be done.

But suppose we want to go further. Recall that $\sqrt{8} = \sqrt{4\cdot 2} =
2\sqrt{2}$.  We would have a hard time deducing this from the above result.
This is where symbolic computation comes in.  With a symbolic computation
system like SymPy, square roots of numbers that are not perfect squares are
left unevaluated by default

```
import sympy
sympy.sqrt(8)
```

    2*sqrt(2)

Furthermore—and this is where we start to see the real power of symbolic
computation—symbolic results can be symbolically simplified

## A more interesting example

The above example starts to show how we can manipulate irrational numbers
exactly using SymPy.  But it is much more powerful than that.  Symbolic
computation systems (which by the way, are also often called computer algebra
systems, or just CASs) such as SymPy are capable of computing symbolic
expressions with variables.

Let us define a symbolic expression, representing the mathematical expression

```
from sympy import symbols
x, y = symbols('x y')
expr = x + 2*y
expr
```

    x + 2*y

Note that we wrote `x + 2*y` just as we would if `x` and `y` were
ordinary Python variables. But in this case, instead of evaluating to
something, the expression remains as just `x + 2*y`.  Now let us play around
with it:

```
expr + 1
```

    x + 2*y + 1

```
expr - x
```

    2*y

```
x*expr
```

    x*(x + 2*y)

Notice something in the above example.  When we typed `expr`, we did not
get $x + 2 y-x$, but rather just $2y$.  The `x` and the `-x`
automatically canceled one another.  This is similar to how `sqrt(8)`
automatically turned into `2*sqrt(2)` above.  This isn’t always the case in
SymPy, however:

```
from sympy import expand, factor
expanded = expand(x*expr)
expanded
```

    x**2 + 2*x*y

```
factor(expanded)
```

    x*(x + 2*y)

# The Power of Symbolic Computation

The real power of a symbolic computation system such as SymPy is the ability
to do all sorts of computations symbolically.  SymPy can simplify expressions,
compute derivatives, integrals, and limits, solve equations, work with
matrices, and much, much more, and do it all symbolically.  It includes
modules for plotting, printing (like 2D pretty printed output of math
formulas, or $\mathrm{\LaTeX}$, code generation, physics, statistics, combinatorics,
number theory, geometry, logic, and more. Here is a small sampling of the sort
of symbolic power SymPy is capable of, to whet your appetite.

```
from sympy import *
x, t, z, nu = symbols('x t z nu')
```

```
#This will make things look pretty
init_printing(use_unicode=True)
```

*Take* the derivative of $sin{(x)}e^x$

```
diff(sin(x)*exp(x), x)
```

Compute $\int(e^x\sin{(x)} + e^x\cos{(x)})\,dx$.

```
integrate(exp(x)*sin(x) + exp(x)*cos(x), x)
```

Compute $\int_{-\infty}^\infty \sin{(x^2)}\,dx$

```
integrate(sin(x**2), (x, -oo, oo))
```

Find $\lim_{x\to 0}\frac{\sin{(x)}}{x}$

```
limit(sin(x)/x, x, 0)
```

Solve $x^2 - 2 = 0$

```
solve(x**2 - 2, x)
```

Solve the differential equation $y'' - y = e^t$

```
y = Function('y')
dsolve(Eq(y(t).diff(t, t) - y(t), exp(t)), y(t))
```

[link text](https://)Print$\int_{0}^{\pi} \cos^{2}{\left (x \right )}\, dx$ in $\mathrm{\LaTeX}$

```
latex(Integral(cos(x)**2, (x, 0, pi)))
```

    '\\int_{0}^{\\pi} \\cos^{2}{\\left (x \\right )}\\, dx'

# Why Sympy

1.   It is free and open source, you can even use it as part of something you sell
2.   SymPy is written entirely in Python, and is executed entirely in Python. Thus it is interoperable
3. It is very lightweight to download and install. Other packages might be Gb in size.
4. It can be used as a library. There is an Application Programming Interface or API for use with other tools.

# Series Expansion:
SymPy can compute asymptotic series expansions of functions around a point. To
compute the expansion of $f(x)$ around the point $x = x_0$ terms of order
$x^n$ use `f(x).series(x,x0,n)`.`x0` and `n` can be omitted, in
which case the defaults `x0=0` and `n=6` will be used.

```
expr = exp(sin(x))
expr.series(x,0,4)
```

The $O\left(x^4\right)$ term at the end represents the Landau order term at
`x=0` (not to be confused with big O notation used in computer science, which
generally represents the Landau order term at $x=\infty$.  It means that all
x terms with power greater than or equal to $x^4$ are omitted.  Order terms
can be created and manipulated outside of `series`.  They automatically
absorb higher order terms.

```
x + x**3 + x**6 + O(x**4)
```

# Lambdafy

The easiest way to convert a SymPy expression to an expression that can be numerically evaluated is to use the `lambdify` function. `lambdify` acts like a `lambda` function, except it converts the SymPy names to the names of the given numerical library, usually NumPy. For example

`Lambda` functions are just small anonymous functions

```
import numpy as np
x = lambda a : a + 10
print(x(np.arange(10)))
```

    [10 11 12 13 14 15 16 17 18 19]

```
x = lambda a, b : a * b
print(x(5,6))
```

    30

```
def myfunc(n):
  return lambda a : a * n

mytripler = myfunc(3)

print(mytripler(11))
```

    33

```
a = np.arange(10)
expr = sin(x)

f = lambdify(x, expr)

f(a)

```

    array([ 0.        ,  0.84147098,  0.90929743,  0.14112001, -0.7568025 ,
           -0.95892427, -0.2794155 ,  0.6569866 ,  0.98935825,  0.41211849])


# The Linear Model II

* linear classification | classification error | perceptron learning algorithm, pocket algorithm, ...
* linear regression | squared error | pseudo-inverse, ...
* third linear model (logistic regression) | cross-entropy error | gradient descent, ...
* nonlinear transforms

## 1. The Logistic Regression Linear Model

### 1.1 Hypothesis Functions

In the case of linear models, inputs are combined linearly using weights, and summed into a signal, $s$:

$$s = \sum\limits_{i=0}^d w_i x_i$$

Next, the signal passes through a function, given by:

* **Linear classification**: $h\left(\mathbf{x}\right) = \text{sign}\left(s\right)$
* **Linear regression**: $h\left(\mathbf{x}\right) = s$
* **Logistic regression**: $h\left(\mathbf{x}\right) = \theta\left(s\right)$

For logistic regression, we use a "soft threshold", by choosing a logistic function, $\theta$, that has a sigmoidal shape.  The sigmoidal function can take on various forms, such as the following:

$$\theta\left(s\right) = \frac{e^s}{1+e^s}$$

This model implements a probability that has a genuine probability interpretation.

### 1.2 Likelihood Measure and Probabilistic Connotations

The likelihood of a dataset, $\mathcal{D} = \left(\mathbf{x_1},y_1\right), \dots, \left(\mathbf{x_N},y_N\right)$, that we wish to maximize is given by:

$$\prod\limits_{n=1}^N P\left(y_n | \mathbf{x_n}\right) = \prod\limits_{n=1}^N \theta\left(y_n \mathbf{w^T x_n}\right)$$

It is possible to derive an error measure (that would *maximise* the above likelihood measure), which **has a probabilistic connotation**, and is called the in-sample "cross-entropy" error.  It is based on assuming the hypothesis (of the logistic regression function) as the target function:

$$E_{in}\left(\mathbf{w}\right) = \frac{1}{N}\sum\limits_{n=1}^N \ln\left[1 + \exp\left(-y_n \mathbf{w^T x_n}\right)\right]$$

$$E_{in}\left(\mathbf{w}\right) = \frac{1}{N}\sum\limits_{n=1}^N e\left[ h\left(\mathbf{x_n}\right), y_n \right]$$

While the above does not have a closed form solution, it is a *convex function* and therefore we can find the weights corresponding to the minimum of the above error measure using various techniques. Such techniques include gradient descent (and its variations, such as stochastic gradient descent and batch gradient descent) and there are others which make use of second order derivatives (such as the conjugate gradient method) or [Hessians](https://en.wikipedia.org/wiki/Hessian_matrix).

### 1.3 Libraries Used

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import minimize
from numpy.random import permutation
from sympy import var, diff, exp, latex, factor, log, simplify
from IPython.display import display, Math, Latex
%matplotlib inline
```

### 1.4 Gradient Descent for Logistic Regression

#### 1.4.1 Gradient of the Cost Function - Derivation (using Sympy)
The Python package, `sympy`, can be used to obtain the form for the gradient of the cost function in logistic regression:

```python
var('x y w')
logistic_cost = log(1 + exp(-y*w*x))
display(Math(latex(logistic_cost)))
```

$$\log{\left (1 + e^{- w x y} \right )}$$

```python
logistic_grad = logistic_cost.diff(w)
display(Math(latex(logistic_grad)))
display(Math(latex(simplify(logistic_grad))))
```

$$- \frac{x y e^{- w x y}}{1 + e^{- w x y}}$$

$$- \frac{x y}{e^{w x y} + 1}$$

#### 1.4.2 Gradient Descent Algorithm

The gradient descent algorithm is a means to find the minimum of a function,
starting from some initial weight, $\mathbf{w}()$.
The weights are adjusted at each iteration, by moving them in the direction of the **steepest descent** ($\nabla E_{in}$).  A **learning rate**, $\eta$, is used to scale the gradient, $\nabla E_{in}$.

$$\mathbf{w}(t+1) = \mathbf{w}(t) - \eta\nabla E_{in}$$

For the case of logistic regression, the gradient of the error measure with respect to the weights, is calculated as:

$$\nabla E_{in}\left(\mathbf{w}\right) = -\frac{1}{N}\sum\limits_{n=1}^N \frac{y_n\mathbf{x_N}}{1 + \exp\left(y_n \mathbf{w^T}(t)\mathbf{x_n}\right)}$$

## 2. Linear Regression Error with Noisy Targets

### 2.1 Effect of Sample Size on In-Sample Errors

Consider a noisy target, $y=\mathbf{w^{*T}x} + \epsilon$ where $\epsilon$ is a noise term with zero mean and variance, $\sigma^2$
The in-sample error on a training set, $\mathcal{D}$,

$$\mathbb{E}_\mathcal{D}\left[E_{in}\left(\mathbf{w_{lin}}\right)\right] = \sigma^2\left(1 - \frac{d+1}{N}\right)$$

```python
def in_sample_err(N, sigma = 0.1, d = 8):
    return (sigma**2)*(1 - (d+1)/N)
```

```python
N_arr = [10, 25, 100, 500, 1000]
err = [ in_sample_err(N) for N in N_arr ]
for i in range(len(N_arr)):
    print("N = {:4}, E_in = {}".format(N_arr[i],err[i]))
```

    N =   10, E_in = 0.001
    N =   25, E_in = 0.006400000000000001
    N =  100, E_in = 0.009100000000000002
    N =  500, E_in = 0.009820000000000002
    N = 1000, E_in = 0.009910000000000002

Here, we can see that, *for a noisy target*, as the number of examples, $N$, increases, the in-sample error also increases.

```python
result = minimize(lambda x: (0.008-in_sample_err(x))**2, x0=[20.0], tol=1e-11)
if result.success is True:
    N = result.x[0]
    print("N = {}".format(N))
    print("err({}) = {}".format(int(N),in_sample_err(int(N))))
    print("err({}) = {}".format(int(N+1),in_sample_err(int(N+1))))
```

    N = 44.99981844579476
    err(44) = 0.007954545454545455
    err(45) = 0.008000000000000002

If we desire an in-sample error of not more than 0.008, then the maximum number of examples we should have is 44.

## 3. Non-linear Transforms

### 3.1 Background

Consider the linear transform $z_i = \phi_i\left(\mathbf{x}\right)$ or $\mathbf{z} = \Phi\left(\mathbf{x}\right)$, with the following mapping:

$$\mathbf{x} = \left(x_0, x_1, \dots, x_d\right) \rightarrow \mathbf{z} = \left(z_0, z_1, \dots, z_{\tilde d}\right)$$

The final hypothesis, $\mathcal{X}$ space is:

$$g\left(\mathbf{x}\right) = \mathbf{\tilde w^T} \Phi\left(\mathbf{x}\right)$$

$$g\left(\mathbf{x}\right) = \left(w_0, w_1, w_2\right) \left(\begin{array}{c}1\\x_1^2\\x_2^2\end{array}\right) = w_0 + w_1 x_1^2 + w_2 x_2^2$$

The non-linear transforms are implemented in the subroutine `add_nonlinear_features()` below.  The contour plots corresponding to the non-linear transforms are implemented in `plot_data_nonlinear()`.

```python
def add_nonlinear_features(X):
    N = X.shape[0]
    X = np.hstack((X,np.zeros((N,3))))
    X[:,3] = X[:,1]*X[:,2]
    X[:,4] = X[:,1]**2
    X[:,5] = X[:,2]**2
    return(X)

def plot_data_nonlinear(fig,plot_id,w_arr,w_colors,titles):
    p = 2.0
    x1 = np.linspace(-p,p,100)
    x2 = np.linspace(-p,p,100)
    X1,X2 = np.meshgrid(x1,x2)
    X1X2 = X1*X2
    X1_sq= X1**2
    X2_sq= X2**2

    for i,w in enumerate(w_arr):
        Y = w[0] + w[1]*X1 + w[2]*X2 + w[3]*X1X2 + \
            w[4]*X1_sq + w[5]*X2_sq
        ax = fig.add_subplot(plot_id[i])
        cp0 = ax.contour(X1,X2,Y,1,linewidth=4, levels=[0.0],
                         colors=w_colors[i])
        ax.clabel(cp0, inline=True, fontsize=14)
        #cp1 = ax.contour(X1,X2,Y,N=1,linewidth=4, levels=[-1.0, 1.0],
        #                 linestyles='dashed', colors=w_colors[i], alpha=0.3)
        cp1 = ax.contourf(X1,X2,Y,1,linewidth=4, linestyles='dashed', alpha=0.8)
        ax.clabel(cp1, inline=True, fontsize=14)

        plt.colorbar(cp1)
        ax.set_title(titles[i])
        #ax.set_axis_off()  #ax.axis('off')
        ax.axes.xaxis.set_ticks([])
        ax.axes.yaxis.set_ticks([])
```

Here we wish to consider the effects of the sign of the weights $\tilde w_1, \tilde w_2$ on the decision boundary.  For simplicity, we choose the weights from [-1, 0, 1], as similar shapes would be obtained if the set of weights were scaled to something like [-2, 0, 2].

```python
w1 = np.array([ 1, 0, 0, 0, 0.0, 1.0])
w2 = np.array([ 1, 0, 0, 0, 1.0, 0.0])
w3 = np.array([ 1, 0, 0, 0, 1.0, 1.0])
w4 = np.array([ 1, 0, 0, 0,-1.0, 1.0])
w5 = np.array([ 1, 0, 0, 0, 1.0,-1.0])
w_arr = [w1,w2,w3,w4,w5]
w_colors = ['red','orange','green','blue','black']
titles = ['(a) $w_1$ = 0, $w_2$ > 0',
          '(b) $w_1$ > 0, $w_2$ = 0',
          '(c) $w_1$ > 0, $w_2$ > 0',
          '(d) $w_1$ < 0, $w_2$ > 0',
          '(e) $w_1$ > 0, $w_2$ < 0']
plot_id_arr = [ 231, 232, 233, 234, 235 ]
fig = plt.figure(figsize=(12,7))
plot_data_nonlinear(fig,plot_id_arr,w_arr,w_colors,titles)
```

In the second last example, $\tilde w_1 <0, \tilde w_2 > 0$, (with $x_0 = 1$), we have:

$$\mathbf{x} = \left(1, x_1, x_2\right) \rightarrow \mathbf{z} = \left(1, x_1^2, x_2^2\right)$$

$$g\left(\mathbf{x}\right) = 1 - x_1^2 + x_2^2$$

## 4. Gradient Descent

### 4.1 Gradient Descent Example Using Sympy

This example provides a demonstration of how the package `sympy` can be used to find the gradient of an arbitrary function, and perform gradient descent to the minimum of the function.

Our arbitrary function in this case is:

$$E\left(u,v\right) = \left(ue^v -2ve^{-u}\right)^2$$

```python
var('u v')
expr = (u*exp(v) -2*v*exp(-u))**2
display(Math(latex(expr)))
```

$$\left(u e^{v} - 2 v e^{- u}\right)^{2}$$

The partial derivative of the function, $E$, with respect to $u$ is:

```python
derivative_u = expr.diff(u)
display(Math(latex(derivative_u)))
display(Math(latex(factor(derivative_u))))
```

$$\left(u e^{v} - 2 v e^{- u}\right) \left(4 v e^{- u} + 2 e^{v}\right)$$

$$2 \left(2 v + e^{u} e^{v}\right) \left(u e^{u} e^{v} - 2 v\right) e^{- 2 u}$$

The partial derivative of the function, $E$, with respect to $v$ is:

```python
derivative_v = expr.diff(v)
display(Math(latex(derivative_v)))
display(Math(latex(factor(derivative_v))))
```

$$\left(u e^{v} - 2 v e^{- u}\right) \left(2 u e^{v} - 4 e^{- u}\right)$$

$$2 \left(u e^{u} e^{v} - 2\right) \left(u e^{u} e^{v} - 2 v\right) e^{- 2 u}$$

Next, the functions to implement the gradient descent are implemented as follows.  In the first case, `err_gradient()`, the derivatives are specified in the code.  In the second case, `err_gradient2()`, the derivatives are calculated using `sympy + evalf`:

```python
def err(uv):
    u = uv[0]
    v = uv[1]
    ev = np.exp(v)
    e_u= np.exp(-u)
    return (u*ev - 2.0*v*e_u)**2

def err_gradient(uv):
    u = uv[0]
    v = uv[1]
    ev = np.exp(v)
    e_u= np.exp(-u)
    return np.array([ 2.0*(ev + 2.0*v*e_u)*(u*ev - 2.0*v*e_u),
                      2.0*(u*ev - 2.0*e_u)*(u*ev - 2.0*v*e_u) ])
def err_gradient2(uv):
    du = derivative_u.subs(u,uv[0]).subs(v,uv[1]).evalf()
    dv = derivative_v.subs(u,uv[0]).subs(v,uv[1]).evalf()
    return np.array([ du, dv ], dtype=float)
```

To follow the gradient to the function minimum, we can either use $\nabla E$ in the gradient descent approach, or we can alternate between the individual derivatives, $\frac{\partial E}{\partial u}$ and $\frac{\partial E}{\partial v}$ in the coordinate descent approach.

```python
def gradient_descent(x0, err, d_err, eta=0.1):
    x = x0
    for i in range(20):
        e = err(x)
        de = d_err(x)
        print("%2d: x = (%8.5f, %8.5f) | err' = (%8.4f, %8.4f) | err = %.3e" %
              (i,x[0],x[1],de[0],de[1],e))
        if e < 1e-14:
            break
        x = x - eta*de

def coordinate_descent(x0, err, d_err, eta=0.1):
    x = x0
    for i in range(15):
        # Step 1: Move along the u-coordinate
        e = err(x)
        de = d_err(x)
        print("%2d: x = (%8.5f, %8.5f) | err' = (%8.4f, --------) | err = %.3e" %
              (i,x[0],x[1],de[0],e))
        x[0] = x[0] - eta*de[0]
        if e < 1e-14: break

        # Step 2: Move along the v-coordinate
        e = err(x)
        de = d_err(x)
        print("%2d: x = (%8.5f, %8.5f) | err' = (--------, %8.4f) | err = %.3e" %
              (i,x[0],x[1],de[1],e))
        x[1] = x[1] - eta*de[1]
        if e < 1e-14: break
```

```python
x0 = np.array([1.0,1.0])
gradient_descent(x0=x0, err=err, d_err=err_gradient)
```

     0: x = ( 1.00000,  1.00000) | err' = ( 13.6954,   7.8608) | err = 3.930e+00
     1: x = (-0.36954,  0.21392) | err' = ( -4.0006,   7.2185) | err = 1.160e+00
     2: x = ( 0.03052, -0.50793) | err' = ( -0.7700,  -3.8572) | err = 1.007e+00
     3: x = ( 0.10752, -0.12221) | err' = (  0.4188,  -1.0704) | err = 9.901e-02
     4: x = ( 0.06564, -0.01517) | err' = (  0.1780,  -0.3366) | err = 8.661e-03
     5: x = ( 0.04784,  0.01849) | err' = (  0.0284,  -0.0501) | err = 1.818e-04
     6: x = ( 0.04500,  0.02350) | err' = (  0.0024,  -0.0043) | err = 1.297e-06
     7: x = ( 0.04476,  0.02392) | err' = (  0.0002,  -0.0003) | err = 7.292e-09
     8: x = ( 0.04474,  0.02396) | err' = (  0.0000,  -0.0000) | err = 4.010e-11
     9: x = ( 0.04474,  0.02396) | err' = (  0.0000,  -0.0000) | err = 2.202e-13
    10: x = ( 0.04474,  0.02396) | err' = (  0.0000,  -0.0000) | err = 1.209e-15

```python
gradient_descent(x0=x0, err=err, d_err=err_gradient2)
```

     0: x = ( 1.00000,  1.00000) | err' = ( 13.6954,   7.8608) | err = 3.930e+00
     1: x = (-0.36954,  0.21392) | err' = ( -4.0006,   7.2185) | err = 1.160e+00
     2: x = ( 0.03052, -0.50793) | err' = ( -0.7700,  -3.8572) | err = 1.007e+00
     3: x = ( 0.10752, -0.12221) | err' = (  0.4188,  -1.0704) | err = 9.901e-02
     4: x = ( 0.06564, -0.01517) | err' = (  0.1780,  -0.3366) | err = 8.661e-03
     5: x = ( 0.04784,  0.01849) | err' = (  0.0284,  -0.0501) | err = 1.818e-04
     6: x = ( 0.04500,  0.02350) | err' = (  0.0024,  -0.0043) | err = 1.297e-06
     7: x = ( 0.04476,  0.02392) | err' = (  0.0002,  -0.0003) | err = 7.292e-09
     8: x = ( 0.04474,  0.02396) | err' = (  0.0000,  -0.0000) | err = 4.010e-11
     9: x = ( 0.04474,  0.02396) | err' = (  0.0000,  -0.0000) | err = 2.202e-13
    10: x = ( 0.04474,  0.02396) | err' = (  0.0000,  -0.0000) | err = 1.209e-15

Here, we can see that in both approaches of gradient descent above, it takes about 10 iterations to get the error below $10^{-14}$.

For comparison, an attempt to find the roots of the minimum via [`scipy.optimize.minimize`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.minimize.html) was made, but it yielded a different result.  This could be due to the fact that another method was used (in this case, conjugate gradient).  At the moment, `scipy.optimize.minimize`, does not appear to have a gradient descent implementation.

```python
err_fn = lambda x: (x[0]*np.exp([1]) - 2.0*x[1]*np.exp(-x[0]))**2
result = minimize(err_fn, x0=np.array([1.0,1.0]), tol=1e-5, method='CG')
if result.success is True:
    x = result.x
    print("x = {}".format(x))
    print("f = {}".format(result.fun))
    print("evalf = {}".format(expr.subs(u,x[0]).subs(v,x[1]).evalf()))
```

    x = [ 0.49714221  1.11083931]
    f = 1.7782471701861512e-17
    evalf = 0.0250910775449603

### 4.2 Coordinate Descent

Using the coordinate descent approach, the error minimization takes place more slowly.  Even after 15 iterations, the error remains at only ~0.15, regardless of implementation.

```python
x0 = np.array([1.0,1.0])
coordinate_descent(x0=x0, err=err, d_err=err_gradient)
```

     0: x = ( 1.00000,  1.00000) | err' = ( 13.6954, --------) | err = 3.930e+00
     0: x = (-0.36954,  1.00000) | err' = (--------,  30.3992) | err = 1.520e+01
     1: x = (-0.36954, -2.03992) | err' = (-67.6202, --------) | err = 3.429e+01
     1: x = ( 6.39247, -2.03992) | err' = (--------,   1.3878) | err = 7.024e-01
     2: x = ( 6.39247, -2.17870) | err' = (  0.1548, --------) | err = 5.341e-01
     2: x = ( 6.37700, -2.17870) | err' = (--------,   1.0477) | err = 5.318e-01
     3: x = ( 6.37700, -2.28347) | err' = (  0.1239, --------) | err = 4.327e-01
     3: x = ( 6.36461, -2.28347) | err' = (--------,   0.8474) | err = 4.311e-01
     4: x = ( 6.36461, -2.36821) | err' = (  0.1033, --------) | err = 3.650e-01
     4: x = ( 6.35428, -2.36821) | err' = (--------,   0.7138) | err = 3.640e-01
     5: x = ( 6.35428, -2.43959) | err' = (  0.0886, --------) | err = 3.165e-01
     5: x = ( 6.34542, -2.43959) | err' = (--------,   0.6178) | err = 3.157e-01
     6: x = ( 6.34542, -2.50138) | err' = (  0.0774, --------) | err = 2.798e-01
     6: x = ( 6.33768, -2.50138) | err' = (--------,   0.5452) | err = 2.792e-01
     7: x = ( 6.33768, -2.55590) | err' = (  0.0687, --------) | err = 2.510e-01
     7: x = ( 6.33081, -2.55590) | err' = (--------,   0.4884) | err = 2.505e-01
     8: x = ( 6.33081, -2.60473) | err' = (  0.0617, --------) | err = 2.278e-01
     8: x = ( 6.32464, -2.60473) | err' = (--------,   0.4425) | err = 2.274e-01
     9: x = ( 6.32464, -2.64898) | err' = (  0.0559, --------) | err = 2.087e-01
     9: x = ( 6.31904, -2.64898) | err' = (--------,   0.4047) | err = 2.083e-01
    10: x = ( 6.31904, -2.68945) | err' = (  0.0511, --------) | err = 1.926e-01
    10: x = ( 6.31393, -2.68945) | err' = (--------,   0.3730) | err = 1.923e-01
    11: x = ( 6.31393, -2.72675) | err' = (  0.0470, --------) | err = 1.789e-01
    11: x = ( 6.30923, -2.72675) | err' = (--------,   0.3460) | err = 1.787e-01
    12: x = ( 6.30923, -2.76135) | err' = (  0.0435, --------) | err = 1.671e-01
    12: x = ( 6.30488, -2.76135) | err' = (--------,   0.3227) | err = 1.670e-01
    13: x = ( 6.30488, -2.79361) | err' = (  0.0404, --------) | err = 1.569e-01
    13: x = ( 6.30084, -2.79361) | err' = (--------,   0.3024) | err = 1.567e-01
    14: x = ( 6.30084, -2.82385) | err' = (  0.0377, --------) | err = 1.478e-01
    14: x = ( 6.29708, -2.82385) | err' = (--------,   0.2845) | err = 1.477e-01

```python
x0 = np.array([1.0,1.0])
coordinate_descent(x0=x0, err=err, d_err=err_gradient2)
```

     0: x = ( 1.00000,  1.00000) | err' = ( 13.6954, --------) | err = 3.930e+00
     0: x = (-0.36954,  1.00000) | err' = (--------,  30.3992) | err = 1.520e+01
     1: x = (-0.36954, -2.03992) | err' = (-67.6202, --------) | err = 3.429e+01
     1: x = ( 6.39247, -2.03992) | err' = (--------,   1.3878) | err = 7.024e-01
     2: x = ( 6.39247, -2.17870) | err' = (  0.1548, --------) | err = 5.341e-01
     2: x = ( 6.37700, -2.17870) | err' = (--------,   1.0477) | err = 5.318e-01
     3: x = ( 6.37700, -2.28347) | err' = (  0.1239, --------) | err = 4.327e-01
     3: x = ( 6.36461, -2.28347) | err' = (--------,   0.8474) | err = 4.311e-01
     4: x = ( 6.36461, -2.36821) | err' = (  0.1033, --------) | err = 3.650e-01
     4: x = ( 6.35428, -2.36821) | err' = (--------,   0.7138) | err = 3.640e-01
     5: x = ( 6.35428, -2.43959) | err' = (  0.0886, --------) | err = 3.165e-01
     5: x = ( 6.34542, -2.43959) | err' = (--------,   0.6178) | err = 3.157e-01
     6: x = ( 6.34542, -2.50138) | err' = (  0.0774, --------) | err = 2.798e-01
     6: x = ( 6.33768, -2.50138) | err' = (--------,   0.5452) | err = 2.792e-01
     7: x = ( 6.33768, -2.55590) | err' = (  0.0687, --------) | err = 2.510e-01
     7: x = ( 6.33081, -2.55590) | err' = (--------,   0.4884) | err = 2.505e-01
     8: x = ( 6.33081, -2.60473) | err' = (  0.0617, --------) | err = 2.278e-01
     8: x = ( 6.32464, -2.60473) | err' = (--------,   0.4425) | err = 2.274e-01
     9: x = ( 6.32464, -2.64898) | err' = (  0.0559, --------) | err = 2.087e-01
     9: x = ( 6.31904, -2.64898) | err' = (--------,   0.4047) | err = 2.083e-01
    10: x = ( 6.31904, -2.68945) | err' = (  0.0511, --------) | err = 1.926e-01
    10: x = ( 6.31393, -2.68945) | err' = (--------,   0.3730) | err = 1.923e-01
    11: x = ( 6.31393, -2.72675) | err' = (  0.0470, --------) | err = 1.789e-01
    11: x = ( 6.30923, -2.72675) | err' = (--------,   0.3460) | err = 1.787e-01
    12: x = ( 6.30923, -2.76135) | err' = (  0.0435, --------) | err = 1.671e-01
    12: x = ( 6.30488, -2.76135) | err' = (--------,   0.3227) | err = 1.670e-01
    13: x = ( 6.30488, -2.79361) | err' = (  0.0404, --------) | err = 1.569e-01
    13: x = ( 6.30084, -2.79361) | err' = (--------,   0.3024) | err = 1.567e-01
    14: x = ( 6.30084, -2.82385) | err' = (  0.0377, --------) | err = 1.478e-01
    14: x = ( 6.29708, -2.82385) | err' = (--------,   0.2845) | err = 1.477e-01

## 5. Logistic Regression

### 5.1 Creating a target function

For simplicity, we choose a target function, $f$, to be a 0/1 probability.
For visualization purposes, we choose the domain of interest to be in 2 dimensions, and choose $\mathbf{x}$ to be picked uniformly from the region $\mathcal{X}=\left[-1,1\right] \times \left[-1,1\right]$,
where $\times$ denotes the [Cartesian Product](https://en.wikipedia.org/wiki/Cartesian_product).

A random line is created, and to ensure that it falls within the region of interest, it is created from two random points, $(x_0,y_0)$ and $(x_1,y_1)$ which are generated within $\mathcal{X}$.  The equation for this line in *slope-intercept* form and in the *hypothesis / weights* can be shown to be:

**Slope-Intercept Form**

$$m = - \frac{w_1}{w_2}, c = - \frac{w_0}{w_2}$$

**Hypothesis Weights Form**

$$\mathbf{w} = \left(-c,-m,1\right)$$

```python
def generate_data(n,seed=None):
    if seed is not None:
        np.random.seed(seed)
    x0 = np.ones(n)
    x1 = np.random.uniform(low=-1,high=1,size=(2,n))
    return np.vstack((x0,x1)).T
```

```python
def get_random_line(seed=None):
    X = generate_data(2,seed=seed)
    x = X[:,1]
    y = X[:,2]
    m = (y[1]-y[0])/(x[1]-x[0])
    c = y[0] - m*x[0]
    return np.array([-c,-m,1])

def draw_line(ax,w,marker='g--',label=None):
    m = -w[1]/w[2]
    c = -w[0]/w[2]
    x = np.linspace(-1,1,20)
    y = m*x + c
    if label is None:
        ax.plot(x,y,marker)
    else:
        ax.plot(x,y,marker,label=label)

def get_hypothesis(X,w):
    h=np.dot(X,w)
    return np.sign(h).astype(int)
```

### 5.2 Plotting the Data

```python
def plot_data(fig,plot_id,X,y=None,w_arr=None,my_x=None,title=None):
    ax = fig.add_subplot(plot_id)
    if y is None:
        ax.plot(X[:,1],X[:,2],'gx')
    else:
        ax.plot(X[y > 0,1],X[y > 0,2],'b+',label='Positive (+)')
        ax.plot(X[y < 0,1],X[y < 0,2],'ro',label='Negative (-)')
    ax.set_xlim(-1,1)
    ax.set_ylim(-1,1)
    ax.grid(True)
    if w_arr is not None:
        if isinstance(w_arr,list) is not True:
            w_arr=[w_arr]
        for i,w in enumerate(w_arr):
            if i==0:
                draw_line(ax,w,'g-',label='Theoretical')
            else:
                draw_line(ax,w,'g--')
    if my_x is not None:
        ax.plot([my_x[0]],[my_x[1]],'kx',markersize=10)
    if title is not None:
        ax.set_title(title)
    ax.legend(loc='best',frameon=True)
```

```python
def create_dataset(N,make_plot=True,seed=None):
    X = generate_data(N,seed=seed)
    w_theoretical = get_random_line()
    y = get_hypothesis(X,w_theoretical)
    if make_plot is True:
        fig = plt.figure(figsize=(7,5))
        plot_data(fig,111,X,y,w_theoretical,title="Initial Dataset")
    return X,y,w_theoretical
```

We choose 100 training points at random from $\mathcal{X}$ and record the outputs, $y_n$, for each of the points, $\mathbf{x_n}$.

```python
N = 100
X,y,w_theoretical = create_dataset(N=N,make_plot=True,seed=127)
```

### 5.3 Gradient Descent

The gradient descent algorithm adjust the weights in the direction of the 'steepest descent' ($\nabla E_{in}$), with the adjustment of a learning rate, $\eta$:

$$\mathbf{w}(t+1) = \mathbf{w}(t) - \eta\nabla E_{in}$$

We thus need to know the gradient of the error measure with respect to the weights, i.e.:

$$\nabla E_{in}\left(\mathbf{w}\right) = -\frac{1}{N}\sum\limits_{n=1}^N \frac{y_n\mathbf{x_N}}{1 + \exp\left(y_n \mathbf{w^T}(t)\mathbf{x_n}\right)}$$

$$E_{in}\left(\mathbf{w}\right) = \frac{1}{N}\sum\limits_{n=1}^N \ln\left[1 + \exp\left(-y_n \mathbf{w^T x_n}\right)\right]$$

```python
w = w_theoretical
def cross_entropy(y_i,w,x):
    return np.log(1 + np.exp(-y_i*np.dot(x,w)))
def gradient(y_i,w,x):
    return -y_i*x/(1+np.exp(y_i*np.dot(x,w)))
assert np.allclose(cross_entropy(y[0],w,X[0,:]),np.log(1 + np.exp(-y[0]*np.dot(X[0,:],w))))
assert np.allclose(gradient(y[0],w,X[0,:]),-y[0]*X[0,:]/(1+np.exp(y[0]*np.dot(X[0,:],w))))
```

```python
np.mean(cross_entropy(y,w,X))
```

    0.15242626832575448

```python
np.set_printoptions(precision=4)
assert np.linalg.norm(np.array([1.0, 2.0, 3.0])) == np.sqrt(1**2 + 2**2 + 3**2)
```

```python
def run_simulation(N=100,eta=0.01,make_plot=None,w0 = np.array([0,0,0],dtype=float)):
    X = generate_data(N)
    w_theoretical = get_random_line()
    y = get_hypothesis(X,w_theoretical)

    w_arr = []
    w_arr2= []
    e_arr = []
    w = w0
    h = get_hypothesis(X,w)
    assert y.dtype == h.dtype
    for t_epoch in range(1000):
        w_epoch = w
        for i,p in enumerate(permutation(N)):
            grad = gradient(y[p],w,X[p,:])
            w = w - eta*grad;
            w_arr2.append(w)

        #Estimate out-of-sample error by re-generating data
        X_out = generate_data(N)
        h = get_hypothesis(X_out,w_theoretical)
        misclassified = np.mean(h != y)
        #E_out = np.mean(cross_entropy(y,w,X))
        E_out = np.mean(cross_entropy(h,w,X_out))
        delta_w = np.linalg.norm(w - w_epoch)
        w_arr.append(w)
        e_arr.append(E_out)
        #if t_epoch % 20 == 0:
        #    print("epoch{:4}: miss={}, delta_w={}, E_out={}, w={}".format(
        #          t_epoch, misclassified, np.round(delta_w,5), E_out, w))
        if delta_w < 0.01: break
    print("Epochs = {}, E_out = {}, w = {}".format(t_epoch, E_out, w))
    if make_plot is not None:
        fig = plt.figure(figsize=(7,5))
        plot_data(fig,111,X,y,[w_theoretical,w],title="Converged")
    return e_arr, np.array(w_arr), X, y, np.array(w_arr2)
```

Due to the randomness of starting with different target functions each time, we run stochastic gradient descent multiple times and consider the statistics in terms of the average number of epochs and the average out-of-sample errors.

```python
t_arr = []
e_arr = []
w_arr = []
for n in range(50):
    e, w, _, _, _ = run_simulation()
    t_arr.append(len(e)-1)  #Should I subtract 1 here?
    e_arr.append(e[-1])
    w_arr.append(w[-1])
```

    Epochs = 344, E_out = 0.1145102478546653, w = [-4.3349 -6.8128  3.151 ]
    Epochs = 330, E_out = 0.13254923381857528, w = [ 3.9176  7.7578  3.0019]
    Epochs = 338, E_out = 0.12272718416074929, w = [-5.1854  5.7707 -0.0663]
    Epochs = 362, E_out = 0.11358961920713569, w = [-5.1431  1.1128  7.6227]
    Epochs = 355, E_out = 0.08452200240499405, w = [-3.4195  7.4099  4.1785]
    Epochs = 387, E_out = 0.10453583621865964, w = [ 1.4044 -7.1103  6.8903]
    Epochs = 382, E_out = 0.11095706431956907, w = [ 1.2906 -6.6268  7.327 ]
    Epochs = 375, E_out = 0.11401376588425498, w = [ 4.3071  8.4367  1.3905]
    Epochs = 313, E_out = 0.08906826626000988, w = [-4.6285  5.0157  4.3839]
    Epochs = 319, E_out = 0.09907279197577364, w = [ 1.8946  7.248   4.4978]
    Epochs = 331, E_out = 0.11951194996454045, w = [-2.1788  7.0753  5.1223]
    Epochs = 345, E_out = 0.08534027589824467, w = [ 4.5074  2.043   6.8368]
    Epochs = 353, E_out = 0.09919776381452064, w = [ 2.6006  8.8545  2.8099]
    Epochs = 386, E_out = 0.10680017454642046, w = [ 2.3637 -7.6321  6.7116]
    Epochs = 327, E_out = 0.07963332746431315, w = [-1.3705  1.8633  8.9342]
    Epochs = 310, E_out = 0.1004902896810616, w = [-4.8985  2.0773  6.2821]
    Epochs = 352, E_out = 0.09435527830870676, w = [ 3.026   8.6722  2.6698]
    Epochs = 177, E_out = 0.06606451292292667, w = [-4.0843 -2.4275  0.7686]
    Epochs = 312, E_out = 0.08993382351733965, w = [-1.6081 -8.5505  0.3119]
    Epochs = 343, E_out = 0.10269877468148719, w = [ 2.1217 -8.803   2.7701]
    Epochs = 272, E_out = 0.0872375327354725, w = [-4.8057 -2.0756  4.3213]
    Epochs = 405, E_out = 0.1150348727541617, w = [-4.831  -8.8921  0.8516]
    Epochs = 322, E_out = 0.10791883242053275, w = [-4.0173  1.1187  7.0841]
    Epochs = 343, E_out = 0.13927435537954996, w = [ 3.7391  6.7424  4.8557]
    Epochs = 324, E_out = 0.11451497060907402, w = [-4.0974  4.2159  6.4424]
    Epochs = 337, E_out = 0.11013500604859183, w = [-1.6868 -3.1915  8.7905]
    Epochs = 311, E_out = 0.1202390667292876, w = [ 1.2721 -8.2782  3.2557]
    Epochs = 425, E_out = 0.1040334437781984, w = [ 1.1855 -8.1509  6.6445]
    Epochs = 324, E_out = 0.1266759412214779, w = [ 1.769  -1.973   8.6323]
    Epochs = 362, E_out = 0.10454353508849133, w = [ 1.6676  3.9335  8.6676]
    Epochs = 331, E_out = 0.07162336922800794, w = [-1.9393  2.237   8.7294]
    Epochs = 347, E_out = 0.11201459689167029, w = [ 2.7126 -6.0879  6.4069]
    Epochs = 324, E_out = 0.10162245585984317, w = [-4.5788 -6.8037  3.3853]
    Epochs = 376, E_out = 0.07224885673385947, w = [ 4.9275 -1.5392  8.1009]
    Epochs = 344, E_out = 0.09418330137125026, w = [-2.2507 -8.0475  4.707 ]
    Epochs = 409, E_out = 0.07914745644057647, w = [ 1.181  -5.8557  8.8743]
    Epochs = 361, E_out = 0.0781523158521627, w = [-4.0031  6.3655  5.5268]
    Epochs = 354, E_out = 0.11836358806105533, w = [-2.5064 -4.4141  8.1074]
    Epochs = 361, E_out = 0.10094928768153809, w = [-1.5559 -5.7539  7.5764]
    Epochs = 363, E_out = 0.07564746211750291, w = [-1.2404 -4.2471  8.6759]
    Epochs = 401, E_out = 0.10600085255274277, w = [-5.4864  6.9466  0.912 ]
    Epochs = 315, E_out = 0.13348176305348317, w = [ 2.0044 -8.0714  2.6051]
    Epochs = 398, E_out = 0.08309106515191791, w = [-4.8958  0.2701  9.4649]
    Epochs = 259, E_out = 0.16735110442009668, w = [-4.3832  0.5935  5.2335]
    Epochs = 294, E_out = 0.10293018590522714, w = [ 0.0834 -0.3418  8.7189]
    Epochs = 384, E_out = 0.08324299232848924, w = [ 3.9464  6.2222  6.9053]
    Epochs = 345, E_out = 0.12940067224923055, w = [-0.0396  5.4435  7.5821]
    Epochs = 342, E_out = 0.11404890276530884, w = [ 1.3681  9.2648  0.4564]
    Epochs = 327, E_out = 0.10240693589397777, w = [ 2.7934 -8.2483  1.8324]
    Epochs = 312, E_out = 0.07759210390392761, w = [-4.5546 -5.4712  4.6846]

The average out of sample error and the average number of epochs from the multiple runs above are:

```python
print("<E_out> = {}".format(np.mean(e_arr)))
print("<Epochs> = {}".format(np.mean(t_arr)))
```

    <E_out> = 0.10325358016261309
    <Epochs> = 342.26

### 5.4 Gradient Descent Visualization

```python
def normalize_weights(w_arr):
    # You can't normalize the weights as this changes the cross entropy.
    w_arr[:,1] = w_arr[:,1] / w_arr[:,0]
    w_arr[:,2] = w_arr[:,2] / w_arr[:,0]
    w_arr[:,0] = 1.0
    return w_arr
```

```python
def calculate_J(w0,w1,w2,X,y):
    J = np.zeros((w1.size,w2.size))
    for j in range(w1.size):
        for i in range(w2.size):
            W = np.array([w0, w1[j], w2[i]])
            J[i,j] = np.mean(cross_entropy(y,W,X))
    return J

def get_WJ(w_arr,X,y,n=100):
    w_arr = np.array(w_arr)

    w1_min = np.min(w_arr[:,1])
    w2_min = np.min(w_arr[:,2])
    w1_max = np.max(w_arr[:,1])
    w2_max = np.max(w_arr[:,2])
    sp = 10.0

    w0 = w_arr[-1,0]  # take a 2D slice through the final value of w_0 in the 3D space [w0,w1,w2]
    w1 = np.linspace(w1_min-sp,w1_max+sp,n)
    w2 = np.linspace(w2_min-sp,w2_max+sp,n)
    W1, W2 = np.meshgrid(w1,w2)
    J = calculate_J(w0,w1,w2,X,y)
    return w_arr,w1,w2,W1,W2,J
```

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
def visualise_SGD_3D(e_arr,w_arr,w_arr2,X,y,epoch_interval,elevation=30,azimuth=75):
    w_arr,w1,w2,W1,W2,J = get_WJ(w_arr,X,y)
    w0 = w_arr[-1,0]  # take a 2D slice through the final value of w_0 in the 3D space [w0,w1,w2]
    z_arr = [ np.mean(cross_entropy(y,[w0,w_i[1],w_i[2]],X)) for w_i in w_arr ]
    z_arr2 = [ np.mean(cross_entropy(y,[w0,w_i[1],w_i[2]],X)) for w_i in w_arr2 ]

    fig = plt.figure(figsize=(14,10))
    ax = fig.gca(projection='3d')
    surf = ax.plot_surface(W1,W2,J, rstride=10, cstride=10, cmap=cm.coolwarm,
           linewidth=0.3, antialiased=True, alpha=0.9) #, zorder=3)
    ax.set_xlabel(r'$w_1$', fontsize=18)
    ax.set_ylabel(r'$w_2$', fontsize=18)
    ax.set_zlabel(r'$E_{in}$', fontsize=18)
    ax.plot(w_arr[:,1],w_arr[:,2],z_arr,'k-',lw=0.8,label="Stochastic Gradient Descent (SGD)")
    ax.plot(w_arr2[:,1],w_arr2[:,2],z_arr2,'k-',lw=1.8,alpha=0.3,label="SGD within epochs")
    ax.plot(w_arr[::epoch_interval,1],w_arr[::epoch_interval,2],z_arr[::epoch_interval],
            'ko',markersize=7,label=r"Intervals of $n$ Epochs")
    ax.scatter([w_arr[-1,1]],[w_arr[-1,2]],[z_arr[-1]], c='r', s=250, marker='x', lw=3);
    #fig.colorbar(surf, shrink=0.5, aspect=12)
    ax.legend(loc='best',frameon=False)
    ax.axes.xaxis.set_ticklabels([])
    ax.axes.yaxis.set_ticklabels([])
    ax.axes.zaxis.set_ticklabels([])
    ax.view_init(elev=elevation, azim=azimuth)

def visualise_SGD_contour(e_arr,w_arr,w_arr2,X,y,epoch_interval):
    w_arr,w1,w2,W1,W2,J = get_WJ(w_arr,X,y)

    fig = plt.figure(figsize=(12,8))
    ax = fig.gca()
    CS = plt.contour(W1,W2,J,20)
    #plt.clabel(CS, inline=1, fontsize=10)
    ax.set_xlabel(r'$w_1$', fontsize=18)
    ax.set_ylabel(r'$w_2$', fontsize=18)
    ax.plot(w_arr[:,1],w_arr[:,2],'k-',lw=0.8,label="Stochastic Gradient Descent (SGD)")
    ax.plot(w_arr2[:,1],w_arr2[:,2],'k-',lw=1.8,alpha=0.3,label="SGD within epochs")
    ax.plot(w_arr[::epoch_interval,1],w_arr[::epoch_interval,2],
            'ko',markersize=7,label=r"Intervals of $n$ Epochs")
    ax.scatter([w_arr[-1,1]],[w_arr[-1,2]], c='r', s=150, marker='x', lw=3);
    ax.legend(loc='best',frameon=False)
    ax.axes.xaxis.set_ticklabels([])
    ax.axes.yaxis.set_ticklabels([])
    plt.title(r'$E_{in}$', fontsize=16);
```

```python
def plot_epochs(e_arr,w_arr,X,y,epoch_interval):
    w_arr,w1,w2,W1,W2,J = get_WJ(w_arr,X,y)
    E_in = [ np.mean(cross_entropy(y,w_i,X)) for w_i in w_arr ]
    epoch = np.array(range(len(e_arr)))

    fig = plt.figure(figsize=(10,10))
    ax = fig.add_subplot(211)
    ax.set_ylabel(r'Error', fontsize=16)
    ax.plot(epoch,e_arr,c='g',markersize=1,marker='+',lw=1,alpha=0.8,label=r'$E_{out}$')
    #ax.scatter(epoch[::epoch_interval],e_arr[::epoch_interval],c='g',s=20,marker='o',lw=3,alpha=0.8)
    ax.plot(epoch,E_in,c='k',linestyle='--',label=r'$E_{in}$')
    ax.legend(loc='best',frameon=False, fontsize=16)
    ax.set_title('"Cross Entropy" Error', fontsize=16);
    ax.axes.xaxis.set_ticklabels([])
    ax.axes.yaxis.set_ticklabels([])
    ax.grid(True)

    ax = fig.add_subplot(212)
    ax.set_xlabel(r'Epoch', fontsize=16)
    ax.set_ylabel(r'Error', fontsize=16)
    ax.loglog(epoch,e_arr,c='g',markersize=1,marker='+',lw=1,alpha=0.8,label=r'$E_{out}$')
    ax.loglog(epoch,E_in,c='k',linestyle='--',label=r'$E_{in}$')
    #ax.loglog(epoch[::epoch_interval],e_arr[::epoch_interval],c='g',markersize=8,marker='o',lw=3,alpha=0.8,ls='None')
    ax.legend(loc='best',frameon=False, fontsize=16)
    ax.axes.xaxis.set_ticklabels([])
    ax.axes.yaxis.set_ticklabels([])
    ax.grid(True)
```

```python
np.random.seed(12345)
e_arr, w_arr, X, y, w_arr2 = run_simulation(N=15,eta=0.8,w0=np.array([2.0, 10.0, -20.0]))
```

    Epochs = 751, E_out = 0.00495140131231788, w = [  4.8336  21.6983  15.9885]

```python
visualise_SGD_3D(e_arr,w_arr,w_arr2,X,y,epoch_interval=100)
```

```python
visualise_SGD_contour(e_arr,w_arr,w_arr2,X,y,epoch_interval=100)
```

```python
plot_epochs(e_arr,w_arr,X,y,epoch_interval=100)
```

### 5.5 Stochastic Gradient Descent vs Perceptron Learning Algorithm

"Consider that you are picking a point at random out of the $N$ points. In PLA, you see if it is misclassified then update using the PLA rule if it is and not update if it isn't. In SGD, you take the gradient of the error on that point w.r.t. $\mathbf{w}$ and update accordingly. Which of the 5 error functions would make these equivalent?

- **(a)**: $e_n\left(\mathbf{w}\right) = \exp\left(-y_n \mathbf{w^T x_n}\right)$
- **(b)**: $e_n\left(\mathbf{w}\right) = -y_n \mathbf{w^T x_n}$
- **(c)**: $e_n\left(\mathbf{w}\right) = \left(y_n - \mathbf{w^T x_n}\right)^2$
- **(d)**: $e_n\left(\mathbf{w}\right) = \ln\left[1 + \exp\left(-y_n \mathbf{w^T x_n}\right)\right]$
- **(e)**: $e_n\left(\mathbf{w}\right) = -\min\left(0, y_n \mathbf{w^T x_n}\right)$

Answer: **(e)**

Notes: an attempt to evaluate the gradients of the above functions using sympy was carried out as follows (the final expression, which contains the function `min` was excluded):

```python
var('y_n w_i x_n')
expr = exp(-y_n * w_i * x_n)
d_expr = expr.diff(w_i)
display(Math(latex(d_expr)))

expr = -y_n * w_i * x_n
d_expr = expr.diff(w_i)
display(Math(latex(d_expr)))

expr = (y_n - w_i * x_n)**2
d_expr = simplify(expr.diff(w_i))
display(Math(latex(d_expr)))

expr = log(1+exp(-y_n * w_i * x_n))
d_expr = simplify(expr.diff(w_i))
display(Math(latex(d_expr)))
```

$$- x_{n} y_{n} e^{- w_{i} x_{n} y_{n}}$$

$$- x_{n} y_{n}$$

$$2 x_{n} \left(w_{i} x_{n} - y_{n}\right)$$

$$- \frac{x_{n} y_{n}}{e^{w_{i} x_{n} y_{n}} + 1}$$

```python
w_final = np.array(w_arr)[-1,:]
e_a = np.mean(np.exp(-y*np.dot(X,w_final)))
e_b = np.mean(-y*np.dot(X,w_final))
e_c = np.mean((y - np.dot(X,w_final))**2)
e_d = np.mean(np.log(1 + np.exp(-y*np.dot(X,w_final))))
e_e = -y*np.dot(X,w_final); e_e[e_e > 0] = 0; e_e = np.mean(e_e)
print("(a) e_n(w) = {}".format(e_a))
print("(b) e_n(w) = {}".format(e_b))
print("(c) e_n(w) = {}".format(e_c))
print("(d) e_n(w) = {}".format(e_d))
print("(e) e_n(w) = {}".format(e_e))
```

    (a) e_n(w) = 0.006921150896126407
    (b) e_n(w) = -15.160717353544413
    (c) e_n(w) = 314.2802497501979
    (d) e_n(w) = 0.006754859029354315
    (e) e_n(w) = -15.160717353544413

An attempt was also made to visualize the gradient descent algorithm when performed on the various error functions.

```python
def my_err_fn(y,W,X):
    #e = np.exp(-y*np.dot(X,W))              # e_a
    #e = -y*np.dot(X,W)                      # e_b
    #e = (y - np.dot(X,W))**2                # e_c
    e = np.log(1 + np.exp(-y*np.dot(X,W)))   # e_d
    #e = -y*np.dot(X,W); e[e > 0] = 0        # e_e
    return np.mean(e)

def calculate_J(w0,w1,w2,X,y,my_err_fn):
    J = np.zeros((w1.size,w2.size))
    for j in range(w1.size):
        for i in range(w2.size):
            W = np.array([w0, w1[j], w2[i]])
            J[i,j] = my_err_fn(y,W,X)
    return J

def get_WJ(w_arr,X,y,my_err_fn,n=100):
    w_arr = np.array(w_arr)
    w1_min = np.min(w_arr[:,1])
    w2_min = np.min(w_arr[:,2])
    w1_max = np.max(w_arr[:,1])
    w2_max = np.max(w_arr[:,2])
    sp = 10.0

    w0 = w_arr[-1,0]  # take a 2D slice through the final value of w_0 in the 3D space [w0,w1,w2]
    w1 = np.linspace(w1_min-sp,w1_max+sp,n)
    w2 = np.linspace(w2_min-sp,w2_max+sp,n)
    W1, W2 = np.meshgrid(w1,w2)
    J = calculate_J(w0,w1,w2,X,y,my_err_fn)
    return w_arr,w1,w2,W1,W2,J

def visualise_SGD_contour2(e_arr,w_arr,X,y,my_err_fn):
    w_arr,w1,w2,W1,W2,J = get_WJ(w_arr,X,y,my_err_fn)

    fig = plt.figure(figsize=(10,7))
    ax = fig.gca()
    CS = plt.contour(W1,W2,J,20)
    plt.clabel(CS, inline=1, fontsize=10)
    ax.set_xlabel(r'$w_1$', fontsize=18)
    ax.set_ylabel(r'$w_2$', fontsize=18)
    ax.plot(w_arr[:,1],w_arr[:,2],'k-',label="Gradient Descent")
    ax.plot(w_arr[::100,1],w_arr[::100,2],'ko',markersize=7,label=r"Intervals of $n$ Epochs")
    ax.scatter([w_arr[-1,1]],[w_arr[-1,2]], c='r', s=150, marker='x', lw=3);
    ax.legend(loc='best',frameon=False)
    ax.axes.xaxis.set_ticklabels([])
    ax.axes.yaxis.set_ticklabels([])
    plt.title(r'$E_{in}$', fontsize=16)
```

```python
np.random.seed(12345)
e_arr, w_arr, X, y, w_arr2 = run_simulation(N=300,eta=0.15)
visualise_SGD_contour2(e_arr,w_arr,X,y,my_err_fn)
```


# Pyomo - Getting started

Pyomo installation: see http://www.pyomo.org/installation

```
pip install pyomo
```

```python
from pyomo.environ import *
```

## Ice cream example

This example is taken from the following book: *Pyomo - Optimization Modeling in Python* by W. E. Hart & al. , Second Edition, Springer (p.19)

$$
\begin{align}
    \max_{x}     & \quad \sum_{i \in \mathcal{A}} h_i (1 - u/d_i^2) x_i \\
    \text{s.t.}  & \quad \sum_{i \in \mathcal{A}} c_i x_i \leq b \\
                 & \quad 0 \leq x_i \leq u_i, \quad i \in \mathcal{A}
\end{align}
$$

### Concrete Model

`ConcHLinScript.py` in https://github.com/Pyomo/pyomo/tree/master/examples/doc/pyomobook/optimization-ch

```python
instance = ConcreteModel(name="Linear (H)")

A = ['I_C_Scoops', 'Peanuts']
h = {'I_C_Scoops': 1,    'Peanuts': 0.1}
d = {'I_C_Scoops': 5,    'Peanuts': 27}
c = {'I_C_Scoops': 3.14, 'Peanuts': 0.2718}
b = 12
u = {'I_C_Scoops': 100,  'Peanuts': 40.6}

def x_bounds(m, i):
    return (0,u[i])

instance.x = Var(A, bounds=x_bounds)

def obj_rule(instance):
    return sum(h[i]*(1 - u[i]/d[i]**2) * instance.x[i] for i in A)

instance.z = Objective(rule=obj_rule, sense=maximize)

instance.budgetconstr = Constraint(expr = sum(c[i] * instance.x[i] for i in A) <= b)

# @tail:
opt = SolverFactory('glpk')

results = opt.solve(instance)    # solves and updates instance

instance.display()
# @:tail
```

### Abstract Model

`AbstHLinScript.py` in https://github.com/Pyomo/pyomo/tree/master/examples/doc/pyomobook/optimization-ch

```python
DATA_STR = """# Pyomo data file for AbstractH.py
set A := I_C_Scoops Peanuts ;
param h := I_C_Scoops 1 Peanuts 0.1 ;
param d :=
  I_C_Scoops 5
  Peanuts 27 ;
param c := I_C_Scoops 3.14 Peanuts 0.2718 ;
param b := 12 ;
param u := I_C_Scoops 100 Peanuts 40.6 ;
"""

with open("AbstractH.dat", "w") as fd:
    print(DATA_STR, file=fd)
```

```python
!cat AbstractH.dat
```

```python
model = AbstractModel(name="Simple Linear (H)")

model.A = Set()

model.h = Param(model.A)
model.d = Param(model.A)
model.c = Param(model.A)
model.b = Param()
model.u = Param(model.A)

def xbounds_rule(model, i):
    return (0, model.u[i])
model.x = Var(model.A, bounds=xbounds_rule)

def obj_rule(model):
    return sum(model.h[i] * (1 - model.u[i]/model.d[i]**2) * model.x[i] for i in model.A)

model.z = Objective(rule=obj_rule, sense=maximize)

def budget_rule(model):
    return summation(model.c, model.x) <= model.b

model.budgetconstr = Constraint(rule=budget_rule)

# @tail:
opt = SolverFactory('glpk')

instance = model.create_instance("AbstractH.dat")
results = opt.solve(instance)     # solves and updates instance

instance.display()
# @:tail
```