## 1. Basic Design of the Model
1. Output of the model: 0 or 1 (Binary Classification)
2. Hypothesis to be tested: $Z = W \cdot X + b$
3. Activation Function: $\frac{1}{1 + e^{-x}} $ (Signmoid Function)

## 2. Import Packages

1. numpy
2. matplotlib
3. seaborn

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

# Next Libraries are unimportant, they just make everyhting look better

import matplotlib.style as style
import seaborn as sns

style.use('seaborn-poster') #sets the size of the charts
style.use('ggplot')
```

## 3. Loading the dataset

```python
dataset = np.load('dataset.npz', encoding='ASCII')
## Get the numpy arrays from the dictionary
X_train = dataset['X_train']
Y_train = dataset['Y_train']
X_test  = dataset['X_test']
Y_test  = dataset['Y_test']
```

```python
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)
```

    (784, 8000)
    (1, 8000)
    (784, 2000)
    (1, 2000)

```python
idx = np.random.randint(X_train.shape[1])

plt.imshow(X_train[:, idx].reshape(28, 28))

label = "cat" if Y_train[:, idx][0] else "bat"
print(f"Label: {label}")
```

## 4. Normalizing the data

Normalizing the data with the following equation:

$$ X_{norm} = \frac {X - X_{min}}{X_{max} - X_{min}} $$

For this pixel data, $X_{max} = 255$ and $X_{min} = 0$

> After running the next cell, go back and view the raw array again

```python
## Normalizing the training and testing data
X_min = 0
X_max = 255
X_train = X_train / X_max
X_test = X_test / X_max
```

## 5. Helper functions for the Model:

### Sigmoid Function and Initialize Parameters function

```python
def sigmoid(z):
    """
    Computes the element sigmoid of scalar or numpy array(element wise)

    Arguments:
        z: Scalar or numpy array

    Returns:
        s: Sigmoid of z (element wise in case of Numpy Array)
    """
    s = 1/(1+np.exp(-z))

    return s
```

```python
def initialize_parameters(n_x):
    """
    Initializes w to a zero vector, and b to a 0 with datatype float

    Arguments:
        n_x: Number of features in each sample of X

    Returns:
        w: Initialized Numpy array of shape (1, n_x) (Weight)
        b: Initialized Scalar (bias)
    """

    w = np.full((1,X_train.shape[0]),0)
    b = 0

    return w, b

```

Here is a summary of the equations for Forward Propagation and Backward Propagation we have used so far:

For m training examples $ X_{train} $ and $ Y_{train} $:

### 5.1 Forward Propagation

$$ Z^{(i)} = w \cdot X_{train}^{(i)} + b $$

$$ \hat Y^{(i)} = A^{(i)} = \sigma(Z^{(i)}) = sigmoid(Z^{(i)}) $$

$$ \mathcal{L}(\hat Y^{(i)}, Y_{train}^{(i)}) = \mathcal{L}(A^{(i)}, Y_{train}^{(i)}) = -[Y_{train}^{(i)} \log(A^{(i)}) + (1 - Y_{train}^{(i)}) \log(1 - A^{(i)})] $$

$$ J = \frac{1}{m} \sum_1^m \mathcal{L} (A^{(i)}, Y_{train}^{(i)}) $$

### 5.2 Backward Propagation - Batch Gradient Descent

$$ \frac{\partial J}{\partial w} = \frac{1}{m} (A - Y) \cdot X^T $$

$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_1^m (A - Y) $$

> Note: $ \frac{\partial J}{\partial w} $ is represented as dw, and $ \frac{\partial J}{\partial b}$ is represented as db

```python
def compute_cost(A, Y, m):
    """
    Calculates the Cost using the Cross Entropy Loss

    Arguments:
        A: Computer Probabilities, numpy array
        Y: Known Labels, numpy array

    Returns:
        cost: The computed Cost
    """
    cost = np.sum(((- np.log(A))*Y + (-np.log(1-A))*(1-Y)))/m

    return np.squeeze(cost)
```

```python
def propagate(w, b, X, Y):
    """
    Performs forward and backward propagation for the Logistic Regression model

    Arguments:
        w: The Weight Matrix of dimension (1, n_x)
        b: Bias
        X: Input Matrix, with shape (n_x, m)
        Y: Label Matrix of shape (1, m)

    Returns:
        dw: Gradient of the weight matrix
        db: Gradient of the bias
        cost: Cost computed on Calculated Probability, and output Label
    """
    m = X.shape[1]

    A = sigmoid((w @ X)+b)
    cost = compute_cost(A, Y, m)
    dw = (np.dot(X,(A-Y).T).T)/m
    db = (np.sum(A-Y))/m

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    return dw, db, cost

```

### 5.3 Optimization

For a parameter $ \theta $, the gradient descent update rule is given by:
$$ \theta := \theta - \alpha \frac{\partial J}{\partial \theta} $$

where $\alpha$ is the learning rate

```python
def fit(w, b, X, Y, num_iterations, learning_rate, print_freq=100):
    """
    Given the parameters of the model, fits the model corresponding to the given Input Matrix aand output labels, by performing batch gradient descent for given number of iterations.

    Arguments:
        w: The Weight Matrix of dimension (1, n_x)
        b: Bias
        X: Input Matrix, with shape (n_x, m)
        Y: Label Matrix of shape (1, m)
        num_iterations: The number of iteratios of bgd to be performed
        print_freq: Frequency of recording the cost
    Returns:
        w: Optimized weight matrix
        b: optimized bias
        costs: print the cost at frequency given by print_freq, no prints if freq is 0
    """

    costs = []
    for i in range(num_iterations):
        ## 1. Calculate Gradients and cost
        dw, db, cost = propagate(w, b, X, Y)

        costs.append(cost)

        if print_freq and i % print_freq == 0:
            print(f"Cost after iteration {i}: {cost}")

        ## 2. Update parameters
        w = w - (learning_rate*dw)
        b = b - (learning_rate*db)

    return w, b, costs

```

### 5.4 Prediction
Using the following equation to determine the class that a given sample belongs to:

$$
\begin{equation}
    Y_{prediction}^{(i)} =
    \begin{cases}
        1 \text{, if } \hat Y^{(i)} \ge 0.5\\
        0 \text{, if } \hat Y^{(i)} \lt 0.5\\
    \end{cases}
\end{equation}
$$

```python
def predict(w, b, X):
    """
    Predicts the class which the given feature vector belongs to given Weights and Bias of the model

    Arguments:
        w: The Weight Matrix of dimension (1, n_x)
        b: Bias
        X: Input Matrix, with X.shape[0] = n_X
    Returns:
        Y_prediction: Predicted labels
    """

    m = X.shape[1]
    Y_prediction = np.full((1,m),0)
    A = sigmoid((w @ X) + b)
    Y_prediction = (A >= 0.5) * 1.0

    return Y_prediction
```

## 6. Building the Model

Now we have assembled all the individual pieces required to create the Logistic Regression model.
Next function is creating the model and calculating its train and test accuracy.

```python
def model(X_train, Y_train, X_test, Y_test, num_iterations, learning_rate, print_freq):
    """
    Creates a model and fit it to the train and test data. Use this model to compute the train and test accuracy after 2500 iterations

    Arguments:
        X_train: Training Data X
        Y_train: Training Data Y
        X_test: Testing Data X
        Y_test: Testing data Y
        num_iterations: Number of iterations of bgd to perform
        learning_rate: Learning Rate of the model
        print_freq: Frequency of recording the cost
    Returns:
        -None-
    """

    w, b = initialize_parameters(X_train.shape[0])
    w, b, costs = fit(w, b, X_train, Y_train, num_iterations, learning_rate, print_freq)

    Y_prediction_train = predict(w, b, X_train)
    Y_prediction_test = predict(w, b, X_test)

    costs = np.squeeze(costs)

    print(f"train accuracy: {100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100} %")
    print(f"test accuracy: {100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100} %")

    plt.plot(costs)

    plt.ylabel('cost')
    plt.xlabel('iterations (per hundreds)')
    plt.title(f"Learning rate = {learning_rate}")
    plt.show()

```

```python
model(X_train, Y_train, X_test, Y_test, num_iterations=2000, learning_rate=0.1, print_freq=100)
```


# ODE

We will solve the following linear Cauchy model

\begin{align}
y^{\prime}(t) &= \lambda y(t)\\
y(0) & = 1
\end{align}

whose exact solution is

$$
y(t) = e^{\lambda t}
$$

```python
%matplotlib inline
from numpy import *
from matplotlib.pyplot import *
import scipy.linalg
import numpy.linalg
```

```python
l = -5.   # lamba (see formula above)
t0 = 0.   # initial time
tf = 10.  # final time
y0 = 1.   # boundary condition

s = linspace(t0,tf,5000)
exact = lambda x: exp(l*x)  # exact solution
```

### Forward Euler

$$
\frac{y_{n}-y_{n-1}}{h} = f(y_{n-1}, t_{n-1})
$$

```python
def fe(l,y0,t0,tf,h):
    timesteps = arange(t0,tf+1e-10, h)
    sol = zeros_like(timesteps)
    sol[0] = y0
    for i in range(1,len(sol)):
        sol[i] = sol[i-1]*(1+l*h)

    return sol, timesteps

y, t = fe(l,y0,t0,tf,0.1)
_ = plot(t,y, 'o-')
_ = plot(s,exact(s))

error = numpy.linalg.norm(exact(t) - y, 2)
print(error)
```

### Backward Euler

$$
\frac{y_{n}-y_{n-1}}{h} = f(y_{n}, t_{n})
$$

```python
def be(l,y0,t0,tf,h):
    timesteps = arange(t0, tf+1e-10, h)
    sol = zeros_like(timesteps)
    sol[0] = y0
    for i in range(1, len(sol)):
        sol[i] = sol[i-1]/(1-l*h)

    return sol, timesteps

y,t = be(l, y0, t0, tf, 0.1)
_ = plot(t, y, 'o-')
_ = plot(s, exact(s))

error = numpy.linalg.norm(exact(t)-y, infty)
print(error)
```

### $\theta$-method

$$
\frac{y_{n}-y_{n-1}}{h} = \theta\, f(y_{n}, t_{n}) + (1-\theta)\,f(y_{n-1}, t_{n-1})
$$

```python
def tm(theta,l,y0,t0,tf,h):
    timesteps = arange(t0, tf+1e-10, h)
    sol = zeros_like(timesteps)
    sol[0] = y0
    for i in range(1, len(sol)):
        sol[i] = theta*sol[i-1]/(1-l*h) + (1. - theta)*sol[i-1]*(1+l*h)

    return sol, timesteps

y,t = tm(0.5, l, y0, t0, tf, 0.1)
_ = plot(t, y, 'o-')
_ = plot(s, exact(s))

error = numpy.linalg.norm(exact(t) - y, infty)
print(error)
```

### Simple adaptive time stepper

For each time step:
- Compute solution with CN
- Compute solution with BE
- Check the difference
- If the difference satisfy a given tolerance:
    - keep the solution of higher order
    - double the step size
    - go to the next step
- Else:
    - half the step size and repeat the time step

```python
def adaptive(l,y0,t0,tf,h0, hmax=0.9,tol=1e-3):
    sol = []
    sol.append(y0)
    t = []
    t.append(t0)
    h = h0
    while t[-1] < tf:
        #print 'current t =', t[-1], '            h=', h
        current_sol = sol[-1]
        current_t = t[-1]
        sol_cn, _ = tm(0.5,l,current_sol,current_t, current_t + h, h)
        sol_be, _ = tm(1.,l,current_sol,current_t, current_t + h, h)

        if (abs(sol_cn[-1] - sol_be[-1]) < tol): #accept
            sol.append(sol_cn[-1])
            t.append(current_t+h)
            h *= 2.
            if h > hmax:
                h=hmax
        else:
            h /= 2.

    return sol, t

y,t = adaptive(l,y0,t0,tf,0.9)
_ = plot(t,y, 'o-')
_ = plot(s,exact(array(s)))

error = numpy.linalg.norm(exact(array(t)) - y, infty)
print(error, len(y))
```


# 逆行列を求める方法

- `np.linalg` に `inv` という関数がある

```python
import numpy as np
```

```python
a = np.array([[3, 1, 1], [1, 2, 1], [0, -1, 1]])
```

```python
np.linalg.inv(a)
```

    array([[ 0.42857143, -0.28571429, -0.14285714],
           [-0.14285714,  0.42857143, -0.28571429],
           [-0.14285714,  0.42857143,  0.71428571]])

# ひとつの連立方程式を解く方法

下記の連立方程式を解くには逆行列を求めるよりも `solve` 関数を使うほうが良い。(高速かつ数値安定的なアルゴリズムを背後で利用しているため)

\begin{equation}
\begin{pmatrix}
3& 1& 1\\
1& 2& 1\\
0& -1& 1
\end{pmatrix}
\begin{pmatrix}
x \\ y\\ z
\end{pmatrix}
=
\begin{pmatrix}
1 \\ 2 \\ 3
\end{pmatrix}
\end{equation}

```python
b = np.array([[3, 1, 1], [1, 2, 1], [0, -1, 1]])
```

```python
c = np.array([1, 2, 3])
```

```python
np.linalg.solve(b, c)
```

    array([-0.57142857, -0.14285714,  2.85714286])

# 同じ係数行列からなく複数の連立方程式を解く方法

\begin{equation}
Ax=b_1, Ax=b_2, \dots, Ax=b_m
\end{equation}
となる連立方程式があったときは、 $A^{-1}$ を計算することで、

\begin{equation}
A^{-1}b_1, A^{-1}b_2, \dots, A^{-1}b_m
\end{equation}

と解が計算できる。しかし、もっと良い方法がある。

## LU分解

$A=PLU$ の形に分解することで連立方程式を高速かつ数値安定的に解くことができる。

ここで $L$ は下三角行列で対角成分が $1$ となるもの、 $U$ は上三角行列、 $P$ は各行に $1$ となる成分がただひとつだけある行列でそのほかの成分は $0$(置換行列)

\begin{equation}
PLUx = b
\end{equation}

という連立方程式は次の3つの方程式を逐次的に解くことで解 $x$ を求めることができる。

\begin{align}
Uz &= b \\
Ly &= z \\
Px &= y
\end{align}

```python
# scipy を利用
from scipy import linalg
```

```python
a = np.array([[3, 1, 1], [1, 2, 1], [0, -1, 1]])
b = np.array([1, 2, 3])
```

```python
lu, p = linalg.lu_factor(a)
linalg.lu_solve((lu, p), b)
```

    array([-0.57142857, -0.14285714,  2.85714286])

```python

```


# Mixed Integer Linear Programming (MILP)

## Introduction

* Some variables are restricted to be integers
* NP-complete
* Applications
    * Production planning
    * Scheduling
    * Many more...

## The standard form

\begin{align}
\text{maximize}\  & \mathbf{c}^T\mathbf{x} + \mathbf{k}^T\mathbf{y} \\
\text{subject to } & \\
& A\mathbf{x} &&\leq \mathbf{b} \\
& D\mathbf{y} &&\leq \mathbf{e} \\
& \mathbf{x},\mathbf{y} &&\geq 0 \\
& \mathbf{x} \in \mathbb{Z}^n
\end{align}
where $A, D \in \mathbb{R}^{m\times n}$ are matrices, $\mathbf{b}, \mathbf{e}\in\mathbb{R}^{m}$ are constants, $\mathbf{c}, \mathbf{k} \in \mathbb{R}^{n}$ objective function coefficients, and $\mathbf{x}, \mathbf{y} \in\mathbb{R}^{n}$ are the decision variables.

## CPLEX basics: Mixed Integer Programming Model
## Mathematical Model
\begin{align}
\text{maximize}\  & 2x + y + 3z \\
\text{subject to } & \\
& x+2y+z &&\leq 4 \\
& 2z + y &&\leq 5 \\
& x + y &&\geq 1 \\
& x &&\in \{0,1\} \\
& y, z \geq 0 \\
& z \in \mathbb{Z}
\end{align}

# Code in Python using docplex
## Step 1: Importing Model from docplex package

```python
from docplex.mp.model import Model
```

## Step 2: Create an optimization model

```python
milp_model = Model(name = "MILP")
```

## Step 3: Add decision variables

```python
x = milp_model.binary_var(name = 'x')
y = milp_model.continuous_var(name = 'y', lb = 0)
z = milp_model.integer_var(name="z", lb=0)
```

## Step 4: Add the constraints

```python
# Add constraint: x + 2 y + z <= 4
c1 = milp_model.add_constraint(x + 2 * y + z <= 4, ctname = "c1")

# Add constraint: 2 z + y <= 5 \\
c2 = milp_model.add_constraint(2 * z + y <= 5, ctname = "c2")

# Add constraint x + y >= 1
c3 = milp_model.add_constraint(x + y >= 1, ctname = "c3")
```

## Step 5: Define the objective function

```python
obj_fn = 2 * x + y + 3 * z
milp_model.set_objective('max', obj_fn)

milp_model.print_information()
```

## Step 6: Solve the model

```python
milp_model.solve()
```

## Step 7: Output the result

```python
milp_model.print_solution()
```

```python

```


# Lab 3
## Introduction
In this lab we will analyse population dynamics under the logisitic model with managed harvesting.

First import the modules we need.

```python
from plotly.figure_factory import create_quiver
from plotly import graph_objs as go
from numpy import meshgrid, arange, sqrt, linspace
from scipy.integrate import odeint
```

## Harvesting of fish
A population of fish in a lake, left to its own devices, is modelled by the logistic differential equation
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y),
\end{align}
where the population $y$ is in units of thousands of fish and time $t$ is measured in years.

First define a function for $\mathrm{d}y/\mathrm{d}x$ in terms of $y$ and $x$.

```python
def diff_eq(y, x):
    return 4 * y * (1 - y)
```

Next define a function that creates a Plotly Figure object that contains a slope field and, optionally, a few solutions to initial value problems.

It automates a few things we did in the last lab.

- `diff_eq` is the differential equation to be plotted
- `x` and `y` should be outputs from `meshgrid`.
- `args` is any additional arguments to `diff_eq` (we will use that below).
- `initial_values` is a list (or array) of starting $y$ values from which approximate solutions will start. The corresponding $x$ value is the minimum element of `x`.

Note that the numerical solutions will plotted for the whole range of $x$ values in `x`, so if they blow up you will probably get a warning and less-than-useful plot.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.8*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=True, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
```

The slope field below should hopefully give you some idea for the fish population dynamics.

Note that we use `arange` rather than `linspace` this week so that we can carefully control the increments between our grid points. `arange(0, 1.1, 0.25)` returns an array that starts with 0 and increments by 0.25 until it exceeds 1.1.

The plot also contains the solution curves for
$y(0) = 1$ and $y(0) = 0.4$. Edit the cell to also include the solution curve for $y(0)=1.4$.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, initial_values=(0.4, 1))
fig.show('png')
```

### Equilibrium solutions
Looking back to our differential equation, $\mathrm{d}y/\mathrm{d}t = 0$ when $y(t) = 0$ or $y(t) = 1$. Looking at the slope field, we see that the equilibrium solution $y(t) = 1$ is stable (this is the carrying capacity here, corresponding to 1000 fish), whereas the equilibrium solution $y(t) = 0$ is unstable. Any non-zero initial population will eventually stabilise at 1000 fish.

### What will happen if harvesting is now commenced at a steady rate?
For the simplest harvesting model, assume that $H$ units (thousands) of fish are taken
continuously (smoothly) over the year, rather than at one instant each year.
Note that the units of $H$ are the same as those of $\mathrm{d}y/\mathrm{d}t$, thousands of fish per year, so we simply subtract $H$ from the RHS of our existing equation to give the DE with harvesting as
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y) - H.
\end{align}
Again, the (constant) equilibrium solutions are found by setting $\mathrm{d}y/\mathrm{d}t = 0$, giving from the quadratic formula (check this),
\begin{align}
y(t) = \frac{4\pm\sqrt{16-16H}}{8} = \frac{1\pm\sqrt{1-H}}{2}.
\end{align}
What happens after harvesting starts will depend on the equilibrium solutions, their
stability and the initial number of fish $y(0)$.

Start by redefining `diff_eq` to include the `H` parameter. Note that defining `diff_eq` again overides our original definition.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
```

Now set $H = 0.6$ and plot the slope field. This is done by setting `args=(0.6,)` when we call `create_slope_field`. This is exactly how you would pass additional arguments like this one to `odeint` if you were calling it directly.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.6,))
fig.show('png')
```

From the solutions to the quadratic equation above, the equilibrium solutions of the DE are found to be $y(t) \approx 0.184$ and $y(t) \approx 0.816$. The previous equilibrium solution with no harvesting at $y(t) = 0$ has moved up to $y(t) \approx 0.184$, while the previous equilibrium solution with no harvesting at $y(t) = 1$ has moved down to $y(t) \approx 0.816$.

From the slope field, we see that the equilibrium solution $y(t) \approx 0.184$ is unstable, whereas the equilibrium solution $y(t) \approx 0.816$ is stable. If the population ever falls below about 0.184, or 184 fish, it will then drop to 0. This is a new feature, introduced by harvesting.

In the cell below, use `create_slope_field` to experiment by plotting the solutions to the initial value problems $y(0)=0.183$ and $y(0)=0.25$. Extend the $x$ range of your slope field until the top line is close to equlibrium. Note that if you extend it too far you will break `odeint` (why?). You may also like to increase the increments in `arange` to make the plot clearer.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
x, y = meshgrid(arange(0, 2.0, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.25))
fig.show('png')
```

## Exercises

In this lab you will experiment with the population dynamics given by the logistic equation with harvesting that we started analysing in the lab.

This week the questions will be a combination of plots and written answers.

1. Assume that the harvest is 600 fish per year. **On the same figure,**
  a. plot the slope field,
  b. plot the equilibrium solutions that we found in above, and
  c. plot the solution curves for $y(0)=1$, $y(0)=0.3$, and $y(0)=0.15$.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
x, y = meshgrid(arange(0, 0.9, 0.05), arange(-0.05, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(1, 0.3, 0.15))
fig.show()
```

1.  d. In the cell below, describe the behaviour of the fish population for each of these five initial numbers of fish.

When y(0)=0.15, the curve tends to diverge.

2. a. i. Assume that $H=0.8$. Plot the slope field and five solutions, one for each equilibrium solution and one for each region between, above, or below them. You can use the equation from the lab to calculate the equilibrium solutions.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.8*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=400,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.5,), initial_values=(0.6,))
fig.show('png')
```

2. a. ii. In the cell below, describe the limiting behaviour of each line.

2. b. i. Assume that $H=1$. Plot the slope field and three solutions for the equilibrium solution and the regions above and below it.

```python
x, y = meshgrid(arange(0, 0.8, 0.1), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(1.0,), initial_values=(1, 0.3, 0.15,0.183, 0.25))
fig.show('png')
```

2. b. ii. Describe the limiting behaviour of each line.

2. c. i. Assume that  𝐻=1.2. Plot the slope field and two or three solutions.

```python
x, y = meshgrid(arange(0.0, 0.7, 0.1), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(1.2,), initial_values=(1, 0.3, 0.15,0.183, 0.25))
fig.show()
```

2. c. iii. Describe the limiting behaviour of the lines.

3. Summarize what happens to the equilibrium solutions and their stability as $H$
is increased from 0 to beyond 1. Refer to your plots to support your answers.

As H value increases, the lower the initial value of y it requires to diverge as seen on the plot.

4. What is a reasonable strategy for sustainable fishing in this case?
Don’t forget to allow qualitatively for minor catastrophes, such as disease or temporary overfishing.

THe best way to sustain fishing is to set a certain threshold on harvesting fish yearly. If we harvest too many on a yearly rate, then there's a strong likelihood that the population will plummet.


## FowardDiff.jl

[The JuliaDiff website](http://www.juliadiff.org/) describes the advantages (and some of the implementation) of autodifferentiation. The basic idea is as follows. For normal derivative approximations, you do calculations like:

```julia
Df = (f(x+ϵ) - f(x)) / ϵ
```

with ϵ sufficiently small. However, when ϵ is small [catestrophic cancellation](https://en.wikipedia.org/wiki/Loss_of_significance) occurs which causes numerical error. This puts numerical differentiation methods in a bind: a small epsilon is required to not have derivative error (since it's defined as the limit), but small epsilon results in floating point errors due to cancelation. The result is that numerical derivative approximations are very error prone.

Symbolic differentiation isn't always possible since the length of the expressions grows exponentially. But a newer technique, called autodifferentiation, can do this. The central idea is that you can directly define the derivatives of basic expressions (like `sin`), and then use the chain rule to know how to propagate derivatives. This allows you to efficiently calculate derivatives through entire codes without using small epsilon values, meaning it doesn't result in the high numerical errors that plague numerical differentiation. The result is fast and accurate derivatives.

ForwardDiff.jl is an implementation of autodifferentiation which uses [dual numbers](https://en.wikipedia.org/wiki/Dual_number).  The idea is you have a two dimensional number, and you define operators like `+` so that way the first part is the value and the second part is the derivative. For example, `sin(Ax) = sin(Ax) + cos(A)ϵ`, and then you can use the chain rule to define how the epsilon gets passed down. The resulting derivative at the end of the calculation is then simply the coefficient on the epsilon part. Here, ϵ is part of a number type, and this is all done by multiple dispatch and defining new dispatches on basic mathematical functions for Dual numbers which also compute the derivative.

### Problem

The ForwardDiff.jl documentation can be found here: http://www.juliadiff.org/ForwardDiff.jl/stable/user/api.html

1. Use the `ForwardDiff.derivative` function to take the derivative of `x^5 + 3x^2`. Compare the result to the analytical solution by plotting both with Plots.jl, overlaying one on top of the other (hint: `plot!` can be helpful!).

2. The transformation from spherical to cartesian coordinates is given by:

$$
\begin{align}
x &= r\sin(\theta)\cos(\phi) \\
y &= r\sin(\theta)\sin(\phi) \\
z &= r\cos(\theta)
\end{align}
$$

Use ForwardDiff.jl to calculate the Jacobian of the transformation. Compare the determinant of the Jacobian against the analytical solution $r^2 \sin(\theta)$.


# ODE boundary value problems

Let's have a look at how the DifferentialEquations.jl package handles boundary value problems. Let's solve the simple harmonic equation
$$
u'' + 3 u = 0
$$
on the interval $t \in [0,2\pi]$ where $u(0)=7$ and $u(2 \pi)=0$. The exact solution is given by
$$
\begin{align}
u(t) = 7 \cos \left(\sqrt{3}t \right) - 7 \cot \left(2\sqrt{3}\pi \right) \sin \left(\sqrt{3}t \right).
\end{align}
$$

The second order equation can be rewritten as the first order system
$$
\begin{align}
\left( u \right)' =& u', \\
\left( u' \right)' =& - 3 u.
\end{align}
$$
So let's solve this two point boundary value problem numerically using Julia.

```julia
#using Pkg
#Pkg.add( "BoundaryValueDiffEq" )
```

```julia
using DifferentialEquations, BoundaryValueDiffEq, Plots
using BenchmarkTools
```

```julia
function harmonic!(du,u,p,t)
  du[1] = u[2]
  du[2] = - 3 * u[1]
end
```

    harmonic! (generic function with 1 method)

```julia
function boundary_conditions!(residual, u, p, t)
    residual[1] = u[1][1] - 7
    residual[2] = u[end][1]
end
```

    boundary_conditions! (generic function with 1 method)

```julia
tspan = (0.0,2*pi)
tstep = 0.05
initial_guess = [7.0,-1.0]
bvp = TwoPointBVProblem( harmonic!, boundary_conditions!, initial_guess, tspan )
sol = solve( bvp, MIRK4(), dt=tstep);
```

```julia
plot( sol, vars=(1), lw=2, label="Numerical solution" )

function exact( t )
     return 7*( cos(sqrt(3)*t) - cot(2*sqrt(3)*pi) * sin(sqrt(3)*t) )
end

plot!( sol.t, t->exact(t), lw=2,ls=:dash, label="Exact solution" )
```

```julia
error = map( exact, sol.t )
for i in 1:length(sol.t)
    error[i] = error[i] - sol.u[i][1]
end
plot( sol.t, error, label="Error", lw=2 )
```

Let's try a slightly more difficult example such as the Blasius equation
$$
f''' + f f'' = 0,
$$
with the conditions $f(0)=f'(0)=0$ and $f'(\infty)=1$. We can rewrite the Blasius equation as the first order system
$$
\begin{align}
\left( f \right)' =& f', \\
\left( f' \right)' =& f'', \\
\left( f'' \right)' =& -f f''.
\end{align}
$$

```julia
function Blasius!(du,u,p,t)
    du[1] = u[2]
    du[2] = u[3]
    du[3] = -u[1] * u[3]
end
```

    Blasius! (generic function with 1 method)

```julia
function Blasius_bcs!(residual, u, p, t)
    residual[1] = u[1][1]
    residual[2] = u[1][2]
    residual[3] = u[end][2] - 1
end
```

    Blasius_bcs! (generic function with 1 method)

```julia
tinf = 20
N = 200
tspan = (0.0,20.0)
tstep = (tspan[2] - tspan[1])/N
# Make an initial guess
t = tspan[1]:tstep:tspan[2]
function f_guess( t )
    return t * ( 1.0 - exp( -t ) )
end
function f_d_guess( t )
    return ( 1.0 - exp( -t ) ) + t * exp( -t )
end
function f_dd_guess( t )
    return 2 * exp( -t ) - t * exp( -t )
end
f = map( f_guess, t )
f_d = map( f_d_guess, t )
f_dd = map( f_dd_guess, t )
#initial_guess = [f,f_d,f_dd]
initial_guess = [1.0,1.0,0.0]
bvp_Blasius = TwoPointBVProblem( Blasius!, Blasius_bcs!, initial_guess, tspan )
@btime sol_Blasius = solve( bvp_Blasius, MIRK4(), dt=tstep );
```

      2.528 s (40327661 allocations: 3.56 GiB)

```julia
plot( sol_Blasius, xlims=(0,10), ylims=(0,2), label=["f" "f'" "f''"] )
```

Adding the initial guesses doesn't seem to help much with speed or the memory allocation. How can the performance be improved?

```julia

```


# Lucas_task 19 - Motor Control
### Introduction to modeling and simulation of human movement
https://github.com/BMClab/bmc/blob/master/courses/ModSim2018.md

### 1) Find the extrema in the function f(x)=x3−7.5x2+18x−10 analytically and determine if they are minimum or maximum.

Import Python libraries

```python
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

Finding the roots

```python
x = sym.symbols('x')
F = x**3 - 7.5*x**2 + 18*x - 10
Fdiff = sym.expand(sym.diff(F, x))
roots = sym.solve(Fdiff, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.0, \quad 3.0\right ]$$

Finding the minimum and maximum

```python
x1 = 2
x2 = 3
F1 = x1**3 - 7.5*x1**2 + 18*x1 - 10
F2 = x2**3 - 7.5*x2**2 + 18*x2 - 10

print("F(x1): ", F1)
print("\nF(x2): ", F2)

```

    F(x1):  4.0

    F(x2):  3.5

```python
print("Minimim:", x2,"\nMaximum:", x1)

```

    Minimim: 3
    Maximum: 2

### 2) Find the minimum in the f(x)=x3−7.5x2+18x−10 using the gradient descent algorithm.

```python
cur_x = 6               # The algorithm starts at x=6
gamma = 0.01            # step size multiplier
precision = 0.00001
step_size = 1           # initial step size
max_iters = 10000       # maximum number of iterations
iters = 0               # iteration counter

f  = lambda x: x**3 - 7.5*x**2 + 18*x - 10  # lambda function for f(x)
df = lambda x: 3*x**2 - 15*x  + 18   # lambda function for the gradient of f(x)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x -= gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local minimum at {} with function value {}.'.format(9/4, f(9/4)))
print('Local minimum by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

    True local minimum at 2.25 with function value 3.921875.
    Local minimum by gradient descent at 3.000323195755751 with function value 3.5000001567170003.

```python

```


```python
import numpy as np
```

```python
# matrix multiplication
a = np.array([[0,6],[2,8],[4,10]])
b = np.array([[1,3,5],[7,9,11]])
print(a)
print(b)
print(np.dot(a,b))
```

    [[ 0  6]
     [ 2  8]
     [ 4 10]]
    [[ 1  3  5]
     [ 7  9 11]]
    [[ 42  54  66]
     [ 58  78  98]
     [ 74 102 130]]

```python
print(np.matmul(a,b))
```

    [[ 42  54  66]
     [ 58  78  98]
     [ 74 102 130]]

```python
# sigmoid activation function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

```python
x1=1
x2=1
score = 4*x1 + 5*x2 - 9
```

```python
sigmoid(score)
```

    0.5

```python
w1=3
w2=5
b=-2.2

sigmoid(w1*0.4 + w2*0.6 + b)
```

    0.8807970779778823

```python
# eigenvector and eigenvalues
A = np.array([[3, 0],[-9, 6]])
x = np.array([1, 3])

np.dot(A, x)
```

    array([3, 9])

```python
# picture size calculation
#     W  = input volume
#     K  = Kernel size
#     P =  padding
#     S = stride

np.int_(((130-3+0)/1)+1)
```

    128

$[(W−K+2*P)/S]+1$

```python
# calculating padding that has kernel_size of 7 for CNN that is the same x-y size as an input array
# see above equation; using P = 3
x = (-7 + 2*3/1) + 1 # results in zero; W - 0 = W
print(np.int_(x))
```

    0

```python
from sympy import Symbol

Symbol('w') + x == Symbol('w') # W = W; CNN has same x-y size as input array
```

    True

```python
# nlp: Subsampling equation
# text with 1 million words in it
# word "learn" appears 700 times in this text
# threshold is 0.0001
```

$p = 1 - \sqrt{\frac{t}{f(x_[i])}}$

```python
# -> calculate probability that we will discard the word "learn"
f = 700
w_i = 1e6
t = 1e-4
```

```python
p = 1 - np.sqrt(t/(f/w_i))
print(p)
```

    0.6220355269907727

```python
# For an input [23, 702, 89, 15, 99] and R=2, what will the returned context be if we pass in the token at idx=1 (702)?
def get_target(words, idx, R):
    start = idx - R if (idx - R) > 0 else 0
    stop = idx + R
    target_words = words[start:idx] + words[idx+1:stop+1]

    return list(target_words)
```

```python
print(get_target(words=[23, 702, 89, 15, 99], idx=1, R=2))
```

    [23, 89, 15]


# sympy

Since we want to work interactively with `sympy`, we will import the complete module. Note that this polutes the namespace, and is not recommended in general.

```python
from sympy import *
```

Enable pretty printing in this notebook.

```python
init_printing()
```

## Expression manipulation

Define a number of symbols to work with, as well as an example expression.

```python
x, y, a, b, c = symbols('x y a b c')
```

```python
expr = (a*x**2 - b*y**2 + 5)/(c*x + y)
```

Check the expression's type.

```python
expr.func
```

    sympy.core.mul.Mul

Although the expression was defined as a divisino, it is represented as a multiplicatino by `sympy`.  The `args` attribute of an expressions stores the operands of the top-level operator.

```python
expr.args
```

Although the first factor appears to be a division, it is in fact a power.  The denominator of this expression would be given by:

```python
expr.args[0].func
```

    sympy.core.power.Pow

```python
expr.args[0].args[0]
```

The expression $\frac{1}{a x + b}$ can alternatively be defined as follows, which highlights the internal representation of expressions.

```python
expr = Pow(Add(Mul(a, x), b), -1)
```

```python
pprint(expr)
```

       1
    ───────
    a⋅x + b

```python
expr.args
```

```python
expr.args[0].args[0]
```

This may be a bit surprising when you look at the mathematical representation of the expression, but the order of the terms is different from its rendering on the screen.

```python
expr.args[0].args
```

Since the addition operation is commutative, this makes no difference mathematically.

```python
expr.args[0].args[1].args
```

```python
expr = x**2 + 2*a*x + y**2
```

```python
expr2 = expr.subs(y, a)
expr2
```

Most expression manipulation algorithms can be called as functions, or as methods on expressions.

```python
factor(expr2)
```

```python
expr2.factor()
```

```python
x, y = symbols('x y', positive=True)
```

```python
(log(x) + log(y)).simplify()
```

## Calculus

### Series expansion

```python
x, a = symbols('x a')
```

```python
expr = sin(a*x)/x
```

```python
expr2 = series(expr, x, 0, n=7)
```

```python
expr2
```

A term of a specific order in a given variable can be selected easily.

```python
expr2.taylor_term(2, x)
```

When the order is unimportant, or when the expression should be used to define a function, the order term can be removed.

```python
expr2.removeO()
```

Adding two series deals with the order correctly.

```python
s1 = series(sin(x), x, 0, n=7)
```

```python
s2 = series(cos(x), x, 0, n=4)
```

```python
s1 + s2
```

### Derivatives and integrals

```python
expr = a*x**2 + b*x + c
```

```python
expr.diff(x)
```

```python
expr.integrate(x)
```


# SymPy
The SymPy package is useful for symbolic algebra, much like the commercial software Mathematica.

We won't make much use of SymPy during the boot camp, but it is definitely useful to know about
for mathematics courses.

```python
import sympy as sp
sp.init_printing()
```

# Symbols and Expressions
We'll define `x` and `y` to be `sympy` symbols, then do some symbolic algebra.

```python
x, y = sp.symbols("x y")
```

```python
expression = (x+y)**4
```

```python
expression
```

```python
sp.expand(expression)
```

```python
expression = 8*x**2 + 26*x*y + 15*y**2
```

```python
expression
```

```python
sp.factor(expression)
```

```python
expression - 20 *x*y - 14*y**2
```

```python
sp.factor(expression - 20*x*y - 14*y**2)
```

# Lambdify: Making python functions from sympy expressions

```python
expression
```

```python
f = sp.lambdify((x,y), expression, 'numpy')
```

```python
f(3,4)
```

```python
8 * 3**2 + 26 * 3 * 4 + 15 * 4**2
```

# Calculus
You can use `sympy` to perform symbolic integration or differentiation.

```python
expression = 5*x**2 * sp.sin(3*x**3)
```

```python
expression
```

```python
expression.diff(x)
```

```python
expression = sp.cos(x)
```

```python
expression.integrate(x)
```

```python
expression.integrate((x, 0, sp.pi / 2))
```

You can also create unevalated integrals or derivatives. These can later be evaluated with their `doit` methods.

```python
deriv = sp.Derivative(expression)
```

```python
deriv
```

```python
deriv.doit()
```

```python
inte = sp.Integral(expression, (x, 0, sp.pi / 2))
```

```python
inte
```

```python
inte.doit()
```

```python

```


# Activity: Logistic Functions

```python
from cyllene import *
from sympy import solve,log
from f_special import logistic_function, logistic_plot_L, second_derivative
```

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
%matplotlib inline

from ipywidgets import interact, interactive
from IPython.display import Image

```

## The definition of a logistic function

A *logistic function* is a function of the form

$f(x) = \dfrac{L}{1+C \, e^{-kx}},$

where $L, C, k$ are *parameters*.

Logistic functions *initially* exhibit growth that is similar to *exponential*, but after a while the growth begins to slow down as a saturation property takes hold of the growth process.

```python
f = function('1/(1+10*exp(-x))')
```

```python
graph(f(x))
```

### The parameter $L$

Let us take a look at the paramater $L$. Use the slider below to change the value of $L$.
What effect does this have on the graph of a logistic function?

```python
interactive_plot = interactive(logistic_plot_L, L=(1, 8, 0.5))
output = interactive_plot.children[-1]
output.layout.height = '300px'
interactive_plot
```

    interactive(children=(FloatSlider(value=4.0, description='L', max=8.0, min=1.0, step=0.5), Output(layout=Layou…

We see that $L$ determines the **height of the graph**. It is the **limiting value** of the logistic function. The values of $f$ will approach $L$ as $x$ goes to infinity. The parameter $L$ is also called the **carrying capacity**.

## The inflection point

Let us again consider a simple logistic function:

```python
f = function('8/(1+10*exp(-2*x))')
```

To find possible inflection points, we need to compute the second derivative:

```python
f2 = second_derivative(f(x))
f2
```

$\displaystyle - \frac{320 e^{- 2 x}}{\left(1 + 10 e^{- 2 x}\right)^{2}} + \frac{6400 e^{- 4 x}}{\left(1 + 10 e^{- 2 x}\right)^{3}}$

Try to find the zeros for this expression.

You can check your answer by running the code below:

```python
p = solve(f2,x)[0]
solve(f2,x)[0]
```

$\displaystyle \log{\left(\sqrt{10} \right)}$

Now plug the solution into the original function:

```python
f(p)
```

$\displaystyle 4$

Repeat this with different values for the parameters $L, C, k$. Do you observe a relation between $L$ and the location of the inflection point?

### **FACT**: The inflection point of a logistic function occurs where $f(x) = L/2$.

```python
Image(filename='logistic_function.png', width=600)
```

---

## Logistic modeling of COVID-19 data

In the following cell, we read the current Coronavirus data from CSSE at Johns Hopkins University. We extract a time series for the US, giving us for each day since January 22 the total number of infected persons.

```python
url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv"
df = pd.read_csv(url)
# country = df[df["Country/Region"] == "Italy"]
# country = df[df["Country/Region"] == "Sweden"]
# country = df[df["Country/Region"] == "Germany"]
country = df[df["Country/Region"] == "US"]

interval_len = 200

x_data = list(range(interval_len))
y_data = list(country.iloc[int(0),int(4):int(interval_len+4)])
```

We run a logistic regression for our data and the logistic model.

```python
fit = curve_fit(logistic_function, x_data, y_data)
```

```python
k = round(fit[0][0],3)
C = round(fit[0][1],3)
L = int(fit[0][2])

print("We estimated the following parameters:")
print("k = ", k)
print("C = ", C)
print("L = ", L)
```

    We estimated the following parameters:
    k =  0.025
    C =  106.736
    L =  8580027

### Question: What does the parameter $L$ reflect in this case?

#### Plotting the data

```python
plt.figure(figsize=(10,8))

plt.scatter(list(range(interval_len)),y_data,label="Real data",color="red")

# Predicted logistic curve
t = np.arange(0, interval_len, 1)
s = L/(1+C*np.exp(-k*t))
plt.plot(t,s)

plt.xlabel("Days since January 22")
plt.ylabel("Total number of infected people")

plt.show()
```

```python

```


## Ejercicios Big O()
### 1. Calcule el costo T(n) para la siguiente función.
        A. Grafique la función junto con otros O() clásicos.
        B. Analice los cambios a medida que el n aumenta.
        C. T(n) esta incluido en la familia de O(n)? Justifique utilizando definición de O().
        D. El costo obtenido es aceptable?

A continuación un ayuda memoria de $O()$ en operaciones sobre Python:

```python
from IPython.display import IFrame
IFrame('https://wiki.python.org/moin/TimeComplexity', width=900, height=350)

```

```python
def dummy_func(lst, n):
    print("primer elemento", lst[0])

    midpoint = int(n / 2)

    for val in lst[:midpoint]:
        print(val)

    for x in range(10):
        print('o_O')
```

```python
dummy_func([1, 2, 3, 4], 4)
```

    primer elemento 1
    1
    2
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O
    o_O

1.A)
Analizando las lineas de la función dada podemos ver que:
- En linea 2 tenemos $O(1)$
- En linea 4 tenemos $O(1)$
- En linea 6 y 7 tenemos $O(n/2)$
- En linea 9 y 10 tenemos $O(10)$

Por lo que nuestro $O(n/2 + 12)$

1.B) Ahora graficaremos $O(n/2 + 12)$ junto con $O(n)$, $O(n \log n)$, $O(n^2)$, $O(2^n)$

```python
from math import log
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('default')

# Set up runtime comparisons

n = np.linspace(1, 100, 1000)
labels = ['Lineal', 'Lineal Log', 'Cuadrática','Exponencial', 'Mi T(n)']
big_o = [n, n* np.log(n), n**2, 2**n, n/2 + 12]

# Plot setup
plt.figure(figsize=(12,10))
plt.ylim(0, 700)
plt.xlim(1, 100)

for i in range(len(big_o)):
    plt.plot(n,big_o[i],label = labels[i])

plt.legend(loc=0)
plt.ylabel('Tiempo relativo de ejecución')
plt.xlabel('n')
```

 1.C) Podemos observar que nuestra función con $n_0 = 24, c=1$ $ \rightarrow T(n) \in O(n)\; \forall\; n \geq 24$:

 \begin{equation}
 n/2 + 12 \leq c.n \; \forall\; n \geq 24 \\
 n/2 + 12 \leq n \; \forall\; n \geq 24 \\
 \end{equation}

1.D)

Imagen de http://bigocheatsheet.com/

### 2. Desarrolle dos funciones en Python para encontrar el número más chico en una lista. La primer función debería comparar cada número entre si ($O(n^2)$). La segunda función debería ser lineal ($O(n)$).
        A. Grafique ambas funciones.
        B. Con que implementación se quedaría?
        C. Hay correspondencia entre el benchmarking y $O()$.

```python
def min_value_n2(lst):
    min_value = lst[0] # O(1)
    for i in lst:      # O(n)
        for j in lst:  # O(n)
            if i < j:  # O(1)
                min_value = i   # O(1)
            elif j < i:         # O(1)
                min_value = j   # O(1)
    return min_value            # O(1)

min_value_n2([3, 2, 6, 1])
# O(4*n^2 + 2)
```

    1

```python
def min_value_n(lst):
    min_value = lst[0]        # O(1)
    for i in lst:     # O(n)
        if i < min_value:     # O(1)
            min_value = i     # O(1)
    return min_value          # O(1)

min_value_n([3, 2, 6, 1])
# O(2*n + 2)
```

    1

2.A)

```python
from math import log
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('default')

# Set up runtime comparisons

n = np.linspace(1, 100, 1000)
labels = ['min_value_n', 'min_value_n2']
big_o = [2*n + 2, 4*n**2 + 2]

# Plot setup
plt.figure(figsize=(12,10))
plt.ylim(0, 700)
plt.xlim(1, 100)

for i in range(len(big_o)):
    plt.plot(n,big_o[i],label = labels[i])

plt.legend(loc=0)
plt.ylabel('Tiempo relativo de ejecución')
plt.xlabel('n')
```

2.C)

```python
assert(min_value_n([3, 2, 6, 1]) == min_value_n2([3, 2, 6, 1]))
```

```python
%timeit min_value_n(range(10000))
```

    1000 loops, best of 3: 362 µs per loop

```python
%timeit min_value_n2(range(10000))
```

    1 loop, best of 3: 5.51 s per loop


## RIHAD VARIAWA, Data Scientist - Who has fun LEARNING, EXPLORING & GROWING
## Linear Equations
The equations in the previous lab included one variable, for which you solved the equation to find its value. Now let's look at equations with multiple variables. For reasons that will become apparent, equations with two variables are known as linear equations.

## Solving a Linear Equation
Consider the following equation:

\begin{equation}2y + 3 = 3x - 1 \end{equation}

This equation includes two different variables, **x** and **y**. These variables depend on one another; the value of x is determined in part by the value of y and vice-versa; so we can't solve the equation and find absolute values for both x and y. However, we *can* solve the equation for one of the variables and obtain a result that describes a relative relationship between the variables.

For example, let's solve this equation for y. First, we'll get rid of the constant on the right by adding 1 to both sides:

\begin{equation}2y + 4 = 3x \end{equation}

Then we'll use the same technique to move the constant on the left to the right to isolate the y term by subtracting 4 from both sides:

\begin{equation}2y = 3x - 4 \end{equation}

Now we can deal with the coefficient for y by dividing both sides by 2:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Our equation is now solved. We've isolated **y** and defined it as 3x-4/2

While we can't express **y** as a particular value, we can calculate it for any value of **x**. For example, if **x** has a value of 6, then **y** can be calculated as:

\begin{equation}y = \frac{3\cdot6 - 4}{2} \end{equation}

This gives the result 14/2 which can be simplified to 7.

You can view the values of **y** for a range of **x** values by applying the equation to them using the following Python code:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the solved equation to x
df['y'] = (3*df['x'] - 4) / 2

#Display the dataframe
df
```

      x
      y

      0
      -10
      -17.0

      1
      -9
      -15.5

      2
      -8
      -14.0

      3
      -7
      -12.5

      4
      -6
      -11.0

      5
      -5
      -9.5

      6
      -4
      -8.0

      7
      -3
      -6.5

      8
      -2
      -5.0

      9
      -1
      -3.5

      10
      0
      -2.0

      11
      1
      -0.5

      12
      2
      1.0

      13
      3
      2.5

      14
      4
      4.0

      15
      5
      5.5

      16
      6
      7.0

      17
      7
      8.5

      18
      8
      10.0

      19
      9
      11.5

      20
      10
      13.0

We can also plot these values to visualize the relationship between x and y as a line. For this reason, equations that describe a relative relationship between two variables are known as *linear equations*:

```python
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey", marker = "o")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.show()
```

In a linear equation, a valid solution is described by an ordered pair of x and y values. For example, valid solutions to the linear equation above include:
- (-10, -17)
- (0, -2)
- (9, 11.5)

The cool thing about linear equations is that we can plot the points for some specific ordered pair solutions to create the line, and then interpolate the x value for any y value (or vice-versa) along the line.

## Intercepts
When we use a linear equation to plot a line, we can easily see where the line intersects the X and Y axes of the plot. These points are known as *intercepts*. The *x-intercept* is where the line intersects the X (horizontal) axis, and the *y-intercept* is where the line intersects the Y (horizontal) axis.

Let's take a look at the line from our linear equation with the X and Y axis shown through the origin (0,0).

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.show()
```

The x-intercept is the point where the line crosses the X axis, and at this point, the **y** value is always 0. Similarly, the y-intercept is where the line crosses the Y axis, at which point the **x** value is 0. So to find the intercepts, we need to solve the equation for **x** when **y** is 0.

For the x-intercept, our equation looks like this:

\begin{equation}0 = \frac{3x - 4}{2} \end{equation}

Which can be reversed to make it look more familar with the x expression on the left:

\begin{equation}\frac{3x - 4}{2} = 0 \end{equation}

We can multiply both sides by 2 to get rid of the fraction:

\begin{equation}3x - 4 = 0 \end{equation}

Then we can add 4 to both sides to get rid of the constant on the left:

\begin{equation}3x = 4 \end{equation}

And finally we can divide both sides by 3 to get the value for x:

\begin{equation}x = \frac{4}{3} \end{equation}

Which simplifies to:

\begin{equation}x = 1\frac{1}{3} \end{equation}

So the x-intercept is 11/3 (approximately 1.333).

To get the y-intercept, we solve the equation for y when x is 0:

\begin{equation}y = \frac{3\cdot0 - 4}{2} \end{equation}

Since 3 x 0 is 0, this can be simplified to:

\begin{equation}y = \frac{-4}{2} \end{equation}

-4 divided by 2 is -2, so:

\begin{equation}y = -2 \end{equation}

This gives us our y-intercept, so we can plot both intercepts on the graph:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.annotate('x-intercept',(1.333, 0))
plt.annotate('y-intercept',(0,-2))
plt.show()
```

The ability to calculate the intercepts for a linear equation is useful, because you can calculate only these two points and then draw a straight line through them to create the entire line for the equation.

## Slope
It's clear from the graph that the line from our linear equation describes a slope in which values increase as we travel up and to the right along the line. It can be useful to quantify the slope in terms of how much **x** increases (or decreases) for a given change in **y**. In the notation for this, we use the greek letter Δ (*delta*) to represent change:

\begin{equation}slope = \frac{\Delta{y}}{\Delta{x}} \end{equation}

Sometimes slope is represented by the variable ***m***, and the equation is written as:

\begin{equation}m = \frac{y_{2} - y_{1}}{x_{2} - x_{1}} \end{equation}

Although this form of the equation is a little more verbose, it gives us a clue as to how we calculate slope. What we need is any two ordered pairs of x,y values for the line - for example, we know that our line passes through the following two points:
- (0,-2)
- (6,7)

We can take the x and y values from the first pair, and label them x1 and y1; and then take the x and y values from the second point and label them x2 and y2. Then we can plug those into our slope equation:

\begin{equation}m = \frac{7 - -2}{6 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{7 + 2}{6 - 0} \end{equation}

That gives us the result 9/6 which is 11/2 or 1.5 .

So what does that actually mean? Well, it tells us that for every change of **1** in x, **y** changes by 11/2 or 1.5. So if we start from any point on the line and move one unit to the right (along the X axis), we'll need to move 1.5 units up (along the Y axis) to get back to the line.

You can plot the slope onto the original line with the following Python code to verify it fits:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# set the slope
m = 1.5

# get the y-intercept
yInt = -2

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

### Slope-Intercept Form
One of the great things about algebraic expressions is that you can write the same equation in multiple ways, or *forms*. The *slope-intercept form* is a specific way of writing a 2-variable linear equation so that the equation definition includes the slope and y-intercept. The generalised slope-intercept form looks like this:

\begin{equation}y = mx + b \end{equation}

In this notation, ***m*** is the slope and ***b*** is the y-intercept.

For example, let's look at the solved linear equation we've been working with so far in this section:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Now that we know the slope and y-intercept for the line that this equation defines, we can rewrite the equation as:

\begin{equation}y = 1\frac{1}{2}x + -2 \end{equation}

You can see intuitively that this is true. In our original form of the equation, to find y we multiply x by three, subtract 4, and divide by two - in other words, x is half of 3x - 4; which is 1.5x - 2. So these equations are equivalent, but the slope-intercept form has the advantages of being simpler, and including two key pieces of information we need to plot the line represented by the equation. We know the y-intecept that the line passes through (0, -2), and we know the slope of the line (for every x, we add 1.5 to y.

Let's recreate our set of test x and y values using the slope-intercept form of the equation, and plot them to prove that this  describes the same line:

```python
%matplotlib inline

import pandas as pd
from matplotlib import pyplot as plt

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Define slope and y-intercept
m = 1.5
yInt = -2

# Add a y column by applying the slope-intercept equation to x
df['y'] = m*df['x'] + yInt

# Plot the line
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# label the y-intercept
plt.annotate('y-intercept',(0,yInt))

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

```python

```

AI Professional/2 - Essential Mathematics For Artificial Intelligence/DAT256x/Module01/01-02-Linear Equations.ipynb


```python
from sympy import *
x, y, z, t = symbols('x y z t')
```

## Linear algebra

A matrix $A \in \mathbb{R}^{m\times n}$ is a rectangular array of real numbers with $m$ rows and $n$ columns.
To specify a matrix $A$, we specify the values for its $mn$ components $a_{11}, a_{12}, \ldots, a_{mn}$
as a list of lists:

```python
A = Matrix( [[ 2,-3,-8, 7],
             [-2,-1, 2,-7],
             [ 1, 0,-3, 6]] )
```

Use the square brackets to access the matrix elements or to obtain a submatrix:

```python
A[0,1]         # row 0, col 1 of A
```

$\displaystyle -3$

```python
A[0:2,0:3]     # top-left 2x3 submatrix of A
```

$\displaystyle \left[\begin{matrix}2 & -3 & -8\\-2 & -1 & 2\end{matrix}\right]$

Some commonly used matrices can be created with shortcut methods:

```python
eye(2)         # 2x2 identity matrix
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

```python
zeros(2, 3)
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

Standard algebraic operations like
addition `+`, subtraction `-`, multiplication `*`,
and exponentiation `**` work as expected for `Matrix` objects.
The `transpose` operation flips the matrix through its diagonal:

```python
A.transpose()  # the same as A.T
```

$\displaystyle \left[\begin{matrix}2 & -2 & 1\\-3 & -1 & 0\\-8 & 2 & -3\\7 & -7 & 6\end{matrix}\right]$

Recall that the transpose is also used to convert row vectors into column vectors and vice versa.

### Row operations

```python
M = eye(3)
M[1,:] = M[1,:] + 3*M[0,:]
M
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\3 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

The notation `M[i,:]` refers to entire rows of the matrix.
The first argument specifies the $0$-based row index,
for example the first row of~`M` is `M[0,:]`.
The code example above implements the row operation $R_2 \gets R_2 + 3R_1$.

To scale a row `i` by constant `c`, use the command `M[i,:] = c*M[i,:]`.

To swap rows `i` and `j`, use can use the `Python` tuple-assignment syntax `M[i,:], M[j,:] = M[j,:], M[i,:]`.

### Reduced row echelon form

The Gauss—Jordan elimination procedure is a sequence of row operations you can perform
on any matrix to bring it to its *reduced row echelon form* (RREF).
In `SymPy`, matrices have a `rref` method that computes their RREF:

```python
A = Matrix( [[2,-3,-8, 7],
             [-2,-1,2,-7],
             [1, 0,-3, 6]])
A.rref()  # RREF of A, location of pivots
```

    (Matrix([
     [1, 0, 0,  0],
     [0, 1, 0,  3],
     [0, 0, 1, -2]]),
     (0, 1, 2))

Note the `rref` method returns a tuple of values:
the first value is the RREF of $A$,
while the second tells you the indices of the leading ones (also known as pivots) in the RREF of $A$.
To get just the RREF of $A$, select the $0^\mathrm{th}$ entry form the tuple: `A.rref()[0]`.

### Matrix fundamental spaces

Consider the matrix $A \in \mathbb{R}^{m\times n}$.
The fundamental spaces of a matrix are its column space $\mathcal{C}(A)$,
its null space $\mathcal{N}(A)$,
and its row space $\mathcal{R}(A)$.
These vector spaces are important when you consider the matrix product
$A\vec{x}=\vec{y}$ as “applying” the linear transformation $T_A:\mathbb{R}^n \to \mathbb{R}^m$
to an input vector $\vec{x} \in \mathbb{R}^n$ to produce the output vector $\vec{y} \in \mathbb{R}^m$.

**Linear transformations** $T_A:\mathbb{R}^n \to \mathbb{R}^m$ (vector functions)
**are equivalent to $m\times n$ matrices**.
This is one of the fundamental ideas in linear algebra.
You can think of $T_A$ as the abstract description of the transformation
and $A \in \mathbb{R}^{m\times n}$ as a concrete implementation of $T_A$.
By this equivalence,
the fundamental spaces of a matrix $A$
tell us facts about the domain and image of the linear transformation $T_A$.
The columns space $\mathcal{C}(A)$ is the same as the image space space $\textrm{Im}(T_A)$ (the set of all possible outputs).
The null space $\mathcal{N}(A)$ is the same as the kernel $\textrm{Ker}(T_A)$ (the set of inputs that $T_A$ maps to the zero vector).
The row space $\mathcal{R}(A)$ is the orthogonal complement of the null space.
Input vectors in the row space of $A$ are in one-to-one correspondence with the output vectors in the column space of $A$.

Okay, enough theory! Let's see how to compute the fundamental spaces of the matrix $A$ defined above.
The non-zero rows in the reduced row echelon form of $A$ are a basis for its row space:

```python
[ A.rref()[0][r,:] for r in A.rref()[1] ]  # R(A)
```

    [Matrix([[1, 0, 0, 0]]), Matrix([[0, 1, 0, 3]]), Matrix([[0, 0, 1, -2]])]

The column space of $A$ is the span of the columns of $A$ that contain the pivots
in the reduced row echelon form of $A$:

```python
[ A[:,c] for c in  A.rref()[1] ]           # C(A)
```

    [Matrix([
     [ 2],
     [-2],
     [ 1]]),
     Matrix([
     [-3],
     [-1],
     [ 0]]),
     Matrix([
     [-8],
     [ 2],
     [-3]])]

Note we took columns from the original matrix $A$ and not its RREF.

To find the null space of $A$, call its `nullspace` method:

```python
A.nullspace()                              # N(A)
```

    [Matrix([
     [ 0],
     [-3],
     [ 2],
     [ 1]])]

### Determinants

The determinant of a matrix,
denoted $\det(A)$ or $|A|$,
is a particular way to multiply the entries of the matrix to produce a single number.

```python
M = Matrix( [[1, 2, 3],
             [2,-2, 4],
             [2, 2, 5]] )
M.det()
```

$\displaystyle 2$

Determinants are used for all kinds of tasks:
to compute areas and volumes,
to solve systems of equations,
and to check whether a matrix is invertible or not.

### Matrix inverse

For every invertible matrix $A$,
there exists an inverse matrix $A^{-1}$ which *undoes* the effect of $A$.
The cumulative effect of the product of $A$ and $A^{-1}$ (in any order)
is the identity matrix: $AA^{-1}= A^{-1}A=\mathbb{1}$.

```python
A = Matrix( [[1,2],
             [3,9]] )
A.inv()
```

$\displaystyle \left[\begin{matrix}3 & - \frac{2}{3}\\-1 & \frac{1}{3}\end{matrix}\right]$

```python
A.inv()*A
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

```python
A*A.inv()
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

The matrix inverse $A^{-1}$ plays the role of division by $A$.

### Eigenvectors and eigenvalues

When a matrix is multiplied by one of its eigenvectors the output
is the same eigenvector multiplied by a constant $A\vec{e}_\lambda =\lambda\vec{e}_\lambda$.
The constant $\lambda$ (the Greek letter *lambda*) is called an *eigenvalue* of $A$.

To find the eigenvalues of a matrix, start from the definition $A\vec{e}_\lambda =\lambda\vec{e}_\lambda$,
insert the identity $\mathbb{1}$,
and rewrite it as a null-space problem:

$$
A\vec{e}_\lambda =\lambda\mathbb{1}\vec{e}_\lambda
\qquad
\Rightarrow
\qquad
\left(A - \lambda\mathbb{1}\right)\vec{e}_\lambda = \vec{0}.
$$

This equation will have a solution whenever $|A - \lambda\mathbb{1}|=0$.(The invertible matrix theorem states
that a matrix has a non-empty null space if and only if its determinant is zero.)
The eigenvalues of $A \in \mathbb{R}^{n \times n}$,
denoted $\{ \lambda_1, \lambda_2, \ldots, \lambda_n \}$,\
are the roots of the *characteristic polynomial* $p(\lambda)=|A - \lambda \mathbb{1}|$.

```python
A = Matrix( [[ 9, -2],
             [-2,  6]] )
A.eigenvals()  # same as solve(det(A-eye(2)*x), x)
               # return eigenvalues with their multiplicity
```

    {10: 1, 5: 1}

```python
A.eigenvects()
```

    [(5,
      1,
      [Matrix([
       [1/2],
       [  1]])]),
     (10,
      1,
      [Matrix([
       [-2],
       [ 1]])])]

Certain matrices can be written entirely in terms of their eigenvectors and their eigenvalues.
Consider the matrix $\Lambda$ (capital Greek *L*) that has the eigenvalues of the matrix $A$ on the diagonal,
and the matrix $Q$ constructed from the eigenvectors of $A$ as columns:

$$
\Lambda =
\begin{bmatrix}
\lambda_1	&  \cdots  &  0 \\
\vdots 	&  \ddots  &  0  \\
0  	&   0      &  \lambda_n
\end{bmatrix}\!,
\ \
Q \: =
\begin{bmatrix}
|  &  & | \\
\vec{e}_{\lambda_1}  & \!  \cdots \! &  \large\vec{e}_{\lambda_n} \\
|  &  & |
\end{bmatrix}\!,
\ \
\textrm{then}
\ \
A = Q \Lambda Q^{-1}.
$$

Matrices that can be written this way are called *diagonalizable*.
To *diagonalize* a matrix $A$ is to find its $Q$ and $\Lambda$ matrices:

```python
Q, L = A.diagonalize()
Q            # the matrix of eigenvectors as columns
```

$\displaystyle \left[\begin{matrix}1 & -2\\2 & 1\end{matrix}\right]$

```python
Q.inv()
```

$\displaystyle \left[\begin{matrix}\frac{1}{5} & \frac{2}{5}\\- \frac{2}{5} & \frac{1}{5}\end{matrix}\right]$

```python
L            # the matrix of eigenvalues
```

$\displaystyle \left[\begin{matrix}5 & 0\\0 & 10\end{matrix}\right]$

```python
Q*L*Q.inv()  # eigendecomposition of A
```

$\displaystyle \left[\begin{matrix}9 & -2\\-2 & 6\end{matrix}\right]$

```python
Q.inv()*A*Q  # obtain L from A and Q
```

$\displaystyle \left[\begin{matrix}5 & 0\\0 & 10\end{matrix}\right]$

Not all matrices are diagonalizable.
You can check if a matrix is diagonalizable by calling its `is_diagonalizable` method:

```python
A.is_diagonalizable()
```

    True

```python
B = Matrix( [[1, 3],
            [0, 1]] )
B.is_diagonalizable()
```

    False

```python
B.eigenvals()  # eigenvalue 1 with multiplicity 2
```

    {1: 2}

```python
B.eigenvects()
```

    [(1,
      2,
      [Matrix([
       [1],
       [0]])])]

The matrix $B$ is not diagonalizable because it doesn't have a full set of eigenvectors.
To diagonalize a $2\times 2$ matrix, we need two orthogonal eigenvectors but $B$ has only a single eigenvector.
Therefore, we can't construct the matrix of eigenvectors $Q$ (we're missing a column!)
and so $B$ is not diagonalizable.

Non-square matrices don't have eigenvectors and therefore don't have an eigendecomposition.
Instead, we can use the *singular value decomposition* to break up a non-square matrix $A$ into
left singular vectors,
right singular vectors,
and a diagonal matrix of singular values.
Use the `singular_values` method on any matrix to find its singular values.


```python
import numpy as np
p = np.poly1d([1,0,0,0,0,0])
print (p)
print (p.integ())
p.integ()(1.0) - p.integ()(-1.0)
```

       5
    1 x
            6
    0.1667 x

    0.0

```python
from sympy import integrate, symbols
x, y = symbols('x y', real=True)
integrate(x**5, x)
```

    x**6/6

```python
integrate(x**5, (x, -1, 1))
```

    0

```python
from sympy import N, exp as Exp, sin as Sin
integrate(Exp(-x) * Sin(x), x)
```

    -exp(-x)*sin(x)/2 - exp(-x)*cos(x)/2

```python
integrate(Exp(-x) * Sin(x), (x, 0, 1))
```

    -exp(-1)*sin(1)/2 - exp(-1)*cos(1)/2 + 1/2

```python
N(_)
```

    0.245837007000237

```python
integrate(Sin(x) / x, x)
```

    Si(x)

```python
integrate(Sin(x) / x, (x, 0, 1))
```

    Si(1)

```python
N(_)
```

    0.946083070367183

```python
integrate(x**1, (x, 0, 1))
```

    1/2

```python
from sympy import oo
integrate(Exp(-x**2), (x,0,+oo))
```

    sqrt(pi)/2

```python

```


# The Gaussian (Normal) Distribution

## Probability Distribution Function

The Normal (Gaussian) distribution probability distribution function is

\begin{equation}
f\left(x; \mu, \sigma\right) = \frac{1}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2},
\end{equation}

normalized to unity,
$$
\int\limits_{-\infty}^{\infty} f\left(x; \mu,\sigma\right)\,dx = 1,
$$
is symmetrically distributed about its mean, $\mu$, with width $\sigma$.

## Full Width at Half Maximum (FWHM)

The full width at half maximum (FWHM) is the distance between points on a curve at which the function reaches half its maximum value. The FWHM is often used to describe the "width" of a distribution.

For a 1-dimensional Gaussian, it is seen that as the maximum value occurs as $x = \mu$ (by definition), half of the maximum value is

\begin{align*}
\left.\frac{1}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\right|_{x = x \text{ of (max/2)}} &= \frac{1}{2} f\left(x_{\text{max}}\right)\\
    &= \frac{1}{2} f\left(\mu\right) = \frac{1}{2} \frac{1}{\sqrt{2\pi}\sigma},
\end{align*}

resulting in the equality

\begin{equation*}
e^{-\left(x-\mu\right)^2/2\sigma^2} = \frac{1}{2},
\end{equation*}

which is (taking the log)

\begin{equation*}
-\frac{\left(x-\mu\right)^2}{2\sigma^2} = -\ln2.
\end{equation*}

Thus, solving the equality,
$$
\left(x-\mu\right)^2 = 2\sigma^2 \ln2,
$$
yields
$$
x_{\pm} = \pm \sigma \sqrt{2 \ln 2} + \mu.
$$

Thus, the FWHM is

\begin{align*}
\text{FWHM} &= x_{+} - x_{-}\\
    &= \left(\sigma \sqrt{2 \ln 2} + \mu\right) - \left(-\sigma \sqrt{2 \ln 2} + \mu\right)\\
    &= \boxed{2\sqrt{2\ln2}\sigma}\,.
\end{align*}

**N.B.:** It is seen that the FWHM for a Gaussian is _independent_ of both the normalization constant and the mean, and is only dependent on the standard devaiation of the Gaussian.

## Probability and the Error Function

The probability that a Normally distributed random variable will lie in a range of values symmetrically integrated over is given by

\begin{align*}
\text{Pr}\left(\mu - y \leq x \leq \mu + y\right) &= \int\limits_{\mu - y}^{\mu + y}\frac{1}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\,dx\\
    &= \int\limits_{\mu}^{\mu + y}\frac{2}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\,dx.
\end{align*}

Making the substitution
$$
t = \frac{\left(x-\mu\right)}{\sqrt{2}\sigma},
$$
then

\begin{align*}
\text{Pr}\left(\mu - y \leq x \leq \mu + y\right) &= \int\limits_{\mu}^{\mu + y}\frac{2}{\sqrt{2\pi}\sigma}\,e^{-\left(x-\mu\right)^2/2\sigma^2}\,dx\\
    &= \boxed{\frac{2}{\sqrt{\pi}} \int\limits_{0}^{y/\sqrt{2}\sigma}e^{-t^2}\,dt \equiv \text{erf}\left(\frac{y}{\sqrt{2}\sigma}\right)}\,.
\end{align*}

## Cumulative Distribution Function (cdf)

For the cumulative distribution function (cdf),
$$
\Phi\left(x\right) = \int\limits_{-\infty}^{x}f\left(t;\mu,\sigma\right)\,dt,
$$
it is seen, noting from the form of the error function, that

\begin{equation*}
\frac{2}{\sqrt{\pi}} \int\limits_{0}^{y/\sqrt{2}\sigma}e^{-t^2}\,dt = \frac{2}{\sqrt{2 \pi}} \int\limits_{0}^{y/\sigma}e^{-t^2/2}\,dt = \text{erf}\left(\frac{y}{\sqrt{2}\sigma}\right),
\end{equation*}

```python
import sympy as sym
sym.init_printing(use_unicode=True, wrap_line=False, no_global=True)
from sympy.abc import sigma
```

```python
t, y, = sym.symbols('t y')
sym.integrate((2/sym.sqrt(sym.pi)) * sym.exp(-t**2), (t, 0, y/(sym.sqrt(2) * sigma)))
```

```python
sym.integrate((2/sym.sqrt(2 * sym.pi)) * sym.exp(-t**2/2), (t, 0, y/sigma))
```

then for the standard Gaussian ($\mu = 0$, $\sigma=1$)

\begin{align*}
\Phi\left(x\right) &= \frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{x} e^{-t^2/2}\,dt\\
    &= \frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{0} e^{-t^2/2}\,dt + \frac{1}{\sqrt{2\pi}}\int\limits_{0}^{x} e^{-t^2/2}\,dt\\
    &= \frac{1}{2} + \frac{1}{2} \text{erf}\left(\frac{x}{\sqrt{2}}\right)\\
    &= \frac{1}{2} \left(1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)
\end{align*}

so it is likewise seen that

\begin{align*}
\text{Pr}\left(\mu - n \sigma \leq x \leq \mu + n\sigma\right) &= \Phi(n) - \Phi(-n)\\
    &= \Phi(n) - \left(1-\Phi(n)\right)\\
    &= \frac{1}{2} \left(1 + \text{erf}\left(\frac{n}{\sqrt{2}}\right)\right) - \left[1-\frac{1}{2} \left(1 + \text{erf}\left(\frac{n}{\sqrt{2}}\right)\right)\right]\\
    &= \text{erf}\left(\frac{n}{\sqrt{2}}\right).
\end{align*}

**Show this for:** generic normal distribution $f$ with mean $\mu$ and std $\sigma$

It is noted that in the case that $\left|y\right|=n\sigma$,
$$
\text{Pr}\left(\mu - y \leq x \leq \mu + y\right) = \text{Pr}\left(\mu - n\sigma \leq x \leq \mu + n\sigma\right) = \text{erf}\left(\frac{n}{\sqrt{2}}\right).
$$

So, for $n=1$,
$$
\text{Pr}\left(\mu - \sigma \leq x \leq \mu + \sigma\right) = \text{erf}\left(\frac{1}{\sqrt{2}}\right)
$$

However, at this point we are at an impass analytically, as the integral of a Gaussian function over a finite range has no analytical solution, and must be evaluated numerically.

```python
import math
from scipy import special as special
```

```python
def prob_n_sigma(n):
    return special.erf(n/math.sqrt(2.))
```

```python
prob_n_sigma(1)
```

## $p$-values

### Two-Tailed  $p$-value

### One-Tailed  $p$-value


$$f(x) = x^2$$

### The probability of getting (k) heads when flipping (n) coins is
\begin{equation*}
P(E)   = {n \choose k} p^k (1-p)^{ n-k}
\end{equation*}

\begin{equation}
\int y\, \mathrm{d}x
\end{equation}

## Mean, Median, Mode

Mean, median, and mode are different measures of center in a numerical data set. They each try to summarize a dataset with a single number to represent a "typical" data point from the dataset.

__Mean:__ The "average" number; found by adding all data points and dividing by the number of data points.

\begin{equation}
\bar{y} = \frac{\sum y}{N}
\end{equation}

- $\sum y$ Take all the values and add them up
- divide by the total number of observations you have

__Median__ The middle number found by ordering all data points and picking out the one in the middle (or if there are two middle numbers, taking the mean of those two numbers).

To get the median

\begin{equation}
y = \tilde{x}
\end{equation}

$ y = [1,2,3,4,4,3,2,1] $

$ y = 4 $

The __mode__ is the most commonly occurring data point in a dataset. The mode is useful when there are a lot of repeated values in a dataset. There can be no mode, one mode, or multiple modes in a dataset.

## Standard Deviation

__Standard deviation__ is a number used to tell how measurements for a group are spread out from the average (mean), or expected value. A low standard deviation means that most of the numbers are close to the average. A high standard deviation means that the numbers are more spread out.

$$s = \sqrt{\frac{1}{N-1} \sum_{i=1}^N (x_i - \overline{x})^2}$$

## Variance

The __Variance__ is defined as: The average of the squared differences from the Mean. To calculate the variance follow these steps: Find the __Mean__, Then for each number: subtract the __Mean__ and square the result (the squared difference).

$$\sigma^2 = \frac{\displaystyle\sum_{i=1}^{n}(x_i - \mu)^2} {n}$$

## What is RMSE

 $RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}{\Big(\frac{d_i -f_i}{\sum_i}\Big)^2}}$

 ## Root mean squared error (RMSE)
RMSE is a quadratic scoring rule that also measures the average magnitude of the error. It’s the square root of the average squared differences between prediction and actual observation.

 $RMSE = \sqrt{\frac{1}{n}\Sigma_{i=1}^{n}{\Big(\frac{d_i -f_i}{\Sigma_i}\Big)^2}}$

 we can do this simply from sklearn:

```python
from sklearn.metrics import mean_squared_error
from math import sqrt
rmse = sqrt(mean_squared_error(y_actual, y_predicted))
print(rmse)

# another way
n = len(predictions)
rmse = np.linalg.norm(predictions - targets) / np.sqrt(n)

# and another way
from sklearn.metrics import mean_squared_error
from math import sqrt
expected = [0.0, 0.5, 0.0, 0.5, 0.0]
predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
mse = mean_squared_error(expected, predictions)
rmse = sqrt(mse)
print('RMSE: %f' % rmse)

```
The mean square root and square root of it will be useful.

```python
#pip install sklearn
```

```python
from sklearn.metrics import mean_squared_error
from math import sqrt
expected = [0.0, 0.5, 0.0, 0.5, 0.0]
predictions = [0.2, 0.4, 0.1, 0.6, 0.2]
mse = mean_squared_error(expected, predictions)
rmse = sqrt(mse)
print('RMSE: %f' % rmse)
```

    RMSE: 0.148324

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/RECAP_DS/01_BASIC_STATISTICS/01_Math_Basics.ipynb


# Generating Functions

Generating functions are functions that encode sequences of numbers as the coefficients of power series.

Consider a set $S$ with $n$ elements.

Pretend there is a picture function' $P(s)\ \forall s\in S$.

The picture function enables one, for example, to write the multiset $\{1,1,2\}$ as an expression $P(1)^2P(2)$.

The enumerating function $E_p(s)\ :s\in S$ is used to list combinations subject to restraints. For example, the enumerating function for all multisets that include either one or two times some element $a$ and between zero and two times some element $b$ is written:

\begin{equation}
\begin{array}{rl}
E_P(s) &= P(a)+ P(a)^2 + P(a)P(b) + P(a)^2P(b) +  + P(a)P(b)^2 + P(a)^2P(b)^2\\
&\left(P(a) + P(a)^2\right)\left(1 + P(b) + P(b)^2\right)
\end{array}
\end{equation}

This hints at how generating functions can make it easier to keep track of combinations.

## Example: Binomial Coefficients

Consider the case of a collection of $n$ indistinguishable objects $s$, and write $P(s) = x$. Then the enumerator for selecting any subset of those $n$ objects is given by:

\begin{equation}
E_P(s) = \prod_i^n (x^0+x^1) = (1+x)^n = \sum_i^n {n \choose i}x^i
\end{equation}

Where $(x^0 + x^1)$ corresponds to including an element zero or one times. The exponent encodes how many objects were included into a particular subset. This is one way of "proving" the binomial theorem, and one can says $(1_x)^n$ is the generating function for the binomial coefficients ${n \choose i}$.

## Example: Basket of Goods

An apple costs $20c$, a pear costs $25c$ and a banana costs $30c$. How many different fruit baskets can be bought for $100c$?

By replacing the picture function $P(s)$ with $x$, it was possible to identify subsets of $n$ objects by looking at the exponent of $x^n$ in the enumerating function. In this case, the exponent is supposed to show the price. This can be done by writing $P(apple) = x^{20}, P(pear) = x^{25}$ and $P(banana) = x^{30}$.

\begin{equation}
E_P(s) = \left( \sum_{i=0}^5 x^{20} \right)\left( \sum_{i=0}^4 x^{25} \right)\left( \sum_{i=0}^3 x^{30} \right)
\end{equation}

Which results in some power series of the form:

\begin{equation}
E_P(s) = 1x^0 + 1x^{20} + 1x^{25} + 1x^{30} + 1x^{40} +... + 2x^{60} + ... + 1x^{290}
\end{equation}

To obtain the number of combinations that correspond to a cost of exactly $100c$, one can apply the operator $\frac{1}{n!}\frac{d^n}{dx^n}$ and set $x=0$ to obtain the desired term. This is what is done with moment generating functions in statistics.

But actually it is easier to think through what the coefficients will be so that:

\begin{equation}
\sum_{l=0}^{n+m+h}d_l x^l = \left(\sum_{i=0}^n a_i x^i\right)\left(\sum_{j=0}^m b_j x^j\right)\left(\sum_{k=0}^h c_k x^k\right)
\end{equation}

\begin{equation}
d_l = \sum_{\begin{array}{c}i,j,k\\i+j+k=l\end{array}} a_i b_j c_k
\end{equation}

If the coefficients are all '1', which is the case for these combinations, then:

\begin{equation}
d_l = \sum_{\begin{array}{c}i,j,k\\i+j+k=l\end{array}} 1 = {l + 3 - 1 \choose l}
\end{equation}

Which is the number of all multisets of size $l$ and 3 classes. In this case the coefficients are equal to 1 for i=20, j=25, and k=30, and zero otherwise, so that there are 4 ways of summing to 100. Hence, $d_{100} = 4$ for the basket of goods above.

The sum can also be rewritten:

\begin{equation}
d_l = \sum^l_{i=0}\sum^{l-i}_{j=0}\sum^{l-i-j}_{k= l-i-j} a_i b_j c_k
\end{equation}

Where the last sum is actually just over a single term.

Of course, this is the discrete version of a convolution. So, it is no surprise that the addition of random variables winds up being a convolution.

## Example: Dice

How many ways are there for $n$ dice with $k$ faces to show $s$ eyes?

\begin{equation}
E_P = \left(\sum_{i=0}^\infty a_i x^i\right)^n = \sum_{i=0}^{\infty}d_i x^i
\end{equation}

Where $a_i = 1\ \forall i\in(1,k)$ and $a_i = 0$ otherwise.

\begin{equation}
E_P = \left(\sum_{i=1}^k x^i\right)^n = \left(x(1-x^k)\sum_{i=0}^\infty x^i\right)^n = x^n\left(\frac{1-x^k}{1-x}\right)^n = x^n \left(\sum_{i=0}^n (-1)^i {n \choose i} x^{ik}\right)\left(\sum_{j=0}^{\infty} (-1)^j {-n \choose j} x^j\right)
\end{equation}

The coefficient for $x^s$ is given the sum:

\begin{equation}
d_s = \sum_{ki+j=s-n} (-1)^{i+j}{n \choose i}{-n \choose j}\\
i \in [0,n] \\
j \in [0,\infty]
\end{equation}

Where the indices $i$ and $j$ satisfy $ki+j = s-n$. For example, for $s=7$, $n=2$ and $k=6$:

\begin{equation}
6i+j = 7-2 = 5
\end{equation}

Holds for $i=0, j=5$:

\begin{equation}
d_s = (-1)^5 {2 \choose 0}{-2 \choose 5} = (-1)^{10} 1 {2+5-1 \choose 5} = {6 \choose 5} = \frac{6!}{5!1!} = 6
\end{equation}

Indeed, there are 6 ways for two d6 to add to 7:

\begin{equation}
[6,1],[5,2],[4,3],[3,4],[2,5],[1,6]
\end{equation}

```python
import warnings

def memoize(func):
    """
    memoizing wrapper to speed up recursion by keeping track of previously calculated values.
    """
    S = {}
    def wrappingfunction(*args):
        if args not in S:
            S[args] = func(*args)
        return S[args]
    return wrappingfunction

@memoize
def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
        return res

@memoize
def binomial(n,k):
    if n > 40 or k > 40:
        warnings.warn('careful with large n or k - unresolved numerical stability issues.')

    if n >= k and k>= 0:
        return int(factorial(n)/(factorial(n-k)*factorial(k)))
    elif n < 0 and k >= 0:
        return int((-1)**k * binomial(-n+k-1,k))
    elif n < 0 and k <= n:
        return int((-1)**(n-k) * binomial(-k-1,n-k))
    else:
        return 0

f = lambda x: str(x) if x != 0 else '-'
N = 5

print('\t'.join(['     ']+['k=%i' % k for k in range(-N,N+1)]))
for n in range(-N,N+1)[::-1]:
    print('\t'.join(['n=%i' % n]+[f(binomial(n,k)) for k in range(-N,N+1)]))
```

         	k=-5	k=-4	k=-3	k=-2	k=-1	k=0	k=1	k=2	k=3	k=4	k=5
    n=5	-	-	-	-	-	1	5	10	10	5	1
    n=4	-	-	-	-	-	1	4	6	4	1	-
    n=3	-	-	-	-	-	1	3	3	1	-	-
    n=2	-	-	-	-	-	1	2	1	-	-	-
    n=1	-	-	-	-	-	1	1	-	-	-	-
    n=0	-	-	-	-	-	1	-	-	-	-	-
    n=-1	1	-1	1	-1	1	1	-1	1	-1	1	-1
    n=-2	-4	3	-2	1	-	1	-2	3	-4	5	-6
    n=-3	6	-3	1	-	-	1	-3	6	-10	15	-21
    n=-4	-4	1	-	-	-	1	-4	10	-20	35	-56
    n=-5	1	-	-	-	-	1	-5	15	-35	70	-126

```python
def d_s(s,n=2,k=6):
    """
    Number of Dice Combinations

    n = number of dice
    k = number of dice faces (faces: 1,2,3,4,5,...,k)
    s = sum of dice throw

    Beware of numerical stability issues for N >~ 45
    """
    res = 0
    for i in range(0,int((s-n)/k)+1):
        j = s-n-k*i
        res += (-1)**(i+j) * binomial(n,i) * binomial(-n,j)
    return res

print('2 d6')
for s in range(1,14):
    print('s = %i\t' % s, 'd = %i' % d_s(s,n=2,k=6))
```

    2 d6
    s = 1	 d = 0
    s = 2	 d = 1
    s = 3	 d = 2
    s = 4	 d = 3
    s = 5	 d = 4
    s = 6	 d = 5
    s = 7	 d = 6
    s = 8	 d = 5
    s = 9	 d = 4
    s = 10	 d = 3
    s = 11	 d = 2
    s = 12	 d = 1
    s = 13	 d = 0

```python
import matplotlib.pyplot as plt
import numpy as np

k = 6

N = 31
ss = list(range(1,N*(k-2)))
nn = list(range(1,N))

plt.figure(figsize=(12,5))
for n in nn:
    d = [d_s(s,n=n,k=k) for s in ss]
    d = np.array(d)/np.sum(d)

    plt.plot(ss,d,'.-',linewidth=2,markersize=8)
    plt.text(x=3.5*n+0.2,y=max(d)+0.0025,s='%i' % n)

plt.xlabel('s')
plt.ylabel('p(s)')
plt.title('Probability of d6 Throw')
plt.savefig('./img/d6throw.png')

xlim = plt.xlim()

k = 20

N = 31
ss = list(range(1,N*(k-2)))
nn = list(range(1,N))

plt.figure(figsize=(12,5))
for n in nn:
    d = [d_s(s,n=n,k=k) for s in ss]
    d = np.array(d)/np.sum(d)

    plt.plot(ss,d,'.-',linewidth=2,markersize=8)
    if 10.5*n <= max(xlim):
        plt.text(x=10.5*n+0.2,y=max(d)+0.00075,s='%i' % n)

plt.xlim(*xlim)
plt.xlabel('s')
plt.ylabel('p(s)')
plt.title('Probability of d20 Throw')
plt.savefig('./img/d20throw.png')
```

```python

```

Combinatorics - Generating Functions.ipynb


# Basic Naive Bayes

## Theory

Naive Bayes classifiers attempt to divine the probability of each class given a data set.  If we represent the predicted class as a random variable $Y$ and the set of features as a random variable $X$, then naive Bayes predicts a class from data $x_i$ by selecting the class $y_i$ that have the highest value of $P(Y=y_i|X=x_i)$.  In statistical nomenclature, the class with the highest posterior probability.  So training consists of figuring out how we can estimate these posterior probabilities from a set of training data.

To begin, let us assume that we are trying to predict the chance of borrower default given some historical data:

| married | diet | defaulted |
| --- | ---| ---|
| yes | conventional | no |
| no | conventional | yes |
| no | vegan | yes |
| yes | vegetarian | no |
| no | gluten-free | no |
| yes | West Coast | yes |

Given this data, what is the probability that a married vegan will default?

We can attempt to solve this with the standard conditional probability definition, $$P(default | (married \cap vegan)) = \frac{P(default \cap married \cap vegan)}{P(married \cap vegan)} $$ but find that this leaves us trying to approximate the joint probability of our features.  We likely didn't gather data with this estimation in mind (we are attempting to correlate attributes with our predicted class, not other attributes).  If we assume indepedence of attributes, our denominator is always 0.  Instead, we would like to estimate $P(default | (married \cap vegan))$ based on the relationships between our attributes and our predicted class.  We can accomplish this using Bayes' formula, which is the consequence of the following equalities:
$$ P(Y|X) = \frac{P(Y \cap X)}{P(X)} $$
$$ P(X|Y) = \frac{P(X \cap Y)}{P(Y)} $$
$$P(X \cap Y) = P(Y \cap X)$$

Solving the first two for their joint probabilities, equating them, and then dividing by $P(X)$ yields Bayes formula:

$$ P(Y | X) = \frac{P(X|Y)P(Y)}{P(X)} $$

As noted above, we can make classify data $x_i$ by selecting the class $y_i$ that has the highest value of $P(Y=y_i|X=x_i)$.  Since $P(X)$ is constant across classes, we can ignore it when searching for the highest posterior.  Estimating $P(Y=y_i)$ from the training data is easy as it is just the proportion of labels that belong to a particular class.  Estimating $P(X | Y)$ is more difficult, since X is a vector of attributes and therefore $P(X=x_i)$ is shorthand for $P(x_0 = x_{i0} \cap x_1 = x_{i1} \cap ... \cap x_m = x_{im})$.  With __naive bayes__, this calculation is greatly simplified by assuming that the attributes are conditionally independent.  This assumption yields:

$$P((x_0 = x_{i0} \cap x_1 = x_{i1} \cap ... \cap x_m = x_{im}) | Y=y) = \prod_{j=0}^{m}{P(x_j=x_{ij} | Y=y)}$$

Consequently, we can estimate our posterior (dropping the class constant denominator) as:

$$P(Y=y | X=x_i) \sim = \prod_{j=0}^{m}{P(x_j=x_{ij} | Y=y)}*P(Y=y)$$

### Training

The learned 'model' is a set of prior probabilities (i.e. $P(Y=y)$ for all classes $y$) and a set of conditional probabilities (i.e. $P(x_j=x_{ij} | Y=y$).  We learn the priors based on class proportion in the training set and calculate the conditional probabilities based on the training data.

### Classifying

To classify, we calculate the posterior (i.e. $P(Y=y | X=x_i)$) for every class y using our learned priors and conditional probabilities.  Our label is the class with the highest posterior.

### Misc

Arguably the most useful characteristic of a naive bayes classifier is that it returns a probability for each class, which can be used to guage classification confidence.

### Exercise

Given our borrower default data, calculate the naive bayes 'model' and use it to predict if a married vegan will default on their loan.

---
Our model:

| model param | value |
|---|---|
| $P(defaulted=yes)$ | 0.5|
| $P(defaulted=no)$ | 0.5|
| $P(married=yes | defaulted=yes)$ | 0.333|
| $P(married=no | defaulted=yes)$ | 0.666|
| $P(married=yes | defaulted=no)$ | 0.666|
| $P(married=no | defaulted=no)$ | 0.333|
| $P(diet=conventional | defaulted=yes)$ | 0.333|
| $P(diet=conventional | defaulted=no)$ | 0.333|
| ... | ... |
| $P(diet=West Coast | defaulted=yes)$ | 0.333|
| $P(diet=West Coast | defaulted=no)$ | 0|

Our posteriors:

$\begin{align}
 P(default=yes|(married=yes \cap diet=vegan)) &= \\
 & =  P(married=yes | defaulted = yes) * P(diet=vegan | defaulted = yes) * P(defaulted=yes)\\
 & = 0.333 * 0.333 * 0.5 \\
 & = 0.055
\end{align}$

$\begin{align}
 P(default=no|(married=yes \cap diet=vegan)) &= \\
 & =  P(married=yes | defaulted = no) * P(diet=vegan | defaulted = no) * P(defaulted=no)\\
 & = 0.666 * \rho * 0.5 \\
 & = 0.166\rho
\end{align}$

We see how to actually deal with 0 conditional probabilities later but for now lets just say that $\rho=0.333$, so we have the posterior for default=yes  is 0.055 and for default=no is 0.0833.  Therefore, our model would predict that a married vegan would not default.

### Edge cases

#### Conditionals of zero
As we saw in our example above, a zero conditional can occur when there is too little data (esp. with regard to a single attribute).  Since a zero conditional will zero out the product of conditionals, it is important to come up with a heuristic for dealing with them. A common approach is to use the m-estimate:

$$P(x_i|y_i)=\frac{n_c+mp}{n+m}$$

where $m$ (equivalent sample size) and $p$ are parameters, $n_c$ is the number of records with attribute value $x_i$ and class $y_i$, and n is the number of records with class $y_i$.  We note that if there is no training set then $P(x_i|y_i)=p$, so $p$ can be conceived of as the prior probability of attribute value $x_i$ given class $y_i$.

#### Estimating conditionals for continuous attributes
The astute student might notice that directly calculating conditionals for continuous variables would be impossible since any single point is unlikely to occur multiple times in our training set.  The simplest way to deal with this issue is discretization; divide up the range of values into regions, map the values to regions, and treat the region 'labels' as categorical variables.  The problem with this approach is that the analyst must select the regions and this is prone to error.

Alternatively, we can assume that the data fits a specific distribution (e.g. Gaussian) and use the class conditional probability of that distribution as our conditional.  Assuming a Gaussian distribution, we have:

$$P(X_i=x|Y=y_j)=\frac{1}{\sqrt{2\pi}\sigma_{ij}}e^{-\frac{(x-\mu_{ij})^2}{2\sigma^2_{ij}}}$$

Consequently, during the training phase we calculate the sample mean and sample variance of the attribute $X_i$'s values for records with class $y_j$ and use those values to estimate $\mu_{ij}$ and $\sigma_{ij}$ respectively.  At that point, we have a function for determining the conditional for a particular attribute value and class.

## Implementation

```python
#Assumptions : Only string (nominal values) or numeric fields
import pandas as pd
import math

class naive_bayes:

    def __init__(self, equivalent_sample_size=1, default_attr_prior=0.1):
        self._prior = default_attr_prior
        self._eq_sample_size = equivalent_sample_size

    def _m_cond(self, class_cnt, class_attr_cnt):
        return (class_attr_cnt + (self._eq_sample_size*self._prior)) / (class_cnt + self._eq_sample_size)

    def _calc_priors(self, labels):
        return labels.value_counts() / len(labels)

    def _categorical_conditionals(self, df):
        "Assumes 2 columns, attribute and label, and the attribute values are categorical"
        attr_col = df.columns[0]
        class_col = df.columns[1]

        conds = []
        for y, df in data.groupby(class_col):
            class_count = len(df)
            conds += [[val,self._m_cond(class_count, cnt),y] for val,cnt in df[attr_col].value_counts().items()]

        return pd.DataFrame(conds,columns=['Value','Probability','Class'])

    def _continuous_conditionals(self, df):
        "Assumes 2 columns, attribute and label, and the attribute values are numeric"
        attr_col = df.columns[0]
        class_col = df.columns[1]

        stats = []
        for y, df in data.groupby(class_col):
            stats += [[df[attr_col].mean(),df[attr_col].std(),y]]

        return pd.DataFrame(stats,columns=['Mean','Std','Class'])

    def fit(self, data,class_column):
        result = {}

        for col,is_numeric in ((data.dtypes == 'float64') | (data.dtypes == 'int64')).items():
            if col != class_column:
                result[col] = {'type' : 'continuous', 'conditionals' : self._continuous_conditionals(data[[col,class_column]])} if is_numeric else {'type' : 'categorical', 'conditionals' : self._categorical_conditionals(data[[col,class_column]])}

        self._model = result
        self._priors = self._calc_priors(data[class_column])

    def _gaussian(self, val, params):
        std = params['Std']
        return (1.0/(math.sqrt(2*math.pi) * std)) * math.exp(-((val-params['Mean'])**2)/(2*(std**2)))

    def _calc_posterior(self, record, col):
        model_params = self._model[col]
        x = record[col]
        conds = model_params['conditionals']

        if model_params['type'] == 'categorical':
            return conds[conds['Value'] == x][['Probability','Class']]

        #else continuous
        gauss = lambda stats : pd.Series([self._gaussian(x,stats), stats['Class']])
        tmp = conds.apply(gauss,axis=1)
        return tmp.rename(columns={0 : 'Probability', 1 : 'Class'})

    def predict(self, record):
        if not hasattr(self, '_model'):
            raise Exception('The model has not been fitted, prediction is impossible')

        tmp_df = pd.DataFrame()
        for col in record.index:
            tmp_df = tmp_df.append(self._calc_posterior(record,col),ignore_index=True)

        posteriors = tmp_df.groupby('Class').prod()['Probability']*self._priors
        print(posteriors)

        return posteriors.idxmax()

```

```python
data = pd.DataFrame({'Diet' : ['conventional','conventional','vegan','vegetarian','gluten-free','west-coast'],
                    'Married' : ['yes','no','no','yes','no','yes'],
                    'Salary' : [120000.0,34000.0,54000.0,75000.0,90000.0,65000.0],
                    'Default' : ['no','yes','yes','no','no','yes']})

nb = naive_bayes()

nb.fit(data,'Default')

new_record = pd.Series(['east-coast','no',50000.00],index=['Diet','Married','Salary'])

nb.predict(data.iloc[0,:-1])
nb.predict(data.iloc[1,:-1])
nb.predict(new_record)

```

    no     6.930807e-07
    yes    6.260629e-11
    dtype: float64
    no     1.902869e-08
    yes    1.020828e-06
    dtype: float64
    no     3.479887e-07
    yes    6.649849e-06
    dtype: float64

    'yes'