###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C. Cooper. Based on [CFDPython](https://github.com/barbagroup/CFDPython), (c)2013 L.A. Barba, also under CC-BY license.

# Space & Time

## 1-D Diffusion

Welcome back! This is the third Jupyter Notebook of the series *Space and Time — Introduction of Finite-difference solutions of PDEs*, the second module of ["Practical Numerical Methods with Python"](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about).

In the previous Jupyter notebooks of this series, we studied the numerical solution of the linear and non-linear convection equations using the finite-difference method, and learned about the CFL condition. Now, we will look at the one-dimensional diffusion equation:

$$
\begin{equation}
\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}
\end{equation}
$$

where $\nu$ is a constant known as the *diffusion coefficient*.

The first thing you should notice is that this equation has a second-order derivative. We first need to learn what to do with it!

### Discretizing 2nd-order derivatives

The second-order derivative can be represented geometrically as the line tangent to the curve given by the first derivative.  We will discretize the second-order derivative with a Central Difference scheme: a combination of forward difference and backward difference of the first derivative.  Consider the Taylor expansion of $u_{i+1}$ and $u_{i-1}$ around $u_i$:

$$
u_{i+1} = u_i + \Delta x \frac{\partial u}{\partial x}\big|_i + \frac{\Delta x^2}{2!} \frac{\partial ^2 u}{\partial x^2}\big|_i + \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\big|_i + {\mathcal O}(\Delta x^4)
$$

$$
u_{i-1} = u_i - \Delta x \frac{\partial u}{\partial x}\big|_i + \frac{\Delta x^2}{2!} \frac{\partial ^2 u}{\partial x^2}\big|_i - \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\big|_i + {\mathcal O}(\Delta x^4)
$$

If we add these two expansions, the odd-numbered derivatives will cancel out.  Neglecting any terms of ${\mathcal O}(\Delta x^4)$ or higher (and really, those are very small), we can rearrange the sum of these two expansions to solve for the second-derivative.

$$
u_{i+1} + u_{i-1} = 2u_i+\Delta x^2 \frac{\partial ^2 u}{\partial x^2}\big|_i + {\mathcal O}(\Delta x^4)
$$

And finally:

$$
\begin{equation}
\frac{\partial ^2 u}{\partial x^2}=\frac{u_{i+1}-2u_{i}+u_{i-1}}{\Delta x^2} + {\mathcal O}(\Delta x^2)
\end{equation}
$$

The central difference approximation of the 2nd-order derivative is 2nd-order accurate.

### Back to diffusion

We can now write the discretized version of the diffusion equation in 1D:

$$
\begin{equation}
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\nu\frac{u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n}}{\Delta x^2}
\end{equation}
$$

As before, we notice that once we have an initial condition, the only unknown is $u_{i}^{n+1}$, so we re-arrange the equation to isolate this term:

$$
\begin{equation}
u_{i}^{n+1}=u_{i}^{n}+\frac{\nu\Delta t}{\Delta x^2}(u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n})
\end{equation}
$$

This discrete equation allows us to write a program that advances a solution in time—but we need an initial condition. Let's continue using our favorite: the hat function. So, at $t=0$, $u=2$ in the interval $0.5\le x\le 1$ and $u=1$ everywhere else.

### Stability of the diffusion equation

The diffusion equation is not free of stability constraints. Just like the linear and non-linear convection equations, there are a set of discretization parameters $\Delta x$ and $\Delta t$ that will make the numerical solution blow up. For the diffusion equation and the discretization used here, the stability condition for diffusion is

$$
\begin{equation}
\nu \frac{\Delta t}{\Delta x^2} \leq \frac{1}{2}
\end{equation}
$$

### And solve!

 We are ready to number-crunch!

The next two code cells initialize the problem by loading the needed libraries, then defining the solution parameters and initial condition. This time, we don't let the user choose just *any* $\Delta t$, though; we have decided this is not safe: people just like to blow things up. Instead, the code calculates a value of $\Delta t$ that will be in the stable range, according to the spatial discretization chosen! You can now experiment with different solution parameters to see how the numerical solution changes, but it won't blow up.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
# Set parameters.
nx = 41  # number spatial grid points
L = 2.0  # length of the domain
dx = L / (nx - 1)  # spatial grid size
nu = 0.3  # viscosity
sigma = 0.2  # CFL limit
dt = sigma * dx**2 / nu  # time-step size
nt = 20  # number of time steps to compute

# Get the grid point coordinates.
x = numpy.linspace(0.0, L, num=nx)

# Set the initial conditions.
u0 = numpy.ones(nx)
mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
u0[mask] = 2.0
```

```python
# Integrate in time.
u = u0.copy()
for n in range(nt):
    u[1:-1] = u[1:-1] + nu * dt / dx**2 * (u[2:] - 2 * u[1:-1] + u[:-2])
```

```python
# Plot the solution after nt time steps
# along with the initial conditions.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, label='Initial',
            color='C0', linestyle='--', linewidth=2)
pyplot.plot(x, u, label='nt = {}'.format(nt),
            color='C1', linestyle='-', linewidth=2)
pyplot.legend(loc='upper right')
pyplot.xlim(0.0, L)
pyplot.ylim(0.5, 2.5);
```

## Animations

Looking at before-and-after plots of the wave in motion is helpful, but it's even better if we can see it changing!

First, let's import the `animation` module of `matplotlib` as well as a special IPython display method called `HTML` (more on this in a bit).

##### Note

You will also have to install a video encoder/decoder named `ffmpeg`.

If you use Linux or OSX, you can install ffmpeg using conda:
```
conda install -c conda-forge ffmpeg
```

If you use Windows, installation instructions can be found [here](http://adaptivesamples.com/how-to-install-ffmpeg-on-windows/).

```python
from matplotlib import animation
from IPython.display import HTML
```

We are going to create an animation.
This takes a few steps, but it's actually not hard to do!

First, we define a function, called `diffusion`, that computes the numerical solution of the 1D diffusion equation over the time steps.
(The function returns a list with `nt` elements, each one being a Numpy array.)

```python
def diffusion(u0, sigma=0.5, nt=20):
    """
    Computes the numerical solution of the 1D diffusion equation
    over the time steps.

    Parameters
    ----------
    u0 : numpy.ndarray
        The initial conditions as a 1D array of floats.
    sigma : float, optional
        The value of nu * dt / dx^2;
        default: 0.5.
    nt : integer, optional
        The number of time steps to compute;
        default: 20.

    Returns
    -------
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    u_hist = [u0.copy()]
    u = u0.copy()
    for n in range(nt):
        u[1:-1] = u[1:-1] + sigma * (u[2:] - 2 * u[1:-1] + u[:-2])
        u_hist.append(u.copy())
    return u_hist
```

We now call the function to store the history of the solution:

```python
# Compute the history of the numerical solution.
u_hist = diffusion(u0, sigma=sigma, nt=nt)
```

Next, we create a Matplotlib figure that we want to animate.
For now, the figure contains the initial solution (our top-hat function).

```python
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
line = pyplot.plot(x, u0,
                   color='C0', linestyle='-', linewidth=2)[0]
pyplot.xlim(0.0, L)
pyplot.ylim(0.5, 2.5)
fig.tight_layout()
```

**Note**: `pyplot.plot()` can (optionally) return several values. Since we're only creating one line, we ask it for the "zeroth" (and only...) line by adding `[0]` after the `pyplot.plot()` call.

Now that our figure is initialized, we define a function `update_plot` to update the data of the line plot based on the time-step index.

```python
def update_plot(n, u_hist):
    """
    Update the line y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    u_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line.set_ydata(u_hist[n])
```

Next, we create an `animation.FuncAnimation` object with the following arguments:

* `fig`: the name of our figure,
* `diffusion`: the name of our solver function,
* `frames`: the number of frames to dra (which we set equal to `nt`),
* `fargs`: extra arguments to pass to the function `diffusion`,
* `interval`: the number of milliseconds each frame appears for.

```python
# Create an animation.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(u_hist,),
                               interval=100)
```

Ok! Time to display the animation.
We use the `HTML` display method that we imported above and the `to_html5_video` method of the animation object to make it web compatible.

```python
# Display the video.
HTML(anim.to_html5_video())
```

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>

```python

```

```python

```

```python

```


# Orthogonal polynomials

Copyright (C) 2020 Andreas Kloeckner

<details>
<summary>MIT License</summary>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
</details>

```python
import numpy as np
import numpy.linalg as la
import matplotlib.pyplot as pt
```

## Mini-Introduction to `sympy`

```python
import sympy as sym

# Enable "pretty-printing" in IPython
sym.init_printing()
```

Make a new `Symbol` and work with it:

```python

```

```python
myexpr = (x**2-3)**2
myexpr
myexpr.expand()
```

```python
sym.integrate(myexpr, x)
```

```python
sym.integrate(myexpr, (x, -1, 1))
```

## Orthogonal polynomials

Now write a function `inner_product(f, g)`:

```python

```

Show that it works:

```python

```

```python

```

Next, define a `basis` consisting of a few monomials:

```python

```

And run Gram-Schmidt on it:

```python
orth_basis = []

for q in basis:
    for prev_q in orth_basis:
        q = q - inner_product(prev_q, q)*prev_q / inner_product(prev_q,prev_q)
    orth_basis.append(q)

legendre_basis = [orth_basis[0],]

#to compute Legendre polynomials need to normalize so that q(1)=1 rather than ||q||=1
for q in orth_basis[1:]:
    q = q / q.subs(x,1)
    legendre_basis.append(q)
```

```python
legendre_basis
```

These are called the *Legendre polynomials*.

--------------------
What do they look like?

```python
mesh = np.linspace(-1, 1, 100)

pt.figure(figsize=(8,8))
for f in legendre_basis:
    f = sym.lambdify(x, f)
    pt.plot(mesh, [f(xi) for xi in mesh])
```

-----
These functions are important enough to be included in `scipy.special` as `eval_legendre`:

```python
import scipy.special as sps

for i in range(10):
    pt.plot(mesh, sps.eval_legendre(i, mesh))
```

What can we find out about the conditioning of the generalized Vandermonde matrix for Legendre polynomials?

```python
#keep
n = 20
xs = np.linspace(-1, 1, n)
V = np.array([
    sps.eval_legendre(i, xs)
    for i in range(n)
]).T

la.cond(V)
```

The Chebyshev basis can similarly be defined by Gram-Schmidt, but now with respect to a different inner-product weight function,
$$w(x) = 1/\sqrt{1-x^2}.$$

```python

```

```python
for i in range(10):
    pt.plot(mesh, np.cos(i*np.arccos(mesh)))
```

Chebyshev polynomials achieve similar good, but imperfect conditioning on a uniform grid (but perfect conditioning on a grid of Chebyshev nodes).

```python
#keep
n = 20
xs = np.linspace(-1, 1, n)
V = np.array([
    np.cos(i*np.arccos(xs))
    for i in range(n)
]).T

la.cond(V)
```

```python

```


# Assignment 1: Part 1 - kNN Basics
In this assignment you will implement a kNN model from scratch. The objectives are:
- To familiarise yourself with Python and some aspects of Numpy if you are not accustomed to it
- To gain some hands-on experience with a simple machine learning model
- Classify a set of test data

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
```

# Data exploration
We can use sklearn to generate random data. For now we will classify data with two input features that may belong to one of two classes.
The full dataset is split 50/50 into a training and test set.
For now we will not do any pre-processing on the data.

At a later stage we will look at real-world datasets and some of the problems that might be experienced with real-world data.

```python
X, Y = make_classification(n_samples=500, n_features=2, n_redundant=0, n_informative=1, n_classes=2, n_clusters_per_class=1)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.5)
```

It is very useful to know the shape of the data you are working with. Very often it serves as a quick way to debug parts of your code. If you are working with Python lists you can use the `len()` function, but you would have to use it for each dimension of your data. e.g.\
`dim1 = len(my_list)` ,
`dim2 = len(my_list[0])` ,
`dim3 = len(my_list([0][0])`\
Numpy provides easy access to the shape of n-dimensional arrays `ndarrays` e.g. \
`my_array.shape => (dim1, dim2, dim3)` \
You will notice that I have provided clues about the shape of the data for each function where necessary.

You can use the cell block below to examine the data however you wish. It will not be graded. I suggest confirming that the data shapes make sense. For example the data matix $\mathbf{X}$ would be given by
\begin{align}
\mathbf{X} &= \begin{bmatrix}
			\mathbf{x_1}^T \\
			\mathbf{x_2}^T \\
			\vdots \\
			\mathbf{x_n}^T
		\end{bmatrix}
		=
		\begin{bmatrix}
			x_{1}^{[1]} & x_{2}^{[1]}  & \dots  & x_{m}^{[1]} \\
			x_{1}^{[2]} & x_{2}^{[2]}  & \dots  & x_{m}^{[2]} \\
			\vdots & \vdots & \ddots & \vdots \\
			x_{1}^{[n]} & x_{2}^{[n]}  & \dots  & x_{m}^{[n]}
		\end{bmatrix}
\end{align}
Where there are $n$ number of examples and $m$ number of features

```python
# You can use this block to play around. n = number of examples, m = number of features => each row is an example
# Just make sure not to change the generated data as these are global variables.
# You can re-run the data generating block to create new data
Y_train.shape
```

    (250,)

# Data Visualisation
It's useful to visualise the data. Here it is easy because we only have two features. High-dimensional inputs need additional tools in order to help visualise them.

```python
fig = plt.figure(figsize=(6,6))
plt.scatter(X_train[:, 0], X_train[:, 1], c=Y_train, cmap='cool')
plt.ylabel('$x_2$')
plt.xlabel('$x_1$')
plt.show()
```

```python
# You can use this block play around
# notation: superscript = example, subscript = feature
x_a = [0, 1]
x_b = [0, 5]
```

# Task 1: Distance Function
Your first task is to develop a function that takes in two feature vectors (numpy arrays), $\mathbf{x}^{[a]}$, and $\mathbf{x}^{[b]}$ and ouputs the Euclidean distance between them

$$
d\left(\mathbf{x}^{[a]}, \mathbf{x}^{[b]}\right) = \sqrt{\sum\limits_{j=1}^m\left(x_j^{[a]} - x_j^{[b]}\right)^2}
$$

Some helpful functions:\
`np.sqrt()`, `np.sum()` and `np.pow()`

```python
def euclideanDistance(x_a, x_b):
    """
    Calculates the Euclidean distance between two vectors

    Arguments:
        x_a (array): shape [m_features, ] a single vector a
        x_b (array): shape [m_features, ] a single vector b

    Returns:
        distance (float): distance between vectors x_a and x_b
    """

    # YOUR CODE HERE
    # Each x_a, x_b are examples in D. They each have m features
    # Steps:
    # 1. find the difference between each corresponding element in x_a and x_b. Store in array
    # 2. square each of these values
    # 3. Sum all these elements together. And take the root

    distance = np.sum((x_a - x_b)**2)**(1/2)

    #raise NotImplementedError()
    return distance
```

```python
# Free cell
D = np.array([[1, 4], [2, 5], [2, 2], [3, 1], [4, 3], [5, 1]])
xt = np.array([3, 2])

dist_list = []

for example in D:
    dist_list.append(euclideanDistance(example, xt))

print(dist_list)
```

    [2.8284271247461903, 3.1622776601683795, 1.0, 1.0, 1.4142135623730951, 2.23606797749979]

```python
# The following tests are visible to you
x1_grade = np.array((-1.0, 2.0))
x2_grade = np.array((2.5, -2.0))
assert euclideanDistance(x1_grade, x2_grade) == 5.315072906367325

x1_grade = np.array((1.0, -1.0, 0))
x2_grade = np.array((-2.0, 2.6, 1.8))
assert euclideanDistance(x1_grade, x2_grade) == 5.019960159204453

```

# Task 2: Calculate list of distances
For the kNN algorithm you need to generate the distances between a single test example and all possible training examples.To do this you will need to write a function that takes in a test example and a list of examples, calculates the distance between the test example and each of the other example, and outputs a list of distances. The distances should be in the correct order as they correspond to a specific training example. To give you an idea what the output should be:

$$
\mathcal{distance\_list} = \left[d\left(\mathbf{x}^{[t]}, \mathbf{x}^{[1]}\right), \ldots, d\left(\mathbf{x}^{[t]}, \mathbf{x}^{[n]}\right)\right]
$$

where $d\left(\mathbf{x}^{[t]}, \mathbf{x}^{[1]}\right)$ is the distance function from task 1

The distances must be stored in a Python list, not a numpy array.

Some helpful functions:\
Python's built-in `append()` function

```python
def calculateDistances(x_test, X_in):
    """
    Calculates the distance between a single test example, x_test,
    and a list of examples X_in.

    Args:
        x_test (array): shape [n_features,] a single test example
        X_in (array): shape [n_samples, n_features] a list of examples to compare against.

    Returns:
        distance_list (list of float): The list containing the distances
    """

    distance_list = []
    for example in X_in:
        distance_list.append(euclideanDistance(example, x_test))
    return distance_list
```

```python
# Free cell
D = np.array([[1, 4], [2, 5], [2, 2], [3, 1], [4, 3], [5, 1]])
xt = np.array([3, 2])

dist_list = calculateDistances(xt, D)

print(dist_list)
```

    [2.8284271247461903, 3.1622776601683795, 1.0, 1.0, 1.4142135623730951, 2.23606797749979]

```python
# The following tests are visible to you
x1_grade = np.array((1.0, -1.0))
x2_grade = np.array([(2.0, -1.0),
              (-1.5, 2.5),
              (-2, -2),
              (0, 0)])

assert calculateDistances(x1_grade, x2_grade) == [1.0, 4.301162633521313, 3.1622776601683795, 1.4142135623730951]

```

# Task 3: Determine k Nearest Neighbours
This task is broken into subtasks that will create a set of the k nearest neighbours to a single test example.
$$
\mathcal{D}_k = \{(\mathbf{x}^{[i]}, \mathbf{y}^{[i]}), \ldots, (\mathbf{x}^{[k]}, \mathbf{y}^{[k]})\}
$$
## Task 3.1 Sorting the distances and returning indices
To find the k nearest neighbours you first need to sort the list of distances in ascending order. For a naive kNN we don't care about the actual distances, so we only need to know which examples are responsible for the k smallest distances. We can do this by sorting the list of distances while keeping track of the corresponding indices so that we can pull the examples from the original training data at those indices.

Your function must return a numpy array (it will make indexing the original dataset easier compared to python lists)

Some helpful functions:\
`np.argsort()`

```python
def kNearestIndices(distance_list, k):
    """
    Determines the indices of the k nearest neighbours

    Arguments:
        distance_list (list of float): list of distances between a test point
            and every training example
        k (int): the number of nearest neighbours to consider

    Returns:
        k_nearest_indices (array of int): shape [k,] array of the indices
            corresponding to the k nearest neighbours
    """

    # YOUR CODE HERE
    k_nearest_indices = np.array( np.argsort(distance_list)[:k] )
    return k_nearest_indices
```

```python
# Free cell
D = np.array([[1, 4], [2, 5], [2, 2], [3, 1], [4, 3], [5, 1]])
xt = np.array([3, 2])

dl = calculateDistances(xt, D)

kNearestIndices(dl, 3) + 1
```

    array([3, 4, 5])

```python
# The following tests are visible to you
distance_list_grade = [5.0, 3.5, 2.5, 1.0]
k_grade = 3
assert kNearestIndices(distance_list_grade, k_grade).tolist() == [3, 2, 1]

distance_list_grade = [5.0, 3.0, 3.5, 1.0, 10.0]
k_grade = 4
assert kNearestIndices(distance_list_grade, k_grade).tolist() == [3, 1, 2, 0]

```

## Task 3.2: Create $\mathcal{D}_k$
Now write a function that samples the original training set to produce the set of k nearest neighbours.
For now the function should return the `X_k` and `Y_k` data matrices seperately as indicated.

```python
def kNearestNeighbours(k_nearest_indices, X_in, Y_in):
    """
    Creates the dataset of k nearest neighbours

    Arguments:
        k_nearest_indices (array of int): shape [k,] array of the indices
            corresponding to the k nearest neighbours
        X_in (array): shape [n_examples, n_features] the example data matrix to sample from
        Y_in (array): shape [n_examples, ] the label data matrix to sample from

    Returns:
        X_k (array): shape [k, n_features] the k nearest examples
        Y_k (array): shape [k, ] the labels corresponding to the k nearest examples
    """

    X_k = []
    Y_k = []

    for i in k_nearest_indices:
        X_k.append(X_in[i])
        Y_k.append(Y_in[i])

    X_k = np.array(X_k)
    Y_k = np.array(Y_k)
    return X_k, Y_k
```

```python
# Free cell
D = np.array([([1, 4], 1), ([2, 5], 1), ([2, 2], 1), ([3, 1], 0), ([4, 3], 0), ([5, 1], 0)])
xt = np.array([3, 2])

X = []
Y = []
for p in D:
    X.append(p[0])
    Y.append(p[1])
X = np.array(X)
Y = np.array(Y)

dl = calculateDistances(xt, X)
kNN = kNearestIndices(dl, 3)

kNearestNeighbours(kNN, X, Y)
```

    (array([[2, 2],
            [3, 1],
            [4, 3]]),
     array([1, 0, 0]))

```python
# The following tests are visible to you

# dummy dataset for autograding purposes
X_train_grade = np.array([[1, 1],
                       [0, 2],
                       [1, 2],
                       [2, 2],
                       [10, 10],
                       [5, 10],
                       [6, 6],
                       [2, 3]])
Y_train_grade = np.array([0, 0, 0, 0, 1, 1, 1, 0])

X_k_grade, Y_k_grade = kNearestNeighbours([0, 1, 3], X_train_grade, Y_train_grade)
assert  np.equal(X_k_grade, np.array([[1, 1],[0, 2],[2, 2]])).all()
assert  np.equal(Y_k_grade, np.array([0, 0, 0])).all()

```

# Task 4: Predict Class
You can now write a function to predict the class of a test example by choosing the class the appears most frequently in ground truth labels for the k nearest neighbours i.e. the mode.
$$
h(\mathbf{x}^{[t]}) = mode(\{y^{[i]}, \ldots, y^{[k]}\})
$$

Some helpful functions:\
`mode()` function from scipy.

```python
from scipy.stats import mode
def predict(x_test, X_in, Y_in, k):
    """
    Predicts the class of a single test example

    Arguments:
        x_test (array): shape [n_features, ] the test example to classify
        X_in (array): shape [n_input_examples, n_features] the example data matrix to sample from
        Y_in (array): shape [n_input_labels, ] the label data matrix to sample from

    Returns:
        prediction (array): shape [1,] the number corresponding to the class
    """

    distance_list = calculateDistances(x_test, X_in)
    kNN_indices = kNearestIndices(distance_list, k)
    X_k, Y_k = kNearestNeighbours(kNN_indices, X_in, Y_in)
    prediction =  mode(Y_k, axis=None)[0]

    return prediction
```

```python
# Free cell
from scipy.stats import mode
k = 3
D = np.array([([1, 4], 1), ([2, 5], 1), ([2, 2], 1), ([3, 1], 0), ([4, 3], 0), ([5, 1], 0)])
xt = np.array([3, 2])

X = []
Y = []
for p in D:
    X.append(p[0])
    Y.append(p[1])
X = np.array(X)
Y = np.array(Y)

dl = calculateDistances(xt, X)
kNI = kNearestIndices(dl, k)

kNN = kNearestNeighbours(kNI, X, Y)
Dk = list(zip(kNN[0].tolist(), kNN[1].tolist()))

p = predict(xt, X, Y, k)
print( p == 0)
print(type(p))
```

    [ True]
    <class 'numpy.ndarray'>

```python
# The following tests are visible to you

# dummy dataset for autograding purposes
X_train_grade = np.array([[1, 1],
                       [0, 2],
                       [2, 1],
                       [1, 3],
                       [10, 10],
                       [5, 10],
                       [6, 8],
                       [2, 3]])
Y_train_grade = np.array([0, 0, 0, 0, 1, 1, 1, 0])

x1_grade = np.array([1, 2])
k_grade = 3
assert predict(x1_grade, X_train_grade, Y_train_grade, k_grade) == 0

x1_grade = np.array([6, 9])
k_grade = 2
assert predict(x1_grade, X_train_grade, Y_train_grade, k_grade) == 1

```

# Task 5: Predict for an entire batch of test examples
After you can successfully classify a single test example you need to repeat this across an entire batch of examples so we can apply performance metrics to assess the model.
\begin{align}
\hat{\mathbf{Y}}(\mathbf{X}_{test}) &= \{h(\mathbf{x}_{test}^{[i]}), \ldots, h(\mathbf{x}_{test}^{[j]})\} \\
&= \{\hat{y}^{[1]}, \ldots, \hat{y}^{[j]}\}
\end{align}

Sklearn and many other machine learning libraries will provide the data in terms of numpy arrays or similar (i.e. the tensors are commonly used by libraries such as tensorflow and pytorch and in most cases will work similarly to numpy arrays). If you take a slice of label data for example: \
Slicing the first 3 labels from Y, `Y[:3] => array([0, 1, 1])`. The shape of this would be (n_train_labels, ) *more generally (n_train_labels, n_output_features) if your output is a vector. If you had a python list it would appear as `Y[:3] => [0, 1, 1]`

You want to make sure that a batch of predicted outputs from your model matches the same form as a batch of labels for when you start making calculations such as accuracy. e.g, you want `y_hat.shape => (n_test_labels,)` to have the same shape as the lables `y.shape => (n_test_labels,)`.

Most libraries will have this as part of how they create their models, but when building your own algorithms you may have intermediate steps that do not produce it exactly like this. These kinds of errors regarding data shape and dimensions pop up very often in practise.

Some helpful functions:\
To combine multiple arrays: `np.concatenate()` *beware of axis when concatenating or for convenience `np.vstack()`
To change the shape of an array: `ndarray.reshape(shape)` or sometimes conventiently `ndarray.flatten()`

```python
def predictBatch(X_t, X_in, Y_in, k):
    """
    Performs predictions over a batch of test examples

    Arguments:
        X_t (array): shape [n_test_examples, n_features]
        X_in (array): shape [n_input_examples, n_features]
        Y_in (array): shape [n_input_labels, ]
        k (int): number of nearest neighbours to consider

    Returns:
        predictions (array): shape [n_test_examples,] the array of predictions

    """
    predictions = []
    for x_t_i in X_t:
        predictions.append(predict(x_t_i, X_in, Y_in, k)[0])

    return np.array(predictions)
```

```python
X_train_grade = np.array([[1, 1],
                       [0, 2],
                       [2, 1],
                       [1, 3],
                       [10, 10],
                       [5, 10],
                       [6, 8],
                       [2, 3]])
Y_train_grade = np.array([0, 0, 0, 0, 1, 1, 1, 0])

X_test_grade = np.array([[0, 0],
                [0, 1],
                [6, 10],
                [9, 8]])
Y_test_grade = np.array([0, 0, 1, 1])
k_grade=2

p = predictBatch(X_test_grade, X_train_grade, Y_train_grade, k=k_grade)
print(p.shape)
print(type(p))
```

    (4,)
    <class 'numpy.ndarray'>

```python
# The following tests are visible to you
# dummy dataset for grading purposes
X_train_grade = np.array([[1, 1],
                       [0, 2],
                       [2, 1],
                       [1, 3],
                       [10, 10],
                       [5, 10],
                       [6, 8],
                       [2, 3]])
Y_train_grade = np.array([0, 0, 0, 0, 1, 1, 1, 0])

X_test_grade = np.array([[0, 0],
                [0, 1],
                [6, 10],
                [9, 8]])
Y_test_grade = np.array([0, 0, 1, 1])
k_grade=1
assert np.equal(predictBatch(X_test_grade, X_train_grade, Y_train_grade, k=k_grade), Y_test_grade).all()
```

# Task 6: Accuracy metric
In this task you will create a function to measure the overall accuracy of your model.
$$
ACC = \frac{\# correct predictions}{\# total examples}
$$

```python
def accuracy(Y_pred, Y_test):
    """
    Calculates the accuracy of the model

    Arguments:
        Y_pred (array): shape [n_test_examples,] an array of model predictions
        Y_test (array): shape [n_test_labels,] an array of test labels to
            evaluate the predictions against

    Returns:
        accuracy (float): the accuracy of the model
    """
    assert(Y_pred.shape == Y_test.shape)

    correct = 0
    total = len(Y_test)

    for i in range(total):
        if (Y_pred[i] == Y_test[i]):
            correct += 1

    accuracy = correct/total
    return accuracy

```

```python
# Free cell
Yt = np.array([0, 1, 0, 1])
Yp = np.array([1, 0, 1, 1])

accuracy(Yp, Yt)
```

    0.25

```python
# The following tests are visible to you

Y_test_grade = np.array([0, 1, 0, 0])
Y_pred_grade = np.array([0, 1, 1, 0])

assert accuracy(Y_pred_grade, Y_test_grade) == 0.75

Y_pred_grade = np.array([1, 0, 0, 0])
assert accuracy(Y_pred_grade, Y_test_grade) == 0.5

```

# Task 7: Test your model
Now you can combine the rest of the functions you've built into on function run your model with the generated training and test to data evaluate your model. This is really just to make running multiple tests more convenient.

```python
def run(X_train, X_test, Y_train, Y_test, k):
    """
    Evaluates the model on the test data

    Arguments:
        X_train (array): shape [n_train_examples, n_features]
        X_test (array): shape [n_test_examples, n_features]
        Y_train (array): shape [n_train_examples, ]
        Y_test (array): shape [n_test_examples, ]
        k (int): number of nearest neighbours to consider

    Returns:
        test_accuracy (float): the final accuracy of your model
    """
    Y_pred = predictBatch(X_test, X_train, Y_train, k)
    test_accuracy = accuracy(Y_pred, Y_test)

    return test_accuracy
```

```python
X_train_grade = np.array([[1, 1],
                       [0, 2],
                       [2, 1],
                       [1, 3],
                       [10, 10],
                       [5, 10],
                       [6, 8],
                       [2, 3]])
Y_train_grade = np.array([0, 0, 0, 0, 1, 1, 1, 0])

X_test_grade = np.array([[0, 0],
                [0, 1],
                [8, 3],
                [6, 10],
                [9, 8],
                [2, 9]])
Y_test_grade = np.array([0, 0, 0, 1, 1, 1])
k_grade=1 #Outlier in i=3, using k=2 improves

print( run(X_train_grade, X_test_grade, Y_train_grade, Y_test_grade, k_grade) )
```

    0.8333333333333334

```python
# I can't show you an example test because it would give away the answer
```

# End of Part 1
That concludes this part of assignment 1.


# Euler Lagrange - pendulum with  oscillating support

We define Lagrange function as a difference between kinetic and potential energy:

\begin{equation}
\label{eq:Lagangian}
L = E_k - E_p
\end{equation}

Then equation of motion are given by:

\begin{equation}
\label{eq:EL}
\frac{d}{dt} \left (\frac{\partial L}{\partial \dot{\varphi}}\right ) - \frac{\partial L}{\partial \varphi} = 0
\end{equation}

Since this formulation in invariant with respect to the change of system off coordinates, we can use it for many problems in mechanics with constraints.

## System definition

Let us define a system,

```python
load('cas_utils.sage')
```

```python
var('l g w0')
xy_wsp = ['x','y']
uv_wsp = [('phi',r'\varphi')]

to_fun, to_var = make_symbols(xy_wsp, uv_wsp)
```

### Horizontal oscillations of a support point

We parametrize the system similarily to mathematical pendulum,

\begin{eqnarray}
\label{eq:parametic}
x = a \sin\left(\omega t\right) + l \sin\left({\varphi}\right) \\
y =  -l \cos\left({\varphi}\right)
\end{eqnarray}

```python
# horizontal
var('a omega t')
x2u = {x:l*sin(phi)+a*sin(omega*t),y:-l*cos(phi)}
showmath(x2u)
```

**Step 1: Kinetic energy**

We have to write kinetic energy in terms of generalized coordinates:

\begin{equation}
\label{eq:Ekin}
    E_k = \frac{1}{2}(\dot x^2 + \dot y^2)
\end{equation}

Using transformation dictionary, to generalized coordinates we have $E_k(\varphi,,\dot\varphi)$:

```python
Ek = 1/2*sum([x_.subs(x2u).subs(to_fun).diff(t).subs(to_var)^2 for x_ in [x,y]])
Ek = Ek.trig_simplify()
showmath(Ek)
```

**Step 2: Potential energy**

Similarily we have to express potential energy:

\begin{equation}
\label{eq:Ekin}
   E_p = g y
\end{equation}

as a function of $E_P(\varphi)$

```python
Ep = g*y.subs(x2u)
showmath(Ep)
```

**Step 3: Langranian**

Now we have Lagrangian $L(\varphi,\dot\varphi)$:

```python
L = Ek - Ep
showmath(L)
```

## Derivation of equations of motion

Using Euler-Lagrange formulas \ref{eq:EL} we write equation of motion in generalized coordinate $\varphi$. Note, we can differentiate over $\varphi$ and $\dot\varphi$. However to perform time derivative  we first replace symbols representing  variables with functions (i.e. Sage symbolic functions) of time. We have ``to_fun`` dictionary which automatizes this step. Then we use symbolic differenctiation ``diff``. After this operation we bring thhe result back to symbolic variable $\varphi$ and $\dot\varphi$ with ``to_var`` dictionary.

```python
EL1 = L.diff(phid).subs(to_fun).diff(t).subs(to_var) - L.diff(phi)
```

```python
showmath(EL1)
```

## Analysis

### Small angle approximation

Let see what happens if oscillations are small. We can expand in Taylor seried the equations of motion:

```python
eq_lin = EL1.taylor(phi,0,1)
showmath(eq_lin)
```

```python
var('alpha,omega0')
eq_lin2  = (eq_lin/l^2).expand().subs({a:l*alpha,g:l*omega0^2})
showmath(eq_lin2)
```

We see that the equations are essentially equivalent to forced harmonic oscillator. The difference might be that the effective amplitude of forcing depends on forcing frequency.

```python
assume(g>0)
assume(omega0>0)
phi_anal = desolve((eq_lin/l^2).expand()\
                   .subs({a:l*alpha,g:l*omega0^2})\
                   .subs(to_fun).subs({l:1}),\
                   dvar=Phi,ivar=t,contrib_ode=True)
showmath(phi_anal)
```

```python
showmath(eq_lin2)
```

### Numerical  integration

We can numerically compare if the linear approximation works for selected initial conditions and parameters. For this purpose we need to solve Euler-Lagrange equation for $\ddot\varphi$, and for following system od 1st order ODEs:

\begin{eqnarray}
\label{eq:ode}
\frac{d\varphi}{dt} &=& \dot\varphi\\
\frac{d\dot\varphi}{dt} &=& \frac{a \omega^{2} \cos\left({\varphi}\right) \sin\left(\omega t\right)}{l} - \frac{g \sin\left({\varphi}\right)}{l}
\end{eqnarray}

Note that we threat $\varphi$ and $\dot\varphi$ as independent variables. Since in Sage we use formulas where there are represented by different symbolic variables: `phi` and `dphi`, there will be no confusion of "dot" and derivative operator.

```python
rhs = EL1.solve(phidd)[0].rhs()
showmath(rhs().expand())
```

Linear system can be derived in similar way:

```python
rhs_lin = eq_lin.solve(phidd)[0].rhs()
showmath(rhs_lin)
```

```python
pars = {l:1,g:1,a:.03,omega:1.31}
t_end = 60
w0 = sqrt(g/l).subs(pars)
```

Now we can plug the system of ODE into ``desolve_odeint`` solver:

```python
ode = [phid, rhs.subs(pars)]
times = srange(0,t_end,0.1)
ics = [0.0, 0.1]
sol = desolve_odeint(ode, ics, times, [phi, phid])
line( zip(times,sol[::1,0]),figsize=(6,2), )
```

```python
ode_lin = [phid, rhs_lin.subs(pars)]
times = srange(0,t_end,0.1)
ics = [0.0, 0.1]
sol_lin = desolve_odeint(ode_lin, ics, times, [phi,phid])

line( zip(times[0:],sol[0:,0]),figsize=(6,2) )\
 +line( zip(times[0:],sol_lin[0:,0]),color='red')

```

We see that for small oscillations the for some time. Then they diverge. One can experiment and simulate both systems for longer times to see that the divergence grows. Also larger amplitudes of driving will make them differ significantly.

### Vertical oscillations

In the case of vertical oscillations of a support point, the transformation to generalized coordinates reads:

\begin{eqnarray}
\label{eq:oscil_vert}
x = l \sin\left({\varphi}\right)\\
y= -a \cos\left(\omega t\right) - l \cos\left({\varphi}\right),
\end{eqnarray}

```python
# vertical
var('a omega t')
x2u = {x:l*sin(phi), y:-l*cos(phi)-a*cos(omega*t)}
showmath(x2u)
```

Let us once again calculate Lagrangian and derive symbolically equations of motion:

```python
Ek = 1/2*sum([x_.subs(x2u).subs(to_fun).diff(t).subs(to_var)^2 for x_ in [x,y]])
Ek = Ek.trig_simplify()
Ep = g*y.subs(x2u)
L = Ek - Ep
EL1 = L.diff(phid).subs(to_fun).diff(t).subs(to_var) - L.diff(phi)
showmath(EL1)
```

```python
rhs = EL1.solve(phidd)[0].rhs()
showmath( rhs.expand().collect(sin(phi)) )
```

Let's try to obtain a linear approximation for small $\varphi$, as in previous case:

```python
showmath(EL1.taylor(phi,0,1) )
```

We see that in this case we do not obtain a harmonic oscillator. Forcing term is multiplied by $\varphi$, i.e. the linearized equation is fundamentally different.

### Stable inverted pendulum

Vertical driving of a support point as a remarkable property - under some conditions the upper steady state can become a stable one.

For example for  following parameters and initial condition:

```python
pars = {l:1,a:.2,g:1,omega:10.}
w0 = sqrt(g/l).subs(pars)

ode = [phid, rhs.subs(pars) ]
times = srange(0, 60, 0.1)
ics = [pi-1e-1, .0]

sol = desolve_odeint(ode, ics, times, [phi,phid])

#plt = line( zip(times,sol[::1,0]),figsize=(8,3), ticks=[None,pi/4],\
#     tick_formatter=[None,pi],gridlines=[[],[pi.n()*i for i in range(-100,100,1)]])
plt = line( zip(times,sol[::1,0]),figsize=(6,2) )

plt.show()
```

We observe that for above inital conditions the pendulum oscillates around **$x=\pi$ state** which is normally unstable fix point.

```python
pendulum = [vector([x,y]).subs(x2u).subs(pars).subs({phi:phi_,t:t_})\
            for t_,phi_ in zip(times,sol[:,0])]
o_point = [vector([x,y]).subs(x2u).subs(l==0).subs(pars).subs(t==t_)\
           for t_ in times ]
```

```python
#@interact
def draw_pendulum(ith = slider(0, len(pendulum)-1,1)):
    p1,p2 = pendulum[ith], o_point[ith]
    plt = line( [p1,p2],xmin=-1, xmax=1, ymin=-1.4, ymax=1.4,\
               aspect_ratio=1,figsize=2,axes=False, title='t=%0.2f'%times[ith])
    plt += points([p1,p2],color='red',size=30,gridlines=[None,[0]],\
                 figsize=3,axes=False)
    plt += line(pendulum[:ith],thickness=0.9,color='gray',zorder=-10)
    return plt
```

```python
#draw_pendulum(1200).save('inverted_pend.png',figsize=8)
```

Time evolution of the inverted stable pendulum:

### System with damping

Adding damping to the system will make inverted state an stable atractor.

$$\ddot\varphi  = -2\gamma \dot\varphi + ( -\omega_0^2 - \frac{a}{l} \omega^2 \cos(\omega t))\sin(\varphi)$$

```python
var('omega,omega0,gama,t,a')
pars = {l:1,a:0.152,omega0:1,omega:14.,gama:.1}
ode = [phid,\
       (-2*gama*phid+(-omega0^2-a/l*omega^2*cos(omega*t))*sin(phi)).subs(pars)]
times = srange(0,30,0.01)
ics = [0,2.1]
sol = desolve_odeint(ode,ics,times,[phi,phid])
line( zip(times,sol[::1,0]),figsize=(8,2), ticks=[None,pi/4],\
     tick_formatter=[None,pi],gridlines=[[],[pi.n()*i for i in range(-100,100,1)]])
```

\newpage


---
title: "The Monte Carlo Approach"
summary: "Simulation is easier than algebra"
date: 2020-05-10
source: jupyter
---

In *Monte Carlo* approaches, we use random simulations to answer questions
that might otherwise require some difficult equations.
Confusingly, they're also known in some fields
as *numerical* approaches, and are contrasted with *analytic* approaches,
where you just work out the correct equation.
[Wikipedia tells us](https://en.wikipedia.org/wiki/Monte_Carlo_method#History) that,
yes, Monte Carlo methods are named after the casino.

The best-known Monte Carlo method is
[Markov Chain Monte Carlo](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo),
which comes up a lot in Bayesian statistics.
In this post, I cover a much simpler example.
Here's a simple Monte Carlo example.
Let's say you want to know the area of a circle
with a radius of $r$.
We'll use a unit circle, $r=1$, in this example.

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams
import seaborn as sns
sns.set_style('whitegrid')
rcParams['figure.figsize'] = (6, 4)
rcParams['font.size'] = 18

def circle_plot():
    fig, ax = plt.subplots(figsize=(5, 5))
    plt.hlines([-1, 1], -1, 1)
    plt.vlines([-1, 1], -1, 1)
    plt.plot([0, 1], [0, 0], color='k')
    plt.scatter(0, 0, marker='+', color='k')
    plt.xlim(-1.05, 1.05)
    plt.ylim(-1.05, 1.05)
    circle = plt.Circle((0, 0), 1, facecolor='None', edgecolor='r')
    ax.add_artist(circle)
    return fig, ax

circle_plot();
```

Analytically, you know that the answer is

$$\text{Area} = \pi r^2$$

What if we didn't know this equation?
The Monte Carlo solution is as follows.
We know that the area of the bounding square is $2r \times 2r = 4r^2$
We need to figure out what proportion of this square is taken up by the circle.
To find out, we randomly select a large number of points in the square,
and check if they're within $r$ of the center point $[0, 0]$.

```python
n = 1000 # Number of points to simulate
x = np.random.uniform(low=-1, high=1, size=n)
y = np.random.uniform(low=-1, high=1, size=n)
# Distance from center (Pythagoras)
dist_from_origin = np.sqrt(x**2 + y**2)
# Check is distance is less than radius
is_in_circle = dist_from_origin < 1

# Plot results
circle_plot()
plt.scatter(x[is_in_circle], y[is_in_circle], color='b', s=2)     # Points in circle
plt.scatter(x[~is_in_circle], y[~is_in_circle], color='k', s=2);  # Points outside circle

m = is_in_circle.mean()
print('%.4f of points are in the circle' % m)
```

Since the area of the square is $4r^2$,
and the circle takes up ~$0.78$ of the square,
the area of the circle is roughly

$$
\begin{align}
\text{Area}
 &\approx 0.78 \times 4r^2 \newline
 &= 3.14r^2 \newline
 &\approx \pi r^2
\end{align}
$$

We've discovered $\pi$.


# Part 1 - Scalars and Vectors

For the questions below it is not sufficient to simply provide answer to the questions, but you must solve the problems and show your work using python (the NumPy library will help a lot!) Translate the vectors and matrices into their appropriate python  representations and use numpy or functions that you write yourself to demonstrate the result or property.

```
# importing the libraries used in class

import matplotlib.pyplot as plt
import math
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
```

## 1.1 Create a two-dimensional vector and plot it on a graph

```
# Done in class
# plt.arrow(x,y,dx,dy) draws an arrow from (x,y) to (dx,dy)
# xlim and ylim is range of the borders of the graph, title sets the title

twod_vector = [0.5,0.5]
plt.arrow(0,0,twod_vector[0],twod_vector[1],head_width=0.02,head_length=0.02,color="green")
plt.xlim(0,1)
plt.ylim(0,1)
plt.title("Two Dimensional Vector")
plt.show()
```

## 1.2 Create a three-dimensional vecor and plot it on a graph

```
# Done in class

vector_3d = [0.6, 0.8, 0.3]
vector_3d_v = np.array([[0,0,0,0.6,0.8,0.3]])

# From what I understand, quiver creates a vector from point (X,Y,Z) to (U,V,W) so in this case the shown vector is from (0,0,0) to (0.6,0.8,0.3)
# The line below makes X, Y, Z, U, V, W into 1 element tuples with the corresponding value in the array, don't know why "X, Y, Z, U, V, W = 0,0,0,0.6,0.8,0.3" wasn't used instead
X, Y, Z, U, V, W = zip((*vector_3d_v))
fig = plt.figure()
ax = fig.add_subplot(111,projection='3d')
ax.quiver(X,Y,Z,U,V,W,length=1)
ax.set_xlim([0,1])
ax.set_ylim([0,1])
ax.set_zlim([0,1])
ax.set_xlabel("X")
ax.set_ylabel("Y")
ax.set_zlabel("Z")
plt.show()
```

```
# First 3 lines of the above code block is replaced by the 1st line in this code block, same output
X, Y, Z, U, V, W = 0,0,0,0.6,0.8,0.3
fig = plt.figure()
ax = fig.add_subplot(111,projection='3d')
ax.quiver(X, Y, Z, U, V, W,length=1)
ax.set_xlim([0,1])
ax.set_ylim([0,1])
ax.set_zlim([0,1])
ax.set_xlabel("X")
ax.set_ylabel("Y")
ax.set_zlabel("Z")
plt.show()
```

## 1.3 Scale the vectors you created in 1.1 by $5$, $\pi$, and $-e$ and plot all four vectors (original + 3 scaled vectors) on a graph. What do you notice about these vectors?

```
# I use the math library to get e and pi.
# I need to plot them from biggest to smallest so the smallest ones don't get covered.
# These vectors are all in the same direction. Each of these vectors can be multiplied by a factor to be equal to any of these vectors.

plt.arrow(0,0,5*twod_vector[0],5*twod_vector[1],head_width=0.02,head_length=0.02,color="green")
plt.arrow(0,0,math.pi*twod_vector[0],math.pi*twod_vector[1],head_width=0.02,head_length=0.02,color="red")
plt.arrow(0,0,twod_vector[0],twod_vector[1],head_width=0.02,head_length=0.02,color="blue")
plt.arrow(0,0,-math.e*twod_vector[0],-math.e*twod_vector[1],head_width=0.02,head_length=0.02,color="yellow")
plt.xlim(-3,5)
plt.ylim(-3,5)
plt.title("Scaled Vectors")
plt.show()
```

## 1.4 Graph vectors $\vec{a}$ and $\vec{b}$ and plot them on a graph

\begin{align}
\vec{a} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}
\qquad
\vec{b} = \begin{bmatrix} 3 \\4 \end{bmatrix}
\end{align}

```
# Similar to 1.1 except the vectors are given

a = [5,7]
b = [3,4]
plt.arrow(0,0,a[0],a[1],head_width=0.02,head_length=0.1,color="red",width=0.05)
plt.arrow(0,0,b[0],b[1],head_width=0.02,head_length=0.1,color="blue",width=0.05)
plt.xlim(0,8)
plt.ylim(0,8)
```

## 1.5 find $\vec{a} - \vec{b}$ and plot the result on the same graph as $\vec{a}$ and $\vec{b}$. Is there a relationship between vectors $\vec{a} \thinspace, \vec{b} \thinspace \text{and} \thinspace \vec{a-b}$

```
# The relationship between a,b, and a-b is that a-b+b = a. a-b = [2,3], if I add b to it, I'll get: a-b+b = [2+3,3+4] = [5,7] which is a
# Also, (a-b)+b = b+(a-b), they all end up at the same place [5,7]
plt.arrow(0,0,a[0],a[1],head_width=0.02,head_length=0.1,color="red",width=0.05)
plt.arrow(0,0,b[0],b[1],head_width=0.02,head_length=0.1,color="blue",width=0.05)
plt.arrow(0,0,a[0]-b[0],a[1]-b[1],head_width=0.02,head_length=0.1,color="purple",width=0.05)
plt.xlim(0,8)
plt.ylim(0,8)
plt.show()
```

## 1.6 Find $c \cdot d$

\begin{align}
\vec{c} = \begin{bmatrix}7 & 22 & 4 & 16\end{bmatrix}
\qquad
\vec{d} = \begin{bmatrix}12 & 6 & 2 & 9\end{bmatrix}
\end{align}

```
# dot product is the sum of the products of the corresponding entries

def dot(x,y): # x and y are lists with the same length, representing vectors
  Dot = 0
  for n in range(len(x)):
    Dot+=x[n]*y[n]
  return Dot
```

```
dot([7,22,4,16],[12,6,2,9])
```

    368

```
#np.dot returns the dot product of 2 arrays

c = np.array([7,22,4,16])
d = np.array([12,6,2,9])

np.dot(c,d)
```

    368

##  1.7 Find $e \times f$

\begin{align}
\vec{e} = \begin{bmatrix} 5 \\ 7 \\ 2 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\4 \\ 6 \end{bmatrix}
\end{align}

```
# Cross Product
# if p = [a,b,c] and q = [x,y,z] p cross q would be [bz-cy,-(az-cx),ay-bx]
# these components are the determinants of the 2x2 matrix that excludes a column, for example: bz - cy is the determinant of the matrix below, it excludes the 1st column (a z)
# b c
# y z
# so if e = [5,7,2] and f = [3,4,6], e cross f would be [(7)(6)-(2)(4),-((5)(6)-(3)(2)), (5)(4)-(3)(7)] = [42 - 8, -(30 - 6), 20-21] = [34,-24,1]
```

```
def cross(x,y): # x and y are vectors with 3 dimensions (lists with 3 numbers) in this instance
  a = x[1]*y[2] - x[2]*y[1]
  b = -(x[0]*y[2] - y[0]*x[2])
  c = x[0]*y[1] - x[1]*y[0]
  return [a,b,c]
```

```
cross([5,7,2],[3,4,6])
```

    [34, -24, -1]

```
# np.cross returns the cross product of 2 vectors

e = np.array([5,7,2])
f = np.array([3,4,6])

np.cross(e,f)
```

    array([ 34, -24,  -1])

## 1.8 Find $||g||$ and then find $||h||$. Which is longer?

\begin{align}
\vec{g} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 8 \end{bmatrix}
\qquad
\vec{h} = \begin{bmatrix} 3 \\3 \\ 3 \\ 3 \end{bmatrix}
\end{align}

```
# ||x|| is the sqrt of the sum of squares of the values
# ||g|| = sqrt(1^2 + 1^2 + 1^2 + 8^2) = sqrt(1+1+1+64) = sqrt(67)
```

```
def mag(vec): #vec is vector which is a list
  leng = 0
  for n in vec:
    leng+=n**2
  return leng**0.5
```

```
print("||g|| is", mag([1,1,1,8])) #sqrt(67)
print("||h|| is", mag([3,3,3,3])) #sqrt(36)

```

    ||g|| is 8.18535277187245
    ||h|| is 6.0

```
# np.linalg.norm can also be used to figure out ||x||

g = np.array([1,1,1,8])
h = np.array([3,3,3,3])

print("||g|| is", np.linalg.norm(g))
print("||h|| is", np.linalg.norm(h))
```

    ||g|| is 8.18535277187245
    ||h|| is 6.0

# Part 2 - Matrices

## 2.1 What are the dimensions of the following matrices? Which of the following can be multiplied together? See if you can find all of the different legal combinations.
\begin{align}
A = \begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
\qquad
B = \begin{bmatrix}
2 & 4 & 6 \\
\end{bmatrix}
\qquad
C = \begin{bmatrix}
9 & 6 & 3 \\
4 & 7 & 11
\end{bmatrix}
\qquad
D = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\qquad
E = \begin{bmatrix}
1 & 3 \\
5 & 7
\end{bmatrix}
\end{align}

```
# to multiply matrices, with dimensions rows by columns, the columns of 1st matrix = rows of 2nd matrix
# and the product has rows of 1st matrix by columns of 2nd matrix
# m*n matrix times n*p matrix equals m*p matrix

# Purpose of the code below is to answer the question "Which of the following [matrices] can be multiplied together?"
# Value is set to dimensions of corresponding Key matrix
# Using dictionary so I can print out the letters.

Dimdict = {"A":[3,2],
           "B":[1,3],
           "C":[2,3],
           "D":[3,3],
           "E":[2,2]}

# testings all 25 combinations, x[0] is rows, x[1] is columns,

for m in Dimdict:
  for n in Dimdict:
    if Dimdict[m][1]==Dimdict[n][0]:
      print(f'{m}{n} is a legal combination. The dimensions are {Dimdict[m]} * {Dimdict[n]}.')

```

    AC is a legal combination. The dimensions are [3, 2] * [2, 3].
    AE is a legal combination. The dimensions are [3, 2] * [2, 2].
    BA is a legal combination. The dimensions are [1, 3] * [3, 2].
    BD is a legal combination. The dimensions are [1, 3] * [3, 3].
    CA is a legal combination. The dimensions are [2, 3] * [3, 2].
    CD is a legal combination. The dimensions are [2, 3] * [3, 3].
    DA is a legal combination. The dimensions are [3, 3] * [3, 2].
    DD is a legal combination. The dimensions are [3, 3] * [3, 3].
    EC is a legal combination. The dimensions are [2, 2] * [2, 3].
    EE is a legal combination. The dimensions are [2, 2] * [2, 2].

## 2.2 Find the following products: CD, AE, and BA. What are the dimensions of the resulting matrices? How does that relate to the dimensions of their factor matrices?

```

```

## 2.3  Find $F^{T}$. How are the numbers along the main diagonal (top left to bottom right) of the original matrix and its transpose related? What are the dimensions of $F$? What are the dimensions of $F^{T}$?

\begin{align}
F =
\begin{bmatrix}
20 & 19 & 18 & 17 \\
16 & 15 & 14 & 13 \\
12 & 11 & 10 & 9 \\
8 & 7 & 6 & 5 \\
4 & 3 & 2 & 1
\end{bmatrix}
\end{align}

# Part 3 - Square Matrices

## 3.1 Find $IG$ (be sure to show your work) 😃

You don't have to do anything crazy complicated here to show your work, just create the G matrix as specified below, and a corresponding 2x2 Identity matrix and then multiply them together to show the result. You don't need to write LaTeX or anything like that (unless you want to).

\begin{align}
G=
\begin{bmatrix}
13 & 14 \\
21 & 12
\end{bmatrix}
\end{align}

## 3.2 Find $|H|$ and then find $|J|$.

\begin{align}
H=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\qquad
J=
\begin{bmatrix}
0 & 1 & 2 \\
7 & 10 & 4 \\
3 & 2 & 0
\end{bmatrix}
\end{align}

## 3.3 Find $H^{-1}$ and then find $J^{-1}$

## 3.4 Find $HH^{-1}$ and then find $J^{-1}J$. Is $HH^{-1} == J^{-1}J$? Why or Why not?

Please ignore Python rounding errors. If necessary, format your output so that it rounds to 5 significant digits (the fifth decimal place).

# Go Beyond:

A reminder that these challenges are optional. If you finish your work quickly we welcome you to work on them. If there are other activities that you feel like will help your understanding of the above topics more, feel free to work on that. Topics from the Stretch Goals sections will never end up on Sprint Challenges. You don't have to do these in order, you don't have to do all of them.

- Write a function that can calculate the dot product of any two vectors of equal length that are passed to it.
- Write a function that can calculate the norm of any vector
- Prove to yourself again that the vectors in 1.9 are orthogonal by graphing them.
- Research how to plot a 3d graph with animations so that you can make the graph rotate (this will be easier in a local notebook than in google colab)
- Create and plot a matrix on a 2d graph.
- Create and plot a matrix on a 3d graph.
- Plot two vectors that are not collinear on a 2d graph. Calculate the determinant of the 2x2 matrix that these vectors form. How does this determinant relate to the graphical interpretation of the vectors?


# Gaussian Distribution (Normal or Bell Curve)

Think of a Jupyter Notebook file as a Python script, but with comments given the seriousness they deserve, meaning inserted Youtubes if necessary.  We also adopt a more conversational style with the reader, and with Python, pausing frequently to take stock, because we're telling a story.

One might ask, what is the benefit of computer programs if we read through them this slowly?  Isn't the whole point that they run blazingly fast, and nobody needs to read them except those tasked with maintaining them, the programmer cast?

First, lets point out the obvious:  even when reading slowly, we're not keeping Python from doing its part as fast as it can, and what it does would have taken a single human ages to do, and would have occupied a team of secretaries for ages.  Were you planning to pay them?  Python effectively puts a huge staff at your disposal, ready to do your bidding.  But that doesn't let you off the hook.  They need to be managed, told what to do.

Here's what you'll find at the top of your average script.  A litany of players, a congress of agents, need to be assembled and made ready for the job at hand.  But don't worry, as you remember to include necessary assets, add them at will as you need them.  We rehearse the script over and over while building it.  Nobody groans, except maybe you, when the director says "take it from the top" once again.

```python
import numpy as np
import scipy.stats as st
import matplotlib.pyplot as plt
import math
```

You'll be glad to have np.linspace as a friend, as so often you know exactly what the upper and lower bounds, of a domain, might be.  You'll be computing a range.  Do you remember these terms from high school?  A domain is like a pile of cannon balls that we feed to our cannon, which them fires them, testing our knowledge of ballistics.  It traces a parabola.  We plot that in our tables.  A lot of mathematics traces to developing tables for battle field use.  Leonardo da Vinci, a great artist, was also an architect of defensive fortifications.

Anyway, np.linspace lets to give exactly the number of points you would like of this linear one dimensional array space, as a closed set, meaning -5 and 5 are included, the minimum and maximum you specify.  Ask for a healthy number of points, as points are cheap.  All they require is memory.  But then it's up to you not to overdo things.  Why waste CPU cycles on way too many points?

I bring up this niggling detail about points as a way of introducing what they're calling "hyperparameters" in Machine Learning, meaning settings or values that come from outside the data, so also "metadata" in some ways.  You'll see in other notebooks how we might pick a few hyperparameters and ask scikit-learn to try all combinations of same.

Here's what you'll be saying then:

from sklearn.model_selection import GridSearchCV  #CV = cross-validation

```python
domain = np.linspace(-5, 5, 100)
```

I know mu sounds like "mew", the sound a kitten makes, and that's sometimes insisted upon by sticklers, for when we have a continuous function, versus one that's discrete.  Statisticians make a big deal about the difference between digital and analog, where the former is seen as a "sampling" of the latter.  Complete data may be an impossibility.  We're always stuck with something digital trying to approximate something analog, or so it seems.  Turn that around in your head sometimes:  we smooth it over as an approximation, because a discrete treatment would require too high a level of precision.

The sticklers say "mu" for continuous, but "x-bar" (an x with a bar over it) for plain old "average" of discrete sets.  I don't see this conventions holding water necessarily, for one thing because it's inconvenient to always reach for the most fancy typography.  Python does have full access to Unicode, and to LaTex, but do we have to bother?  Lets leave that question for another day and move on to...

## The Guassian (Binomial if Discrete)

```python
mu = 0  # might be x-bar if discrete
sigma = 1  # standard deviation, more below
```

What we have here (below) is a typical Python numeric function, although it does get its pi from numpy instead of math.  That won't matter.  The sigma and mu in this function are globals and set above.  Some LaTex would be in order here, I realize.  Let me scavange the internet for something appropriate...

$pdf(x,\mu,\sigma) = \frac{1}{ \sigma \sqrt{2 \pi}} e^{\left(-\frac{{\left(\mu - x\right)}^{2}}{2 \, \sigma^{2}}\right)}$

Use of dollar signs is key.

Here's another way, in a code cell instead of a Markdown cell.

```python
from IPython.display import display, Latex

ltx = '$ pdf(x,\\mu,\\sigma) = \\frac{1}{ \\sigma' + \
 '\\sqrt{2 \\pi}} e^{\\left(-\\frac{{\\left(\\mu - ' + \
 'x\\right)}^{2}}{2 \\, \\sigma^{2}}\\right)} $'
display(Latex(ltx))
```

$ pdf(x,\mu,\sigma) = \frac{1}{ \sigma\sqrt{2 \pi}} e^{\left(-\frac{{\left(\mu - x\right)}^{2}}{2 \, \sigma^{2}}\right)} $

I'm really tempted to try out [PrettyPy](https://github.com/charliekawczynski/prettyPy).

```python
def g(x):
    return (1/(sigma * math.sqrt(2 * np.pi))) * math.exp(-0.5 * ((mu - x)/sigma)**2)
```

What I do below is semi-mysterious, and something I'd like to get to in numpy in more detail.  The whole idea behind numpy is every function, or at least the unary ones, are vectorized, meaning the work element-wise through every cell, with no need for any for loops.

My Gaussian formula above won't natively understand how to have relations with a numpy array, unless we store it in vectorized form.  I'm not claiming this will make it run any faster than under the control of for loops, we can test that.  Even without a speedup, here we have a recipe for shortening our code.

As many have proclaimed around numpy: one of its primary benefits is it allows one to "lose the loops".

```python
%timeit vg = np.vectorize(g)
```

    The slowest run took 5.55 times longer than the fastest. This could mean that an intermediate result is being cached.
    100000 loops, best of 3: 4.1 µs per loop

At any rate, this way, with a list comprehension, is orders of magnitude slower:

```python
%timeit vg2 = np.array([g(x) for x in domain])
```

    1000 loops, best of 3: 263 µs per loop

```python
vg = np.vectorize(g)
```

```python
%matplotlib inline
%timeit plt.plot(domain, vg(domain))
```

I bravely built my own version of the Gaussian distribution, a continuous function (any real number input is OK, from negative infinity to infinity, but not those (keep it in between). The thing about a Gaussian is you can shrink it and grow it while keeping the curve itself, self similar.  Remember "hyperparamters"?  They control the shape.  We should be sure to play around with those parameters.

Of course the stats.norm section of scipy comes pre-equipped with the same PDF (probability distribution function).  You'll see this curve called many things in the literature.

```python
%timeit plt.plot(domain, st.norm.pdf(domain))
```

```python
mu = 0
sigma = math.sqrt(0.2)
plt.plot(domain, vg(domain), color = 'blue')
sigma = math.sqrt(1)
plt.plot(domain, vg(domain), color = 'red')
sigma = math.sqrt(5)
plt.plot(domain, vg(domain), color = 'orange')
mu = -2
sigma = math.sqrt(.5)
plt.plot(domain, vg(domain), color = 'green')
plt.title("Gaussian Distributions")
```

[see Wikipedia figure](https://en.wikipedia.org/wiki/Gaussian_function#Properties)

These are Gaussian PDFs or Probability Density Functions.

68.26% of values happen within -1 and 1.

```python
from IPython.display import YouTubeVideo
YouTubeVideo("xgQhefFOXrM")
```

```python
a = st.norm.cdf(-1)  #  Cumulative distribution function
b = st.norm.cdf(1)
b - a
```

    0.68268949213708585

```python
a = st.norm.cdf(-2)
b = st.norm.cdf(2)
b - a
```

    0.95449973610364158

```python
# 99.73% is more correct than 99.72%
a = st.norm.cdf(-3)
b = st.norm.cdf(3)
b - a
```

    0.99730020393673979

```python
# 95%
a = st.norm.cdf(-1.96)
b = st.norm.cdf(1.96)
b - a
```

    0.95000420970355903

```python
# 99%
a = st.norm.cdf(-2.58)
b = st.norm.cdf(2.58)
b - a
```

    0.9901199684844586

```python
from IPython.display import YouTubeVideo
YouTubeVideo("zZWd56VlN7w")
```

What are the chances a value is less than -1.32?

```python
st.norm.cdf(-1.32)
```

    0.093417508993471787

What are the chances a value is between -0.21 and 0.85?

```python
1 - st.norm.sf(-0.21) # filling in from the right (survival function)
```

    0.41683383651755768

```python
a = st.norm.cdf(0.85) # filling in from the left
a
```

    0.80233745687730762

```python
b = st.norm.cdf(-0.21) # from the left
b
```

    0.41683383651755768

```python
a-b # getting the difference (per the Youtube)
```

    0.38550362035974994

Lets plot the integral of the Bell Curve.  This curve somewhat describes the temporal pattern whereby a new technology is adopted, first by early adopters, then comes the bandwagon effect, then come the stragglers.  Not the every technology gets adopted in this way.  Only some do.

```python
plt.plot(domain, st.norm.cdf(domain))
```

[Standard Deviation](https://en.wikipedia.org/wiki/Standard_deviation)

Above is the Bell Curve integral.

Remember the derivative is obtain from small differences: (f(x+h) - f(x))/x

Given x is our entire domain and operations are vectorized, it's easy enough to plot said derivative.

```python
x = st.norm.cdf(domain)
diff = st.norm.cdf(domain + 0.01)
plt.plot(domain, (diff-x)/0.01)
```

```python
x = st.norm.pdf(domain)
diff = st.norm.pdf(domain + 0.01)
plt.plot(domain, (diff-x)/0.01)
```

```python
x = st.norm.pdf(domain)
plt.plot(domain, x, color = "red")

x = st.norm.pdf(domain)
diff = st.norm.pdf(domain + 0.01)
plt.plot(domain, (diff-x)/0.01, color = "blue")
```

# Integrating the Gaussian

Apparently there's no closed form, however sympy is able to do an integration somehow.

```python
from sympy import var, Lambda, integrate, sqrt, pi, exp, latex

fig = plt.gcf()
fig.set_size_inches(8,5)
var('a b x sigma mu')
pdf = Lambda((x,mu,sigma),
  (1/(sigma * sqrt(2*pi)) * exp(-(mu-x)**2 / (2*sigma**2)))
)
cdf = Lambda((a,b,mu,sigma),
  integrate(
    pdf(x,mu,sigma),(x,a,b)
  )
)
display(Latex('$ cdf(a,b,\mu,\sigma) = ' + latex(cdf(a,b,mu,sigma)) + '$'))
```

$ cdf(a,b,\mu,\sigma) = - \frac{1}{2} \operatorname{erf}{\left (\frac{\sqrt{2} \left(a - \mu\right)}{2 \sigma} \right )} + \frac{1}{2} \operatorname{erf}{\left (\frac{\sqrt{2} \left(b - \mu\right)}{2 \sigma} \right )}$

    <matplotlib.figure.Figure at 0x1a197a4978>

Lets stop right here and note the pdf and cdf have been defined, using sympy's Lambda and integrate, and the cdf will be fed a lot of data, one hundred points, along with mu and sigma.  Then it's simply a matter of plotting.

What's amazing is our ability to get something from sympy that works to give a cdf, independently of scipy.stats.norm.

```python
x = np.linspace(50,159,100)
y = np.array([cdf(-1e99,v,100,15) for v in x],dtype='float')
plt.grid(True)
plt.title('Cumulative Distribution Function')
plt.xlabel('IQ')
print(type(plt.xlabel))
plt.ylabel('Y')
plt.text(65,.75,'$\mu = 100$',fontsize=16)
plt.text(65,.65,'$\sigma = 15$',fontsize=16)
plt.plot(x,y,color='gray')
plt.fill_between(x,y,0,color='#c0f0c0')
plt.show()
```

The above is truly a testament to Python's power, or the Python ecosystem's power.  We've brought in sympy, able to do symbolic integration, and talk LaTeX at the same time.  That's impressive.  Here's [the high IQ source](https://arachnoid.com/IPython/normal_distribution.html) for the original version of the above code.

There's no indefinite integral of the Gaussian, but there's a definite one.  sympy comes with its own generic sympy.stats.cdf function which produces Lambdas (symbolic expressions) when used to integrate different types of probability spaces, such as Normal (a continuous PDF).  It also accepts discrete PMFs as well.

<pre>
Examples
========

>>> from sympy.stats import density, Die, Normal, cdf
>>> from sympy import Symbol

>>> D = Die('D', 6)
>>> X = Normal('X', 0, 1)

>>> density(D).dict
{1: 1/6, 2: 1/6, 3: 1/6, 4: 1/6, 5: 1/6, 6: 1/6}
>>> cdf(D)
{1: 1/6, 2: 1/3, 3: 1/2, 4: 2/3, 5: 5/6, 6: 1}
>>> cdf(3*D, D > 2)
{9: 1/4, 12: 1/2, 15: 3/4, 18: 1}

>>> cdf(X)
Lambda(_z, -erfc(sqrt(2)*_z/2)/2 + 1)
</pre>

## LAB:  convert the Normal Distribution Below to IQ Curve...

That means domain is 0-200, standard deviation 15, mean = 100.

```python
domain = np.linspace(0, 200, 3000)
IQ = st.norm.pdf(domain, 100, 15)
plt.plot(domain, IQ, color = "red")
```

```python
domain = np.linspace(0, 200, 3000)
mu = 100
sigma = 15
IQ = vg(domain)
plt.plot(domain, IQ, color = "green")
```