# Lecture Mathematical Foundations
Horacio Gomez-Acevedo

BMIG 6120

UAMS

# Basic Concepts of Linear Algebra

Mathematical quantities are normally associated with sets of numbers. For instance points on the plane have two coordinates $x$ and $y$, whereas on the space they will have three coordinates $x$,$y$ and $z$. We further extend this idea to say that $(x_1,\ldots,x_n)$ is a point (or vector) in the $n$-dimensional space (normally denoted as $\mathbb{R}^n)$. We will denote vectors by bold face symbols, for instance $\pmb{x}=(x_1,\ldots,x_n)$

Some basic operations on vectors are:

\begin{align*}
\pmb{x}+\pmb{y}&=
(x_1,x_2,\ldots,x_n)+(y_1,y_2,\ldots,y_n)=(x_1+y_1,x_2+y_2,\ldots,x_n+y_n)\\
\pmb{x}-\pmb{y}&=
(x_1,x_2,\ldots,x_n)-(y_1,y_2,\ldots,y_n)=(x_1-y_1,x_2-y_2,\ldots,x_n-y_n)\\
\end{align*}
The **dot product** of two vectors $\pmb{x}$ and $\pmb{y}$ is defined as

\begin{align*}
\pmb{x}\cdot\pmb{y}&=(x_1,x_2,\ldots,x_n)\cdot(y_1,y_2,\ldots,y_n)=x_1\cdot y_1+ x_2\cdot y_2,\ldots+x_n\cdot y_n\\
&=\sum_{i=1}^n x_iy_i\\
\end{align*}
The length of an $n$-vector is given by
\begin{equation*}
\| \pmb{x} \| = \sqrt{ \pmb{x} \cdot \pmb{x}}  = \sqrt{x_1^2+x_2^2+\ldots+x_n^2}= \sqrt{\sum_{i=1}^n x_i^2}
\end{equation*}

## Matrices

A matrix is a collection of numbers, arranged in a particular way, for instance
\begin{equation*}
\left(
    \matrix{1 & 2& 3\\ 4&5&6}
    \right)
\end{equation*}

We can think of a matrix as a square section of a table.
A matrix of size $n\times p$ is a table with $n$ rows and $p$ columns like
\begin{equation*}
\left(
\matrix{ x_{1,1} & x_{1,2} & \ldots & x_{1,p} \\
x_{2,1} & x_{2,2} & \ldots & x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1} & x_{n,2} & \cdots & x_{n,p}
}
\right)
\end{equation*}

We will denote matrices with capital bold variables such as $\pmb{X}$.

Matrices can be represented as either column or row vectors.

## Matrices operations

The matrices have similar operations as vectors

\begin{equation*}
\pmb{X}+\pmb{Y}= \left(
\matrix{ x_{1,1} & x_{1,2} & \ldots & x_{1,p} \\
x_{2,1} & x_{2,2} & \ldots & x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1} & x_{n,2} & \cdots & x_{n,p}
}
\right) +
\left(
\matrix{ y_{1,1} & y_{1,2} & \ldots & y_{1,p} \\
y_{2,1} & y_{2,2} & \ldots & y_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
y_{n,1} & y_{n,2} & \cdots & y_{n,p}
}
\right)=
\left(
\matrix{ x_{1,1}+y_{1,1} & x_{1,2}+ y_{1,2} & \ldots & x_{1,p}+y_{1,p} \\
x_{2,1}+y_{2,1} & x_{2,2}+y_{2,2} & \ldots & x_{2,p} +y_{2,p}\\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1}+y_{n,1} & x_{n,2}+y_{n,2} & \cdots & x_{n,p}+y_{n,p}
}
\right)
\end{equation*}

\begin{equation*}
\pmb{X}-\pmb{Y}= \left(
\matrix{ x_{1,1} & x_{1,2} & \ldots & x_{1,p} \\
x_{2,1} & x_{2,2} & \ldots & x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1} & x_{n,2} & \cdots & x_{n,p}
}
\right) -
\left(
\matrix{ y_{1,1} & y_{1,2} & \ldots & y_{1,p} \\
y_{2,1} & y_{2,2} & \ldots & y_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
y_{n,1} & y_{n,2} & \cdots & y_{n,p}
}
\right)=
\left(
\matrix{ x_{1,1}-y_{1,1} & x_{1,2}- y_{1,2} & \ldots & x_{1,p}-y_{1,p} \\
x_{2,1}-y_{2,1} & x_{2,2}-y_{2,2} & \ldots & x_{2,p} -y_{2,p}\\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1}-y_{n,1} & x_{n,2}-y_{n,2} & \cdots & x_{n,p}-y_{n,p}
}
\right)
\end{equation*}

For any real number $\alpha \in \mathbb{R}$
\begin{equation*}
\alpha \pmb{X}= \alpha\left(
\matrix{ x_{1,1} & x_{1,2} & \ldots & x_{1,p} \\
x_{2,1} & x_{2,2} & \ldots & x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1} & x_{n,2} & \cdots & x_{n,p}
}
\right)=
\left(
\matrix{\alpha x_{1,1} & \alpha x_{1,2} & \ldots & \alpha x_{1,p} \\
\alpha x_{2,1} & \alpha x_{2,2} & \ldots & \alpha x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
\alpha x_{n,1} & \alpha x_{n,2} & \cdots & \alpha x_{n,p}
}
\right)
\end{equation*}

## Transpose of a Matrix

The transpose of a matrix is obtained by swapping rows and columns.
\begin{equation*}
\pmb{X}= \left(
\matrix{ x_{1,1} & x_{1,2} & \ldots & x_{1,p} \\
x_{2,1} & x_{2,2} & \ldots & x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1} & x_{n,2} & \cdots & x_{n,p}
}
\right)
\Rightarrow
\;\pmb{X}^t=
 \left(
\matrix{ x_{1,1} & x_{2,1} & \ldots & x_{n,1} \\
x_{1,2} & x_{2,2} & \ldots & x_{n,2} \\
\vdots & \vdots & \ddots & \vdots \\
x_{1,p} & x_{2,p} & \cdots & x_{n,p}
}
\right)
\end{equation*}

## Matrix multiplication

This operation is a bit different than the other operations.

Let $\pmb{X}$ be an $n\times p$ matrix represented as an arrange of row vectors, and $\pmb{Y}$ be a $p \times k$ matrix represented as an arrange of column vectors

\begin{equation*}
\pmb{X}=
\left(
\matrix{ x_{1,1} & x_{1,2} & \ldots & x_{1,p} \\
x_{2,1} & x_{2,2} & \ldots & x_{2,p} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n,1} & x_{n,2} & \cdots & x_{n,p}
}
\right)=
\left(
\matrix{ \pmb{x}_1  \\
\pmb{x}_2 \\
\vdots \\
\pmb{x}_n
}
\right) \; \textrm{and}\;
\pmb{Y} =
\left(
\matrix{ y_{1,1} & y_{1,2} & \ldots & y_{1,k} \\
y_{2,1} & y_{2,2} & \ldots & y_{2,k} \\
\vdots & \vdots & \ddots & \vdots \\
y_{p,1} & y_{n,2} & \cdots & y_{n,k}
}
\right)=
\left(
\matrix{\pmb{y}_1 & \pmb{y}_2 & \ldots & \pmb{y}_k \\}
\right)
\end{equation*}

The multiplication of $\pmb{X}$ and $\pmb{Y}$

\begin{equation*}
\pmb{X}\cdot\pmb{Y}= \left(
\matrix{ \pmb{x}_1\cdot\pmb{y}_1  & \pmb{x}_1\cdot \pmb{y}_2 & \ldots & \pmb{x}_1 \cdot \pmb{y}_k \\
\pmb{x}_2\cdot\pmb{y}_1  & \pmb{x}_2\cdot \pmb{y}_2 & \ldots & \pmb{x}_2 \cdot \pmb{y}_k \\
\vdots & \vdots & \ddots & \vdots \\
\pmb{x}_n\cdot\pmb{y}_1  & \pmb{x}_n\cdot \pmb{y}_2 & \ldots & \pmb{x}_n \cdot \pmb{y}_k \\
}
\right)
\end{equation*}

> Check the size of the matrix under multiplication! Multiplying a matrix of size $n\times k$ by a matrix of size $k \times p$ gives you a matrix of size $n \times p$ (middle number $k$ must coincide but cancells out)

## Example

Consider the following matrices

\begin{equation*}
\pmb{A}=\left(  \matrix{2 & -5 \\ 1 & 2} \right) \; \textrm{and } \; \pmb{B}= \left( \matrix{3 & 2 \\ -1 & 4} \right)
\end{equation*}

\begin{equation*}
\pmb{A}\cdot \pmb{B}= \left( \matrix{
2*3 + (-5)*(-1) & 2*2+(-5)*4 \\
1*3 + 2*(-1) & 1*2+2*4 }
\right)= \left( \matrix{11 & -16 \\ 1 & 10 } \right)
\end{equation*}

## Systems of Equations in Matrix Form

Suppose you want to solve the following system of equations

\begin{align}
2x -5y &= 3\\
x + 2y &= 2
\end{align}

At this point we write these equations as

\begin{equation}
\left(
\matrix{2 & -5\\ 1 & 2}
\right) \cdot \left(
\matrix{x \\ y}
\right)
= \left(
\matrix{3 \\ 2}
\right)
\end{equation}

If we call $A=\left(
\matrix{2 & -5\\ 1 & 2}
\right)  $, $\pmb{x}= \left( \matrix{x \\y} \right) $ and $\pmb{b}=\left( \matrix{3\\2}\right)$, we can rewrite our system as

\begin{equation*}
\pmb{A}\cdot \pmb{x}=\pmb{b}
\end{equation*}

Following our basic algebra knowledge, we would like to solve this equation as

\begin{equation*}
\pmb{x}= \pmb{A}^{-1} \cdot \pmb{b}
\end{equation*}

But, how do we find $\pmb{A}^{-1}$?

```python
import numpy as np
from numpy.linalg import inv
a = np.array([[2., -5.], [1., 2.]])
ainv = inv(a)
print(ainv)
x=np.dot(ainv,b=np.array([3,2]))
```

    [[ 0.22222222  0.55555556]
     [-0.11111111  0.22222222]]

Thus,
\begin{equation*}
\pmb{x}= \left(
\matrix{
0.222 & 0.556 \\
-0.111 & 0.222 } \right)
\cdot \left(
\matrix{ 3 \\ 2} \right)=
\left(\matrix{1.778\\ 0.111} \right)
\end{equation*}

And we can verify that $\pmb{A}\cdot \pmb{x}= \pmb{b}$

```python
print(np.dot(a,x))
```

    [3. 2.]

> Note. Not all the systems of equations have a solution and some have infinite number of solutions.

For instance,
\begin{align*}
x_1-x_2 +x_3 -x_4 &=1\\
x_1-x_2-x_3+x_4 &=0\\
x_1-x_2-2x_3+2x_4&= -\frac{1}{2}
\end{align*}
or in matrix form
\begin{equation*}
\left(
\begin{matrix}
1 & -1 & 1 & -1 \\
1 & -1 & -1 & 1 \\
1 & -1 & -2 & 2
\end{matrix} \right)
\cdot \pmb{x}= \left(
\begin{matrix}
1 \\
0\\
-\frac{1}{2}
\end{matrix}
\right)
\end{equation*}

In this case any expression that satisfies $x_2=x_1 -1/2$ and $x_3=x_4+1/2$ for arbitrary values of $x_1$ and $x_4$ will satisfy the system.
So, what is the problem?

First, there are more $x$'s than equations. But not only that, we have to make sure that the equations that we add make rows *linearly dependent*. This means that they cannot be additions(substractions) or constant multiplication of the previous rows.

There is a simple way to find out wheter the system has a solution and if so this one would be unique. We apply the **determinant** function. If it is different from zero, we are Ok.

Let's check quickly the what is the definition of the **determinant** of a matrix $A$ of type $n\times n$.

Case $n=2$

\begin{equation*}
\mathrm{det}\left[
\begin{matrix}
a_{11} & a_{12} \\
a_{21} & a_{22}
\end{matrix}
\right]= (a_{11}* a_{22}) - (a_{21}* a_{12})
\end{equation*}

Case n=3

\begin{equation*}
\mathrm{det}
\begin{pmatrix}
a_{11} & a_{12} & a_{13} \\
a_{21} & a_{22} & a_{23} \\
a_{31} & a_{32} & a_{33}
\end{pmatrix}
= a_{11} \cdot \mathrm{det}
\begin{pmatrix}
a_{22} & a_{23} \\
a_{32} & a_{33}
\end{pmatrix}
- a_{12}\cdot \mathrm{det}
\begin{pmatrix}
a_{21} & a_{13} \\
a_{31} & a_{33}
\end{pmatrix}
+a_{13} \cdot \mathrm{det}
\begin{pmatrix}
a_{21} & a_{22} \\
a_{31} & a_{32}
\end{pmatrix}
\end{equation*}

Matrices of type $n\times n$ that have determinant different from zero are called **non-singular**. These matrices will have an **inverse matrix** that satisfies

\begin{equation*}
A \cdot A^{-1} = A^{-1} \cdot A = I_n,
\end{equation*}
where $I_n$ is the identity matrix (a matrix of type $n\times n$ that has 1's in the diagonal and zeros elsewhere)

### Example

Let $A$ be the $3\times3$ matrix

\begin{equation}
A=
\begin{pmatrix}
4 & 3 & 2 \\
3& 5 & 2 \\
2 & 2 &1 \\
\end{pmatrix}
\end{equation}

What is the inverse matrix $A^{-1}$?

```python
import numpy as np
A=np.array([[4.,3.,2.],[3.,5.,2.],[2.,2.,1.]])
from numpy.linalg import inv

inv(A)
```

    array([[-1.00000000e+00, -1.00000000e+00,  4.00000000e+00],
           [-1.00000000e+00, -3.22973971e-16,  2.00000000e+00],
           [ 4.00000000e+00,  2.00000000e+00, -1.10000000e+01]])

## Matrices as transformations

Among the different functions say from $f \colon \mathbb{R}^2\to \mathbb{R}^2$, the so-called **linear transformation** play a very important role in mathematics. Such transformations can be represented by a matrix (defined by the transformation itself). Let's suppose we have a vector $\pmb{x}=(x_1,x_2)^t$ (note that we write $\pmb{x}$ as a column vector) and a matrix $\pmb{A}$ (of type $2\times2$ in this case), so we can define a linear transformation
\begin{equation}
T_A(\pmb{x}) = \pmb{A} \cdot \pmb{x}
\end{equation}
What does the transformation $T_A$ do to any vector $\pmb{x}$?

Here are some possibilitites

+ It keeps the same vector (trivial transformation)

\begin{equation}
\pmb{I}_2 \cdot \pmb{x}= \pmb{x}
\end{equation}

+ Expands the vector in a given direction
$\pmb{x}=\pmatrix{x_1\\x_2}$
\begin{equation}
\begin{pmatrix}
2 & 0\\
0 & 1
\end{pmatrix} \cdot \pmb{x} = (2x_1,x_2)^t= \pmatrix{2x_1\\x_2}
\end{equation}

+ Rotates the vector $180^\circ$

\begin{equation}
\begin{pmatrix}
-1 & 0\\
0 & -1
\end{pmatrix} \cdot \pmb{x} =\pmatrix{-x_1\\ -x_2}
\end{equation}

+ Rotates the vector with a specific angle $\theta$

\begin{equation}
\begin{pmatrix}
\cos(\theta) & \sin (\theta)\\
-\sin (\theta) & \cos(\theta)
\end{pmatrix} \cdot \pmb{x} =\pmatrix{x_1 \cos(\theta) +x_2 \sin(\theta)\\ -x_1\sin(\theta)+ x_2 \cos(\theta)
}
\end{equation}

+ It can project the vector to each coordinate axis

\begin{equation}
\begin{pmatrix}
0 &0 \\
0 & 1 \\
\end{pmatrix} \cdot \pmb{x}= \pmatrix{0 \\ x_2}
\end{equation}

## Rigid transformations

From basic algebra we know that the equation $y=mx +b$ represents a line in the plane with slope $m$ and $y$-intercept $b$. However, a similar expression such as
\begin{equation}
\pmb{A} \cdot \pmb{x} + \pmb{b}
\end{equation}
does not represent a linear transformation! This transformation is a translation of every vector by the vector $\pmb{b}$

For instance, the rigid-transformation $T$ defined as
\begin{equation}
T(\pmb{x})= \pmatrix{
3 & 0 \\
2 & 1 } \cdot \pmb{x}+ \pmatrix{1\\1}
\end{equation}

For $e_1=(1,0)$ and $e_2=(0,1)$

## Composition of transformations

In mathematics, a composition of functions refers to the operation of putting functions in tandem. That is, applying a function to a point (vector) and taking the output to put it as input of another function and so on.

Let $T_A \colon \mathbb{R}^m\to \mathbb{R}^k$ and $T_B\colon \mathbb{R}^n \to \mathbb{R}^m$ two linear transformations, then the composition $T_A$ with $T_B$ is the (linear) transformation $T_{A B}\colon \mathbb{R}^n \to \mathbb{R}^k$ defined as

\begin{equation}
T_{A B} ( \pmb{x})= T_A\circ T_B(\pmb{x}))= T_A(T_B(\pmb{x}))= \pmb{A} \cdot \pmb{B} \cdot \pmb{x}
\end{equation}

# Geometric properties

The **dot** product on arrays (or vectors) have a nice geometric interpretation

\begin{equation*}
\pmb{x} \cdot \pmb{y} = \| \pmb{x}\| \| \pmb{y} \| \cos( \theta)
\end{equation*}
where $\theta$ is the angle between the two vectors. This property is very handy when we want to find **perpendicular vectors**, in which case $\theta =90^\circ$ and the dot product will be zero!)

There is another operation between vectors in higher dimensions, the so-called **cross-product**. Whereas the previous product results in a number, the cross-product represents another vector.

\begin{equation*}
\pmb{x} \times \pmb{y}= \textrm{det }
\pmatrix{ \pmb{i} & \pmb{j} & \pmb{k} \\
x_1 & x_2 & x_3\\
y_1 & y_2 & y_3 }
= \pmb{i}(x_2y_3 - y_2 x_3)+ \pmb{j}(x_3y_1-x_1 y_3)+ \pmb{k}(x_1 y_2 -x_2y_1)
\end{equation*}
where $\pmb{i}= (1,0,0)$, $\pmb{j}=(0,1,0)$ and $\pmb{k}=(0,0,1)$.

```python
import numpy as np

x = [1, 2, 0]
y = [4, 5, 6]
np.cross(x, y)
```

    array([12, -6, -3])

# Derivatives and Gradients

## Case $n=1$
We will begin with elementary calculus. Let $f\colon \mathbb{R} \to \mathbb{R}$ be a smooth (differentiable) function at the point $x_0$.

What does the derivative of $f$ at $x_0$ represents? In theory, it represents the slope of the tangent line of $f(x)$ at $x_0$ (for points *close* to $x_0$).

\begin{equation}
f'(x_0) \approx \frac{f(x)-f(x_0)}{x-x_0}
\end{equation}

We can isolate $f(x)$ from the previous expressin and conclude that the derivate provide a linear approximation of a function a the given point for points *close* to $x_0$. More specifically

\begin{equation}
f(x) \approx f(x_0) + f'(x_0) (x-x_0)
\end{equation}
for points close to $x_0$.

## Case $n\ge2$

Let us first recall the concept of **level surfaces** of a function. If $f\colon \mathbb{R}^2 \to \mathbb{R}$, we can represent the graph of $f$ in a 3-d plot as a surface. If the function reaches a level $c$ (i.e., there is $\pmb{x}_0$ such that $f(\pmb{x}_0)=c$), then the set of all points in the plane that have the same value $c$ are called the level surface at level $c$.

Another example is the temperature map

Let's suppose now that $f\colon \mathbb{R}^n \to \mathbb{R}$ a smooth (differentiable) function at the point (vector) $\pmb{x}_0=(x_1^0,x_2^0,\ldots,x_n^0)$. The **gradient** of the function $f$ (in cartesian coordinates) is defined as

\begin{equation}
\textrm{grad }f(\pmb{x}_0)= \nabla f(\pmb{x}_0)= \left( \frac{\partial f (\pmb{x_0})}{\partial x_1},\frac{\partial f (\pmb{x_0})}{\partial x_2},\ldots,\frac{\partial f (\pmb{x_0})}{\partial x_n}  \right)
\end{equation}

Recall that the partial derivatives represent derivatives of the function $f$ with respect to a given variable while considering the rest of them constant.

How can we interpret the gradient?

+ It is a vector whose direction is always perpendicular to the direction of the level surface passing through the point.

+ The gradient vector always points to the direction in which the function $f$ is increasing.

### Example.

Consider $f(x_1,x_2)= 20 \cos(36*(x_1^2+x_2^2))+30$. The surface levels are depicted in the following animation.

Let's suppose we have a value $c=40$. The level-surface $L_c$ is defined as

\begin{equation}
L_c= \{ (x_1,x_2): f(x_1,x_2)=40\}= \{(x_1,x_2): x_1^2+x_2^2= \frac\pi{108} \}
\end{equation}
Thus, $L_c$ represents a circle of radius $\sqrt{\pi/108}$ centered at the origin. The gradient would be
\begin{equation}
\nabla f(x_1,x_2)= \left(
-1440 \sin(36*(x_1^2+x_2^2))x_1 , -1440 \sin(36*(x_1^2+x_2^2))x_2
\right)
\end{equation}
For a point $(x_1^0,x_2^0) \in L_c$,
\begin{equation}
\nabla f(x_1^0,x_2^0)= -720\sqrt{3} (x_1^0,x_2^0)
\end{equation}
Thus, the gradient points towards the origin for every point in the level surface $L_c$.

# Random Numbers

Can a deterministic machine produce random numbers?
In principle, any program in a deterministic machine will produce an output that is entirely predictable, thus not random.

Our computers produce *pseudo*-random numbers. At the minimum, we expect that a random number generator produces different sequences which are statistically uncorrelated to each other.

When we invoke a random number generator, we have access to a large sequence of numbers. The term **seed** refers to the initialization point of such sequence. Consecutive calls to the generator will give us the desired sequence or "random" numbers.

The basic module *random* generates uniform random numbers, but it is possible to use *numpy*.

```python
import random
random.seed(2021)
print("Uniformly Distributed Random numbers between (0,1) ")
print("%.4f"%random.uniform(0,1))
print("%.4f"%random.uniform(0,1))
```

    Uniformly Distributed Random numbers between (0,1)
    0.8363
    0.8583

```python
import numpy as np
np.random.seed(2021)
print("Uniformly Distributed Random numbers between (0,1) ")
print("%.4f"%np.random.uniform(0,1))
print("%.4f"%np.random.uniform(0,1))
```

    Uniformly Distributed Random numbers between (0,1)
    0.6060
    0.7334

Let's verify emprically, how accurate are our random number generators

```python
import math
my_mean=0; my_stdev=1.0; n=5000
u=np.random.normal(my_mean,my_stdev,n)
m=sum(u)/n; s =math.sqrt(sum((u-my_mean)**2)/(n-1))
print("Mean = %.5f and Emprical mean= %.5f"%(my_mean,m))
print("Standard Deviation = %.5f and Empirical Stdev = %.5f"%(my_stdev,s))

```

    Mean = 0.00000 and Emprical mean= -0.00063
    Standard Deviation = 1.00000 and Empirical Stdev = 0.99494

# Numerical Python

As we saw in our previous discussion, one of the most often used package for mathematical computations in Python is called *Numerical Python* abbreviates **NumPy**

In **Numpy** arrays with one index are called vectors.

```python
import numpy as np
a=np.zeros(10)
print(a)
type(a)
```

    [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

    numpy.ndarray

In the previous example $\verb+a+$ is not a list per se, it has been converted into a *numpy* array.

If we want to find in an interval, say $[1,3]$, 10 uniformly distributed values.

```python
b=np.linspace(1,3,10)
print(b)
```

    [1.         1.22222222 1.44444444 1.66666667 1.88888889 2.11111111
     2.33333333 2.55555556 2.77777778 3.        ]


# Introduction to Neural Networks

## The Perceptron

To get an intuitive idea about Neural Networks, we will code an elementary perceptron. In this example we will illustrate some of the concepts you have just seen, build a small perceptron and make a link between Perceptron and linear classifier.

### Generating some data

Before working with the MNIST dataset, you'll first test your perceptron implementation on a "toy" dataset with just a few data points. This allows you to test your implementations with data you can easily inspect and visualise without getting lost in the complexities of the dataset itself.

Start by loading two basic libraries: `matplotlib` and numpy

```python
# Load the libraries ...
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

```

Then let us generate some points in a 2D space that will form our dataset (you can add points later if you'd like)

```python
crosses = np.array([[0.5, 1.0], [1.0, 1.5], [1.5, 1.5], [2.0, 1.2], [3.0, 1.7], [1.5, 1.1],[2.1, 1.7]])
circles = np.array([[3.0, 0.5], [4.0, 1.0], [5.0, 0.7], [4.0, 0.2], [5.1, 0.3], [4.2, 0.7]])
```

### Visualising the data

Using `matploblib`, you can display the crosses as crosses (use `marker='x'`) and the circles as circles (use `marker='o'`). You will need to specify that you don't want a line using `linestyle='none'`. You can observe that the points are very easily separable.

```python
plt.plot(crosses[:,0], crosses[:,1], marker='x', linestyle='none')
plt.plot(circles[:,0], circles[:,1], marker='o', linestyle='none')
plt.ylim((0, 2))
plt.xlim((0, 6))

```

### Computing the output of a Perceptron

Let us consider the problem of building a classifier that for a given **new** point will return whether it belongs to the crosses (class 1) or circles (class 0). So for example it would take `(2, 1.5)` and return `1`.

Define a function `outPerceptron` which takes a 2d vector `x`, a 2d weight vector `w` and a bias `b` and returns the output following the step rule:

$$
\text{output} = \left\{\begin{align} 1\,\, &\text{if}\,\, \langle x, w\rangle -b \, >\,0 \\ 0\,\, &\text{otherwise}\end{align}\right.
$$

```python
def outPerceptron(x, w, b):
    innerProd = np.dot(x, w)
    output = 0
    if innerProd > b:
        output = 1
    return output

```

You can then enrich the function so that it can take a **sequence of inputs** (in the form of a matrix where each line of the matrix is one input vector) and return the corresponding **sequence of outputs**.

One way of doing this is to loop over the rows of `X` and for each of them, use the function `outPerceptron` that you just wrote. Store the results in an array `outputs` and return that. Call that function `multiOutPerceptron`/

Once you have that, you can try optimising the function by using a matrix-vector product; call the resulting function `multiOutPerceptron2` (and make sure it leads to the same results!)

```python
def multiOutPerceptron(X,w,b):
    nInstances = X.shape[0]
    outputs    = np.zeros(nInstances)
    for i in range(0,nInstances):
        outputs[i] = outPerceptron(X[i,:],w,b)
    return outputs

def multiOutPerceptron2(X,w,b):
    return (np.dot(X,w)>b).astype(float)

```

## = checkpoint 1 =

here, you should copy-paste the following code. If it returns `True` you're good to go on.

```python
np.random.seed(1234)
X = np.random.randn(10, 5)
w = np.random.randn(5)
b = np.random.randn()
np.all(multiOutPerceptron2(X, w, b) == np.array([ 1.,  1.,  0.,  0.,  0.,  1.,  0.,  0.,  1.,  0.]))
```

### Trying different weights and biases

You now have a method that can compute the outputs predicted by an **untrained** perceptron. Can you try picking different weights and biases and see how well you can classify the crosses and circles?

**Note**: to join the crosses and circles into one `instances` matrix, you can use `np.concatenate((crosses, circles), axis=0)`.

You can maybe start with `w=[1, 1]` and `b=1` and output the result of `multiOutPerceptron`. What is your analysis?

```python
test_w1 = [1., 1.]
test_b1 = 1.
instances = np.concatenate((crosses, circles), axis=0)
print(multiOutPerceptron(instances, test_w1, test_b1))

```

With the suggested weights and biases (`([1, 1],1)`), you should see something like

> `[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]`

which is clearly not great! Now try with `w=[-0.5, 1]` and `b=-0.2`, what do you observe?

```python
test_w2 = [-0.5, 1.]
test_b2 = -0.2
print (multiOutPerceptron(instances, test_w2, test_b2))

```

### How did we get there?

This is much better (100% correct on the training data).
To obtain these values, we found a **separating hyperplane** (here a line) between the points.
The equation of the line is

$ y = 0.5x-0.2 $

**Quiz**
- **Can you explain why this line corresponds to the weights and bias we used?**
- **Is this separating line unique? does it matter?**

### Illustrating the output of the Perceptron and the separating line

Copy-paste your code to visualise the crosses and circles above and overlay the separating line in red.

Can you modify the parameters of the line a little bit and still find a separating line that "works"?

```python
xx  = np.linspace(0, 6)
yy1 = 0.5 * xx - 0.2
yy2 = 0.4 * xx - 0.3

plt.plot(xx, yy1, color='red')
plt.plot(xx, yy2, color='orange')
plt.plot(crosses[:, 0], crosses[:, 1], marker='x', linestyle='none', label='sp1')
plt.plot(circles[:, 0], circles[:, 1], marker='o', linestyle='none', label='sp2')
plt.ylim((0, 2))
plt.xlim((0, 6))
plt.legend()

```

### Testing a few new points

Can you add the following `testPoints` on the plot and discuss what happens to them?

```python
testPoints = np.array([[1, 0.5],[5, 1.5],[3, 1.1]])
```

```python
plt.plot(xx, yy1, color='red')
plt.plot(xx, yy2, color='orange')
plt.plot(crosses[:, 0], crosses[:, 1], marker='x', linestyle='none')
plt.plot(circles[:, 0], circles[:, 1], marker='o', linestyle='none')
plt.ylim((0, 2))
plt.xlim((0, 6))

# the points
plt.plot(1, 0.5, marker='s', color='blue', markersize=10)
plt.plot(5, 1.5, marker='s', color='green', markersize=10)
plt.plot(3, 1.1, marker='^', color='black', markersize=10)

```

## = checkpoint 2 =

# Gradient Descent (remember?)

## Coding a simple gradient descent

### Considering some function

Let's consider the following arbitrary function and its gradient

$f(x) = \exp(-\sin(x))x^2$

$f'(x) = -x \exp(-\sin(x)) (x\cos(x)-2)$

It is convenient to define python functions which return the value of the function and its gradient at an arbitrary point $x$. Can you define a function `function` and a function `gradient`?

* use `np.exp`, `np.sin`, `np.cos` and remember that `x**2` is the squared of `x`

```python
def function(x):
    return np.exp(-np.sin(x))*(x**2)

def gradient(x):
    return -x*np.exp(-np.sin(x))*(x*np.cos(x)-2)

```

### Visualising the function

Can you write a simple code that shows what the function looks like over the interval `[-10,10]`? use at least `100` points in order to have a high enough definition of the line.

```python
x = np.linspace(-10, 10, 500)
plt.plot(x, function(x))

```

### Implementing a simple GD

Now let us implement a simple Gradient Descent that uses constant stepsizes. Define two functions:

1. simplest version which doesn't store the intermediate steps that are taken.
2. a version which does store the steps (useful to visualize what is going on and explain some of the typical behaviour of GD).

Let's call them `simpleGD` and `simpleGD2`. The parameters of both functions will be the initial point `x0`, the stepsize, and the number of steps to be taken.

```python
def simpleGD(x0, stepsize, nsteps):
    x    = x0
    for k in range(0,nsteps):
        x -= stepsize*gradient(x)
    return x

def simpleGD2(x0, stepsize, nsteps):
    x    = np.zeros(nsteps+1)
    x[0] = x0
    for k in range(0,nsteps):
        x[k+1] = x[k]-stepsize*gradient(x[k])
    return x

```

### Testing different situations

Try your algorithm `simpleGD` in the following cases:

* $x_0=1, \delta=0.1, n=100$
* $x_0=6, \delta=0.1, n=100$
* $x_0=8, \delta=0.01, n=100$

Can you discuss the results you obtained by having a look at the plot of the function?

### Visualising the cases

We suggest below a function `viz` which shows the path taken by the gradient descent when computed using `simpleGD2`.

Use it in the different cases above in order to see what the Gradient Descent does. Try to interpret the different cases.

```python
def viz(x, a=-10, b=10):
    xx  = np.linspace(a, b, 100)
    yy  = function(xx)
    ygd = function(x)
    plt.plot(xx, yy)
    plt.plot(x, ygd, color='red')
    plt.plot(x[0], ygd[0], marker='o', color='green', markersize=10)
    plt.plot(x[len(x)-1], ygd[len(x)-1], marker='o', color='red', markersize=10)
    plt.show()

```

```python
x1 = simpleGD2(3, 0.1, 100)
x2 = simpleGD2(6, 0.1, 100)
x3 = simpleGD2(8, 0.01, 100)
x4 = simpleGD2(3, 0.5, 100)

viz(x1)
viz(x2)
viz(x3)
viz(x4)

```


# Final project
## Author: Eros Fabrici
The Allen–Cahn equation (after John W. Cahn and Sam Allen) is a reaction–diffusion equation of mathematical physics which describes the process of phase separation in multi-component alloy systems, including order-disorder transitions.

The equation describes the time evolution of a scalar-valued state variable $\eta$  on a domain $\Omega=[0,1]$  during a time interval $[0,T]$, and is given (in one dimension) by:

$$
\frac{\partial \eta}{\partial t} - \varepsilon^2 \eta'' + f'(\eta) = 0, \qquad \eta'(0, t) = \eta'(1, t) = 0,\qquad\eta(x,0) = \eta_0(x)
$$

where $f$ is a double-well potential, $\eta_0$ is the initial condition, and $\varepsilon$ is the characteristic width of the phase transition.

This equation is the L2 gradient flow of the Ginzburg–Landau free energy functional, and it is closely related to the Cahn–Hilliard equation.

A typical example of double well potential is given by the following function

$$
f(\eta) = \eta^2(\eta-1)^2
$$

which has two minima in $0$ and $1$ (the two wells, where its value is zero), one local maximum in $0.5$, and it is always greater or equal than zero.

The two minima above behave like "attractors" for the phase $\eta$. Think of a solid-liquid phase transition (say water+ice) occupying the region $[0,1]$. When $\eta = 0$, then the material is liquid, while when $\eta = 1$ the material is solid (or viceversa).

Any other value for $\eta$ is *unstable*, and the equation will pull that region towards either $0$ or $1$.

Discretisation of this problem can be done by finite difference in time. For example, a fully explicity discretisation in time would lead to the following algorithm.

We split the interval $[0,T]$ in `n_steps` intervals, of dimension `dt = T/n_steps`. Given the solution at time `t[k] = k*dt`, it i possible to compute the next solution at time `t[k+1]` as

$$
\eta_{k+1} = \eta_{k} + \Delta t \varepsilon^2 \eta_k'' - \Delta t f'(\eta_k)
$$

Such a solution will not be stable. A possible remedy that improves the stability of the problem, is to treat the linear term $\Delta t \varepsilon^2 \eta_k''$ implicitly, and keep the term $-f'(\eta_k)$ explicit, that is:

$$
\eta_{k+1} - \Delta t \varepsilon^2 \eta_k'' = \eta_{k} - \Delta t f'(\eta_k)
$$

Grouping together the terms on the right hand side, this problem is identical to the one we solved in the python notebook number 9, with the exception of the constant $\Delta t \varepsilon^2$ in front the stiffness matrix.

In particular, given a set of basis functions $v_i$, representing $\eta = \eta^j v_j$ (sum is implied), we can solve the problem using finite elements by computing

$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} = \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_k)\big)
$$
where a sum is implied over $j$ on both the left hand side and the right hand side. Let us remark that while writing this last version of the equation we moved from a forward Euler scheme to a backward Euler scheme for the second spatial derivative term: that is, we used $\eta^j_{k+1}$ instead of $\eta^j_{k}$.

This results in a linear system

$$
A x = b
$$

where

$$
A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij} = \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big)
$$

and

$$
b_i = M_{ij} \big(\eta_k^j - \Delta t f'(\eta_k^j)\big)
$$

where we simplified the integration on the right hand side, by computing the integral of the interpolation of $f'(\eta)$.

## Step 1

Write a finite element solver, to solve one step of the problem above, given the solution at the previous time step, using the same techniques used in notebook number 9.

In particular:

1. Write a function that takes in input a vector representing $\eta$, an returns a vector containing $f'(\eta)$. Call this function `F`.

2. Write a function that takes in input a vector of support points of dimension `ndofs` and the degree `degree` of the polynomial basis, and returns a list of basis functions (piecewise polynomial objects of type `PPoly`) of dimension `ndofs`, representing the interpolatory spline basis of degree `degree`

3. Write a function that, given a piecewise polynomial object of type `PPoly` and a number `n_gauss_quadrature_points`, computes the vector of global_quadrature_points and global_quadrature_weights, that contains replicas of a Gauss quadrature formula with `n_gauss_quadrature_points` on each of the intervals defined by `unique(PPoly.x)`

4. Write a function that, given the basis and the quadrature points and weights, returns the two matrices $M$ and $K$

## Step 2

Solve the Allen-Cahan equation on the interval $[0,1]$, from time $t=0$ and time $t=1$, given a time step `dt`, a number of degrees of freedom `ndofs`, and a polynomial degree `k`.

1. Write a function that takes the initial value of $\eta_0$ as a function, eps, dt, ndofs, and degree, and returns a matrix of dimension `(int(T/dt), ndofs)`  containing all the coefficients $\eta_k^i$ representing the solution, and the set of basis functions used to compute the solution

2. Write a function that takes all the solutions `eta`, the basis functions, a stride number `s`, and a resolution `res`, and plots on a single plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$, computed on `res` equispaced points between zero and one

## Step 3

Solve the problem for all combinations of

1. eps = [01, .001]

2. ndofs = [16, 32, 64, 128]

3. degree = [1, 2, 3]

3. dt = [.25, .125, .0625, .03125, .015625]

with $\eta_0 = \sin(2 \pi x)+1$.

Plot the final solution at $t=1$ in all cases. What do you observe? What happens when you increase ndofs and keep dt constant?

## Step 4 (Optional)

Instead of solving the problem explicitly, solve it implicitly, by using backward euler method also for the non linear term. This requires the solution of a Nonlinear problem at every step. Use scipy and numpy methods to solve the non linear iteration.

```python
%pylab inline
import sympy as sym
import scipy
from scipy.interpolate import *
from scipy.integrate import *
```

    Populating the interactive namespace from numpy and matplotlib

```python
# Step 1.1

def F(eta):
    """
    Derivative of f(eta)
    """
    return 2*eta*(eta-1)*(2*eta-1)
```

```python
# Step 1.2

def compute_basis_functions(support_points, degree):
    basis_functions = []
    for i in range(len(support_points)):
        temp = support_points*0
        temp[i] = 1
        bi = PPoly.from_spline(splrep(support_points, temp, k=degree))
        basis_functions.append(bi)
    return basis_functions
```

```python
# Step 1.3

def compute_global_quadrature(basis, n_gauss_quadrature_points):
    quad_points, weights = numpy.polynomial.legendre.leggauss(n_gauss_quadrature_points+1)
    quad_points = (quad_points+1)/2
    weights /= 2
    # extract intervals
    intervals = unique(basis[0].x)
    # compute the first order difference points
    discr_difference = diff(intervals)
    discr_diff_length = len(discr_difference)

    gloabl_quadrature = array([intervals[i] + discr_difference[i] * quad_points
                               for i in range(discr_diff_length)]).reshape((-1,))
    global_weights = array([weights * discr_difference[i]
                            for i in range(discr_diff_length)]).reshape((-1,))
    return gloabl_quadrature, global_weights
```

```python
# Step 1.4

def compute_system_matrices(basis, gloabl_quadrature, global_weights):
    n_basis_funct = len(basis)
    derivative_basis = []
    for i in range(n_basis_funct):
        derivative_basis.append(basis[i].derivative(1))
    basis_quad = array([basis[i](gloabl_quadrature)
                        for i in range(n_basis_funct)]).T
    derivative_basis_quad = array([derivative_basis[i](gloabl_quadrature)
                                   for i in range(n_basis_funct)]).T
    M = einsum('qi, q, qj', basis_quad, global_weights, basis_quad)
    K = einsum('qi, q, qj', derivative_basis_quad, global_weights, derivative_basis_quad)
    return M, K
```

```python
# Step 2.1

def solve_allen_cahan(eta_0_function, eps, dt, ndofs, degree):
    '''
    Forward Euler solution.
    '''
    points = linspace(0, 1, ndofs)
    basis = compute_basis_functions(points, degree)
    Q, W = compute_global_quadrature(basis, degree+1)
    M, K = compute_system_matrices(basis, Q, W)
    A = M + (dt*eps**2)*K
    steps = int(1/dt) + 1
    t_interval = [step*dt for step in range(steps)]
    # matrix where the results will be stored
    eta = zeros((len(t_interval), ndofs))
    # setting the initial function
    eta[0, :] = eta_k = eta_0_function(points)
    for t in range(1, len(t_interval)):
        b = M.dot((eta_k - dt*F(eta_k)))
        eta_k = linalg.solve(A, b)
        eta[t, :] = eta_k

    return eta, basis
```

```python
# Step 2.2

def plot_solution(eta, basis, stride, resolution):
    x = linspace(0, 1, resolution)
    B = zeros((resolution, len(basis)))
    for i in range(len(basis)):
        B[:,i] = basis[i](x)

    n_t = shape(eta)[0]
    t = ['t ='+str(round((i/(n_t-1)), 2)) for i in range(n_t)]
    for eta, label_t in zip(eta[::-stride], t[::-stride]):
        plot(x, eta.dot(B.T), label=label_t)

    _ = legend(fontsize='x-large')
    _ = title('Allen–Cahn equation $\eta(x,t)$')
    _ = xlabel('$x$')
    _ = ylabel('$\eta$')
```

```python
#initial function choosen
def eta_0(x):
    return sin(2*pi*x)+1

figure(figsize=(20,6))
eta, basis = solve_allen_cahan(eta_0, eps=0.01, dt=0.1, ndofs=64, degree=1)
_= plot_solution(eta, basis, stride=3, resolution=1025)
```

## Step 3

```python
def plot_increasing_eps(dt, degree, ndofs, eps):
    #eps = [.1, .01, .001]
    fig = figure(figsize=(20, 7))
    for i in range(len(eps)):
        eta, basis = solve_allen_cahan(eta_0, eps[i], dt, ndofs, degree)
        subplot(1,3,i+1)
        plot_solution(eta, basis, int(1/dt+1), 1024)
        xlabel('x')
        ylabel('$\eta$')
        title('eps = '+ str(eps[i]))

def plot_increasing_ndofs(eps, degree, dt, ndofs):
    #ndofs = [16, 32, 64, 128]
    fig = figure(figsize=(22, 11))
    for i in range(len(ndofs)):
        eta, basis = solve_allen_cahan(eta_0, eps, dt, ndofs[i], degree)
        subplot(2, 2, i+1)
        plot_solution(eta, basis, int(1/dt+1), 1024)
        xlabel('x')
        ylabel('$\eta$')
        title('ndofs = '+ str(ndofs[i]))

def plot_increasing_degree(eps, degree, ndofs, dt):
    #dt = [.25, .125, .0625, .03125, .015625]
    fig = figure(figsize=(18,28))
    for i in range(len(degree)):
        eta, basis = solve_allen_cahan(eta_0, eps, dt, ndofs, degree[i])
        subplot(5,3,i+1)
        plot_solution(eta, basis, int(1/dt+1), 1024)
        xlabel('x')
        ylabel('$\eta$')
        title('degree = '+str(degree[i]))

def plot_increasing_dt(eps, degree, ndofs, dt):
    #dt = [.25, .125, .0625, .03125, .015625]
    fig = figure(figsize=(18,28))
    for i in range(len(dt)):
        eta, basis = solve_allen_cahan(eta_0, eps, dt[i], ndofs, degree)
        subplot(5,3,i+1)
        plot_solution(eta, basis, int(1/dt[i]+1), 1024)
        xlabel('x')
        ylabel('$\eta$')
        title('dt = '+ str(dt[i]))

```

```python
resolution = 1024
eps = [.01, .001]
ndofs = [16, 32, 64, 128]
degree = [1, 2, 3]
dt = [.25, .125, .0625, .03125, .015625]

plot_increasing_eps(eps = eps, degree = 1, dt = 0.1, ndofs = 128)
```

```python
plot_increasing_ndofs(eps = 0.01, degree = 1, dt = 0.1, ndofs=ndofs)
```

The more we increase the `ndofos`, the smoother the line becomes. This is due to the fact that we are increasing the support points for computing the basis functions.

```python
plot_increasing_degree(eps=0.01, degree=degree, ndofs=128, dt=0.1)
```

```python
plot_increasing_dt(eps=0.01, degree=1, ndofs=128, dt=dt)
```

It is possible to observe an abnormal behaviour when $dt>0.1$. This is due to the fact that the Forward Euler method is conditionally stable. More precisely, it is stable when the time step is smaller than a specific value, which depends on the problem you are solving.

## Step 4
As stated in the exercise description, if we use a backward Euler method we have:
$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} = \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_k)\big)
$$
Putting everything on the left-end side and rewriting with the matrix notation we obtain
$$
A_{i,j}\eta^j_{k+1} - M_{i,j}\eta^j_k + \Delta tM_{i,j}f'(\eta_{k+1}) = 0
$$
Thus we need to find $\eta^j_{k+1}$ such that the equation above is satisfied, namely we have to find the roots of a non-linear equation.
The solution in terms of code is pretty straight forward, as we just need to change the code inside the for-loop.
For solving the non-linear problem I will use scipy.optimize.fsolve that takes a function and an initial estimate of the root.

```python
def solve_allen_cahan_Backward(eta_0_function, eps, dt, ndofs, degree):
    points = linspace(0, 1, ndofs)
    basis = compute_basis_functions(points, degree)
    Q, W = compute_global_quadrature(basis, degree+1)
    M, K = compute_system_matrices(basis, Q, W)
    A = M + (dt*eps**2)*K
    steps = int(1/dt) + 1
    t_interval = [step*dt for step in range(steps)]
    # matrix where the results will be stored
    eta = zeros((len(t_interval), ndofs))
    # setting the initial function
    eta[0, :] = eta_k = eta_0_function(points)
    for t in range(1, len(t_interval)):
        eta_k = scipy.optimize.fsolve(
            lambda x: A.dot(x) - M.dot(eta_k) + dt*M.dot(F(x)),
            eta_k # starting estimate of the root
        )
        eta[t, :] = eta_k

    return eta, basis
```

Now we confront the results of the two methods with $\Delta t=0.25$

```python
figure(figsize=(20,6))

plt.subplot(1, 2, 1)
eta, basis = solve_allen_cahan_Backward(eta_0, eps=0.01, dt=0.25, ndofs=64, degree=1)
_ = plot_solution(eta, basis, stride=2, resolution=1025)
_ = title("Backward Euler Solution")

plt.subplot(1, 2, 2)
eta, basis = solve_allen_cahan(eta_0, eps=0.01, dt=0.25, ndofs=64, degree=1)
_ = plot_solution(eta, basis, stride=2, resolution=1025)
_ = title("Forward Euler Solution")
```

As stated by theory, Backward Euler method is unconditionally stable.


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
# Solution

alpha, beta = symbols('alpha beta')
```

```python
# Solution

eq3 = Eq(diff(f(t), t), alpha*f(t) + beta*f(t)**2)
```

```python
# Solution

solution_eq = dsolve(eq3)
```

```python
# Solution

general = solution_eq.rhs
```

```python
# Solution

at_0 = general.subs(t, 0)
```

```python
# Solution

solutions = solve(Eq(at_0, p_0), C1)
value_of_C1 = solutions[0]
```

```python
# Solution

particular = general.subs(C1, value_of_C1)
particular.simplify()
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```


## Introduction to Sympy

What if you would like to perform some calculus alongside your other computations? For that. you would need a computer algebra system (CAS). Luckily, the sympy package can provide you with the tools to perform symbolic computations and then can help you numerically evaluate those results.

Let's get started...

```python
# Typical import for numpy
# We will use a utility function or two for now...
import numpy as np
```

```python
#Typical import for Matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Import sympy tools
from sympy import *
init_printing(use_latex=True)
```

Note that I imported sympy, but then also initialized the sympy printing facilities to utilize \LaTeX. The reason for this will be obvious soon, if you haven't already figured it out.

Rather than have x be a vector, I need it to be a symbol. In particular, I need it to be a symbol that one would like to manipulate similar to how one uses symbols in mathematical expressions. Unlike variables which point to specific data structures, a symbol is more like how a variable is used in mathematics, and can take on values at a later time.

```python
# Make x a symbol
x = symbols('x')
```

```python
# Let's write an expression
y = cos(x)
```

```python
# Just provide the expression by itself,
# and it will be printed with LaTeX!
y
```

So, you can define mathematical expressions in sympy, and they will be rendered using \LaTeX.

Additionally, we can perform symbolic math on that expression. For example, let's take the derivative with respect to x using the `diff()` function.

```python
dydx = y.diff(x)
```

```python
dydx
```

Now we have the derivative of the function with respect to x, and have solved it symbolically using sympy! Sympy has it's own matplotlib functions for plotting expressions as well...

```python
plot(dydx)
```

However, we may want more control over the sampling and plot itself. This can sometimes be better done by evaluating the function _numerically_ instead. Let's do that now...

```python
x_vals = np.linspace(-2*np.pi,2*np.pi,101)
y_vals = np.array([dydx.evalf(subs=dict(x=x_val)) for x_val in x_vals])
print('The length of x is %d'%(len(x_vals)))
print('The length of y is %d'%(len(y_vals)))
```

    The length of x is 101
    The length of y is 101

Here we have used a python list comprehension to evaluate our derivative (dydx) at each of the 101 points in the $-2\pi$ to $2\pi$ range created by `linspace()`. The `evalf()` function allows us plug in values for our symbols. In particular, we pass the subs= argument a python dictionary object which contains a mapping from a symbol to a particular value we would like to associate with that symbol. Multiple symbols can be passed into the function using the dictionary object, so that functions with more than one symbol can be evaluated numerically.

```python
plt.plot(x_vals,y_vals)
plt.title('$y=%s$'%(latex(dydx)))
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

You can also now see how the `latex()` function can be used to convert the expression we were storing in dydx into a string form recognized by \LaTeX math mode, and therefore the `title()` function from matplotlib. It's usually much easier to control how you want your plots to look using numerical evaluation instead of using sympy's built-in plotting tools, so keep that in mind in the future.

```python

```


```
#Developer : Vinay Venkatesh

import matplotlib.pyplot as plt
import numpy as np
from sympy import *
```

```
def sigmoid():

  '''
  In the context of artificial neural networks, the Sigmoid Function is a type of activation function. The Sigmoid Function is often referred
  to as a squashing function because it limits its outputs to a range between 0 and 1. The Sigmoid Function has an "S" - shaped curve or sigmoid curve.

  This function was intended to be only for visualizing the graph of a sigmoid function.

  Learn More: https://deepai.org/machine-learning-glossary-and-terms/sigmoid-function
  '''

  x = np.linspace(-10, 10, 100)
  z = 1/(1 + np.exp(-x))

  plt.plot(x, z)
  plt.xlabel("x")
  plt.ylabel("Sigmoid(X)")

  plt.show()
```

```
sigmoid()
```

```
def relu():

  '''
  In the context of artificial neural networks, the ReLu Function is a type of activation function. The ReLu function directly outputs its input
  if it is positive. Otherwise, it outputs 0.

  This function was intended to be only for visualizing the graph of a ReLu function.

  Learn More: https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/#.
  '''

  x = np.linspace(-10, 10, 100)
  z = np.maximum(0, x)

  plt.plot(x, z)
  plt.xlabel("x")
  plt.ylabel("ReLu(X)")

  plt.show()
```

```
relu()
```

```
def constant(c):

    '''
    A constant function is a function whose value is the same for every input value.
    With a constant function, for any two points in the interval, a change in x results in a zero change in f(x).

    Learn More: https://www.varsitytutors.com/hotmath/hotmath_help/topics/constant-function
    '''

    fig = plt.figure()
    # Hold activation for multiple lines on same graph
    # Set x-axis range
    ax = fig.add_subplot(1, 1, 1)
    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')
    ax.spines['right'].set_color('none')
    ax.spines['top'].set_color('none')
    ax.xaxis.set_ticks_position('bottom')
    ax.yaxis.set_ticks_position('left')
    plt.xlim((-1 * (c + c), c + c))
    # Set y-axis range
    plt.ylim((-1 * (c + c), c + c))
    # Draw lines to split quadrants

    plt.axhline(y = c)

    plt.title('Constant Graph')

    plt.show()
```

```
constant(100)
```

```
def linear(m, b):

  '''
  A linear function is a function whose graph is a straight line.
  A linear function has one independent variable and one dependent variable.
  The independent variable is x and the dependent variable is y.
  The inputs, m and b stand for the slope and y intercept.

  The difference between `linear()` and `psflinear()` is the forms of the linear functions.
  `psflinear()` is meant to be in a y - y1 = m(x - x1) form and `linear()` is meant to be in a y = mx+b form.

  Learn More: https://www.mathsisfun.com/algebra/linear-equations.html
  '''

  #y = mx + b
  x = np.linspace(-5,5,100)
  y = m*x+b
  plt.plot(x, y, '-r', label=f'y={m}x+{b}')
  plt.title(f'Linear Graph')
  plt.xlabel('x', color='#1C2833')
  plt.ylabel('y', color='#1C2833')
  plt.legend(loc='upper left')
  plt.grid()
  plt.show()

  x, y = symbols('x y')

  equation = Eq(y, m*x+b)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))

  for i in range(0, len(result)):
    result[i] = round(result[i].simplify().evalf(),3)

  print(f"Slope: {m}")
  print(f"Y-Intercept: (0, {b})")
  print(f"X-Intercept: ")
  for i in range(0, len(result)):
    print(f"({result[i]}, 0)")
```

```
linear(5,2)
```

```
def psflinear(y1, m, x1):

  '''
  A linear function is a function whose graph is a straight line.
  A linear function has one independent variable and one dependent variable.
  The independent variable is x and the dependent variable is y.
  The inputs, y1 and m and x1 stand for a y coordinate, slope, and x coordinate.

  The difference between `psflinear()` and `linear()` is the forms of the linear functions.
  `psflinear()` is meant to be in a y - y1 = m(x - x1) form and `linear()` is meant to be in a y = mx+b form.

  Learn More: https://www.mathsisfun.com/algebra/linear-equations.html
  '''

  #y - y1 = m(x - x1)
  x = np.linspace(-5,5,100)
  y = m * (x - x1) + y1
  plt.plot(x, y, '-r', label=f'y-{y1}={m}(x-{x1})')
  plt.title(f'Linear Graph')
  plt.xlabel('x', color='#1C2833')
  plt.ylabel('y', color='#1C2833')
  plt.legend(loc='upper left')
  plt.grid()
  plt.show()

  x, y = symbols('x y')

  equation = Eq(y, m*(x-x1)+y1)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))

  for i in range(0, len(result)):
    result[i] = round(result[i].simplify().evalf(),3)

  print(f"Slope: {m}")
  print(f"Y-Intercept: (0, {y1})")
  print(f"X-Intercept: ")
  for i in range(0, len(result)):
    print(f"({result[i]}, 0)")
```

```
psflinear(6, 0.5, 20)
```

```
def quadratic(a, b, c):

  '''
  In algebra, a quadratic function, a quadratic polynomial, a
  polynomial of degree 2, or simply a quadratic, is a polynomial
  function with one or more variables in which the highest-degree term is of the second degree.

  The difference between `quadratic()` and `vtquadratic()` is the forms of the quadratic functions.
  `quadratic()` is meant to be in a ax^2 + bx + c form and `vtquadratic()` is meant to be in a a(x - h)^2 + k form.

  Learn More: https://www.mathsisfun.com/algebra/quadratic-equation.html
  '''

  #Y = AX^2 + BX + C
  if a == 0:
    print("a cannot be 0")
  x = np.linspace(-6,6,100)

  # the function, which is y = x^2 here
  y = a*x**2 + b*x + c
  vertex = (-1 * b) / (2 * a)
  vertex = a * vertex ** 2 + b * vertex + c

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('zero')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')
  if c < 0 and a < 0:
    plt.ylim((vertex + (2 * vertex), vertex * -1 + 1))
  if c < 0:
    plt.ylim((vertex -1, vertex * -1 + 1))
  elif c > 0 and c <= 999:
    plt.ylim((-5,c+c+5))
  elif c == 0:
    plt.ylim((-5,5))
  elif c >= 1000 and c <= 5000:
    plt.ylim((0,c*10))
    x = np.linspace(-20,20,100)
  elif c >= 5001 and c <= 20000:
    plt.ylim((0,c*10))
    x = np.linspace(-20,20,100)
  elif c >= 20001:
    plt.ylim((0,c*2))
    x = np.linspace(-20,20,100)
  # plot the function

  else:
    numList = []
    numList.append(a)
    numList.append(b)
    numList.append(c)
    for i in range(0, len(numList)):
      if numList[i] < 0:
        numList[i] = -1 * numList[i]

    plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  plt.plot(x,y, 'r', label=f'{a}x^2+{b}x+{c}')
  plt.title(f'Quadratic Graph')
  # show the plot

  plt.show()

  print(f"Vertex: ({(-1 * b)/(2 * a)}, {vertex})")
  print(f"Y-Intercept: (0, {c})")

  x, y = symbols('x y')

  equation = Eq(y, a*x**2 + b*x + c)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))

  for i in range(0, len(result)):
    result[i] = round(result[i].simplify().evalf(),3)

  #print(result)
  if len(result) == 2:
    print(f"X-Intercept: ({result[0]}, 0), ({result[1]}, 0)")
  elif len(result) == 1:
    print(f"X-Intercept: ({result[0]}, 0)")
```

```
quadratic(1, 5, 4)
```

```
def vtquadratic(a, h, k):

  '''
  In algebra, a quadratic function, a quadratic polynomial, a
  polynomial of degree 2, or simply a quadratic, is a polynomial
  function with one or more variables in which the highest-degree term is of the second degree.

  The difference between `quadratic()` and `vtquadratic()` is the forms of the quadratic functions.
  `quadratic()` is meant to be in a ax^2 + bx + c form and `vtquadratic()` is meant to be in a a(x - h)^2 + k form.

  Learn More: https://www.mathsisfun.com/algebra/quadratic-equation.html
  '''

  #y = a(x - h)^2 + k
  if a == 0:
    print("a cannot be 0")

  if h < 0:
    x = np.linspace(h-5, (-1 * h) + 5,100)
  elif k < 0:
    x = np.linspace(k*2, -k*3,100)
  else:
    x = np.linspace(-6,6,100)
  y = a * (x - h)**2 + k
  vertexX = h
  vertexY = k
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('zero')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')

  if k < 0 and h < 0:
    plt.ylim((k + (2 * k), k * -1 + 1))
  elif a < 0 and k < 0:
    plt.ylim((k*3, 0))
  elif k < 0:
    plt.ylim((k * 1.5, -k * 1.5))
  elif k > 0 and k <= 999:
    plt.ylim((-k*3,k*3))
    #x = np.linspace(-20,20,100)
  elif k == 0:
    plt.ylim((-5,5))
  elif k >= 1000 and k <= 5000:
    plt.ylim((0,k*5))
    x = np.linspace(-100,100,100)
  elif k >= 5001 and k <= 20000:
    plt.ylim((0,k*3))
    x = np.linspace(-100,100,100)
  elif k >= 20001:
    plt.ylim((0,k*3))
    x = np.linspace(-100,100,100)

  else:
    numList = []
    numList.append(a)
    numList.append(h)
    numList.append(k)
    for i in range(0, len(numList)):
      if numList[i] < 0:
        numList[i] = -1 * numList[i]

    plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  # plot the function
  plt.plot(x,y, 'r', label=f'{a}(x-{h})^2+{k}')
  plt.title(f'Quadratic Graph')

  plt.show()

  print(f"Vertex: ({round(h,3)}, {round(k,3)})")
  print(f"Y-Intercept: (0, {round(a*(0-h)**2+k,3)})")

  x, y = symbols('x y')

  equation = Eq(y, a * (x - h)**2 + k)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))
  #print(result)
  for i in range(0, len(result)):
    result[i] = result[i].simplify().evalf()
    try:
      result[i] = round(result[i], 3)
    except:
      pass

  if len(result) == 2:
    print(f"X-Intercept: ({result[0]}, 0), ({result[1]}, 0)")
  elif len(result) == 1:
    print(f"X-Intercept: ({result[0]}, 0)")
```

```
vtquadratic(2, -6, 20)
```

```
def cubic(a, b, c, d):

  '''
  In mathematics, a cubic function is a function of the form f(x)=ax^3+bx^2+cx+d
  where the coefficients a, b, c, and d are real numbers, and the variable x takes real values,
  and a ≠ 0. In other words, it is both a polynomial function of degree three, and a real function.

  Learn More: https://www.varsitytutors.com/hotmath/hotmath_help/topics/cubic-functions
  '''

  #ax^3+bx^2+cx+d
  if a == 0:
    print("a cannot be 0")

  x = np.linspace(-6,6,100)
  if d >= 1000:
    x = np.linspace(-100,100,100)

  # the function, which is y = x^2 here
  y = a * x**3 + b * x**2 + c * x + d

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('zero')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')

  if d == 0:
    plt.ylim((-5,5))
  elif b < 0 and c < 0 and d < 0:
    if -b > -c and -b > -d:
      plt.ylim((b*5,-b*5))
    elif -c > -b and -c > -d:
      plt.ylim((c*5,-c*5))
    elif -d > -c and -d > -b:
      plt.ylim((d*5,-d*5))
  elif d > 0 and c < 0:
    plt.ylim((-d*5,d*5))
  elif c > 0 and d < 0:
    if c > -d:
      plt.ylim((c*4,-c*4))
    else:
      plt.ylim((d*4,-d*4))
  elif d < 0:
    plt.ylim((d*4,-d*4))
  elif d > 0 and c > 0:
    if d > c or d == c:
      plt.ylim((-d*5,d*5))
  elif d > 0 and d <= 999:
    plt.ylim((-d*3.5,d*3.5))
  elif d >= 1000 and d <= 5000:
    plt.ylim((-d*5,d*5))
    #x = np.linspace(-100,100,100)
  elif d >= 5001 and d <= 20000:
    plt.ylim((0,d*3))
    #x = np.linspace(-100,100,100)
  elif d >= 20001:
    plt.ylim((0,d*3))
    #x = np.linspace(-100,100,100)

  else:
    numList = []
    numList.append(a)
    numList.append(b)
    numList.append(c)
    numList.append(d)
    for i in range(0, len(numList)):
      if numList[i] < 0:
        numList[i] = -1 * numList[i]

    plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  # plot the function
  plt.plot(x,y, 'r', label=f'{a}x^3+{b}x^2+{c}x+{d}')
  plt.title(f'Cubic Graph')

  plt.show()

  '''vertX = (-b) / (2 * a)
  vertY = a * vertX**3 + b * vertX**2 + c*vertX + d
  print(f"Vertex: ({vertX}, {vertY})")'''
  print(f"Y-Intercept: (0, {d})")

  x, y = symbols('x y')

  equation = Eq(y, a * x**3 + b * x**2 + c * x + d)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))
  for i in range(0, len(result)):
    result[i] = result[i].simplify().evalf()
    try:
      result[i] = round(result[i], 3)
    except:
      pass

  if len(result) == 1:
    print(f"X-Intercept(s): ({result[0]}, 0)")
  if len(result) == 2:
    print(f"X-Intercept(s): ({result[0]}, 0), ({result[1]}, 0)")
  if len(result) == 3:
    print(f"X-Intercept(s): ({result[0]}, 0), ({result[1]}, 0), ({result[2]}, 0)")
```

```
cubic(2, -3, -3, -35)
```

```
def trigsin(z, b):

  '''
  In mathematics, the trigonometric functions are real functions which relate
  an angle of a right-angled triangle to ratios of two side lengths.

  Learn More: https://www.mathsisfun.com/sine-cosine-tangent.html
  '''

  x = np.linspace(-np.pi,np.pi,100)

  # the function, which is y = sin(x) here
  y = z * np.sin(b * x)
  yint = z * np.sin(b * 0)

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('center')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')
  if z < 0:
    plt.ylim((z*1.5,-z*1.5))
  else:
    plt.ylim((-z*1.5,z*1.5))
  # plot the functions
  plt.plot(x,y, 'b', label=f'y={z}sin({b}x)')
  plt.title('Sine Graph')
  plt.legend(loc='upper left')

  # show the plot
  plt.show()
```

```
trigsin(1, 0.5)
```

```
def trigcos(z, b):

  '''
  In mathematics, the trigonometric functions are real functions which relate
  an angle of a right-angled triangle to ratios of two side lengths.

  Learn More: https://www.mathsisfun.com/sine-cosine-tangent.html
  '''

  x = np.linspace(-np.pi,np.pi,100)

  # the function, which is y = sin(x) here
  y = z * np.cos(b * x)

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('center')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')
  if z < 0:
    plt.ylim((z*1.5,-z*1.5))
  else:
    plt.ylim((-z*1.5,z*1.5))
  # plot the functions
  plt.plot(x,y, 'b', label=f'y={z}cos({b}x)')
  plt.title('Cosine Graph')
  plt.legend(loc='upper left')

  # show the plot
  plt.show()
```

```
trigcos(2, 1)
```

```
def trigtan(z, b):

  '''
  In mathematics, the trigonometric functions are real functions which relate
  an angle of a right-angled triangle to ratios of two side lengths.

  Learn More: https://www.mathsisfun.com/sine-cosine-tangent.html
  '''

  x = np.linspace(-np.pi,np.pi,100)

  # the function, which is y = sin(x) here
  y = z * np.tan(b * x)

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('center')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')
  if z < 0:
    plt.ylim((z*1.5,-z*1.5))
  else:
    plt.ylim((-z*1.5,z*1.5))
  # plot the functions
  plt.plot(x,y, 'b', label=f'y={z}tan({b}x)')
  plt.title('Tangent Graph')
  plt.legend(loc='upper left')

  # show the plot
  plt.show()
```

```
trigtan(5, 2)
```

```
def quartic(a, b, c, d, e):

  '''
  In algebra, a quartic function is a function of the form f(x)=ax^{4}+bx^{3}+cx^{2}+dx+e,
  where a is nonzero, which is defined by a polynomial of degree four, called a quartic polynomial.

  Learn More: https://www.calculushowto.com/types-of-functions/quartic-function/
  '''

  if a == 0:
    print("a cannot be 0")

  x = np.linspace(-6,6,100)
  if b <= -18 and b >= -29:
    x = np.linspace(b,-b,100)
  elif b <= -30:
    x = np.linspace(b/2, -b/2, 100)
  elif b >= 10 and b <= 29:
    x = np.linspace(b,-b,100)
  elif b >= 30:
    x = np.linspace(b/2, -b/2, 100)

  # the function, which is y = x^2 here
  y = a * x**4 + b * x**3 + c * x**2 + d*x + e

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('zero')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')

  #positive constraints
  if b >= -5 and b <= 0:
    plt.ylim((-6,6))
  elif b < -5 and b >= -9:
    plt.ylim((b*3,-b*3))
  elif b < -9 and b > -16:
    plt.ylim((b*-b,-b*-b))
  elif b <= -16 and b >= -27:
    plt.ylim((b*-b*3,-b*-b*3))
  elif b <= -28 and b >= -35:
    plt.ylim((b*-b*5,-b*-b*5))
  elif b <= -35 and b >= -120:
    plt.ylim((b*-b*-b/2,-b*-b*-b/2))
  elif b <= -121:
    plt.ylim((b*-b*-b,-b*-b*-b))

  #negative constraints
  elif b <= 5 and b >= 0:
    plt.ylim((-6,6))
  elif b > 5 and b <= 9:
    plt.ylim((-b*3,b*3))
  elif b > 9 and b < 16:
    plt.ylim((b*-b,-b*-b))
  elif b >= 16 and b <= 27:
    plt.ylim((b*-b*3,-b*-b*3))
  elif b >= 28 and b <= 35:
    plt.ylim((b*-b*5,-b*-b*5))
  elif b >= 35 and b <= 120:
    plt.ylim((b*b*-b/2,-b*b*-b/2))
  elif b >= 121:
    plt.ylim((b*-b*-b,-b*-b*-b))

  if e == 0:
    plt.ylim((-5,5))
  elif e > b:
    plt.ylim((-e*2,e*2))
  elif e * -1 > b * -1:
    plt.ylim((e*2,-e*2))

  else:
    numList = []
    numList.append(a)
    numList.append(b)
    numList.append(c)
    numList.append(d)
    numList.append(e)
    for i in range(0, len(numList)):
      if numList[i] < 0:
        numList[i] = -1 * numList[i]

    plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  #plt.ylim((-5,5))
  # plot the function
  plt.plot(x,y, 'r', label=f'{a}x^4+{b}x^3+{c}x^2+{d}x+{e}')
  plt.title('Quartic Graph')

  plt.show()

  print(f"Y-Intercept: (0, {e})")

  x, y = symbols('x y')

  equation = Eq(y, a * x**4 + b * x**3 + c * x**2 + d*x + e)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))
  for i in range(0, len(result)):
    result[i] = result[i].simplify().evalf()
    try:
      result[i] = round(result[i], 3)
    except:
      pass

  print(f"X-Intercept(s): ")
  for i in range(0, len(result)):
    print(f"({result[i]}, 0)")
```

```
quartic(10, 10, 5, 5, -150)
```

```
def quintic(a, b, c, d, e, f):

  '''
  In algebra, a quintic function is a function of the form g(x)=ax^{5}+bx^{4}+cx^{3}+dx^{2}+ex+f,
  where a, b, c, d, e and f are members of a field, typically the rational numbers, the real numbers or the complex numbers,
  and a is nonzero. In other words, a quintic function is defined by a polynomial of degree five.

  Learn More: https://www.calculushowto.com/quintic-function-polynomial/
  '''

  if a == 0:
    print("a cannot be 0")

  x = np.linspace(-6,6,100)
  if b <= -18 and b >= -29:
    x = np.linspace(b,-b,100)
  elif b <= -30:
    x = np.linspace(b/2, -b/2, 100)
  elif b >= 10 and b <= 29:
    x = np.linspace(b,-b,100)
  elif b >= 30:
    x = np.linspace(b/2, -b/2, 100)

  # the function, which is y = x^2 here
  y = a * x**5 + b * x**4 + c * x**3 + d*x**2 + e*x + f

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('zero')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')

  if b >= 10 and b <= 29:
    plt.ylim((-b*5,b*5))

  else:
    numList = []
    numList.append(a)
    numList.append(b)
    numList.append(c)
    numList.append(d)
    numList.append(e)
    numList.append(f)
    for i in range(0, len(numList)):
      if numList[i] < 0:
        numList[i] = -1 * numList[i]

    plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  plt.plot(x,y, 'r', label=f'{a}x^5+{b}x^4+{c}x^3+{d}x^2+{e}x+{f}')
  plt.title('Quintic Graph')

  plt.show()

  print(f"Y-Intercept: (0, {f})")

  x, y = symbols('x y')

  equation = Eq(y, a * x**5 + b * x**4 + c * x**3 + d*x**2 + e*x + f)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))
  for i in range(0, len(result)):
    result[i] = result[i].simplify().evalf()
    try:
      result[i] = round(result[i], 3)
    except:
      pass

  print(f"X-Intercept(s): ")
  for i in range(0, len(result)):
    print(f"({result[i]}, 0)")
```

```
quintic(2, 4, 0, 2, 10, 4)
```

```
def sextic(a, b, c, d, e, f, g):

  '''
  A sextic function is a function defined by a sextic polynomial. Because they have an even degree,
  sextic functions appear similar to quartic functions when graphed, except they may possess an additional local
  maximum and local minimum each. The derivative of a sextic function is a quintic function.

  Learn More: https://www.calculushowto.com/sextic-function/
  '''

  if a == 0:
    print("a cannot be 0")

  x = np.linspace(-6,6,100)
  if b <= -18 and b >= -29:
    x = np.linspace(b,-b,100)
  elif b <= -30:
    x = np.linspace(b/2, -b/2, 100)
  elif b >= 10 and b <= 29:
    x = np.linspace(b,-b,100)
  elif b >= 30:
    x = np.linspace(b/2, -b/2, 100)

  # the function, which is y = x^2 here
  y = a * x**6 + b * x**5 + c * x**4 + d*x**3 + e*x**2 + f*x + g

  # setting the axes at the centre
  fig = plt.figure()
  ax = fig.add_subplot(1, 1, 1)
  ax.spines['left'].set_position('center')
  ax.spines['bottom'].set_position('zero')
  ax.spines['right'].set_color('none')
  ax.spines['top'].set_color('none')
  ax.xaxis.set_ticks_position('bottom')
  ax.yaxis.set_ticks_position('left')

  numList = []
  numList.append(a)
  numList.append(b)
  numList.append(c)
  numList.append(d)
  numList.append(e)
  numList.append(f)
  numList.append(g)
  for i in range(0, len(numList)):
    if numList[i] < 0:
      numList[i] = -1 * numList[i]

  plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  plt.plot(x,y, 'r', label=f'{a}x^6+{b}x^5+{c}x^4+{d}x^3+{e}x^2+{f}x+{g}')
  plt.title('Sextic Graph')

  plt.show()

  print(f"Y-Intercept: (0, {g})")

  x, y = symbols('x y')

  equation = Eq(y, a * x**6 + b * x**5 + c * x**4 + d*x**3 + e*x**2 + f*x + g)

  # Use sympy.subs() method
  result = solve(equation.subs(y, 0))
  for i in range(0, len(result)):
    result[i] = result[i].simplify().evalf()
    try:
      result[i] = round(result[i], 3)
    except:
      pass

  print(f"X-Intercept(s): ")
  for i in range(0, len(result)):
    print(f"({result[i]}, 0)")
```

```
sextic(2, 4, 0, 2, 10, 4, 2)
```

```
def logFun(a, b, c, d):

  '''
  Logarithm (log) In mathematics, the logarithm is the inverse function to exponentiation.
  That means the logarithm of a given number x is the exponent to which another fixed number,
  the base b, must be raised, to produce that number x.

  Learn More: https://www.mathsisfun.com/sets/function-logarithmic.html
  '''

  if a == 0:
    print("a cannot be 0")
  elif b == 0:
    print("b cannot be 0")

  x = np.linspace(-6,6,100)

  # the function, which is y = x^2 here
  y = a * np.log(b * x ** c) + d

  fig = plt.figure()

  numList = []
  numList.append(a)
  numList.append(b)
  numList.append(c)
  numList.append(d)
  for i in range(0, len(numList)):
    if numList[i] < 0:
      numList[i] = -1 * numList[i]

  plt.ylim((-1 * max(numList) * 5, max(numList) * 5))

  plt.plot(x,y, 'r') #label=f'{a}x^6+{b}x^5+{c}x^4+{d}x^3+{e}x^2+{f}x+{g}')
  plt.title('Log Graph')

  plt.show()
```

```
logFun(4, 1, -2, 10)
```

```
def absVal(a, b, c):

  '''
  An absolute value function is a function that contains an algebraic expression within absolute value symbols.

  Learn More: https://www.varsitytutors.com/hotmath/hotmath_help/topics/absolute-value-functions
  '''

  if a == 0:
    print("a cannot be 0")
  elif b == 0:
    print("b cannot be 0")

  numList = []
  numList.append(a)
  numList.append(b)
  numList.append(c)
  for i in range(0, len(numList)):
    if numList[i] < 0:
      numList[i] = -1 * numList[i]

  fig = plt.figure()

  plt.ylim((-1 * max(numList) * 5, max(numList) * 5))
  x = np.linspace(-5 * max(numList),max(numList)*5,100)
  # the function, which is y = x^2 here
  y = a * abs(b + x) + c

  plt.plot(x,y, 'r') #label=f'{a}x^6+{b}x^5+{c}x^4+{d}x^3+{e}x^2+{f}x+{g}')
  plt.title('Absolute Value Graph')

  plt.show()

```

```
absVal(1, -5, 12)
```


# The Process
Not everytime is there a situation where $Ax = b$ has exact solutions , such is the case when we are trying to find a best fit line for a given set of datapoints , there is no direct solution for it , instead we are in search for the best possible solution.So instead of finding solutions for $Ax = b$ , we can project the $b$ vector onto the column space of $A$ so that we can get the best possible solution.The solution to the equation $A^TA\hat{X} = A^Tb$ , gives us the best possible solution that we are looking for , which can be simplified to , $\hat{X} = (A^TA)^{-1}A^Tb$ understanding all this with an example would give us a better idea.

Let us find the best possible line passing/fitting through $(1,1),(2,2) ,(3,2)$
taking the equation to be of the form $y = mx + c$

\begin{equation}
\begin{pmatrix}
1 & 1 \\
2 & 2 \\
3 & 2
\end{pmatrix}
\begin{pmatrix}
m \\
c
\end{pmatrix}
=
\begin{pmatrix}
1 \\
2 \\
2
\end{pmatrix}
\end{equation}
Which has no solution,so our $\hat{X}$ is given by
\begin{equation}
\begin{pmatrix}
1 & 1 \\
2 & 2 \\
3 & 2
\end{pmatrix}^{T}
\begin{pmatrix}
1 & 1 \\
2 & 2 \\
3 & 2
\end{pmatrix}
\hat{X}
=
\begin{pmatrix}
1 & 1 \\
2 & 2 \\
3 & 2
\end{pmatrix}^{T}
\begin{pmatrix}
1 \\
2 \\
2
\end{pmatrix}
\end{equation}
which on solving gives
\begin{equation}
\begin{pmatrix}
\hat{m} \\
\hat{c}
\
\end{pmatrix}
=
\begin{pmatrix}
\frac{1}{2} \\
\frac{2}{3}
\
\end{pmatrix}
\end{equation}
Thus the best fitting line is given by
$ y = \frac{1}{2}x+\frac{2}{3} $

This exact process is done by the below given python fxn.

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
```

```python
def LeastSquares(points):
    A = []
    b = []
    #Extracting the A,b matrix from the given set of points
    for p in points:
        A.append([1,p[0]])
        b.append([p[1]])
    A = np.array(A)
    b = np.array(b)
    points = np.array(points)
    #finding the parameters m and c
    x = np.matmul(np.matmul(np.linalg.inv(np.matmul(A.T,A)),A.T),b)
    #drawing the fitting line
    inp_space = np.linspace(points[:,0].min(),points[:,0].max(),1000)
    out = (inp_space)*x[1]+x[0]
    plt.plot(inp_space,out)
    for i in points:
        plt.scatter(i[0],i[1])
    plt.title(f"Y = {x[1]}x+{x[0]}")
```

```python
A = LeastSquares([(0,0),(-1,1),(2,3),(3,4)]) # takes in list of points to fit line on.
```

# Learn more on
[MIT OCW LECTURE](https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/lecture-16-projection-matrices-and-least-squares)

[Least square Approximations](http://math.mit.edu/~gs/linearalgebra/ila0403.pdf)


```python
import numpy as np
from numpy.linalg import det, inv, matrix_rank, eig
from sympy import Matrix, symbols

a = np.array([[1, 2], [3, 4]])
a
```

    array([[1, 2],
           [3, 4]])

# Common matrix operations

## Multiplication
***With a number*** (aka scalar, because it scales the matrix).

```python
5 * a
```

    array([[ 5, 10],
           [15, 20]])

**With another matrix.**

It can be viewed as a linear transformation of a coordinate system. Example of 90 degree rotation transform:

```python
transform = np.array([[0,-1],[1,0]])
i = np.array([1,0])
np.matmul(transform,i)
```

    array([0, 1])

Multiplying two square matrices can be viewed as composing two transforms. The one on the right is applied first. Example of rotation then shear transforms:

```python
transform = np.matmul(np.array([[1,1],[0,1]]), np.array([[0,-1],[1,0]]))
transform
```

    array([[ 1, -1],
           [ 1,  0]])

**Identity matrix** multiplication preserves the original matrix

```python
np.matmul(a,np.eye(2))
```

    array([[1., 2.],
           [3., 4.]])

## Dot product
Dot product between two vectors (1d matrices or tensors) gives you an idea about their orientation:

- if dot product is zero -> perpendicular, negative ->opposite directions (angle > 90 degrees)

It is also a way to map vectors into a different space (e.g. lower/higher dimensional). For example the projection of a 2d vector to a 1d line. Dot product is just a shorthand for multiply a and b transponse: $A \bullet B = AB^T$

```python
a * a
```

    array([[ 1,  4],
           [ 9, 16]])

## Transpose
Rows become columns and vice versa. If $A=A^T$ then A is **symmetric** (implies that A is square).

```python
a.T
```

    array([[1, 3],
           [2, 4]])

## Determinant
If determinant is zero, the matrix is called **singular**. It means that the matrix vectors are linearly dependent and that dimensionality of space is reduced. Negative determinant means that the orientation of space is inverted (e.g. flipped). The absolute value of the determinant shows how a shape's size will change.

```python
det(a) # a flips space and increases area twofold
```

## Inverse matrices
Non-singular matrices (det != 0) are invertible: $A^{-1}A=I$, where $I$ is the identity matrix (np.eye)

```python
inv(a)
```

    array([[-2. ,  1. ],
           [ 1.5, -0.5]])

```python
np.matmul(a,inv(a)) #shall give the identity matrix, but there is some quantization error
```

    array([[1.0000000e+00, 0.0000000e+00],
           [8.8817842e-16, 1.0000000e+00]])

## Rank
The max number of linearly independent rows or columns in the matrix

```python
#matrix_rank(a) #2
#matrix_rank(np.array([[1,0],[1,0]])) #1
det(np.array([[15.000000000000001,1],[30,2]]))
```

## Echelon form
Numerical algebra is not very suitable for finding exact solutions to the reduced row echelon form. Symbolic algebra with sympy might be a better fit here.

http://numpy-discussion.10968.n7.nabble.com/Reduced-row-echelon-form-td16486.html

http://docs.sympy.org/0.7.5/tutorial/matrices.html

```python
A = Matrix([[1, -1], [3, 4], [0, 2]])
A.rref()
```

## Cross product
In 3D space, cross product of two vectors, v1 and v2, yields another vector that is perpendicular to the two. The resulting vector is computed by constructing the determinant with the identity vector $(\hat{i}, \hat{j}, \hat{k})$ in the first column and v1 and v2 in the second and third respectively.

$A \times B$

```python
v1 = Matrix([1,2,4])
v2 = Matrix([3,2,5])

i,j,k = symbols('i j k')
result = Matrix([[i,v1[0],v2[0]],[j,v1[1],v2[1]],[k,v1[2],v2[2]]]).det()
print(result)
[result.subs([(i,1),(j,0),(k,0)]), result.subs([(i,0),(j,1),(k,0)]), result.subs([(i,0),(j,0),(k,1)])]
```

## Eigenvalues & Eigenvectors

Given a transformation matrix $A$ if there exists a vector $\vec{v}$ and a scalar $\lambda$, such that:

$A\vec{v} = \lambda\vec{v} \Rightarrow A\vec{v} = \lambda I\vec{v} \Rightarrow (A - \lambda I)\vec{v} = 0 \Rightarrow $

$$det(A - \lambda I) = 0$$

Then all possible values of $\lambda$ that satisfy the equation are called eigenvalues. Each of these values corresponds to one or more eigenvectors $\vec{v}$, which won't change their span after the transformation with matrix $A$.

**Eigenbasis** with respect to the transformation A is the set of new basis vectors (e.g. at least two in 2d) that are also eigenvectors. Transforming the original transformation matrix $A$ to $A'$ in the new basis guarantees that $A'$ will be diagonal.

```python
#eig(np.array([[3,1],[0,2]]))
m = Matrix([[3,1],[0,2]])
print(m.eigenvals()) # value:algebraic multiplicity
m.eigenvects() # eigenvalue, algebraic multiplicity, eigenvector
```

## Null space
In non-full rank matrices, this is the set of vectors that will get reduced to a line or a point. In other words, this is the space of all possible solutions of the system of equations (no single solution exists because det=0 & rank< full rank)

## Frobenius normal form
## Matrix equivalence
## Matrix congruence
## Singular value decomposition

https://www.youtube.com/watch?v=P5mlg91as1c

## PCA
## SVD
## Jacobian matrix
Jacobian is the matrix of partial derivatives of a $ R^n \longrightarrow R^m $ function. It has $m$ columns - one for each output (dependent) variable and $n$ rows - one for each input (independent) variable. Thus each column is the gradient of the respective dependent variable.

https://www.value-at-risk.net/functions/

```python

```

## Conjugate transponse
Conjugate transponse (Hermitian transponse) of a matrix with complex entries is obtained by first taking the transponse of the matrix and then taking the complex conjugate of each entry:

$ A^* = \overline{A^T}$, where complex conjugate is the element-wise operation: $a + ib \Rightarrow a - ib$.

# Some interesting matrix properties

## Similar
Two **square** matrices $A$ and $B$ are similar if $B=P^{-1}AP$ and $P$ is invertible.

## Diagonalizable

## Normal
A complex square matrix is normal if: $A^*A=AA^*$. For real matrices this reduces to: $A^TA=AA^T$

# Unitary matrices
A **complex square** matrix is unitary if its inverse is equal to its conjugate transponse: $А^{*} = A^{-1} \implies A^*A=AA^*=I$

For real valued matrices, the unitary is called orthogonal: $A^{T}=A^{-1} \implies A^TA=AA^T=I$

Unitary matrices preserve the euclidian norm (length) of a vector x during multiplication: $\lVert x \rVert = \lVert Ax \rVert$

```python
np.conj(a).T
```

    array([[1, 3],
           [2, 4]], dtype=int32)

```python
a = np.array([[2,-1],[0.5,-0.5]])
np.matmul(a,a.T)
```

    array([[5. , 1.5],
           [1.5, 0.5]])

```python
x = np.array([[1,2],[3,4],[5,6]])
np.matmul(x,x.T) # symmetrize
```

    array([[ 5, 11, 17],
           [11, 25, 39],
           [17, 39, 61]])

```python
#
```

```python

```

```python

```


### Computing for Mathematics - Mock individual coursework

This jupyter notebook contains questions that will resemble the questions in your individual coursework.

**Important** Do not delete the cells containing:

```
### BEGIN SOLUTION

### END SOLUTION
```

write your solution attempts in those cells.

**If you would like to** submit this notebook:

- Change the name of the notebook from `main` to: `<student_number>`. For example, if your student number is `c1234567` then change the name of the notebook to `c1234567`.
- Write all your solution attempts in the correct locations;
- Save the notebook (`File>Save As`);
- Follow the instructions given in class/email to submit.

#### Question 1

Output the evaluation of the following expressions exactly.

a. \\(\frac{(9a^2bc^4) ^ {\frac{1}{2}}}{6ab^{\frac{3}{2}}c}\\)

```python
### BEGIN SOLUTION
import sympy as sym
a, b, c = sym.Symbol("a"), sym.Symbol("b"), sym.Symbol("c")

sym.expand((9 * a ** 2 * b * c ** 4) ** (sym.S(1) / 2) / (6 * a * b ** (sym.S(3) / 2) * c))
### END SOLUTION
```

$\displaystyle \frac{\sqrt{a^{2} b c^{4}}}{2 a b^{\frac{3}{2}} c}$

b. \\((2 ^ {\frac{1}{2}} + 2) ^ 2 - 2 ^ {\frac{5}{2}}\\)

```python
### BEGIN SOLUTION
sym.expand((sym.S(2) ** (sym.S(1) / 2) + 2) ** 2 - 2 ** (sym.S(5) / 2))
### END SOLUTION
```

$\displaystyle 6$

3. \\((\frac{1}{8}) ^ {\frac{4}{3}}\\)

```python
### BEGIN SOLUTION
(sym.S(1) / 8) ** (sym.S(4) / 3)
### END SOLUTION
```

$\displaystyle \frac{1}{16}$

### Question 2

Write a function `expand` that takes a given mathematical expression and returns the expanded expression.

```python
def expand(expression):
    ### BEGIN SOLUTION
    """
    Take a symbolic expression and expands it.
    """
    return sym.expand(expression)
    ### END SOLUTION
```

### Question 3

The matrix \\(D\\) is given by \\(D = \begin{pmatrix} 1& 2 & a\\ 3 & 1 & 0\\ 1 & 1 & 1\end{pmatrix}\\) where \\(a\ne 2\\).

a. Create a variable `D` which has value the matrix \\(D\\).

```python
### BEGIN SOLUTION
a = sym.Symbol("a")
D = sym.Matrix([[1, 2, a], [3, 1, 0], [1, 1, 1]])
### END SOLUTION
```

b. Create a variable `D_inv` with value the inverse of \\(D\\).

```python
### BEGIN SOLUTION
D_inv = D.inv()
### END SOLUTION
```

c. Using `D_inv` **output** the solution of the following system of equations:

\\[
\begin{array}{r}
    x + 2y + 4z = 3\\
    3x + y = 4\\
    x + y + z = 1\\
\end{array}
\\]

```python
### BEGIN SOLUTION
b = sym.Matrix([[3], [4], [1]])
sym.simplify(D.inv() @ b).subs({a: 4})
### END SOLUTION
```

$\displaystyle \left[\begin{matrix}\frac{7}{3}\\-3\\\frac{5}{3}\end{matrix}\right]$

### Question 4

During a game of frisbee between a handler and their dog the handler chooses to randomly select if they throw using a backhand or a forehand: 25% of the time they will throw a backhand.

Because of the way their dog chooses to approach a flying frisbee they catch it with the following probabilities:

- 80% of the time when it is thrown using a backhand
- 90% of the time when it is thrown using a forehand

a. Write a function `sample_experiment()` that simulates a given throw and returns the throw type (as a string with value `"backhand"` or `"forehand"`) and whether it was caught (as a boolean: either `True` or `False`).

```python
import random

def sample_experiment():
    """
    Returns the throw type and whether it was caught
    """
    ### BEGIN SOLUTION
    if random.random() < .25:
        throw = "backhand"
        probability_of_catch = .8
    else:
        throw = "forehand"
        probability_of_catch = .9

    caught = random.random() < probability_of_catch
    ### END SOLUTION
    return throw, caught
```

b. Using 1,000,000 samples create a variable `probability_of_catch` which has value an estimate for the probability of the frisbee being caught.

```python
### BEGIN SOLUTION
number_of_repetitions = 1_000_000
random.seed(0)
samples = [sample_experiment() for repetition in range(number_of_repetitions)]
probability_of_catch = sum(catch is True for throw, catch in samples) / number_of_repetitions
### END SOLUTION
```

c. Using the above, create a variable `probability_of_forehand_given_drop` which has value an estimate for the probability of the frisbee being thrown with a forehand given that it was not caught.

```python
### BEGIN SOLUTION
samples_with_drop = [(throw, catch) for throw, catch in samples if catch is False]
number_of_drops = len(samples_with_drop)
probability_of_forehand_given_drop = sum(throw == "forehand" for throw, catch in samples_with_drop) / number_of_drops
### END SOLUTION
```


# Exercise 2
Write a function to compute the roots of a mathematical equation of the form
\begin{align}
  ax^{2} + bx + c = 0.
\end{align}
Your function should be sensitive enough to adapt to situations in which a user might accidentally set $a=0$, or $b=0$, or even $a=b=0$.  For example, if $a=0, b\neq 0$, your function should print a warning and compute the roots of the resulting linear function.  It is up to you on how to handle the function header: feel free to use default keyword arguments, variable positional arguments, variable keyword arguments, or something else as you see fit.  Try to make it user friendly.

Your function should return a tuple containing the roots of the provided equation.

**Hint:** Quadratic equations can have complex roots of the form $r = a + ib$ where $i=\sqrt{-1}$ (Python uses the notation $j=\sqrt{-1}$).  To deal with complex roots, you should import the `cmath` library and use `cmath.sqrt` when computing square roots.  `cmath` will return a complex number for you.  You could handle complex roots yourself if you want, but you might as well use available libraries to save some work.

```python
import numpy as np
import cmath
```

```python
def find_roots(a, b, c):
    if 0 == a:
        if 0 == b:
            print('Warning: wrong input! a and b cannot be zero simultaneously!')
            if 0 == c:
                print('All x')
            return None
        else:
            print('Warning: a = 0; it is a linear function!')
            return [-c/b]
    else:
        return [(-b+cmath.sqrt(b*b-4*a*c))/(2*a), (-b-cmath.sqrt(b*b-4*a*c))/(2*a)]
```

```python
# Test 1
roots = find_roots(1, 1, 1)
if roots:
    print('The root(s) is (are) %s.' %', '.join([str(root) for root in roots]))
```

    The root(s) is (are) (-0.5+0.8660254037844386j), (-0.5-0.8660254037844386j).

```python
# Test 2
roots = find_roots(1, 2, 1)
if roots:
    print('The root(s) is (are) %s.' %', '.join([str(root) for root in roots]))
```

    The root(s) is (are) (-1+0j), (-1+0j).

```python
# Test 3
roots = find_roots(1, 2, 1)
if roots:
    print('The root(s) is (are) %s.' %', '.join([str(root) for root in roots]))
```

    The root(s) is (are) (-1+0j), (-1+0j).

```python
# Test 4
roots = find_roots(0, 0, 0)
if roots:
    print('The root(s) is (are) %s.' %', '.join([str(root) for root in roots]))
```

    Warning: wrong input! a and b cannot be zero simultaneously!
    All x

```python
# Test 5
roots = find_roots(0, 1, 1)
if roots:
    print('The root(s) is (are) %s.' %', '.join([str(root) for root in roots]))
```

    Warning: a = 0; it is a linear function!
    The root(s) is (are) -1.0.

```python
# Test 6
roots = find_roots(0, 0, 1)
if roots:
    print('The root(s) is (are) %s.' %', '.join([str(root) for root in roots]))
```

    Warning: wrong input! a and b cannot be zero simultaneously!