# K-Means Clustering algorithm

* *What is clustering?*

* The K-means algorithm is summarized as:
     * Set number of clusters, M
     * Initialize cluster centers
     * Do until Change in cluster centers is small:
         * FOR i = 1 to N
             * Determine the closest representative, $\Theta_j$, for $\mathbf{x}_i$
             * Set label for data point $i$ to $j$
         * FOR j = 1 to M
             * Update cluster representative $\Theta_j$ to the mean of the points with cluster label $j$

* The objective function for the K-means clustering algorithm is
\begin{equation}
J(\Theta, U) = \sum_{i=1}^N \sum_{j=1}^m u_{ij}\left\| \mathbf{x}_i - \theta_j \right\|^2
\end{equation}
where $u_{ij} \in \{0,1\}$ and is '1' for the $j$ index corresponding to the class label assigned to data point $\mathbf{x}_i$ and zero otherwise. The $\theta_j$ vector is the $j^{th}$ cluster representative.
* *How would you optimize this objective function?*

* Does the K-means algorithm find the *globally optimal* solution (i.e., the cluster centers and assignments that globally minimize the objective function)?
* Does the K-means algorithm make any assumptions on cluster shape?
* Given a data set with an unknown number of clusters, come up with a strategy for determining the ``right'' number of clusters.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from scipy import spatial
%matplotlib inline

def KMeans(X, C):
    MaxIter     = 10000;
    StopThresh  = 1e-5;

    #Initialize Cluster Centers by drawing randomly from input data (can use other
    # methods for initialization...)
    N       = X.shape[0] #number of data points
    d       = X.shape[1] #dimensionality
    rp      = np.random.permutation(N); #random permutation of numbers 1:N
    centers = X[rp[0:C],:]; #select first M data points sorted according to rp

    diff    = 1e100;
    iter    = 0;
    while((diff > StopThresh) & (iter < MaxIter)):
        #Assign data to closest cluster representative (using Euclidean distance)
        D   = spatial.distance.cdist(X, centers)
        L   = np.argmin(D, axis=1)

        #Update cluster centers
        centersPrev = centers.copy()
        for i in range(C):
            centers[i,:] = np.mean(X[L == i,:], axis=0)

        #Update diff & iteration count for stopping criteria
        diff = np.linalg.norm(centersPrev - centers)
        iter = iter+1
    return centers, L

```

```python
n_samples = 1500
n_clusters = 3

# Make Blob Data
X, y_blobs = datasets.make_blobs(n_samples=n_samples)

#Cluster
centers, L = KMeans(X,n_clusters)

#Plot Results
plt.figure(figsize=(12, 12))
plt.subplot(221)
plt.scatter(X[:, 0], X[:, 1], c=y_blobs)
plt.title("Blobs with True Labels")
plt.subplot(222)
plt.scatter(X[:, 0], X[:, 1], c=L)
plt.title("Clustered Blobs")

```

```python
# Anisotropicly distributed data
#some examples from: http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_assumptions.html

n_samples = 1500
n_clusters = 3

#generate data
transformation = [[ 0.60834549, -0.63667341], [-0.40887718, 0.85253229]]
X, y = datasets.make_blobs(n_samples=n_samples)
X = np.dot(X, transformation)

#cluster data
centers, L = KMeans(X,n_clusters)

#plot data
plt.figure(figsize=(12, 12))
plt.subplot(221)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("Data with True Labels")
plt.subplot(222)
plt.scatter(X[:, 0], X[:, 1], c=L)
plt.title("Clustered Results")

```

```python
# Data with different variances

n_samples = 1500
n_clusters = 3

#generate data
X, y = datasets.make_blobs(n_samples=n_samples,cluster_std=[1.0, 2.5, 0.5])

#cluster data
centers, L = KMeans(X,n_clusters)

#plot data
plt.figure(figsize=(12, 12))
plt.subplot(221)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("Data with True Labels")
plt.subplot(222)
plt.scatter(X[:, 0], X[:, 1], c=L)
plt.title("Clustered Results")

```

```python
# Uneven sized blobs

n_samples = 1500
n_clusters = 3

#generate data
X, y = datasets.make_blobs(n_samples=n_samples)
X = np.vstack((X[y == 0][:500], X[y == 1][:100], X[y == 2][:10]))
y = np.hstack((np.ones(500), 2*np.ones(100), 3*np.ones(10)))

#cluster data
centers, L = KMeans(X,n_clusters)

#plot data
plt.figure(figsize=(12, 12))
plt.subplot(221)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("Data with True Labels")
plt.subplot(222)
plt.scatter(X[:, 0], X[:, 1], c=L)
plt.title("Clustered Results")

```

```python
# Moons

n_samples = 1500
n_clusters = 3

#generate data
X, y = datasets.make_moons(n_samples=n_samples, noise=.05)

#cluster data
centers, L = KMeans(X,n_clusters)

#plot data
plt.figure(figsize=(12, 12))
plt.subplot(221)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("Data with True Labels")
plt.subplot(222)
plt.scatter(X[:, 0], X[:, 1], c=L)
plt.title("Clustered Results")

```

```python
# Circles

n_samples = 1500
n_clusters = 3

#generate data
X, y = datasets.make_circles(n_samples=n_samples, factor=.5, noise=.05)

#cluster data
centers, L = KMeans(X,n_clusters)

#plot data
plt.figure(figsize=(12, 12))
plt.subplot(221)
plt.scatter(X[:, 0], X[:, 1], c=y)
plt.title("Data with True Labels")
plt.subplot(222)
plt.scatter(X[:, 0], X[:, 1], c=L)
plt.title("Clustered Results")

```

# Cluster Validity

* *How would you evaluate clustering results?*  As discussed and illustrated in our first lecture, clustering results can be subjective and/or the desired result can be application dependent. One approach is the use of *cluster validity indices*

* There are many cluster validity indices in the literature.

* Cluster validity measures are used for a number of different goals.  For example, cluster validity metrics can be used to compare clustering results, try to determine the *correct* number of clusters, try to select the *correct* parameter settings, try to evaluate the appropriateness of the clustering result based on the data only (and not using another result or "ground truth" data).

* *External index* is used to measure how well a clustering result matches a set of supplied class labels.  This can be used to compare to "ground truth" as well as to compare different clustering results to see how similar they are (and how stable a particular clustering is on a data set across parameter settings and/or algorithms).  An example of an external index is the Rand index:

Given a set of $n$ data points, $\mathbf{X} = \{x_1, \ldots, x_n\}$ and two partitions (i.e., clustering results) of $\mathbf{X}$ to compare, $C = \{C_1, \ldots, C_r\}$, a partition of $\mathbf{X}$ into $r$ partitions, and $D = \{D_1, \ldots, D_s\}$, a partition of $\mathbf{X}$ into $s$ partitions, define the following:
* $a$, the number of pairs of elements in $X$ that are in the same subset in $C$ and in the same subset in $D$
* $b$, the number of pairs of elements in  $X$  that are in different subsets in $C$  and in different subsets in $D$
* $c$, the number of pairs of elements in $X$ that are in the same subset in $C$  and in different subsets in $D$
* $d$, the number of pairs of elements in  $X$  that are in different subsets in $C$  and in the same subset in $D$

The Rand index, $R$, is:
$R = \frac{a+b}{a+b+c+d}$
Intuitively,  $a + b$ can be considered as the number of agreements between $C$ and $D$ and $c + d$ as the number of disagreements between $C$ and $D$. The numerator is the number of agreements and the denominator is the total number of pairs (agreements and disagreements)

* There are many other external cluster validity indices, the Rand index is just one example!

* An *Internal index* is used to measure how well a clustering result is without using any external labels or other results.

* Many internal indices are based on measuring within-cluster vs. between-cluster variation with the idea being within cluster variation should be small and between cluster variation should be large.

* One example of an internal index is the Dunn's index:

Let $C_i$ be a cluster of vectors. Let $x$ and $y$ be any two $n$ dimensional feature vectors assigned to the same cluster $C_i$.

$\Delta_i =  \text{max}_{x,y \in C_i} d(x,y)$  calculates the maximum distance in a cluster.

 Let $\delta(C_i,C_j)$ be the intercluster distance metric, between clusters $C_i$ and $C_j$.   This intercluster distance can be computed in a number of ways.  For example, the minumum distance between any two points in the cluster or, alternatively, the distance between the cluster means.

With the above notation, if there are $m$ clusters, then the Dunn Index for the set is defined as:

$\mathit{D}_m = \frac{ \underset{ 1 \leqslant i < j \leqslant m}{\text{min}} \left.\delta(C_i,C_j)\right.}{ \underset{ 1 \leqslant k \leqslant m}{\text{max}} \left.\Delta_k\right.}$

Like the intercluster distance, alternative measures for the internal cluster distance can be used.  Some examples are:

$\Delta_i =   \dfrac{1}{|C_i| (|C_i| - 1)} {\sum}_{x , y \in C_i, x \neq y} d(x,y)$ calculates the mean distance between all pairs in a cluster.

$\Delta_i =   \dfrac{\underset{x \in C_i}{\sum} d(x,\mu)}{|C_i|} , \mu =   \dfrac{\underset{x \in C_i}{\sum} x}{|C_i|}$ calculates the distance of all the points from the mean.

```python

```


# The Iterative Relaxation Method
> Written by Ryan Soklaski

## Understanding Fixed-Points
In mathematics, a function $f(x)$ is said to have a "fixed-point" solution if there exists a value $x_{*}$ such that $f(x_{*}) = x_{*}$. That is, $f$ maps $x_{*}$ to itself. As a simple example, given $f(x) = x^{2}$, check that $f$ has a fixed-points at $x = 0$ and $x = 1$.

For a less trivial example, let's see if the function $f(x) = x^{2} - 1$ has any fixed points. That is, we want to find all solutions to $f(x) = x$:

\begin{equation}
x^{2} - 1 = x \\\
x^{2} - x - 1 = 0 \\\
x = -0.61803...,\; x = 1.61803...
\end{equation}

We made use of the [quadratic formula](https://en.wikipedia.org/wiki/Quadratic_formula) to find the two fixed-points for $f(x) = x^{2} - 1$.

Consider that solving the fixed-point equation $f(x) = x$ is tantamount to finding where the values of $x$ where $f(x)$ intersects the line $y = x$:

There are many functions such that one cannot simply solve for $x$. For example, neither of the following equations are amenable to any algebraic manipulation that would reveal their solutions :
\begin{equation}
\sin{x} = x \\\
e^{x} - 1 = x
\end{equation}

Such equations are thus known as *transcendental* equations. How, then, can we find the fixed-point solutions (if they exist) for such functions? There is a simple numerical method, known as the relaxation method, that can be used towards this end.

## The Relaxation Method
Suppose for now that $f(x)$ has one fixed-point solution, $x_{*}$. The relaxation method allows us to "guess" a fixed-point solution, and then iteratively improve upon this guess until you have arrived at a value that is sufficiently close to the true fixed-point, $x_{*}$.

Specifically, given your initial guess of $f$'s fixed-point, $x_{0}$, you can generate a better guess by simply feeding $x_{0}$ to $f$, and using the output as your updated guess, $x_{1}$:
\begin{equation}
x_{1} = f(x_{0})
\end{equation}

You can then improve this guess by feeding $x_{1}$ to $f$ and using the output as the next guess. Repeating this process $n$ times will produce $n$ consecutively-improved guesses at the true fixed-point, $x_{*}$:
\begin{equation}
x_{1} = f(x_{0}) \\\
x_{2} = f(x_{1}) \\\
x_{3} = f(x_{2}) \\\
... \\\
x_{n} = f(x_{n-1}) \\\
\\\
x_{n} \approx x_{*}
\end{equation}

For example, let's find a fixed point for $f(x) = \tanh{5x}$ taking an initial guess of $0.5$.
\begin{equation}
-0.9866143 = f(0.5) \\\
-0.9998962 = f(-0.9866143) \\\
-0.99990912 = f(-0.99990911) \\\
-0.99990912 = f(-0.99990912) \\\
\end{equation}

We arrived at a fixed-point (within 7 decimal-places of precision) after four iterations!

### Caveats to the relaxation method
For all its simplicity, the relaxation method is not a completely robust solution for finding fixed points, in which case this method can only potentially find one, for a given initial guess. It will fail if your function does not have fixed points. It is also very much possible for a function to have multiple fixed points. Additionally, this iterative process can "blow up" and lead you to ever-growing numbers if you use a "bad" starting guess. It is possible for the relaxation method to get stuck in a loop. If you try to find the fixed points for $x^2 - 1$ using an initial guess of $x_{o} = 0.5$, you will find that you eventually repeatedly guess 0, -1, 0, -1, 0, .... Be aware of these pitfalls when you are testing your code - they are a fundamental issue of the relaxation method, and not a symptom of bad code.

You need not worry about accounting for these issues in your code. You will never be given pathological functions/guesses that would cause these issues, in this homework.

### Problem #1
Write a relaxation-method function that accepts three arguments:
- a python function, which accepts a number as an input, and returns a float as an output
- an initial guess for the fixed-point, $x_{0}$, a floating-point number
- the number of iterations, $n$, to perform the relaxation method on the provided function

Your function should return a list of the $n+1$ numbers: the initial guess and the $n$ guesses that you generate using the relaxation method.

So, in the context of the preceding relaxation example, I could define the function:
```python
from math import tanh
def f(x):
    return tanh(5*x)
```
and then calling your relaxation function, passing it this function, an initial guess of $x_{o}=0.5$, and instructing it to perform 5 iterations, should produce the following list:
```python
>>> relaxation_method(f, xo=-.5, num_it=5)
[-0.5,
 -0.98661429815143031,
 -0.99989620032332682,
 -0.99990910997226823,
 -0.99990912170456125,
 -0.99990912171522284]
```
That is, your `relaxation_method` function should call `f(xo)` to obtain the updated-guess for the fixed point, as the first iteration, and so on. Use the parameters provided in this example to test your code.

```python
def relaxation_method1(func, xo, num_it):
    """ Performs the relaxation method to find a fixed-point for `func`,
        given the initial guess `xo`. The relaxation process is carried out for
        `num_it` steps.

        Parameters
        ----------
        func : Callable[[float], float]
            The function whose fixed point is being found.
        xo : float
            The initial "guess" value.
        num_it : int
            The number of relaxation-iterations to perform.

        Returns
        -------
        List[float]
            A list of the initial guess, and all of the subsequent guesses generated
            by the relaxation method. """
    output = []
    output.append(xo)
    curr = xo
    i = 0
    while i != num_it:
        curr = func(curr)
        output.append(curr)
        i += 1
    return output

```

```python
# run this cell to grade your work
from bwsi_grader.python.relaxation_method import grader1
grader1(relaxation_method1)
```

    Finding fixed-points for the function: f(x) = x**2
    Finding fixed-points for the function: f(x) = tanh(4*x)

    ============================== ALL TESTS PASSED! ===============================
    Your submission code: bwb8e4b8ae3afecb32279a3941952049fff73d1181e93d77b45f39506c
    ================================================================================

## Problem #2
Our current implementation of the relaxation method is quite crude in that we must specify the number of iterations that it performs, and then simply look at the output to see if we have converged to a fix-point. It would instead be better if we could have our algorithm check its own numbers to see if they are converging to a single value, and then terminate itself if it has converged.

We can measure how close our most recent guess is to a fixed-point by looking at our most-recent three guesses $x_{n-2}, x_{n-1}, x_{n}$ , and seeing if $x_{n-1}$ and $x_{n}$ are closer to one another than are $x_{n-1}$ and $x_{n-2}$. Skipping a formal derivation, the following formula gives an upper-bound estimate on how close $x_{n}$ is to a true fixed-point:

\begin{equation}
\epsilon_{n} = \lvert\frac{(x_n - x_{n-1})^2}{2x_{n-1} - x_{n-2} - x_{n}}\rvert
\end{equation}

That is, if your previous three guesses were $1.0$, then $1.63$, and then $1.80$, plugging these into the preceding formula produces an error bound of $\epsilon = 0.06$. This means that the guess $1.80$ is within $0.06$ of the true fixed-point. To prevent division-by-zero errors, if your denominator is equal to 0.0, replace it with the value `1e-14`.

Armed with this formula, we can now write a much better algorithm, which can operate based on a tolerance rather than a strict number of iterations.

Write a second version of the relaxation-method. This function should accept four arguments:
- a python function, which accepts a number as an input, and returns a float as an output
- an initial guess for the fixed-point, $x_{0}$, a floating-point number
- a tolerance value, a positive-valued floating-point number
- a max number of iterations that your algorithm is permitted to run

Your algorithm should produce guesses until $\epsilon_{n}$ is smaller than the specified tolerance value, or until the number of guesses produced (including the initial guess) matches/exceeds the max number of iterations. Like the last function, it should return a list of all the guesses. You will need to have three guesses before you can assess the tolerance.

```python
def relaxation_method2(func, xo, tol, max_it):
    """ Performs the relaxation method to find a fixed-point for `func`,
        given the initial guess `xo`. The relaxation process is carried out for
        `num_it` steps.

        Parameters
        ----------
        func : Callable[[float], float]
            The function whose fixed point is being found.
        xo : float
            The initial "guess" value.
        tol : float
            A positive value that sets the maximum permissable error
            in the final fixed-point estimate.
        max_it : int
            The maximum number relaxation-guesses (i.e. the length of the
            list you are creating) allotted before the
            algorithm will end. The length of the list you return should
            never exceed this number.

        Returns
        -------
        List[float]
            A list of the initial guess, and all of the subsequent guesses generated
            by the relaxation method. """
    output = []
    output.append(xo)
    curr = xo
    i = 0
    tolerance = 0
    while i < max_it - 1:
        if i < 2:
            curr = func(curr)
            output.append(curr)
            i += 1
        else:
            denominator = (2*(output[len(output)-2]))-(output[len(output)-3])-(output[len(output)-1])
            if denominator == 0.0:
                denominator = 1e-14
            tolerance = abs((((output[len(output)-1])-(output[len(output)-2]))**2)/denominator)
            if tolerance <= tol:
                return output
            else:
                curr = func(curr)
                output.append(curr)
                i += 1
    return output
```

```python
# run this cell to grade your work
from bwsi_grader.python.relaxation_method import grader2
grader2(relaxation_method2)
```

    Finding fixed-points for the function: f(x) = 2 - exp(-x)

    ============================== ALL TESTS PASSED! ===============================
    Your submission code: bw7ab135590053bf38cdb655fff9447e2553dfc315e04ed295c498190c
    ================================================================================

## A fun application of the relaxation method
The relaxation method is not just a parlor trick, nor are fixed-point equations a relic of pure-mathematics. The following will put the relaxation-function that you wrote to use, in order to solve a very real physics problem. You don't need to do any work here, just follow along and enjoy!

You have likely held a bar-magnet before - a special kind of metal that can use its magnetic field to push or pull on other magnets. Such a material is known as a ferromagnet, and it's magnetic properties are created by coordinated behavior among the electrons belonging to its atoms. The electrons in a ferromagnet naturally coordinate in such a way to create an overall magnetic field throughout and around the material. However, heating up a ferromagnet will jostle its atoms and electrons around, disturbing the coordination of the electrons and thus weakening the net magnetic field of the material. If you set out to describe the statistical behavior of a ferromagnetic material's electrons, you will eventually find that the strength of its magnetization, $M$, depends on temperature, $T$, according to the following equation:

\begin{equation}
M = \mu\tanh{\frac{JM}{k_{B}T}}
\end{equation}

where $\mu$ and $J$ are physical constants particular to the specific ferromagnetic material we are interested in, and $k_B$ is a fundamental constant from statistical mechanics. If $M = 0$, then the material is completely non-magnetic. For simplicity's sake, we'll set these constants to 1, without changing the essence of the problem at hand. Thus our equation for the magnetization of our material becomes:

\begin{equation}
M = \tanh{\frac{M}{T}}
\end{equation}

This is a fixed-point equation! We can pick a value of $T$, and then use the relaxation method to solve for $M$. By varying $T$, we can measure the magnetization for each value of $T$, and thus understand how the material's magnetization depends on temperature.

In the following code, we will pick a value of $T$, and then solve for $M$ (within a given tolerance). Then we pick our next value of $T$ and repeat the process. Ultimately, we will have a collection of temperature values and corresponding magnetization values. We will plot $M$ vs $T$ to understand the temperature dependence for a ferromagnetic material.

```python
# just run this cell - you don't need to change any of this code

import numpy as np
# `temps` is 1000 evenly-spaced values within [0, 1.5]
temps = np.linspace(0, 1.5, 1000)

mags = []
for T in temps:

    # define the magnetization function, given
    # the current temperature value
    def mag_func(m, temp=T):
        return np.tanh(m / temp) if temp > 0. else 1.

    # Use the relaxation value to compute M within an error
    # of 1e-6.
    mag = relaxation_method2(mag_func, 1., 1e-6, 1000)[-1]
    mags.append(mag)

print("number of magnetization-values computed: {}".format(len(mags)))
```

    number of magnetization-values computed: 1000

```python
# just run this cell - you don't need to change any of this code

# Plotting M vs T
import matplotlib.pyplot as plt
%matplotlib notebook
fig, ax = plt.subplots()
ax.plot(temps, mags)
ax.grid(True)
ax.set_ylabel("Magnetization")
ax.set_xlabel(r"$T$")
ax.set_title("Magnetization vs Temperature");
```

    <IPython.core.display.Javascript object>

See that $M = 0$ for high temperatures, meaning that the material is non-magnetic when it is at a temperature greater than 1 (no actual units, since we set all those constants to be $1$). However, once the material is cooled to $T \leq 1$, the material suddenly magnetizes, and its magnetization strengthens as you cool it further.

This sudden magnetization at $T = 1$ is a *phase transition*. This is very similar to water freezing: water's atoms molecules will rapidly begin to form a crystal once they are cooled to 0-celsius or below. Similarly, we see that a ferromagnetic material's electrons will suddenly be able to coordinate and produce a net magnetic field throughout the material, once they are cooled to $T=1$ and below.

This is a no-joke physics problem that we were able to solve thanks to the relaxation method! If you look up the constants, $\mu$ and $J$, for a specific material, you can repeat this computation to produce its actual magnetic phase diagram. This rules!


```python
# symbols and expressions
from sympy import *
init_printing(use_unicode=True)
```

```python
x = Symbol('x')
y,z = symbols('y,z')
```

```python
f = Function('f')
```

```python
f = x**2 + y**2 + z**2
```

```python
print(f)
```

    x**2 + y**2 + z**2

```python
pprint(f)
```

     2    2    2
    x  + y  + z

```python
f = x**2 - 3*x + 2
```

```python
pprint(f)
```

     2
    x  - 3⋅x + 2

```python
# evaluate f for a particular value of x, say x=1.0
pprint(f.subs(x,1.0))
```

    0

```python
pprint(f.subs(x,3.0))
```

    2.00000000000000

```python
f = (x**2 - 3*x + 2)/(x**2-x)
```

```python
pprint(f)
```

     2
    x  - 3⋅x + 2
    ────────────
        2
       x  - x

```python
# simplify expressions
simplify(f)
```

```python
# expand an expression
f = (x+2)**3*(x-3)**2
```

```python
pprint(f)
```

           2        3
    (x - 3) ⋅(x + 2)

```python
expand(f)
```

```python
# finding roots
f = x**2 - 3*x + 2
```

```python
solve(f,x)
```

```python
g = (x+2)**3*(x-3)**2
```

```python
solve(g,x)
```

```python
f = x**2 + x + 1
```

```python
y=solve(f,x)
```

```python
pprint(y)
```

    ⎡  1   √3⋅ⅈ    1   √3⋅ⅈ⎤
    ⎢- ─ - ────, - ─ + ────⎥
    ⎣  2    2      2    2  ⎦

```python
# show the values are numerals
y[0].evalf()
```

```python
y[1].evalf()
```

```python
for z in y:
    pprint(z.evalf())
```

    -0.5 - 0.866025403784439⋅ⅈ
    -0.5 + 0.866025403784439⋅ⅈ

```python
# solution of a set of equations
x,y,z = symbols('x y z')
```

```python
eq1 = Eq(x+y+z,0)
```

```python
print(eq1)
```

    Eq(x + y + z, 0)

```python
eq2 = Eq(2*x-y-z,10)
eq3 = Eq(y+2*z,5)
```

```python
solve([eq1, eq2, eq3],[x,y,z])
```


# Lecture 2

We now look at solving second-order ordinary differential equations using a computer algebra system.

### Initialising SymPy

To use Sympy, we first need to import it and call `init_printing()` to get nicely typeset equations:

```
from sympy import *

# This initialises pretty printing
init_printing()
from IPython.display import display

# This command makes plots appear inside the browser window
%matplotlib inline
```

## Mass-spring-damper system

The differential equation that governs an unforced, single degree-of-freedom mass-spring-damper system is

$$
m \frac{d^{2}y}{dx^{2}} + \lambda \frac{dy}{dx} + ky = 0
$$

To solve this problem using SymPy, we first define the symbols $t$ (time), $m$ (mass), $\lambda$ (damper coefficient) and $k$ (spring stiffness), and the function $y$ (displacement):

```
t, m, lmbda, k = symbols("t m lambda k")
y = Function("y")
```

Note that we mis-spell $\lambda$ as `lmbda` because `lambda` is a protected keyword in Python.

Next, we define the differential equation, and print it to the screen:

```
eqn = Eq(m*Derivative(y(t), t, t) + lmbda*Derivative(y(t), t) + k*y(t), 0)
display(eqn)
```

Checking the order of the ODE:

```
print("This order of the ODE is: {}".format(ode_order(eqn, y(t))))
```

    This order of the ODE is: 2

and now classifying the ODE:

```
print("Properties of the ODE are: {}".format(classify_ode(eqn)))
```

    Properties of the ODE are: ('nth_linear_constant_coeff_homogeneous', '2nd_power_series_ordinary')

we see as expected that the equation is linear, constant coefficient, homogeneous and second order.

The `dsolve` function solves the differential equation:

```
y = dsolve(eqn, y(t))
display(y)
```

The solution looks very complicated because we have not specified values for the constants $m$, $\lambda$ and $k$. The nature of the solution depends heavily on the relative values of the coefficients, as we will see later. We have four constants because the most general case the solution is complex, with two complex constants having four real coefficients.

Note that the solution is make up of expoential functions and sinusoidal functions. This is typical of second-order ODEs.

## Second order, constant coefficient equation

We'll now solve

$$
\frac{d^{2}y}{dx^{2}} + 2 \frac{dy}{dx} - 3 y = 0
$$

The solution for this problem will appear simpler because we have concrete values for the coefficients.

Entering the differential equation:

```
y = Function("y")
x = symbols("x")
eqn = Eq(Derivative(y(x), x, x) + 2*Derivative(y(x), x) - 3*y(x), 0)
display(eqn)
```

Solving this equation,

```
y1 = dsolve(eqn)
display(y1)
```

which is the general solution. As expected for a second-order equation, there are two constants.

Note that the general solution is of the form

$$
y = C_{1} e^{\lambda_{1} x} + C_{2} e^{\lambda_{2} x}
$$

The constants $\lambda_{1}$ and $\lambda_{2}$ are roots of the \emph{characteristic} equation

$$
\lambda^{2} + 2\lambda - 3 = 0
$$

This quadratic equation is trivial to solve, but for completeness we'll look at how to solve it using SymPy. We first define the quadratic equation:

```
eqn = Eq(lmbda**2 + 2*lmbda -3, 0)
display(eqn)
```

and then compute the roots:

```
solve(eqn)
```

which as expected are the two exponents in the solution to the differential equattion.


# Linear Equations
The equations in the previous lab included one variable, for which you solved the equation to find its value. Now let's look at equations with multiple variables. For reasons that will become apparent, equations with two variables are known as linear equations.

## Solving a Linear Equation
Consider the following equation:

\begin{equation}2y + 3 = 3x - 1 \end{equation}

This equation includes two different variables, **x** and **y**. These variables depend on one another; the value of x is determined in part by the value of y and vice-versa; so we can't solve the equation and find absolute values for both x and y. However, we *can* solve the equation for one of the variables and obtain a result that describes a relative relationship between the variables.

For example, let's solve this equation for y. First, we'll get rid of the constant on the right by adding 1 to both sides:

\begin{equation}2y + 4 = 3x \end{equation}

Then we'll use the same technique to move the constant on the left to the right to isolate the y term by subtracting 4 from both sides:

\begin{equation}2y = 3x - 4 \end{equation}

Now we can deal with the coefficient for y by dividing both sides by 2:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Our equation is now solved. We've isolated **y** and defined it as 3x-4/2

While we can't express **y** as a particular value, we can calculate it for any value of **x**. For example, if **x** has a value of 6, then **y** can be calculated as:

\begin{equation}y = \frac{3\cdot6 - 4}{2} \end{equation}

This gives the result 14/2 which can be simplified to 7.

You can view the values of **y** for a range of **x** values by applying the equation to them using the following Python code:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the solved equation to x
df['y'] = (3*df['x'] - 4) / 2

#Display the dataframe
df
```

      x
      y

      0
      -10
      -17.0

      1
      -9
      -15.5

      2
      -8
      -14.0

      3
      -7
      -12.5

      4
      -6
      -11.0

      5
      -5
      -9.5

      6
      -4
      -8.0

      7
      -3
      -6.5

      8
      -2
      -5.0

      9
      -1
      -3.5

      10
      0
      -2.0

      11
      1
      -0.5

      12
      2
      1.0

      13
      3
      2.5

      14
      4
      4.0

      15
      5
      5.5

      16
      6
      7.0

      17
      7
      8.5

      18
      8
      10.0

      19
      9
      11.5

      20
      10
      13.0

We can also plot these values to visualize the relationship between x and y as a line. For this reason, equations that describe a relative relationship between two variables are known as *linear equations*:

```python
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey", marker = "o")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.show()
```

In a linear equation, a valid solution is described by an ordered pair of x and y values. For example, valid solutions to the linear equation above include:
- (-10, -17)
- (0, -2)
- (9, 11.5)

The cool thing about linear equations is that we can plot the points for some specific ordered pair solutions to create the line, and then interpolate the x value for any y value (or vice-versa) along the line.

## Intercepts
When we use a linear equation to plot a line, we can easily see where the line intersects the X and Y axes of the plot. These points are known as *intercepts*. The *x-intercept* is where the line intersects the X (horizontal) axis, and the *y-intercept* is where the line intersects the Y (horizontal) axis.

Let's take a look at the line from our linear equation with the X and Y axis shown through the origin (0,0).

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.show()
```

The x-intercept is the point where the line crosses the X axis, and at this point, the **y** value is always 0. Similarly, the y-intercept is where the line crosses the Y axis, at which point the **x** value is 0. So to find the intercepts, we need to solve the equation for **x** when **y** is 0.

For the x-intercept, our equation looks like this:

\begin{equation}0 = \frac{3x - 4}{2} \end{equation}

Which can be reversed to make it look more familar with the x expression on the left:

\begin{equation}\frac{3x - 4}{2} = 0 \end{equation}

We can multiply both sides by 2 to get rid of the fraction:

\begin{equation}3x - 4 = 0 \end{equation}

Then we can add 4 to both sides to get rid of the constant on the left:

\begin{equation}3x = 4 \end{equation}

And finally we can divide both sides by 3 to get the value for x:

\begin{equation}x = \frac{4}{3} \end{equation}

Which simplifies to:

\begin{equation}x = 1\frac{1}{3} \end{equation}

So the x-intercept is 11/3 (approximately 1.333).

To get the y-intercept, we solve the equation for y when x is 0:

\begin{equation}y = \frac{3\cdot0 - 4}{2} \end{equation}

Since 3 x 0 is 0, this can be simplified to:

\begin{equation}y = \frac{-4}{2} \end{equation}

-4 divided by 2 is -2, so:

\begin{equation}y = -2 \end{equation}

This gives us our y-intercept, so we can plot both intercepts on the graph:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.annotate('x-intercept',(1.333, 0))
plt.annotate('y-intercept',(0,-2))
plt.show()
```

The ability to calculate the intercepts for a linear equation is useful, because you can calculate only these two points and then draw a straight line through them to create the entire line for the equation.

## Slope
It's clear from the graph that the line from our linear equation describes a slope in which values increase as we travel up and to the right along the line. It can be useful to quantify the slope in terms of how much **x** increases (or decreases) for a given change in **y**. In the notation for this, we use the greek letter Δ (*delta*) to represent change:

\begin{equation}slope = \frac{\Delta{y}}{\Delta{x}} \end{equation}

Sometimes slope is represented by the variable ***m***, and the equation is written as:

\begin{equation}m = \frac{y_{2} - y_{1}}{x_{2} - x_{1}} \end{equation}

Although this form of the equation is a little more verbose, it gives us a clue as to how we calculate slope. What we need is any two ordered pairs of x,y values for the line - for example, we know that our line passes through the following two points:
- (0,-2)
- (6,7)

We can take the x and y values from the first pair, and label them x1 and y1; and then take the x and y values from the second point and label them x2 and y2. Then we can plug those into our slope equation:

\begin{equation}m = \frac{7 - -2}{6 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{7 + 2}{6 - 0} \end{equation}

That gives us the result 9/6 which is 11/2 or 1.5 .

So what does that actually mean? Well, it tells us that for every change of **1** in x, **y** changes by 11/2 or 1.5. So if we start from any point on the line and move one unit to the right (along the X axis), we'll need to move 1.5 units up (along the Y axis) to get back to the line.

You can plot the slope onto the original line with the following Python code to verify it fits:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# set the slope
m = 1.5

# get the y-intercept
yInt = -2

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

### Slope-Intercept Form
One of the great things about algebraic expressions is that you can write the same equation in multiple ways, or *forms*. The *slope-intercept form* is a specific way of writing a 2-variable linear equation so that the equation definition includes the slope and y-intercept. The generalised slope-intercept form looks like this:

\begin{equation}y = mx + b \end{equation}

In this notation, ***m*** is the slope and ***b*** is the y-intercept.

For example, let's look at the solved linear equation we've been working with so far in this section:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Now that we know the slope and y-intercept for the line that this equation defines, we can rewrite the equation as:

\begin{equation}y = 1\frac{1}{2}x + -2 \end{equation}

You can see intuitively that this is true. In our original form of the equation, to find y we multiply x by three, subtract 4, and divide by two - in other words, x is half of 3x - 4; which is 1.5x - 2. So these equations are equivalent, but the slope-intercept form has the advantages of being simpler, and including two key pieces of information we need to plot the line represented by the equation. We know the y-intecept that the line passes through (0, -2), and we know the slope of the line (for every x, we add 1.5 to y.

Let's recreate our set of test x and y values using the slope-intercept form of the equation, and plot them to prove that this  describes the same line:

```python
%matplotlib inline

import pandas as pd
from matplotlib import pyplot as plt

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Define slope and y-intercept
m = 1.5
yInt = -2

# Add a y column by applying the slope-intercept equation to x
df['y'] = m*df['x'] + yInt

# Plot the line
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# label the y-intercept
plt.annotate('y-intercept',(0,yInt))

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

```python

```

```python

```

Basics Of Algebra by Hiren/01-02-Linear Equations.ipynb


# Quadratic Equations

Consider the following equation:

\begin{equation}y = 2(x - 1)(x + 2)\end{equation}

If you multiply out the factored ***x*** expressions, this equates to:

\begin{equation}y = 2x^{2} + 2x - 4\end{equation}

Note that the highest ordered term includes a squared variable (x2).

Let's graph this equation for a range of ***x*** values:

```R
# Create a dataframe with an x column containing values to plot
df = data.frame(x = seq(-9, 8))

# Add a y column by applying the quadratic equation to x
df$y = 2*df$x**2 + 2 *df$x - 4

## Plot the parabola
library(ggplot2)
library(repr)
options(repr.plot.width=4, repr.plot.height=4)
ggplot(df, aes(x,y)) +
       geom_line(color = 'blue', size = 1) +
       geom_hline(yintercept=0) + geom_vline(xintercept=0)
```

Note that the graph shows a *parabola*, which is an arc-shaped line that reflects the x and y values calculated for the equation.

Now let's look at another equation that includes an ***x2*** term:

\begin{equation}y = -2x^{2} + 6x + 7\end{equation}

What does that look like as a graph?:

```R
# Create a dataframe with an x column containing values to plot
df = data.frame(x = seq(-8,11))

# Add a y column by applying the quadratic equation to x
df$y = -2*df$x**2 + 6*df$x + 7

## Plot the parabola
library(ggplot2)
library(repr)
options(repr.plot.width=4, repr.plot.height=4)
ggplot(df, aes(x,y)) +
       geom_line(color = 'blue', size = 1) +
       geom_hline(yintercept=0) + geom_vline(xintercept=0)
```

Again, the graph shows a parabola, but this time instead of being open at the top, the parabola is open at the bottom.

Equations that assign a value to ***y*** based on an expression that includes a squared value for ***x*** create parabolas. If the relationship between ***y*** and ***x*** is such that ***y*** is a *positive* multiple of the ***x2*** term, the parabola will be open at the top; when ***y*** is a *negative* multiple of the ***x2*** term, then the parabola will be open at the bottom.

These kinds of equations are known as *quadratic* equations, and they have some interesting characteristics. There are several ways quadratic equations can be written, but the *standard form* for quadratic equation is:

\begin{equation}y = ax^{2} + bx + c\end{equation}

Where ***a***, ***b***, and ***c*** are numeric coefficients or constants.

Let's start by examining the parabolas generated by quadratic equations in more detail.

## Parabola Vertex and Line of Symmetry
Parabolas are symmetrical, with x and y values converging exponentially towards the highest point (in the case of a downward opening parabola) or lowest point (in the case of an upward opening parabola). The point where the parabola meets the line of symmetry is known as the *vertex*.

Run the following cell to see the line of symmetry and vertex for the two parabolas described previously (don't worry about the calculations used to find the line of symmetry and vertex - we'll explore that later):

```R
plot_parabola = function(a, b, c){
    # get the x value for the line of symmetry
    vx = (-1*b)/(2*a)

    # get the y value when x is at the line of symmetry
    vy = a*vx**2 + b*vx + c

    # Create a dataframe with an x column containing values from x-10 to x+10
    minx = as.integer(vx - 10)
    maxx = as.integer(vx + 10)
    df = data.frame(x = seq(minx, maxx))

    # Add a y column by applying the quadratic equation to x
    df$y = a*df$x**2 + b*df$x + c

    # get min and max y values
    miny = min(df$y)
    maxy = max(df$y)

    ## data frame for line of symmetry
    symmetry = data.frame(sx = c(vx,vx), sy = c(miny,maxy))

    ## Plot the parabola
    ggplot(df, aes(x,y)) +
           geom_line(color = 'blue', size = 1) +
           geom_line(data = symmetry, aes(sx,sy), color = 'magenta', size = 1) +
           geom_point(data = symmetry, aes(sx,sy), color = 'magenta', size = 2)  +
           annotate("text", x = vx, y = -10, label = "Vertex") +
           geom_hline(yintercept=0) + geom_vline(xintercept=0)
    }

plot_parabola(2, 2, -4)

plot_parabola(-2, 3, 5)
```

## Parabola Intercepts
Recall that linear equations create lines that intersect the **x** and **y** axis of a graph, and we call the points where these intersections occur *intercepts*. Now look at the graphs of the parabolas we've worked with so far. Note that these parabolas both have a y-intercept; a point where the line intersects the y axis of the graph (in other words, when x is 0). However, note that the parabolas have *two* x-intercepts; in other words there are two points at which the line crosses the x axis (and y is 0). Additionally, imagine a downward opening parabola with its vertex at -1, -1. This is perfectly possible, and the line would never have an x value greater than -1, so it would have *no* x-intercepts.

Regardless of whether the parabola crosses the x axis or not, other than the vertex, for every ***y*** point in the parabola, there are *two* ***x*** points; one on the right (or positive) side of the axis of symmetry, and one of the left (or negative) side. The implications of this are what make quadratic equations so interesting. When we solve the equation for ***x***, there are *two* correct answers.

Let's take a look at an example to demonstrate this. Let's return to the first of our quadratic equations, and we'll look at it in its *factored* form:

\begin{equation}y = 2(x - 1)(x + 2)\end{equation}

Now, let's solve this equation for a ***y*** value of 0. We can restate the equation like this:

\begin{equation}2(x - 1)(x + 2) = 0\end{equation}

The equation is the product of two expressions **2(x - 1)** and **(x + 2)**. In this case, we know that the product of these expressions is 0, so logically *one or both of the expressions must return 0*.

Let's try the first one:

\begin{equation}2(x - 1) = 0\end{equation}

If we distrbute this, we get:

\begin{equation}2x - 2 = 0\end{equation}

This simplifies to:

\begin{equation}2x = 2\end{equation}

Which gives us a value for *x* of **1**.

Now let's try the other expression:

\begin{equation}x + 2 = 0\end{equation}

This gives us a value for *x* of **-2**.

So, when *y* is **0**, *x* is **-2** or **1**. Let's plot these points on our parabola:

```R
plot_parabola_limits = function(a, b, c){
    # get the x value for the line of symmetry
    vx = (-1*b)/(2*a)

    # get the y value when x is at the line of symmetry
    vy = a*vx**2 + b*vx + c

    # Create a dataframe with an x column containing values from x-10 to x+10
    minx = as.integer(vx - 10)
    maxx = as.integer(vx + 10)
    df = data.frame(x = seq(minx, maxx))

    # Add a y column by applying the quadratic equation to x
    df$y = a*df$x**2 + b*df$x + c

    # get min and max y values
    miny = min(df$y)
    maxy = max(df$y)

    ## data frame for line of symmetry
    symmetry = data.frame(sx = c(vx,vx), sy = c(miny,maxy))

    ## data frame for xlimits
    xlims = data.frame(x = c(-2,1), y = c(0,0))

    ## Plot the parabola
    ggplot(df, aes(x,y)) +
           geom_line(color = 'blue', size = 1) +
           geom_line(data = symmetry, aes(sx,sy), color = 'magenta', size = 1) +
           geom_point(data = symmetry, aes(sx,sy), color = 'magenta', size = 2)  +
           geom_point(data = xlims, aes(x,y), color = 'red', size = 2) +
           geom_text(data = xlims, aes(label=c('x1','x2')),hjust=-0.5, vjust=0) +
           annotate("text", x = vx, y = -10, label = "Vertex") +
           geom_hline(yintercept=0) + geom_vline(xintercept=0)
    }

plot_parabola_limits(2, 2, -4)
```

So from the plot, we can see that both of the values we calculated for ***x*** align with the parabola when ***y*** is 0. Additionally, because the parabola is symmetrical, we know that every pair of ***x*** values for each ***y*** value will be equidistant from the line of symmetry, so we can calculate the ***x*** value for the line of symmetry as the average of the ***x*** values for any value of ***y***. This in turn means that we know the ***x*** coordinate for the vertex (it's on the line of symmetry), and we can use the quadratic equation to calculate ***y*** for this point.

## Solving Quadratics Using the Square Root Method
The technique we just looked at makes it easy to calculate the two possible values for ***x*** when ***y*** is 0 if the equation is presented as the product two expressions. If the equation is in standard form, and it can be factored, you could do the necessary manipulation to restate it as the product of two expressions. Otherwise, you can calculate the possible values for x by applying a different method that takes advantage of the relationship between squared values and the square root.

Let's consider this equation:

\begin{equation}y = 3x^{2} - 12\end{equation}

Note that this is in the standard quadratic form, but there is no *b* term; in other words, there's no term that contains a coeffecient for  ***x*** to the first power. This type of equation can be easily solved using the square root method. Let's restate it so we're solving for ***x*** when ***y*** is 0:

\begin{equation}3x^{2} - 12 = 0\end{equation}

The first thing we need to do is to isolate the ***x2*** term, so we'll remove the constant on the left by adding 12 to both sides:

\begin{equation}3x^{2} = 12\end{equation}

Then we'll divide both sides by 3 to isolate x2:

\begin{equation}x^{2} = 4\end{equation}

No we can isolate ***x*** by taking the square root of both sides. However, there's an additional consideration because this is a quadratic equation. The ***x*** variable can have two possibe values, so we must calculate the *principle* and *negative* square roots of the expression on the right:

\begin{equation}x = \pm\sqrt{4}\end{equation}

The principle square root of 4 is 2 (because 22 is 4), and the corresponding negative root is -2 (because -22 is also 4); so *x* is **2** or **-2**.

Let's see this in R, and use the results to calculate and plot the parabola with its line of symmetry and vertex:

```R
y = 0
x1 = as.integer(-sqrt(y + 12 / 3))
x2 = as.integer(sqrt(y + 12 / 3))

# Create a dataframe with an x column containing some values to plot
df = data.frame(x = seq(x1-10, x2+10))

# Add a y column by applying the quadratic equation to x
df$y = 3*df$x**2 - 12

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = 3*vx**2 - 12

# get min and max y values
miny = min(df$y)
maxy = max(df$y)

## data frame for line of symmetry
symmetry = data.frame(sx = c(vx,vx), sy = c(miny,maxy))

## data frame for xlimits
xlims = data.frame(x = c(x1,x2), y = c(0,0))

## Plot the parabola
ggplot(df, aes(x,y)) +
        geom_line(color = 'blue', size = 1) +
        geom_line(data = symmetry, aes(sx,sy), color = 'magenta', size = 1) +
        geom_point(data = symmetry, aes(sx,sy), color = 'magenta', size = 2)  +
        geom_point(data = xlims, aes(x,y), color = 'red', size = 2) +
        geom_text(data = xlims, aes(label=c('x1','x2')),hjust=-0.5, vjust=0) +
        annotate("text", x = vx, y = -20, label = "Vertex") +
        geom_hline(yintercept=0) + geom_vline(xintercept=0)

```

## Solving Quadratics Using the Completing the Square Method
In quadratic equations where there is a *b* term; that is, a term containing **x** to the first power, it is impossible to directly calculate the square root. However, with some algebraic manipulation, you can take advantage of the ability to factor a polynomial expression in the form *a2 + 2ab + b2* as a binomial *perfect square* expression in the form *(a + b)2*.

At first this might seem like some sort of mathematical sleight of hand, but follow through the steps carefull and you'll see that there's nothing up my sleeve!

The underlying basis of this approach is that a trinomial expression like this:

\begin{equation}x^{2} + 24x + 12^{2}\end{equation}

Can be factored to this:

\begin{equation}(x + 12)^{2}\end{equation}

OK, so how does this help us solve a quadratic equation? Well, let's look at an example:

\begin{equation}y = x^{2} + 6x - 7\end{equation}

Let's start as we've always done so far by restating the equation to solve ***x*** for a ***y*** value of 0:

\begin{equation}x^{2} + 6x - 7 = 0\end{equation}

Now we can move the constant term to the right by adding 7 to both sides:

\begin{equation}x^{2} + 6x = 7\end{equation}

OK, now let's look at the expression on the left: *x2 + 6x*. We can't take the square root of this, but we can turn it into a trinomial that will factor into a perfect square by adding a squared constant. The question is, what should that constant be? Well, we know that we're looking for an expression like *x2 + 2**c**x + **c**2*, so our constant **c** is half of the coefficient we currently have for ***x***. This is **6**, making our constant **3**, which when squared is **9** So we can create a trinomial expression that will easily factor to a perfect square by adding 9; giving us the expression *x2 + 6x + 9*.

However, we can't just add something to one side without also adding it to the other, so our equation becomes:

\begin{equation}x^{2} + 6x + 9 = 16\end{equation}

So, how does that help? Well, we can now factor the trinomial expression as a perfect square binomial expression:

\begin{equation}(x + 3)^{2} = 16\end{equation}

And now, we can use the square root method to find x + 3:

\begin{equation}x + 3 =\pm\sqrt{16}\end{equation}

So, x + 3 is **-4** or **4**. We isolate ***x*** by subtracting 3 from both sides, so ***x*** is **-7** or **1**:

\begin{equation}x = -7, 1\end{equation}

Let's see what the parabola for this equation looks like in R:

```R
x1 = as.integer(-sqrt(16) - 3)
x2 = as.integer(sqrt(16) - 3)

# Create a dataframe with an x column containing some values to plot
df = data.frame(x = seq(x1-10, x2+10))

# Add a y column by applying the quadratic equation to x
df$y = ((df$x + 3)**2) - 16

# Get x at the line of symmetry (halfway between x1 and x2)
vx = (x1 + x2) / 2

# Get y when x is at the line of symmetry
vy = ((vx + 3)**2) - 16

# get min and max y values
miny = min(df$y)
maxy = max(df$y)

## data frame for line of symmetry
symmetry = data.frame(sx = c(vx,vx), sy = c(miny,maxy))

## data frame for xlimits
xlims = data.frame(x = c(x1,x2), y = c(0,0))

## Plot the parabola
ggplot(df, aes(x,y)) +
        geom_line(color = 'blue', size = 1) +
        geom_line(data = symmetry, aes(sx,sy), color = 'magenta', size = 1) +
        geom_point(data = symmetry, aes(sx,sy), color = 'magenta', size = 2)  +
        geom_point(data = xlims, aes(x,y), color = 'red', size = 2) +
        geom_text(data = xlims, aes(label=c('x1','x2')),hjust=-0.5, vjust=0) +
        annotate("text", x = vx, y = -20, label = "Vertex") +
        geom_hline(yintercept=0) + geom_vline(xintercept=0)

```

## Vertex Form
Let's look at another example of a quadratic equation in standard form:

\begin{equation}y = 2x^{2} - 16x + 2\end{equation}

We can start to solve this by subtracting 2 from both sides to move the constant term from the right to the left:

\begin{equation}y - 2 = 2x^{2} - 16x\end{equation}

Now we can factor out the coefficient for x2, which is **2**. 2x2 is 2 • x2, and -16x is 2 • 8x:

\begin{equation}y - 2 = 2(x^{2} - 8x)\end{equation}

Now we're ready to complete the square, so we add the square of half of the -8x coefficient on the right side to the parenthesis. Half of -8 is -4, and -42 is 16, so the right side of the equation becomes *2(x2 - 8x + 16)*. Of course, we can't add something to one side of the equation without also adding it to the other side, and we've just added 2 • 16 (which is 32) to the right, so we must also add that to the left.

\begin{equation}y - 2 + 32 = 2(x^{2} - 8x + 16)\end{equation}

Now we can simplify the left and factor out a perfect square binomial expression on the right:

\begin{equation}y + 30 = 2(x - 4)^{2}\end{equation}

We now have a squared term for ***x***, so we could use the square root method to solve the equation. However, we can also isolate ***y*** by subtracting 30 from both sides. So we end up restating the original equation as:

\begin{equation}y = 2(x - 4)^{2} - 30\end{equation}

Let's just quickly check our math with Python:

```R
x = sample.int(100, 1)

2*x**2 - 16*x + 2 == 2*(x - 4)**2 - 30
```

TRUE

So we've managed to take the expression ***2x2 - 16x + 2*** and change it to ***2(x - 4)2 - 30***. How does that help?

Well, when a quadratic equation is stated this way, it's in *vertex form*, which is generically described as:

\begin{equation}y = a(x - h)^{2} + k\end{equation}

The neat thing about this form of the equation is that it tells us the coordinates of the vertex - it's at ***h,k***.

So in this case, we know that the vertex of our equation is 4, -30. Moreover, we know that the line of symmetry is at ***x = 4***.

We can then just use the equation to calculate two more points, and the three points will be enough for us to determine the shape of the parabola. We can simply choose any ***x*** value we like and substitute it into the equation to calculate the corresponding ***y*** value. For example, let's calculate ***y*** when x is **0**:

\begin{equation}y = 2(0 - 4)^{2} - 30\end{equation}

When we work through the equation, it gives us the answer **2**, so we know that the point 0, 2 is in our parabola.

So, we know that the line of symmetry is at ***x = h*** (which is 4), and we now know that the ***y*** value when ***x*** is 0 (***h*** - ***h***) is 2. The ***y*** value at the same distance from the line of symmetry in the negative direction will be the same as the value in the positive direction, so when ***x*** is ***h*** + ***h***, the ***y*** value will also be 2.

The following Python code encapulates all of this in a function that draws and annotates a parabola using only the ***a***, ***h***, and ***k*** values from a quadratic equation in vertex form:

```R
plot_parabola_from_vertex_form = function(a, h, k){
    # Create a dataframe with an x column containing values from x=-10 to x=10)
    df = data.frame(x = seq(h-10, h+10))

    # Add a y column by applying the quadratic equation to x
    df$y = (a*(df$x - h)**2) + k

    # get min and max y values
    miny = min(df$y)
    maxy = max(df$y)

    # calculate y when x is 0 (h+-h)
    y = a*(0 - h)**2 + k

    ## data frame for line of symmetry
    symmetry = data.frame(sx = c(h,h), sy = c(miny,maxy))

    ## data frame for xlimits
    xlims = data.frame(x = c(h-h, h+h), y = c(y,y))

    ## Plot the parabola
    ggplot(df, aes(x,y)) +
           geom_line(color = 'blue', size = 1) +
           geom_line(data = symmetry, aes(sx,sy), color = 'magenta', size = 1) +
           geom_point(data = symmetry, aes(sx,sy), color = 'magenta', size = 2)  +
           geom_point(data = xlims, aes(x,y), color = 'red', size = 2) +
           geom_text(data = xlims, aes(label=c(toString(h-h),toString(h+h))),hjust=-0.5, vjust=0) +
           annotate("text", x = h, y = -40, label = paste('v = ',toString(h),',',toString(miny))) +
           geom_hline(yintercept=0) + geom_vline(xintercept=0)
    }

# Call the function for the example discussed above
plot_parabola_from_vertex_form(2, 4, -30)
```

```R
plot_parabola_from_vertex_form(3, -1, -1)
```

## Shortcuts for Solving Quadratic Equations
We've spent some time in this notebook discussing how to solve quadratic equations to determine the vertex of a parabola and the ***x*** values in relation to ***y***. It's important to understand the techniques we've used, which incude:
- Factoring
- Calculating the Square Root
- Completing the Square
- Using the vertex form of the equation

The underlying algebra for all of these techniques is the same, and this consistent algebra results in some shortcuts that you can memorize to make it easier to solve quadratic equations without going through all of the steps:

### Calculating the Vertex from Standard Form
You've already seen that converting a quadratic equation to the vertex form makes it easy to identify the vertex coordinates, as they're encoded as ***h*** and ***k*** in the equation itself - like this:

\begin{equation}y = a(x - \textbf{h})^{2} + \textbf{k}\end{equation}

However, what if you have an equation in standard form?:

\begin{equation}y = ax^{2} + bx + c\end{equation}

There's a quick and easy technique you can apply to get the vertex coordinates.

1. To find ***h*** (which is the x-coordinate of the vertex), apply the following formula:
\begin{equation}h = \frac{-b}{2a}\end{equation}
2. After you've found ***h***, use it in the quadratic equation to solve for ***k***:
\begin{equation}\textbf{k} = a\textbf{h}^{2} + b\textbf{h} + c\end{equation}

For example, here's the quadratic equation in standard form that we previously converted to the vertex form:

\begin{equation}y = 2x^{2} - 16x + 2\end{equation}

To find ***h***, we perform the following calculation:

\begin{equation}h = \frac{-b}{2a}\;\;\;\;=\;\;\;\;\frac{-1 \cdot16}{2\cdot2}\;\;\;\;=\;\;\;\;\frac{16}{4}\;\;\;\;=\;\;\;\;4\end{equation}

Then we simply plug the value we've obtained for ***h*** into the quadratic equation in order to find ***k***:

\begin{equation}k = 2\cdot(4^{2}) - 16\cdot4 + 2\;\;\;\;=\;\;\;\;32 - 64 + 2\;\;\;\;=\;\;\;\;-30\end{equation}

Note that a vertex at 4,-30 is also what we previously calculated for the vertex form of the same equation:

\begin{equation}y = 2(x - 4)^{2} - 30\end{equation}

### The Quadratic Formula
Another useful formula to remember is the *quadratic formula*, which makes it easy to calculate values for ***x*** when ***y*** is **0**; or in other words:

\begin{equation}ax^{2} + bx + c = 0\end{equation}

Here's the formula:

\begin{equation}x = \frac{-b \pm \sqrt{b^{2} - 4ac}}{2a}\end{equation}

Let's apply that formula to our equation, which you may remember looks like this:

\begin{equation}y = 2x^{2} - 16x + 2\end{equation}

OK, let's plug the ***a***, ***b***, and ***c*** variables from our equation into the quadratic formula:

\begin{equation}x = \frac{--16 \pm \sqrt{-16^{2} - 4\cdot2\cdot2}}{2\cdot2}\end{equation}

This simplifes to:

\begin{equation}x = \frac{16 \pm \sqrt{256 - 16}}{4}\end{equation}

This in turn (with the help of a calculator) simplifies to:

\begin{equation}x = \frac{16 \pm 15.491933384829668}{4}\end{equation}

So our positive value for ***x*** is:

\begin{equation}x = \frac{16 + 15.491933384829668}{4}\;\;\;\;=7.872983346207417\end{equation}

And the negative value for ***x*** is:

\begin{equation}x = \frac{16 - 15.491933384829668}{4}\;\;\;\;=0.12701665379258298\end{equation}

The following Python code uses the vertex formula and the quadtratic formula to calculate the vertex and the -x and +x for y = 0, and then plots the resulting parabola:

```R
plot_parabola_from_formula = function(a, b, c){
    # Get vertex
    print('CALCULATING THE VERTEX')
    print('vx = -b / 2a')

    nb = -b
    a2 = 2*a
    print(paste('vx = ', toString(nb), ' / ', toString(a2)))

    vx = -b/(2*a)
    print(paste('vx = ', toString(vx)))

    cat('\n')
    print('vy = ax^2 + bx + c')
    print(paste('vy =', toString(a), '(', toString(vx), '^2) + ',
                toString(b), '(', toString(vx), ') + ', toString(c)))

    avx2 = a*vx**2
    bvx = b*vx
    print(paste('vy =', toString(avx2), ' + ', toString(bvx), ' + ', toString(c)))

    vy = avx2 + bvx + c
    print(paste('vy = ', toString(vy)))

    cat('\n')
    print (paste('v = ', toString(vx), ',', toString(vy)))

    # Get +x and -x (showing intermediate calculations)
    cat('\n')
    print('CALCULATING -x AND +x FOR y=0')
    print('x = -b +- sqrt(b^2 - 4ac) / 2a')

    b2 = b**2
    ac4 = 4*a*c
    print(paste('x = ', toString(nb), '+-sqrt(', toString(b2),
          ' - ', toString(ac4), ')/', toString(a2)))

    sr = sqrt(b2 - ac4)
    print(paste('x = ', toString(nb), ' +- ', toString(sr), ' / ', toString(a2)))
    print(paste('-x = ', toString(nb), ' - ', toString(sr), ' / ', toString(a2)))
    print(paste('+x = ', toString(nb), ' + ', toString(sr), ' / ', toString(a2)))

    posx = (nb + sr) / a2
    negx = (nb - sr) / a2
    print(paste('-x = ', toString(negx)))
    print(paste('+x = ', toString(posx)))

    cat('\n')
    print('PLOTTING THE PARABOLA')

    # Create a dataframe with an x column containing values from x=-10 to x=10)
    df = data.frame(x = seq(round(vx)-10, round(vx)+10))

    # Add a y column by applying the quadratic equation to x
    df$y = a*df$x**2 + b*df$x + c

    # get min and max y values
    miny = min(df$y)
    maxy = max(df$y)

    ## data frame for line of symmetry
    symmetry = data.frame(sx = c(vx,vx), sy = c(miny,maxy))

    ## data frame for xlimits
    xlims = data.frame(x = c(negx, posx), y = c(0,0))

    ## Plot the parabola
    ggplot(df, aes(x,y)) +
           geom_line(color = 'blue', size = 1) +
           geom_line(data = symmetry, aes(sx,sy), color = 'magenta', size = 1) +
           geom_point(data = symmetry, aes(sx,sy), color = 'magenta', size = 2)  +
           geom_point(data = xlims, aes(x,y), color = 'red', size = 2) +
           geom_text(data = xlims, aes(label=c(toString(round(negx,4)),toString(round(posx,4)))),hjust=-0.2, vjust=0) +
           annotate("text", x = 4, y = -40, label = paste('v = ',toString(vx),',',toString(vy))) +
           geom_hline(yintercept=0) + geom_vline(xintercept=0)
    }

plot_parabola_from_formula (2, -16, 2)
```


sergazy.nurbavliyev@gmail.com © 2021

## Is the coin biased?

Question: A coin is flipped 1000 times and 560 times heads show up. Do you think the coin is biased?

https://stats.stackexchange.com/questions/282786/a-job-interview-question-on-flipping-a-coin

## Answer

It is already answered pretty nice in the stackexchange website where I give the link above. I would definitely try exactly the same way.

Since the 1000 sample size is large enough, we can apply the Central Limit Theorem. Assume probability of head is $p$ and probability of tails as $q$. Note $p+q=1$. We want to know if $p=1/2$ or not.

Assuming each trial are independent of each other, if $p=1/2$ then we should have expected to see 500 heads.
and the variance is given by:
\begin{equation}
\sigma^2= np(1-p)=1000\frac{1}{2}\frac{1}{2}=250
\end{equation}
then the standard deviation would be $\sqrt{250}=15.81$. Remembering the 68-95-99.7 rule for a Normal Distribution
we can calculate the z-score for 560 heads:
\begin{equation}
z = \frac{560-500}{15.81}=3.79
\end{equation}
Remember that 99.7% of the data lies within 3 standard deviation (15.81) of the mean 500. However, we see that here z lies outside of the 99.7%. In other words, if our coin was fair, then probibility of seeing 560 heads is less than 0.03. This means that, most probably our coin is biased coin.

```python
60/250**(1/2)
```

    3.794733192202055

```python
60
```

## Python code for simulation

```python
import random
trial_list= [sum([random.randint(0,1) for i in range(1000)]) for j in range(1000)]
```

```python
from matplotlib import pyplot as plt
plt.plot(trial_list)
plt.axhline(y=560, color='r', linestyle='-')
plt.show()
```

```python

```

```python

```


## Shanon Information Content

First, let's consider storing an integer that isn't random. Let's say we have an integer that is from $0,1,\dots ,63$. Then the number of bits needed to store this integer is $\log _{2}(64)=6$ bits: you tell me $6$ bits and I can tell you exactly what the integer is.

A different way to think about this result is that we don't a priori know which of the $64$ outcomes is going to be stored, and so each outcome is equally likely with probability $\frac{1}{64}$. Then the number of bits needed to store an event $\mathcal{A}$ is given by what's called the “Shannon information content" (also called self-information):

$$\log _{2}\frac{1}{\mathbb {P}(\mathcal{A})}.$$

In particular, for an integer $x\in \{ 0,1,\dots ,63\}$, the Shannon information content of observing $x$ is

$$\log _{2}\frac{1}{\mathbb {P}(\text {integer is }x)}=\log _{2}\frac{1}{1/64}=\log _{2}64=6\text { bits}.$$

If instead, the integer was deterministically $0$ and never equal to any of the other values $1,2,\dots ,63,$ then the Shannon information content of observing integer $0$ is

$$\log _{2}\frac{1}{\mathbb {P}(\text {integer is }0)}=\log _{2}\frac{1}{1}=0\text { bits}.$$

This is not surprising in that a outcome that we deterministically always observe tells us no new information. Meanwhile, for each integer $x\in \{ 1,2,\dots ,63\},$

$$\log _{2}\frac{1}{\mathbb {P}(\text {integer is }x)}=\log _{2}\frac{1}{0}=\infty \text { bits}.$$

How could observing one of the integers $\{ 1,2,\dots ,63\}$ tell us infinite bits of information?! Well, this isn't an issue since the event that we observe any of these integers has probability $0$ and is thus impossible. An interpration of Shannon information content is how surprised we would be to observe an event. In this sense, observing an impossible event would be infinitely surprising.

It is possible to have the Shannon information content of an event be some fractional number of bits (e.g., $0.7$ bits). The interpretation is that from many repeats of the underlying experiment, the average number of bits needed to store the event is given by the Shannon information content, which can be fractional.

### Exercise: Shannon Information Content

I have an integer in mind, uniformly distributed between $0$ and $127$. You can keep guessing what my number is until you get it right (and each time you guess, I tell you whether you got it right). Each time you guess a number wrong, you discard that number so as to not guess it again.

For example, if you guess wrong the first time, then the Shannon information content of guessing wrong is

$$\log _2\frac{1}{\frac{127}{128}}=\log _{2}\frac{128}{127}=0.0113\dots \text { bits}.$$

If you guess wrong the second time, then the Shannon information content of the second guess is

$$\log _2\frac{1}{\frac{126}{127}}=\log _{2}\frac{127}{126}=0.0114\dots \text { bits}.$$

If you guess right on the third time, then the Shannon information content of the third guess is

$$\log _2\frac{1}{\frac{1}{126}}=\log _{2}\frac{126}{1}=6.9772\dots \text { bits}.$$

**Question:** For the above example where the third guess is right, what is the sum of the Shannon information content of the three guesses?

**Solution:**
$$\begin{eqnarray}
&&
\log_2 \frac1{\frac{127}{128}}
+\log_2 \frac1{\frac{126}{127}}
+\log_2 \frac1{\frac{1}{126}} \\
&&=
\log_2 \frac{128}{127}
+\log_2 \frac{127}{126}
+\log_2 126 \\
&&=
\log_2 128 - \log_2 127
+ \log_2 127 - \log_2 126
+ \log_2 126 \\
&&=
\log_2 128 \\
&&=
\boxed{7\text{ bits}}.
\end{eqnarray}$$

**Question:** Suppose you guessed right only after 5 tries. What is the sum of the Shannon information content of these 5 guesses?

**Solution:**

$$\begin{eqnarray}
&&\log_2 \frac1{\frac{127}{128}}
+\log_2 \frac1{\frac{126}{127}}
+\log_2 \frac1{\frac{125}{126}}
+\log_2 \frac1{\frac{124}{125}}
+\log_2 \frac1{\frac{1}{124}} \\
&&=
\log_2 \frac{128}{127}
+\log_2 \frac{127}{126}
+\log_2 \frac{126}{125}
+\log_2 \frac{125}{124}
+\log_2 124 \\
&&=
\log_2 128 - \log_2 127
+ \log_2 127 - \log_2 126
+ \log_2 126 \\
&&\quad
- \log_2 125
+ \log_2 125 - \log_2 124
+ \log_2 124 \\
&&=
\log_2 128 \\
&&=
\boxed{7\text{ bits}}.
\end{eqnarray}$$

**Question:** Suppose you guess right after $k$ tries $(k \in \{ 1, 2, \dots , 128\})$. If you sum up the Shannon information content for all the guesses up and including the one in which you guess right, does this total number of bits depend on $k$? (While we aren't asking for you to justify your answer, we encourage you to be able to do so! For example, if your answer is "Yes" then you should be able to come up with two specific cases for two different number of tries before guessing right that yield different number of total bits, and if your answer is "No" then you should be able to show why the total number of bits gained is always the same.)

**Solution:** The answer is no. The previous two parts provide a clue: the sum we're computing is a telescoping sum where all the terms cancel out except for the first one: $\log _2 128 = 7.$

In general, if we guess right after $k$ tries, then the total amount of information “learned" is:

$$\begin{eqnarray}
&&
\left[
\sum_{i=1}^{k-1}
  \log_2 \frac1{\frac{128 - i}{128 - (i - 1)}}
\right]
+ \log_2 (128 - (k - 1)) \\
&&=
\left[
\sum_{i=1}^{k-1}
  \log_2 \frac{128 - (i - 1)}{128 - i}
\right]
+ \log_2 (128 - (k - 1)) \\
&&=
\left[
\sum_{i=1}^{k-1}
  \log_2 (128 - (i - 1)) - \log_2(128 - i)
\right]
+ \log_2 (128 - (k - 1)) \\
&&= \log_2 128 - \log_2 127 + \log_2 127 - \cdots \\
&&\quad
- \log_2 (128 - (k - 1)) + \log_2 (128 - (k - 1)) \\
&&= \log_2 128 \\
&&= 7\text{ bits}.
\end{eqnarray}$$

Put another way, $7$ bits of information are needed before you know the number, and with wrong guesses, you learn very few bits of information (although as the number of possibilities shrinks, wrong guesses provide more and more bits of information).

### Shanon Entropy

To go from the number of bits contained in an event to the number of bits contained in a random variable, we simply take the expectation of the Shannon information content across the possible outcomes. The resulting quantity is called the entropy of a random variable:

$$H(X)=\sum _{x}p_{X}(x)\underbrace{\log _{2}\frac{1}{p_{X}(x)}}_{\text {Shannon information content of event }X=x}.$$

The interpretation is that on average, the number of bits needed to encode each i.i.d. sample of a random variable X is $H(X)$. In fact, if we sample n times i.i.d. from $p_{X}$, then two fundamental results in information theory that are beyond the scope of this course state that: (a) there's an algorithm that is able to store these n samples in $nH(X)$ bits, and (b) we can't possibly store the sequence in fewer than $nH(X)$ bits!

**Example:** If $X$ is a fair coin toss “heads" or “tails" each with probability $1/2$, then

$$\begin{eqnarray}
H(X)
&=& p_X(\text{heads}) \log_2 \frac1{p_X(\text{heads})}
+ p_X(\text{tails}) \log_2 \frac1{p_X(\text{tails})} \\
&=& \frac12 \cdot \underbrace{\log_2 \frac1{\frac{1}{2}}}_1
+ \frac12 \cdot \underbrace{\log_2 \frac1{\frac{1}{2}}}_1 \\
&=& 1 \text{ bit}.
\end{eqnarray}$$

**Example:** If $X$ is a biased coin toss where heads occurs with probability $1$ then

$$\begin{eqnarray}
H(X)
&=& p_X(\text{heads}) \log_2 \frac1{p_X(\text{heads})}
+ p_X(\text{tails}) \log_2 \frac1{p_X(\text{tails})} \\
&=& 1 \cdot \underbrace{\log_2 \frac11}_0
+ 0 \cdot \cdot \underbrace{\log_2 \frac10}_1 \\
&=& 0 \text{ bits},
\end{eqnarray}$$

where $0 \log _2 \frac10 = 0 \log _2 1 - 0 \log _2 0 = 0$ using the convention that $0 \log _2 0 \triangleq 0$. (Note: You can use l'Hopital's rule from calculus to show that $\lim _{x\rightarrow 0} x \log x = 0 and \lim _{x\rightarrow 0} x \log \frac1x = 0$.)

Notation: Note that entropy $H(X) = \sum _ x p_ X(x) \log _2 \frac{1}{p_ X(x)}$ is in the form of an expectation! So in fact, we can write an expectation:

$$H(X) = \mathbb {E}\left[\log _2 \frac{1}{p_ X(X)}\right].$$

```python
def H(pX):
    """
    Retrun Shanon entropy of given probability.
    >>> pX = {-1: 999999/1000000, 999   : 1/1000000}
    >>> H(pX)
    2.137426288890686e-05

    >>> coin = {'H': 1/2, 'T': 1/2}
    >>> H(coin)
    1.0
    """
    from math import log
    return - sum([value * log(value,2) for key, value in pX.items()])

if __name__ == "__main__":
    import doctest
    doctest.testmod()
```

### Exercise: Shannon Entropy

Entropy gives a very different measure for uncertainty than variance, which we saw previously. Whereas variance $\text {var}(X) = \mathbb {E}[(X-\mathbb {E}[X])^2]$ measures how far a random variable is expected to deviate from its expected value $\mathbb {E}[X]$, entropy measures how many bits are needed on average to store each i.i.d. sample of a random variable $X$.

Let's return to the three lotteries from earlier. Here, random variables $L_1, L_2,$ and $L_3$ represent the amount won (accounting for having to pay \$1):

|$L_1$      |            $p$           |        $L_2$ |            $p$           |   $L_3$ |       $p$      |
|----------:|:------------------------:|-------------:|:------------------------:|--------:|:--------------:|
|        -1 | $\frac{999999}{1000000}$ |           -1 | $\frac{999999}{1000000}$ |      -1 | $\frac{9}{10}$ |
|   -1+1000 |    $\frac{1}{1000000}$   |   -1+1000000 |    $\frac{1}{1000000}$   |   -1+10 | $\frac{1}{10}$ |

Compute the following entropies in bits. Please input just the number and do not include the text "bits" at the end. (Please be precise with at least $7$ decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

- $H(L_1) =$ {{H_1}}
- $H(L_2) =$ {{H_2}}
- $H(L_3) =$ {{H_3}}

```python
p_L1 = {-1: 999999/1000000, 999   : 1/1000000}
p_L2 = {-1: 999999/1000000, 999999: 1/1000000}
p_L3 = {-1:      9/10     , 9     : 1/10     }

H_1 = '{0:0.7f}'.format(H(p_L1))
H_2 = '{0:0.7f}'.format(H(p_L2))
H_3 = '{0:0.7f}'.format(H(p_L3))
```

Now that you've computed some entropies, let's answer some general questions about entropy.

**Question:** In computing the entropy of a random variable $X$, we look at the probabilities in the probability table of $X.$ Did we have to look at the labels in the probability table (note that the labels correspond to the alphabet of $X$)?

[$\times    $] Yes

[$\checkmark$] No

**Solution:** The answer is no. You can see this in the Python code: we don't need to look at the alphabet $\mathcal{X}$.

Suppose for a random variable $X$, we take its probability table and shuffle the ordering of the probabilities (but otherwise keep the labels the same). For example, if $X$ is a biased coin flip with probability of heads $3/4$, suppose we took its table and instead have the probability of tails be $3/4$ and the probability of heads be $1/4$.

**Question:** By shuffling the ordering of the probabilities, does the entropy of the random variable change?

[$\times    $] Yes

[$\checkmark$] No

**Solution:** The answer is no. Since what the labels are does not actually matter (of course, we cannot have the same label appear twice in the table though), we can permute the labels (or the probabilities) and get the same entropy.

Finally, let's think about how small and large entropy could be.

**Question:** We've seen an example where the entropy is $0$ bits. Can entropy be negative? (While there is an intuitive answer for this, see if you can show it mathematically.)

[$\times    $] Yes

[$\checkmark$] No

**Solution:** The answer is no. First, note that for any probability $p\in [0,1]$, the Shannon information content corresponding to this probability is $\log_2 (1/p)$, which has a value from $0$ to infinity, i.e., Shannon information content is always nonnegative. Shannon entropy is just a weighted average of Shannon information content, where the weights are nonnegative. A nonnegative weighted sum of a collection of nonnegative numbers remains nonnegative.

**Question:** For a random variable $X$ that takes on one of two values, one with probability $p$ and the other with probability $1-p$, plot the entropy $H(p)$ as a function of $p$ in Python. For what value of $p$ is the entropy maximized (i.e., what value of $p$ yields the largest $H(X))$? Please provide an exact answer.

```python
%matplotlib inline
import numpy as np
# Ignore divide by zero error
# Ref: http://stackoverflow.com/questions/14861891/runtimewarning-invalid-value-encountered-in-divide
np.seterr(divide='ignore', invalid='ignore')
import matplotlib.pyplot as plt

entropy = lambda p: np.sum(p * np.log2(1 / p))

p_list = np.linspace(0, 1, 50)
plt.figure()
plt.plot(p_list,
         [entropy(np.array([p, 1-p])) for p in p_list])
plt.xlabel('p')
plt.ylabel('H(Ber(p))')
plt.show()
```

## Information divergence

Information divergence (also called “Kullback-Leibler divergence" or “KL divergence" for short, or also “relative entropy") is a measure of how different two distributions $p$ and $q$ (over the same alphabet) are. To come up with information divergence, first, note that entropy of a random variable with distribution $p$ could be thought of as the expected number of bits needed to encode a sample from $p$ using the information content according to distribution $p$:

$$\underbrace{\sum _{x}p(x)}_{\begin{matrix}\text {expectation } \\ \text{using }p\end{matrix} }\underbrace{\log _{2}\frac{1}{p(x)}}_{\begin{matrix}\text { information content } \\ \text{ according to }p \end{matrix}}\triangleq \mathbb {E}_{X \sim p}\Big[\log _{2}\frac{1}{p(X)}\Big].$$

Here, we have introduced a new notation: $\mathbb {E}_{X \sim p}$ means that we are taking the expectation with respect to random variable $X$ drawn from the distribution $p$. If it's clear which random variable we are taking the expectation with respect to, we will often just abbreviate the notation and write $\mathbb {E}_ p$ instead of $\mathbb {E}_{X \sim p}$.

If instead we look at the information content according to a different distribution $q$, we get

$$\underbrace{\sum _{x}p(x)}_{\begin{matrix}\text {expectation } \\ \text{using }p\end{matrix} }\underbrace{\log _{2}\frac{1}{p(x)}}_{\begin{matrix}\text { information content } \\ \text{ according to }p \end{matrix}}\triangleq \mathbb {E}_{X \sim p}\Big[\log _{2}\frac{1}{p(X)}\Big].$$

It turns out that if we are actually sampling from p but encoding samples as if they were from a different distribution $q$, then we always need to use more bits! This isn't terribly surprising in light of the fundamental result we alluded to that entropy of a random variable with distribution $p$ is the minimum number of bits needed to encode samples from $p$.

Information divergence is the price you pay in bits for trying to encode a sample from $p$ using information content according to $q$ instead of according to $p$:

$$D(p\parallel q)=\mathbb {E}_{X \sim p}\Big[\log _{2}\frac{1}{q(X)}\Big]-\mathbb {E}_{X \sim p}\Big[\log _{2}\frac{1}{p(X)}\Big].$$

Information divergence is always at least $0$, and when it is equal to $0$, then this means that $p$ and $q$ are the same distribution (i.e., $p(x) = q(x)$ for all $x$). This property is called Gibbs' inequality.

Gibbs' inequality makes information divergence seem a bit like a distance. However, information divergence is not like a distance in that it is not symmetric: in general, $D(p \parallel q) \ne D(q \parallel p).$

Often times, the equation for information divergence is written more concisely as

$$D(p\parallel q) = \sum _ x p(x) \log \frac{p(x)}{q(x)},$$

which you can get as follows:

$$\begin{eqnarray}
D(p\parallel q)
&=&
  \mathbb{E}_{X \sim p}\Big[\log_{2}\frac{1}{q(X)}\Big]
- \mathbb{E}_{X \sim p}\Big[\log_{2}\frac{1}{p(X)}\Big] \\
&=&
  \sum_x p(x) \log_2 \frac{1}{q(x)}
- \sum_x p(x) \log_2 \frac{1}{p(x)} \\
&=&
  \sum_x p(x)
  \Big[ \log_2 \frac{1}{q(x)} - \log_2 \frac{1}{p(x)} \Big] \\
&=&
  \sum_x p(x)
  \log_2 \frac{p(x)}{q(x)}.
\end{eqnarray}$$

Example: Suppose $p$ is the distribution for a fair coin flip:

$$\begin{eqnarray}
p(x)
&=&
\begin{cases}
\frac12 & \text{if }x=\text{heads}, \\
\frac12 & \text{if }x=\text{tails}. \\
\end{cases}
\end{eqnarray}$$

Meanwhile, suppose q is a distribution for a biased coin that always comes up heads (perhaps it's double-headed):

$$\begin{eqnarray}
q(x)
&=&
\begin{cases}
1 & \text{if }x=\text{heads}, \\
0 & \text{if }x=\text{tails}. \\
\end{cases}
\end{eqnarray}$$

Then

$$\begin{eqnarray}
D(p \parallel q)
&=&
  p(\text{heads}) \log_2 \frac{p(\text{heads})}{q(\text{heads})}
+ p(\text{tails}) \log_2 \frac{p(\text{tails})}{q(\text{tails})} \\
&=&
  \frac12\log_2 \frac{\frac12}1
+ \underbrace{\frac12\log_2 \frac{\frac12}0}_{\infty} \\
&=&
  \infty\text{ bits}.
\end{eqnarray}$$

This is not surprising: If we are sampling from $p$ (for which we could get tails) but trying to encode the sample using $q$ (which cannot possibly encode tails), then if we get tails, we are stuck: we can't store it! This incurs a penalty of infinity bits.

Meanwhile,

$$\begin{eqnarray}
D(q \parallel p)
&=&
  q(\text{heads}) \log_2 \frac{q(\text{heads})}{p(\text{heads})}
+ q(\text{tails}) \log_2 \frac{q(\text{tails})}{p(\text{tails})} \\
&=&
  1 \log_2 \frac1{\frac12}
+ \underbrace{0 \log_2 \frac0{\frac12}}_0 \\
&=&
  1\text{ bit}.
\end{eqnarray}$$

When we sample from $q$, we always get heads. In fact, as we saw previously, the entropy of the distribution for an always-heads coin flip is $0$ bits since there's no randomness. But here we are sampling from $q$ and storing the sample using distribution $p$. For a fair coin flip, encoding using distribution $p$ would store each sample using on average $1$ bit. Thus, even though a sample from $q$ is deterministically heads, we store it using $1$ bit. This is the penalty we pay for storing a sample from $q$ using distribution $p$.

Notice that in this example, $D(p \parallel q) \ne D(q \parallel p)$. They aren't even close — one is infinity and the other is finite!

### Exercise: Information Divergence

We now look at a different way to think of Shannon entropy for a random variable $X$ with alphabet $\mathcal{X}.$

Let random variable $U$ have what's called a uniform distribution over alphabet $\mathcal{X}$, meaning that

$$p_ U(x) = \frac{1}{|\mathcal{X}|} \qquad \text {for all }x\in \mathcal{X}.$$

Notationally, we can write $U \sim \text {Uniform}(\mathcal{X})$.

In the following problems, suppose the number of labels in $\mathcal{X}$ is given by $k$, i.e., $k = |\mathcal{X}|.$

**Question:** What is $H(U)$ in terms of $k$?

**Solution:**

$$\begin{eqnarray}
H(U)
&=& \sum_{x\in\mathcal{X}} p_U(x) \log_2 \frac1{p_U(x)} \\
&=& \sum_{x\in\mathcal{X}} \frac1k \log_2 \frac1{\frac1k} \\
&=& \sum_{x\in\mathcal{X}} \frac1k \log_2 k \\
&=& (\log_2 k)\Big(\frac1k\Big)
    \underbrace{\sum_{x\in\mathcal{X}} 1}_k \\
&=& \log_2 k.
\end{eqnarray}$$

So using “log" to mean log base 2, the answer is $log(k)$.

Next, we examine the divergence between $p_X$ and the uniform distribution. Show that $D(p_ X \parallel p_ U)$ can be written of the form

$$D(p_ X \parallel p_ U) = f(k) - H(X),$$

for a function $f$ that you will determine:

**Question:** What is $f$?

**Solution:**

$$\begin{eqnarray}
D(p_X \parallel p_U)
&=& \sum_{x\in\mathcal{X}} p_X(x) \log_2 \frac{p_X(x)}{p_U(x)} \\
&=& \sum_{x\in\mathcal{X}} p_X(x) \log_2 \frac{p_X(x)}{1/k} \\
&=& \sum_{x\in\mathcal{X}} p_X(x) \log_2 (k p_X(x)) \\
&=& \sum_{x\in\mathcal{X}} p_X(x) \log_2 k
    + \sum_{x\in\mathcal{X}} p_X(x) \log_2 p_X(x) \\
&=& (\log_2 k) \underbrace{\sum_{x\in\mathcal{X}} p_X(x)}_1
    + \sum_{x\in\mathcal{X}} p_X(x) \log_2 p_X(x) \\
&=& \log_2 k
    - \sum_{x\in\mathcal{X}} p_X(x) \log_2 \frac{1}{p_X(x)} \\
&=& \underbrace{\log_2 k}_{f(k)} - H(X).
\end{eqnarray}$$

In particular, $f$ is $\log$ base $2$, which for this part you answer by just saying log to mean $\log$ base $2$.

Your answers to the previous two parts should tell you how the entropy of a uniform distribution (over an alphabet of size $k$) relates to the entropy of any distribution $p_X$ (over the same alphabet of size $k$).

Fill in the blanks:

**Question:** Because of Gibbs' inequality, the entropy of random variable \_\_\_\_\_\_\_ cannot be larger than the entropy of random variable \_\_\_\_\_\_\_.

**Solution:** Because of Gibbs' inequality, the entropy of random variable $\underline{~X~}$ cannot be larger than the entropy of random variable $\underline{~U~}$.

In particular, notice that from the answers to the previous parts,

$$D(p_ X \parallel p_ U) = H(U) - H(X).$$

By Gibbs' inequality, information divergence is always nonnegative, which means that we must have $H(U) - H(X) \ge 0$, which means that $H(X) \le H(U)$.

## Proof of Gibbs' Inequality

We provide a proof for Gibbs' inequality here for those who are interested. For those of you up for the challenge, try to prove it yourself!

There are various ways to prove Gibbs' inequality. We'll be using a way that relies on the fact that $\ln x\le x-1$ for all $x>0$, with equality if and only if $x=1$, which we provide a proof for at the end of this page, but for which you can also readily see from the following plot:

```python
x = np.linspace(0,2,10000)
plt.figure()
plt.plot(x, np.log(x))
plt.plot(x, x - 1)
plt.xlabel('x')
plt.legend(['ln(x)', 'x - 1'], loc=4)
plt.show()
```

**Gibbs' inequality:** For any two distributions $p$ and $q$ defined over the same alphabet, we have $D(p\parallel q)\ge 0$, where equality holds if and only if $p$ and $q$ are the same distribution, i.e., $p(x)=q(x)$ for all $x$.

Proof: Recall that changing the base of a log just changes the log by a constant factor:

$$\log _{2}x=\frac{\ln x}{\ln 2}.$$

Let $\mathcal{X}$ be the alphabet of distribution p restricted to where the probability is positive, i.e., $\mathcal{X}=\{ a\text { such that }p(a)>0\}.$ (There is no need to look at values a for which $p(a)=0.$)

If $q(a)=0$ for any $a\in \mathcal{X}$, then $D(p\parallel q)=\infty$, so trivially $D(p\parallel q)>0.$

What's left to consider is when $q(a)>0$ for every $a\in \mathcal{X}$. Then

$$\begin{align}D(p\parallel q) &= \sum _{a\in \mathcal{X}}p(a)\log _{2}\frac{p(a)}{q(a)}\\
&= \frac{1}{\ln 2}\sum _{a\in \mathcal{X}}p(a)\ln \frac{p(a)}{q(a)}\\
&= -\frac{1}{\ln 2}\sum _{a\in \mathcal{X}}p(a)\ln \frac{q(a)}{p(a)}.
\end{align}$$

Next, using the fact that $\ln x\le x-1$ for all $x>0,$ and accounting for the minus sign outside the summation,

$$\begin{align}D(p\parallel q) &= -\frac{1}{\ln 2}\sum _{a\in \mathcal{X}}p(a)\ln \frac{q(a)}{p(a)}	\\
&\ge -\frac{1}{\ln 2}\sum _{a\in \mathcal{X}}p(a)\Big(\frac{q(a)}{p(a)}-1\Big)	\\
&= -\frac{1}{\ln 2}\sum _{a\in \mathcal{X}}\big (q(a)-p(a)\big )\\
&=  -\frac{1}{\ln 2}\big (\underbrace{\sum _{a\in \mathcal{X}}q(a)}_{1}-\underbrace{\sum _{a\in \mathcal{X}}p(a)}_{1}\big )\\
&= 0. \end{align}$$

Recall that inequality $\ln x\le x-1$ becomes an equality if and only if $x=1.$ Thus, the inequality above becomes an equality if and only if, for all $a\in \mathcal{X}$, we have $\ln \frac{q(a)}{p(a)}=\frac{q(a)}{p(a)}-1,$ which holds if and only if $\frac{q(a)}{p(a)}=1.$ Thus $D(p\parallel q)=0$ if and only if $p(a)=q(a)$ for all $a\in \mathcal{X}.$ This finishes the proof. $\square$

Claim: $\ln x\le x-1$ for all $x>0$ where equality holds if and only if $x=1.$

Proof: We show that the function $f$ given by $f(x)=x-1-\ln x$ is always at least $0$ and achieves its minimum value at $x=1.$ First, note that $f$ is differentiable for all $x>0$ (which implies that $f$ is continuous on $(0,\infty )$ and doesn't, for example, do some crazy jump midway through). In fact, the derivative of $f$ is given by

$$\frac{d}{dx}f(x)=\frac{d}{dx}(x-1-\ln x)=1-\frac{1}{x}.$$

On the interval $x>0,$ the derivative is $0$ (and so there's a local extremum) precisely when $x=1.$ The question is whether this is a local minimum or a local maximum. We look at the second derivative of $f$ to do a second derivative test:

$$\frac{d^{2}}{dx^{2}}f(x)=\frac{1}{x^{2}},$$

which is strictly positive for all $x>0$. In other words, $x=1$ is a local minimum. The only possible other extrema could happen at the boundaries, but it's easy to check that

$$\lim _{x\rightarrow 0}f(x) = \lim _{x\rightarrow 0} (x - 1 - \ln x) = -1 - \underbrace{\lim _{x\rightarrow 0} \ln x}_{-\infty } = \infty ,$$

and

$$\lim _{x\rightarrow \infty }f(x) = \lim _{x\rightarrow \infty } (x - 1 - \ln x) = \infty ,$$

since $x$ grows faster than $\ln x.$

Hence, $f$ attains its global minimum at $x=1,$ for which we have

$$f(1)=1-1-\ln 1=0.$$

Since this is the global minimum, we know that $f(x)\ge 0$ for all $x>0.$ Furthermore, since there is only one unique global minimum $x=1,$ we further conclude that $f(x)=0$ if and only if $x=1.$ $\square$

```python

```


```python
from sympy import *
import math
```

```python
x,y,z = symbols("x,y,z,")
```

```python
fxyz = x**2*y*z
fxyz
```

$\displaystyle x^{2} y z$

```python
f_x =diff(fxyz,x)
f_x
```

$\displaystyle 2 x y z$

```python
f_y = diff(fxyz,y)
f_y
```

$\displaystyle x^{2} z$

```python
f_z = diff(fxyz,z)
f_z
```

$\displaystyle x^{2} y$

```python
jacobian = Matrix([[f_x,f_y,f_z]])
jacobian
```

$\displaystyle \left[\begin{matrix}2 x y z & x^{2} z & x^{2} y\end{matrix}\right]$

```python
jx_x = diff(f_x,x)
jx_x
```

$\displaystyle 2 y z$

```python
jx_y = diff(f_x,y)
jx_y
```

$\displaystyle 2 x z$

```python
jx_z = diff(f_x,z)
jx_z
```

$\displaystyle 2 x y$

```python
jy_x = diff(f_y,x)
jy_x
```

$\displaystyle 2 x z$

```python
jy_y = diff(f_y,y)
jy_y
```

$\displaystyle 0$

```python
jy_z = diff(f_y,z)
jy_z
```

$\displaystyle x^{2}$

```python
jz_x = diff(f_z,x)
jz_x
```

$\displaystyle 2 x y$

```python
jz_y = diff(f_z,y)
jz_y
```

$\displaystyle x^{2}$

```python
jz_z = diff(f_z,z)
jz_z
```

$\displaystyle 0$

```python
hessian = Matrix([[jx_x,jx_y,jx_z],
                 [jy_x,jy_y,jy_z],
                 [jz_x,jz_y,jz_z]])
hessian
```

$\displaystyle \left[\begin{matrix}2 y z & 2 x z & 2 x y\\2 x z & 0 & x^{2}\\2 x y & x^{2} & 0\end{matrix}\right]$

```python
fxy = x**3*y+x+2*y
fxy
```

$\displaystyle x^{3} y + x + 2 y$

```python
fxy_x = diff(fxy,x)
fxy_x
```

$\displaystyle 3 x^{2} y + 1$

```python
fxy_y = diff(fxy,y)
fxy_y
```

$\displaystyle x^{3} + 2$

```python
jacobian = Matrix([[fxy_x,fxy_y]])
jacobian
```

$\displaystyle \left[\begin{matrix}3 x^{2} y + 1 & x^{3} + 2\end{matrix}\right]$

```python
jx_x = diff(fxy_x,x)
jx_x
```

$\displaystyle 6 x y$

```python
jx_y = diff(fxy_x,y)
jx_y
```

$\displaystyle 3 x^{2}$

```python
jy_x = diff(fxy_y,x)
jy_x
```

$\displaystyle 3 x^{2}$

```python
jy_y = diff(fxy_y,y)
jy_y
```

$\displaystyle 0$

```python
hessian = Matrix([[jx_x, jx_y],
                 [jy_x,jy_y]])
hessian
```

$\displaystyle \left[\begin{matrix}6 x y & 3 x^{2}\\3 x^{2} & 0\end{matrix}\right]$

```python

```


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
# Solution

alpha, beta = symbols('alpha beta')
```

```python
# Solution

eq3 = Eq(diff(f(t), t), alpha*f(t) + beta*f(t)**2)
```

```python
# Solution

solution_eq = dsolve(eq3)
```

```python
# Solution

general = solution_eq.rhs
```

```python
# Solution

at_0 = general.subs(t, 0)
```

```python
# Solution

solutions = solve(Eq(at_0, p_0), C1)
value_of_C1 = solutions[0]
```

```python
# Solution

particular = general.subs(C1, value_of_C1)
particular.simplify()
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

```python

```