# Support Vector Machines

## Motivating Support Vector Machines
### Developing the Intuition

Support vector machines (SVM) are a powerful and flexible class of supervised algorithms. Developed in the 1990s, SVM have shown to perform well in a variety of settings which explains their popularity. Though the underlying mathematics can become somewhat complicated, the basic concept of a SVM is easily understood. Therefore, in what follows we develop an intuition, introduce the mathematical basics of SVM and ultimately look into how we can apply SVM with Python.

As an introductory example, borrowed from VanderPlas (2016), consider the following simplified two-dimensional classification task, where the two classes (indicated by the colors) are well separated.

A linear discriminant classifier as discussed in chapter 8 would attempt to draw a separating hyperplane (which in two dimensions is nothing but a line) in order to distinguish the two classes. For two-dimensional data, we could even do this by hand. However, one problem arises: there are more than one separating hyperplane between the two classes.

There exist an infinite number of possible hyperplanes that perfectly discriminate between the two classes in the training data. In above figure we visualize but three of them. Depending on what hyperplane we choose, a new data point (e.g. the one marked by the red "X") will be assigned a different label. Yet, so far we have no decision criteria established to decide which one of the three hyperplanes we should choose.

How do we decide which line best separates the two classes? The idea of SVM is to add a margin of some width to both sides of each hyperplane - up to the nearest point. This might look something like this:

In SVM, the hyperplane that maximizes the margin to the nearest points is the one that is chosen as decision boundary. In other words, the maximum margin estimator is what we are looking for. Below figure shows the optimal solution for a (linear) SVM. Of all possible hyperplanes, the solid line has the largest margins (dashed lines) - measured from the decision boundary (solid line) to the nearest points (circled points).

### Support Vector

The three circled sample points in above figure represent the nearest points. All three lie along the (dashed) margin line and in terms of perpendicular distance are equidistant from the decision boundary (solid line). Together they form the so called **support vector**. The support vector "supports" the maximal margin hyperplane in the sense that if one of the observations were moved slightly, the maximal margin hyperplane would move as well. In other words, they dictate slope and intercept of  the hyperplane. Interestingly, any points further from the margin that are on the correct side do not modify the decision boundary. For example points at $(x_1, x_2) = (2.5, 1)$ or $(1, 4.2)$ have no effect on the decision boundary. Technically, this is because these points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin (VanderPlas (2016)) . This is an important and helpful property as it simplifies calculations significantly. It is not surprising that computations are a lot faster if a model has only a few data points (in the support vector) to consider (James et al. (2013)).

## Developing the Mathematical Intuition
### Hyperplanes

To start, let us do a brief (and superficial) refresher on hyperplanes. In a $p$-dimensional space, a hyperplane is a flat (affine) subspace of dimension $p - 1$. Affine simply indicates that the subspace need not pass through the origin. As we have seen above, in two dimensions a hyperplane is just a line. In three dimensions it is a plane. For $p > 3$ visualization is hardly possible but the notion applies in similar fashion. Mathematically a $p$-dimensional hyperplane is defined by the expression

\begin{equation}
\beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_p x_p = 0
\end{equation}

If a point $\mathbf{x}^* = (x^*_1, x^*_2, \ldots, x^*_p)^T$ (i.e. a vector of length $p$) satisfies the above equation, then $\mathbf{x}^*$ lies on the hyperplane. If $\mathbf{x}^{*}$ does not satisfy above equation but yields a value $>0$, that is

\begin{equation}
\beta_0 + \beta_1 x^*_1 + \beta_2 x^*_2 + \ldots + \beta_p x^*_p > 0
\end{equation}

then this tells us that $\mathbf{x}^*$ lies on one side of the hyperplane. Similarly,

\begin{equation}
\beta_0 + \beta_1 x^*_1 + \beta_2 x^*_2 + \ldots + \beta_p x^*_p < 0
\end{equation}

tells us that $\mathbf{x}^*$ lies on the other side of the plane.

### Separating Hyperplanes

Suppose our training sample is a $n \times p$ data matrix $\mathbf{X}$ that consists of $n$ observations in $p$-dimensional space,

\begin{equation*}
\mathbf{x}_1 =
\begin{pmatrix}
x_{11} \\
\vdots \\
x_{1p}
\end{pmatrix}, \; \ldots, \; \mathbf{x}_n =
\begin{pmatrix}
x_{n1} \\
\vdots \\
x_{np}
\end{pmatrix}
\end{equation*}

and each observation falls into one of two classes: $y_1, \ldots, y_n \in \{-1, 1\}$. Then a separating hyperplane has the helpful property that

\begin{align}
f(x) = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \ldots + \beta_p x_{ip} \quad \text{is} \quad
\begin{cases}
> 0 & \quad \text{if } y_i =1 \\
< 0 & \quad \text{if } y_i = -1
\end{cases}
\end{align}

Given such a hyperplane exists, it can be used to construct a very intuitive classifier: a test observation is assigned to a class based on the side of the hyperplane it lies. This means we simply calculate $f(x^*)$ and if the result is positive, we assign the test observation to class 1, and to class -1 otherwise.

### Maximal Margin Classifier

If our data can be perfectly separated, then - as alluded to above - there exist an infinite number of separating hyperplanes. Therefore we seek to maximize the margin to the closest training observations (support vector). The result is what we call the *maximal margin hyperplane*.

Let us consider how such a maximal margin hyperplane is constructed. We follow Raschka (2015) in deriving the objective function as this approach is appealing to the intuition. For a mathematically more sound derivation, see e.g. Friedman et al. (2001, chapter 4.5). As before we assume to have a set of $n$ training observations $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n \in \mathbb{R}^p$ with corresponding class labels $y_1, y_2, \ldots, y_n \in \{-1, 1\}$. The hyperplane as our decision boundary we have introduced above. Here is the same in vector notation, where $\mathbf{\beta}$ and $\mathbf{x}$ are vector of dimension $[p \times 1]$:

\begin{equation}
\beta_0 + \mathbf{\beta}^T \mathbf{x}_{\text{hyper}} = 0
\end{equation}

This way of writing is much more concise and therefore we will stick to it moving forward. Let us further define the positive and negative margin hyperplanes, which lie parallel to the decision boundary:
\begin{align}
\beta_0 + \mathbf{\beta}^T \mathbf{x}_{\text{pos}} &= 1 &\text{pos. margin} \\
\beta_0 + \mathbf{\beta}^T \mathbf{x}_{\text{neg}} &= -1 &\text{neg. margin}
\end{align}

Below you find a visual representationof the above. Notice that the two margin hyperplanes are parallel and the values for $\beta_0, \mathbf{\beta}$ are identical

If we subtract the equation for the negative margin from the positive, we get:

\begin{equation}
\mathbf{\beta}^T (\mathbf{x}_{\text{pos}} - \mathbf{x}_{\text{neg}}) = 2
\end{equation}

Let us normalize both sides of the equation by the length of the vector $\mathbf{\beta}$, that is the norm, which is defined as follows:

\begin{equation}
\Vert \mathbf{\beta} \Vert := \sqrt{\sum_{i=1}^p \beta_i^2} = 1
\end{equation}

With that we arrive at the following expression:

\begin{equation}
\frac{\mathbf{\beta}^T (\mathbf{x}_{\text{pos}} - \mathbf{x}_{\text{neg}})}{\Vert \mathbf{\beta}\Vert} = \frac{2}{\Vert \mathbf{\beta} \Vert}
\end{equation}

The left side of the equation can be interpreted as the normalized distance between the positive (upper) and negative (lower) margin. This distance we aim to maximize. Since maximizing the lefthand side of above expression is similar to maximizing the right hand side, we can summarize this in the following optimization problem:

\begin{equation}
\begin{aligned}
& \underset{\beta_0, \beta_1, \ldots, \beta_p}{\text{maximize}}
& & \frac{2}{\Vert \mathbf{\beta} \Vert} \\
& \text{subject to} & & \beta_0 + \mathbf{\beta}^T \mathbf{x}_{i} \geq \;\; 1 \quad \text{if } y_i = 1 \\
&&& \beta_0 + \mathbf{\beta}^T \mathbf{x}_{i} \leq -1 \quad \text{if } y_i = -1 \\
&&& \text{for } i = 1, \ldots, N.
\end{aligned}
\end{equation}

The two constraints make sure that all positive samples ($y_i = 1$) fall on or above the positive side of the positive margin hyperplane and all negative samples ($y_i = -1$) are on or below the negative margin hyperplane. A few tweaks allow us to write the two constraints as one. We show this by transforming the second constraint, in which case $y_i = -1$:

\begin{align}
   \beta_0 + \mathbf{\beta}^T \mathbf{x}_i &\leq -1 \\
   \Leftrightarrow \qquad y_i (\beta_0 + \mathbf{\beta}^T \mathbf{x}_i) &\geq (-1)y_i \\
   \Leftrightarrow \qquad y_i (\beta_0 + \mathbf{\beta}^T \mathbf{x}_i) &\geq 1
\end{align}

The same can be done for the first constraint - it will yield the same expression. Therefore, our maximization problem can be restated in a slightly simpler form:

\begin{equation}
\begin{aligned}
& \underset{\beta_0, \beta_1, \ldots, \beta_p}{\text{maximize}}
& & \frac{2}{\Vert \mathbf{\beta} \Vert} \\
& \text{subject to} & & y_i(\beta_0 + \mathbf{\beta}^T \mathbf{x}_{i}) \geq 1 \quad \text{for } i = 1, \ldots, N.
\end{aligned}
\end{equation}

This is a convex optimization problem (quadratic criterion with linear inequality constraints) and can be solved with Lagrange. For details refer to appendix (D1) of the script.

Note that in practice it is easier to minimize the reciprocal term of the squared norm of  $\mathbf{\beta}$, $\frac{1}{2} \Vert\mathbf{\beta} \Vert^2$. Therefore the objective function is often given as

\begin{equation}
\begin{aligned}
& \underset{\beta_0, \beta}{\text{minimize}}
& & \frac{1}{2}\Vert \mathbf{\beta} \Vert^2 \\
& \text{subject to} & & y_i(\beta_0 + \mathbf{\beta}^T \mathbf{x}_{i}) \geq 1 \quad \text{for } i = 1, \ldots, N.
\end{aligned}
\end{equation}

This transformation does not change the optimization problem yet at the same time is computationally easier to be handled by quadratic programming. A detailed discussion of quadratic programming goes beyond the scope of this course. For details, see e.g. Vapnik (2000) or [Burges (1998)](http://www.cmap.polytechnique.fr/~mallat/papiers/svmtutorial.pdf).**

## Support Vector Classifier

### Non-Separable Data

Given our data is separable into two classes, the maximal margin classifier from before seems like a natural approach. However, it is easy to see that **when the data is not clearly discriminable, no separable hyperplane exists and therefore such a classifier does not exist**. In that case the above maximization problem has no solution. What makes the situation even more complicated is that the maximal margin classifier is very sensitive to changes in the support vectors. This means that this classifier might suffer from inappropriate sensitivity to individual observations and thus it has a substantial risk of overfitting the training data. That is why we might be willing to consider a classifier on a hyperplane that does not perfectly separate the two classes but allows for greater robustness to individual observations and better classification of most of the training observations. In other words it could be worthwhile to misclassify a few training observations in order to do a better job in classifying the test data (James et al. (2013)).

### Details of the Support Vector Classifier

This is where the Support Vector Classifier (SVC) comes into play. It allows a certain number of observations to be on the 'wrong' side of the hyperplane while seeking a solution where the majority of data points are still on the 'correct' side of the hyperplane. The following figure visualizes this.

The SVC still classifies a test observation based on which side of a hyperplane it lies. However, when we train the model, the margins are now somewhat softened. This means that the model allows for a limited number of training observations to be on the wrong side of the margin and hyperplane, respectively.

Let us briefly discuss in general terms how the support vector classifier reaches its optimal solution. For this we extend the optimization problem from the maximum margin classifier as follows:

\begin{equation}
\begin{aligned}
& \underset{\beta_0, \beta}{\text{minimize}}
& & \frac{1}{2}\Vert \mathbf{\beta} \Vert^2 + C \left(\sum_{i=1}^n \epsilon_i \right) \\
& \text{subject to} & & \beta_0 + \mathbf{\beta}^T \mathbf{x}_{i} \geq (1-\epsilon_i) \quad \text{for } i = 1, \ldots, N. \\
& & & \epsilon_i \geq 0 \quad \forall i
\end{aligned}
\end{equation}

This, again, can be solved with Lagrange similar to the way it is shown for the maximum margin classifier (see appendix (D1)) and it is left to the reader as an exercise to derive the Lagrange (primal and dual) objective function. For the impatient readers will find a solution draft in Friedman et al. (2001), section 12.2.1.

Let us now focus on the added term $C \left(\sum_{i=1}^n \epsilon_i \right)$.  Here, $\epsilon_1, \epsilon_2, \ldots, \epsilon_n$ are slack variables that allow the individual observations to be on the wrong side of the margin or the hyperplane. They contain information on where the $i$th observation is located, relative to the hyperplane and relative to the margin.

* If $\epsilon_i = 0$ then the $i$th observation is on the correct side of the margin,
* if $1 \geq \epsilon_i > 0$ it is on the wrong side of the margin but correct side of the hyperplane, and
* if $\epsilon_i > 1$ it is on the wrong side of the hyperplane.

The tuning parameter $C$ can be interpreted as a penalty factor for misclassification. It is defined by the user. Large values of $C$ correspond to a significant error penalty, whereas small values are used if we are less strict about misclassification errors. By controlling for $C$ we indirectly control for the margin and therefore actively tune the bias-variance trade-off. Decreasing the value of $C$ increases the bias but lowers the variance of the model.

Below figure shows how $C$ impacts the decision boundary and its corresponding margin.

### Solving Nonlinear Problems

So far we worked with data that is linearly separable. What makes SVM so powerful and popular is that it can be kernelized to solve nonlinear classification problems. We start our discussion again with illustrations to build an intuition.

Clearly the data is not linear and the resulting (linear) decision boundary is useless. How, then, do we deal with this? With mapping functions. The basic idea is to project the data via some mapping function $\phi$ onto a higher dimension such that a linear separator would be sufficient. The idea is similar to using quadratic and cubic terms of the predictor in linear regression in order to address non-linearity $(y = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + \ldots)$ . For example, for the data in the preceding figure we could use the following mapping function $\phi: \mathbb{R}^2 \rightarrow \mathbb{R}^3$.

\begin{equation}
\phi(x_1, x_2) = (z_1, z_2, z_3) = \left(x_1, x_2, x_1^2 + x_2^2 \right)
\end{equation}

Here we enlarge our feature space from $\mathbb{R}^2 \rightarrow \mathbb{R}^3$ in oder to accommodate a non-linear boundary. The transformed data becomes trivially linearly separable. All we have to do is find a plane in $\mathbb{R}^3$. If we project this decision boundary back onto the original feature space $\mathbb{R}^2$ (with $\phi^{-1}$), we have a nonlinear decision boundary.

Here's an animated visualization of this concept.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('3liCbRZPrZA')
```

### The Problem with Mapping Functions
One could think that this is the recipe to work with nonlinear data: Transform all training data onto a higher-dimensional feature space via some mapping function $\phi$ train a linear SVM model and use the same function $\phi$ to transform new (test) data to classify it.

As attractive as this idea seems, it is unfortunately unfeasible because it quickly becomes computationally very expensive. Here is a hands-on example why: Consider for example a degree-2 polynomial (kernel) transformation of the form $\phi(x_1, x_2) = (x_1^2, x_2^2, \sqrt{2} x_1 x_2, \sqrt{2c} x_1, \sqrt{2c} x_2, c)$. This means that for a dataset in $\mathbb{R}^2$ the transformation adds four additional dimensions ($\mathbb{R}^2 \rightarrow \mathbb{R}^6$). If we generalize this, it means that a $d$-dimensional polynomial (Kernel) transformation maps from $\mathbb{R}^p$ to an ${p + d}\choose{d}$-dimensional space [(Balcan (2011))](http://www.cs.cmu.edu/%7Eninamf/ML11/lect1020.pdf). Thus for datasets with $p$ large, naively performing such transformations will force most computers to its knees.

### The Kernel Trick

Thankfully, not all is lost. It turns out that one does not need to explicitly work in the higher-dimensional space. One can show that when using Lagrange to solve our optimization problem, the training samples are only used to compute the pair-wise dot products $\langle x_i, x_{j}\rangle$ (where $x_i, x_{j} \in \mathbb{R}^{p}$). This is significant because there exist functions that, given two vectors $x_i$ and $x_{j}$ in $\mathbb{R}^p$, implicitly compute the dot product between the two vectors in a higher-dimension $\mathbb{R}^q$ (with $q > p$) without explicitly transforming $x_i, x_{j}$ onto a higher dimension $\mathbb{R}^q$. Such functions are called **Kernel** functions, written $K(x_i, x_{j})$ [(Kim (2013))](http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html#[6]).

Let us show an example of such a Kernel function (following [Hofmann (2006)](http://www.cogsys.wiai.uni-bamberg.de/teaching/ss06/hs_svm/slides/SVM_Seminarbericht_Hofmann.pdf)). For ease of reading we use $x = (x_1, x_2)$ and $z=(z_1, z_2)$ instead of $x_i$ and $x_{j}$. Consider the Kernel function $K(x, z) = (x^T z)^2$ and the mapping function $\phi(x) = (x_1^2, \sqrt{2}x_1 x_2, x_2^2)$. If we were to solve our optimization problem from above with Lagrange, the mapping function appears in the form $\phi(x)^T \phi(z)$.

\begin{align}
\phi(x)^T \phi(z) &= (x_1^2, \sqrt{2}x_1 x_2, x_2^2)^T (z_1^2, \sqrt{2}z_1 z_2, z_2^2) \\
  &= x_1^2 z_1^2 + 2x_1 z_1 x_2 z_2 + x_2^2 z_2^2 \\
  &= (x_1 z_1 + x_2 z_2)^2 \\
  &= (x^T z)^2 \\
  &= K(x, z)
\end{align}

The mapping function would have transformed the data from $\mathbb{R}^2 \rightarrow \mathbb{R}^3$ and back. The Kernel function, however, stays in $\mathbb{R}^2$. This is of course only one (toy) example and far away from a proper proof but it provides the intuition of what can be generalized: that by using a Kernel function where $K(x_i, x_j) = (x^T z)^2 = \phi(x_i)^T \phi(x_j)$, we implicitly transforms our data to a higher-dimension without having to explicitly apply a mapping function $\phi$. This so called "Kernel Trick" allows us to efficiently learn nonlinear decision boundaries for SVM.

### Popular Kernel Functions
Not every random mapping function is also a Kernel function. For a function to be a Kernel function, it needs to have certain properties (see e.g. [Balcan (2011)](http://www.cogsys.wiai.uni-bamberg.de/teaching/ss06/hs_svm/slides/SVM_Seminarbericht_Hofmann.pdf) or [Hofmann (2006)](http://www.cogsys.wiai.uni-bamberg.de/teaching/ss06/hs_svm/slides/SVM_Seminarbericht_Hofmann.pdf) for a discussion). In SVM literature, the following three Kernel functions have emerged as popular choices (Friedman et al. (2001)):

\begin{align}
d\text{th-Degree polynomial} \qquad K(x_i, x_j) &= (r + \gamma \langle x_i, x_j \rangle)^d \\
\text{Radial Basis (RBF)} \qquad K(x_i, x_j) &= \exp(-\gamma \Vert x_i - x_j \Vert^2) \\
\text{Sigmoid} \qquad K(x_i, x_j) &= \tanh(\gamma \langle x_i, x_j \rangle + r)
\end{align}

In general there is no "best choice". With each Kernel having some degree of variability, one has to find the optimal solution by experimenting with different Kernels and playing with their parameter ($\gamma, r, d$).

### Optimization with Lagrange

We have mentioned before that the optimization problem of the maximum margin classifier and support vector classifier can be solved with Lagrange. The details of which are beyond the scope of this notebook. However, the interested reader is encouraged to learn the details in the appendix of the script (and the recommended reference sources) as these are crucial in understanding the mathematics/core of SVM and the application of Kernel functions.

## SVM with Scikit-Learn
### Preparing the Data

Having build an intuition of how SVM work, let us now see this algorithm applied in Python. We will again use the Scikit-learn package that has an optimized class implemented. The data we will work with is called "Polish Companies Bankruptcy Data Set" and was used in Zieba et al. (2014). The full set comprises five data files. Each file contains 64 features plus a class label. The features are ratios derived from the financial statements of the more than 10'000 manufacturing companies considered during the period of 2000 - 2013 (from EBITDA margin to equity ratio to liquidity ratios (quick ratio etc.)). The five files differ in that the first contains data with companies that defaulted/were still running **five** years down the road ('1year.csv'), the second **four** years down the road ('2year.csv') etc. Details can be found in the original publication (Zikeba et al. (2016)) or in the [description provided on the UCI Machine Learning Repository site](https://archive.ics.uci.edu/ml/datasets/Polish+companies+bankruptcy+data) where the data was downloaded from. For our purposes we will use the '5year.csv' file where we should predict defaults within the next year.

```python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
plt.rcParams['font.size'] = 14
```

```python
# Load data
df = pd.read_csv('Data/5year.csv', sep=',')
df.head()
```

<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Attr1</th>
      <th>Attr2</th>
      <th>Attr3</th>
      <th>Attr4</th>
      <th>Attr5</th>
      <th>Attr6</th>
      <th>Attr7</th>
      <th>Attr8</th>
      <th>Attr9</th>
      <th>Attr10</th>
      <th>...</th>
      <th>Attr56</th>
      <th>Attr57</th>
      <th>Attr58</th>
      <th>Attr59</th>
      <th>Attr60</th>
      <th>Attr61</th>
      <th>Attr62</th>
      <th>Attr63</th>
      <th>Attr64</th>
      <th>class</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.088238</td>
      <td>0.55472</td>
      <td>0.01134</td>
      <td>1.0205</td>
      <td>-66.5200</td>
      <td>0.342040</td>
      <td>0.109490</td>
      <td>0.57752</td>
      <td>1.0881</td>
      <td>0.32036</td>
      <td>...</td>
      <td>0.080955</td>
      <td>0.275430</td>
      <td>0.91905</td>
      <td>0.002024</td>
      <td>7.2711</td>
      <td>4.7343</td>
      <td>142.760</td>
      <td>2.5568</td>
      <td>3.2597</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>-0.006202</td>
      <td>0.48465</td>
      <td>0.23298</td>
      <td>1.5998</td>
      <td>6.1825</td>
      <td>0.000000</td>
      <td>-0.006202</td>
      <td>1.06340</td>
      <td>1.2757</td>
      <td>0.51535</td>
      <td>...</td>
      <td>-0.028591</td>
      <td>-0.012035</td>
      <td>1.00470</td>
      <td>0.152220</td>
      <td>6.0911</td>
      <td>3.2749</td>
      <td>111.140</td>
      <td>3.2841</td>
      <td>3.3700</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.130240</td>
      <td>0.22142</td>
      <td>0.57751</td>
      <td>3.6082</td>
      <td>120.0400</td>
      <td>0.187640</td>
      <td>0.162120</td>
      <td>3.05900</td>
      <td>1.1415</td>
      <td>0.67731</td>
      <td>...</td>
      <td>0.123960</td>
      <td>0.192290</td>
      <td>0.87604</td>
      <td>0.000000</td>
      <td>8.7934</td>
      <td>2.9870</td>
      <td>71.531</td>
      <td>5.1027</td>
      <td>5.6188</td>
      <td>0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.089951</td>
      <td>0.88700</td>
      <td>0.26927</td>
      <td>1.5222</td>
      <td>-55.9920</td>
      <td>-0.073957</td>
      <td>-0.089951</td>
      <td>0.12740</td>
      <td>1.2754</td>
      <td>0.11300</td>
      <td>...</td>
      <td>0.418840</td>
      <td>-0.796020</td>
      <td>0.59074</td>
      <td>2.878700</td>
      <td>7.6524</td>
      <td>3.3302</td>
      <td>147.560</td>
      <td>2.4735</td>
      <td>5.9299</td>
      <td>0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.048179</td>
      <td>0.55041</td>
      <td>0.10765</td>
      <td>1.2437</td>
      <td>-22.9590</td>
      <td>0.000000</td>
      <td>0.059280</td>
      <td>0.81682</td>
      <td>1.5150</td>
      <td>0.44959</td>
      <td>...</td>
      <td>0.240400</td>
      <td>0.107160</td>
      <td>0.77048</td>
      <td>0.139380</td>
      <td>10.1180</td>
      <td>4.0950</td>
      <td>106.430</td>
      <td>3.4294</td>
      <td>3.3622</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 65 columns</p>
</div>

```python
# Check for NA values
df.isnull().sum()
```

    Attr1        3
    Attr2        3
    Attr3        3
    Attr4       21
    Attr5       11
    Attr6        3
    Attr7        3
    Attr8       18
    Attr9        1
    Attr10       3
    Attr11       3
    Attr12      21
    Attr13       0
    Attr14       3
    Attr15       6
    Attr16      18
    Attr17      18
    Attr18       3
    Attr19       0
    Attr20       0
    Attr21     103
    Attr22       3
    Attr23       0
    Attr24     135
    Attr25       3
    Attr26      18
    Attr27     391
    Attr28     107
    Attr29       3
    Attr30       0
              ...
    Attr36       3
    Attr37    2548
    Attr38       3
    Attr39       0
    Attr40      21
    Attr41      84
    Attr42       0
    Attr43       0
    Attr44       0
    Attr45     268
    Attr46      21
    Attr47      35
    Attr48       3
    Attr49       0
    Attr50      18
    Attr51       3
    Attr52      36
    Attr53     107
    Attr54     107
    Attr55       0
    Attr56       0
    Attr57       3
    Attr58       0
    Attr59       3
    Attr60     268
    Attr61      15
    Attr62       0
    Attr63      21
    Attr64     107
    class        0
    Length: 65, dtype: int64

```python
# Calculate % of missing values for 'Attr37'
df['Attr37'].isnull().sum()  / (len(df))
```

    0.43113367174280881

Attribute 37 sticks out with 2'548 of 5'910 (43.1%) missing values. This attribute considers *"(current assets - inventories) / long-term liabilities"*. Due to the many missing values we can not use a fill method so let us drop this feature column.

```python
df = df.drop('Attr37', axis=1)
df.iloc[:, 30:38].head()
```

<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Attr31</th>
      <th>Attr32</th>
      <th>Attr33</th>
      <th>Attr34</th>
      <th>Attr35</th>
      <th>Attr36</th>
      <th>Attr38</th>
      <th>Attr39</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.077287</td>
      <td>155.330</td>
      <td>2.3498</td>
      <td>0.24377</td>
      <td>0.135230</td>
      <td>1.4493</td>
      <td>0.32101</td>
      <td>0.095457</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.000778</td>
      <td>108.050</td>
      <td>3.3779</td>
      <td>2.70750</td>
      <td>-0.036475</td>
      <td>1.2757</td>
      <td>0.59380</td>
      <td>-0.028591</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.143490</td>
      <td>81.653</td>
      <td>4.4701</td>
      <td>0.65878</td>
      <td>0.145860</td>
      <td>1.1698</td>
      <td>0.67731</td>
      <td>0.129100</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.138650</td>
      <td>253.910</td>
      <td>1.4375</td>
      <td>0.83567</td>
      <td>0.014027</td>
      <td>1.2754</td>
      <td>0.43830</td>
      <td>0.010998</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.039129</td>
      <td>140.120</td>
      <td>2.6583</td>
      <td>2.13360</td>
      <td>0.364200</td>
      <td>1.5150</td>
      <td>0.51225</td>
      <td>0.240400</td>
    </tr>
  </tbody>
</table>
</div>

As for the other missing values we are left to decide whether we want to remove the corresponding observations (rows) or apply a filling method. The problem with dropping all rows with missing values is that we might lose a lot of valuable information. Therefore in this case we prefer to use a common interpolation technique and impute `NaN` values with the feature mean. Alternatively we could use '`median`' or '`most_frequent`' as strategy. A convenient way to achieve this imputation is to use the `Imputer` class from `sklearn`.

```python
from sklearn.preprocessing import Imputer

# Impute missing values by mean (axis=0 --> along columns)
ipr = Imputer(missing_values='NaN', strategy='mean', axis=0)
ipr = ipr.fit(df.values)
imputed_data = ipr.transform(df.values)

# Assign imputed values to 'df' and check for 'NaN' values
df = pd.DataFrame(imputed_data, columns=df.columns)
df.isnull().sum().sum()
```

    0

Now let us check if we have some categorical features that we need to transform. For this we compare the number of cells in the dataframe with the sum of numeric values (`np.isreal()`). If the result is 0, we do not need to apply a One-Hot-Encoding or LabelEncoding procedure.

```python
df.shape[0] * df.shape[1] - df.applymap(np.isreal).sum().sum()
```

    0

As we see, the dataframe only consists of real values. Therefore, we can proceed by assigning columns 1-63 to variable `X` and column 64 to `y`.

```python
X = df.iloc[:, :-1].values
y = df.iloc[:, -1].values
```

### Applying SVM

Having assigned the data to `X` and `y` we are now ready to divide the dataset into separate training and test sets.

```python
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    random_state=0,
                                                    stratify=y)
```

Unlike e.g. decision tree algorithms SVM are sensitive to the magnitude the data. Therefore scaling our data is recommended.

```python
from sklearn.preprocessing import StandardScaler

# Create StandardScaler object
sc = StandardScaler()

# Standardize features; equal results as if done in two
# separate steps (first .fit() and then .transform())
X_train_std = sc.fit_transform(X_train)

# Transform test set
X_test_std = sc.transform(X_test)
```

With the data standardized, we can finally apply a SVM on the data. We import the `SVC` (for Support Vector Classifier) from the Scikit-learn toolbox and create a `svm_linear` object that represents a linear SVM with `C=0`. Recall that `C` helps us control the penalty for misclassification. Large values of `C` correspond to large error penalties and vice-versa. More parameter can be specified. Details are best explained in the function's [documentation page](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).

```python
from sklearn.svm import SVC
from sklearn import metrics
import matplotlib.pyplot as plt

# Create object
svm_linear = SVC(kernel='linear', C=1.0)
svm_linear
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

With the `svm_linear` object ready we can now fit the object to the training data and check for the model's accuracy.

```python
# Fit linear SVM to standardized training set
svm_linear.fit(X_train_std, y_train)

# Print results
print("Observed probability of default: {:.2f}".format(np.count_nonzero(y==0) / len(y)))
print("Train score: {:.2f}".format(svm_linear.score(X_train_std, y_train)))
print("Test score:  {:.2f}".format(svm_linear.score(X_test_std, y_test)))
```

    Observed probability of default: 0.93
    Train score: 0.93
    Test score:  0.93

```python
# Predict classes
y_pred = svm_linear.predict(X_test_std)

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted': y_pred,
                      'True': y_test})
confm.replace(to_replace={0:'Non-Default', 1:'Default'}, inplace=True)
print(confm.groupby(['True','Predicted'], sort=False).size().unstack('Predicted'))
```

    Predicted    Non-Default  Default
    True
    Non-Default       1096.0      4.0
    Default             82.0      NaN

In the same way we can run a Kernel SVM on the data. We have four Kernel options: one linear as introduced above and three non-linear. All of them have hyperparameter available. If these are not specified, default values are taken. [Check the documentation for details](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html).

* `linear`: linear SVM as shown above with `C` as hyperparameter
* `rbf`: Radial basis function Kernel with `C, gamma` as hyperparameter
* `poly`: Polynomial Kernel with `C, degree, gamma, coef0` as hyperparameter
* `sigmoid`: Sigmoid Kernel with `C, gamma, coef0` as hyperparameter

Let us apply a polynomial Kernel as example.

```python
svm_poly = SVC(kernel='poly', random_state=1)
svm_poly
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='poly',
      max_iter=-1, probability=False, random_state=1, shrinking=True,
      tol=0.001, verbose=False)

Not having specified hyperparameter `C, degree, gamma`, and `coef0` we see that the algorithm has taken default values. For `C` it is equal to 1, default `degree` is 3, `gamma=auto` means that the value will be calculated as $1/n_{\text{features}}$, and `coef0` is set to 0 as default.

```python
# Fit polynomial SVM to standardized training set
svm_poly.fit(X_train_std, y_train)

# Print results
print("Observed probability of default: {:.2f}".format(np.count_nonzero(y==0) / len(y)))
print("Train score: {:.2f}".format(svm_poly.score(X_train_std, y_train)))
print("Test score:  {:.2f}".format(svm_poly.score(X_test_std, y_test)))
```

    Observed probability of default: 0.93
    Train score: 0.94
    Test score:  0.93

```python
# Predict classes
y_pred = svm_poly.predict(X_test_std)

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted': y_pred,
                      'True': y_test})
confm.replace(to_replace={0:'Non-Default', 1:'Default'}, inplace=True)
print(confm.groupby(['True','Predicted'], sort=False).size().unstack('Predicted'))
```

    Predicted    Non-Default  Default
    True
    Non-Default         1096        4
    Default               81        1

As it looks linear and polynomial SVM yield similar results. What is clearly unsatisfactory is the number of true defaults that the SVM missed to detect. Both linear as well as non linear SVM miss to label $\geq$ 80 defaults [sic]. From a financial perspective, this is unacceptable and raises questions regarding
* Class imbalance
* Hyperparameter fine-tuning through cross validation and grid search
* Feature selection
* Noise & dimension reduction

which we want to address in the next section.

## Dealing with Class Imbalance

When we deal with default data sets we observe that the ratio of non-default to default records is heavily skewed towards non-default. This is a common problem in real-world data set: Samples from one class or multiple classes are over-represented. For the present data set we are talking 93% non-defaults vs. 7% defaults. Having an algorithm that predicts non-default 100 out of a 100 times is right in 93% of the cases. Therefore, training a model on such a data set that achieves the same 93% test accuracy (as our SVM above) means nothing else than our model hasn't learned anything informative from the features provided in this data set. Thus, when assessing a classifier on an imbalanced data set we have learned that other metrics such as precision, recall, ROC curve etc. might be more informative.

Having said that, what we have to consider is that a class imbalance might influences a learning algorithm during the model fitting itself. Machine learning algorithms typically optimize a reward or cost function. This means that an algorithm implicitly learns the model that optimizes the predictions based on the most abundant class in the dataset in order to minimize the cost or maximize the reward during the training phase. And this in turn might yield skewed results in case of imbalanced data sets.

There are several options to deal with class imbalance, we will discuss two of them. The first option is to set the `class_weight` parameter to `class_weight='balanced'`. Most classifier hae this option implemented (of the introduced classifiers, KNN, LDA and QDA lack such a parameter). This will assign a larger penalty to wrong predictions on the minority class.

```python
# Initiate and fit a polynomial SVM to training set
svm_poly = SVC(kernel='poly', random_state=1, class_weight='balanced')
svm_poly.fit(X_train_std, y_train)

# Predict classes and print results
y_pred = svm_poly.predict(X_test_std)
print(metrics.classification_report(y_test, y_pred))
print(metrics.confusion_matrix(y_test, y_pred))
print("Test score: {:.2f}".format(svm_poly.score(X_test_std, y_test)))
```

                 precision    recall  f1-score   support

            0.0       0.93      0.99      0.96      1100
            1.0       0.12      0.02      0.04        82

    avg / total       0.88      0.92      0.89      1182

    [[1086   14]
     [  80    2]]
    Test score: 0.92

The second option we want to discuss is up- & downsampling of the minority/majority class. Both up- and downsampling are implemented in Scikit-learn through the `resample` function and depending on the data and given the task at hand, one might be better suited than the other. For the upsampling, scikit-learn will apply a bootstrapping to draw new samples from the datasets with replacement. This means that the function will repeatedly draw new samples from the minority class until it contains the number of samples we define. Here's a code example:

```python
from sklearn.utils import resample

# Upsampling
X_upsampled, y_upsampled = resample(X[y==1], y[y==1],
                                    replace=True,
                                    n_samples=X[y==0].shape[0],
                                    random_state=1)
print('No. of default samples BEFORE upsampling:  {:.0f}'.format(y.sum()))
print('No. of default samples AFTER upsampling:  {:.0f}'.format(y_upsampled.sum()))
```

    No. of default samples BEFORE upsampling:  410
    No. of default samples AFTER upsampling:  5500

Downsampling works in similar fashion.

```python
# Downsampling
X_dnsampled, y_dnsampled = resample(X[y==0], y[y==0],
                                    replace=False,
                                    n_samples=X[y==1].shape[0],
                                    random_state=1)
```

Running the SVM algorighm on the balanced dataset works now as you would expect:

```python
# Combine datasets
X_bal = np.vstack((X[y==1], X_dnsampled))
y_bal = np.hstack((y[y==1], y_dnsampled))

# Train test split
X_train_bal, X_test_bal, y_train_bal, y_test_bal = \
    train_test_split(X_bal, y_bal,
                     test_size=0.2,
                     random_state=0,
                     stratify=y_bal)

# Standardize features; equal results as if done in two
# separate steps (first .fit() and then .transform())
X_train_bal_std = sc.fit_transform(X_train_bal)

# Transform test set
X_test_bal_std = sc.transform(X_test_bal)

# Initiate and fit a polynomial SVM to training set
svm_poly_bal = SVC(kernel='poly', random_state=1)
svm_poly_bal.fit(X_train_bal_std, y_train_bal)

# Predict classes and print results
y_pred_bal = svm_poly_bal.predict(X_test_bal_std)
print(metrics.classification_report(y_test_bal, y_pred_bal))
print(metrics.confusion_matrix(y_test_bal, y_pred_bal))
print("Test score: {:.2f}".format(svm_poly_bal.score(X_test_bal_std, y_test_bal)))
```

                 precision    recall  f1-score   support

            0.0       0.51      1.00      0.68        82
            1.0       1.00      0.05      0.09        82

    avg / total       0.76      0.52      0.39       164

    [[82  0]
     [78  4]]
    Test score: 0.52

By applying a SVM to a balanced set of data we improve our model slightly. Yet there remains some work to be done. The polynomial SVM still misses out on 95.1% (=78/82) of the default cases.

It should be said that in general using an upsampled set is to be preferred over a downsampled set. However, here we are talking 11'000 observations times 63 features for the upsampled set and this can easily take quite some time to run models on, especially if we compute a grid search as in the next section. For this reason the downsampled set was used.

## Hyperparameter Fine-Tuning
### Pipelines

Another tool that is of help in optimizing our model is the `GridSearchCV` function introduced in the previous chapter that finds the best hyperparameter through a brute-force (cross validation) approach. Yet before we simply copy-past the code from the last chapter we ought to address a subtle yet important difference between the decision tree and SVM (or most other ML) algorithms that has implications on the application: Decision tree algorithms are of the few models where data scaling is not necessary. SVM on the other hand are (as most ML algorithms) fairly sensitive to the magnitude of the data. Now you might say that this is precisely why we standardized the data at the very beginning and with that we are good to go. In principle, this is correct. However, if we are precise, we commit a subtle yet possibly significant thought error.

If we decide to apply a grid search using cross validation to find the optimal hyperparameter for e.g. a SVM we unfortunately can not just scale the full data set at the very beginning and then be good for the rest of the process. Conceptually it is important to understand why. Assume we have a data set. As we learned in the chapter on feature scaling and cross validation, applying a scaling on the combined data set and splitting the set into training and holdout set after the scaling is wrong. The reason is that information from the test set found its way into the model and distorts the results. The training set is scaled with not only based on information from that set but also based on information from the test set.

Now the same is true if we apply a gridsearch process with cross validation on a training set. For each fold in the CV, some part of the training set will be declared as the training part, and some the test part. The test part within this split is used to measure the performance of our model trained on the training part. However, if we simply scale the training set and then apply gridsearch-CV on the scaled training set we would commit the same thought error as if we simply scale the full set at the very beginning. The test fold (of the CV split) would no longer be independent but implicitly already be part of the training set we used to fit the model. This is fundamentally different from how new data looks to the model. The test data within each cross validation split would no longer correctly mirrors how new data would look to the modeling process. Information already leaked from the test data into our modeling process. This would lead to overly optimistic results during cross validation, and possibly the selection of suboptimal parameter (Müller & Guido (2017)).

We have not addressed this problem in the chapter on cross validation because so far we have not introduced the tool to deal with it. Furthermore, if our data set is homogeneous and of some size, this is less of an issue. Yet as Scikit-learn provides a fantastic tool to deal with this (and many other) issue(s), we want to introduce it here. The tool is called **pipelines** and allows to combine multiple processing steps in a very convenient and proper way. Let us look at how we can use the `Pipeline` class to express the end-to-end workflow. First we build a pipeline object. This object is provided a list of steps. Each step is a tuple containing a name (you define) and an instance of an estimator.

```python
from sklearn.pipeline import Pipeline

# Create pipeline object with standard scaler and SVC estimator
pipe = Pipeline([('scaler', StandardScaler()),
                 ('svm_poly', SVC(kernel='poly', random_state=0))])
```

Next we define a parameter grid to search over and construct a `GridSearchCV` from the pipeline and the parameter grid. Notice that we have to specify for each parameter which step of the pipeline it belongs to. This is done by calling the name we gave this step, followed by a double underscore and the parameter name. For the present example, let us compare different degrees, and `C` values.

```python
# Define parameter grid
param_grid = {'svm_poly__C': [0.1, 1, 10, 100],
              'svm_poly__degree': [1, 2, 3, 5, 7]}
```

With that we can run a `GridSearchCV` as usual.

```python
from sklearn.model_selection import GridSearchCV

# Run grid search
grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(X_train_bal, y_train_bal)

# Print results
print('Best CV accuracy: {:.2f}'.format(grid.best_score_))
print('Test score:       {:.2f}'.format(grid.score(X_test_bal, y_test_bal)))
print('Best parameters: {}'.format(grid.best_params_))
```

    Best CV accuracy: 0.73
    Test score:       0.78
    Best parameters: {'svm_poly__C': 100, 'svm_poly__degree': 1}

Notice that thanks to the pipeline object, now for each split in the cross validation the `StandardScaler` is refit with only the training splits and no information is leaked from the test split into the parameter search.

Depending on the grid you search, computations might take quite some time. One way to improve speed is by reducing the feature space; that is reducing the number of features. We will discuss feature selection and dimension reduction options in the next section but for the moment, let us just apply a method called Principal Component Analysis (PCA). PCA effectively transforms the feature space from $\mathbb{R}^{p} \rightarrow \mathbb{R}^{q}$ with $q$ being a user specified value (but usually $q < < p$). PCA is similar to other preprocessing steps and can be included in pipelines as e.g. `StandardScaler`.

Here we reduce the feature space from $\mathbb{R}^{63}$ (i.e. $p=63$ features) to $\mathbb{R}^{2}$. This will make the fitting process faster. However, this comes at a cost: by reducing the feature space we might not only get rid of noise but also lose part of the information available in the full dataset. Our model accuracy might suffer as a consequence. Furthermore, the speed that we gain by fitting a model to a smaller subset can be set off by the additional computations it takes to calculate the PCA. In the example of the upsampled data set we would be talking of an $[11'000 \cdot 0.8 \cdot 0.8 \times 63]$ matrix (0.8 for the train/test-split and each cv fold) for which eigenvector and eigenvalues need to be calculated. This means up to 63 eigenvalues per grid search loop.

```python
from sklearn.decomposition import PCA

# Create pipeline object with standard scaler, PCA and SVC estimator
pipe = Pipeline([('scaler', StandardScaler()),
                 ('pca', PCA(n_components=2)),
                 ('svm_poly', SVC(kernel='poly', random_state=0))])

# Define parameter grid
param_grid = {'svm_poly__C': [100],
              'svm_poly__degree': [1, 2, 3]}

# Run grid search
grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(X_train_bal, y_train_bal)

# Print results
print('Best CV accuracy: {:.2f}'.format(grid.best_score_))
print('Test score:       {:.2f}'.format(grid.score(X_test_bal, y_test_bal)))
print('Best parameters: {}'.format(grid.best_params_))
```

    Best CV accuracy: 0.64
    Test score:       0.74
    Best parameters: {'svm_poly__C': 100, 'svm_poly__degree': 2}

Other so called preprocessing steps can be included in the pipeline too. This shows how seamless such workflows can be steered through pipelines. We can even combine multiple models as we show in the next code snippet. By now you are probably aware that trying all possible solutions is not a viable machine learning strategy. Computational power is certainly going to be an issue. Nevertheless, for the record we provide below an example where we apply logistic regression and a SVM with RBF kernel to find the best solution (details see section on PCA below).

```python
from sklearn.linear_model import LogisticRegression

# Create pipeline object with standard scaler, PCA and SVC estimator
pipe = Pipeline([('scaler', StandardScaler()),
                 ('classifier', SVC(random_state=0))])

# Define parameter grid
param_grid = [{'scaler': [StandardScaler()],
               'classifier': [SVC(kernel='rbf')],
               'classifier__gamma': [1, 10],
               'classifier__C': [10, 100]},
              {'scaler': [StandardScaler(), None],
               'classifier': [LogisticRegression()],
               'classifier__C': [10, 100]}]

# Run grid search
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X_train_bal, y_train_bal)

# Print results
print('Best CV accuracy: {:.2f}'.format(grid.best_score_))
print('Test score:       {:.2f}'.format(grid.score(X_test_bal, y_test_bal)))
print('Best parameters: {}'.format(grid.best_params_))
```

    Best CV accuracy: 0.76
    Test score:       0.79
    Best parameters: {'classifier': LogisticRegression(C=100, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False), 'classifier__C': 100, 'scaler': StandardScaler(copy=True, with_mean=True, with_std=True)}

From the above output we see that surprisingly the Logistic regression yields the best accuracy (with `C=100`).

## Feature Selection and Dimensionality Reduction
### Complexity and the Curse of Overfitting

If we observe that a model performs much better on training than on test data, we have an indication that the model suffers from overfitting. The reason for the overfitting is most probably that our model is too complex for the given training data. Common solutions to reduce the generalization error are (Raschka (2015)):
* Collect more (training) data
* Introduce a penalty for complexity via regularization
* Choose a simpler model with fewer parameter
* Reduce the dimensionality of the data

Collecting more data is self explanatory but often not applicable. Regularization via a complexity penalty term is a technique that is primarily applicable to regression settings (e.g. logistic regression). We will not discuss it here but the interested reader will easily find helpful information in e.g. James et al. (2013) chapter 6 or Raschka (2015) chapter 4. Here we will look at one commonly used solution to reduce overfitting: dimensionality reduction via feature selection.

### Feature Selection

A useful approach to select relevant features from a data set is to use information from the random forest algorithm we introduced in the previous chapter. There we elaborated how decision trees rank the feature importance based on a impurity decrease. Conveniently, we can access this feature importance rank directly from the `RandomForestClassifier` object. By executing below code - following the example in Raschka (2015) - we will train a random forest model on the balanced default data set (from before) and rank the features by their respective importance measure.

```python
from sklearn.ensemble import RandomForestClassifier

# Extract feature labels
feat_labels = df.columns[:-1]

# Create Random Forest object, fit data and
# extract feature importance attributes
forest = RandomForestClassifier(random_state=1)
forest.fit(X_train_bal, y_train_bal)
importances = forest.feature_importances_
```

```python
# Sort output (by relative importance) and
# print top 15 features
indices = np.argsort(importances)[::-1]
n = 15
for i in range(n):
    print('{0:2d}) {1:7s} {2:6.4f}'.format(i + 1,
                                           feat_labels[indices[i]],
                                           importances[indices[i]]))
```

     1) Attr39  0.0811
     2) Attr27  0.0684
     3) Attr15  0.0471
     4) Attr13  0.0447
     5) Attr16  0.0421
     6) Attr21  0.0355
     7) Attr26  0.0339
     8) Attr11  0.0310
     9) Attr23  0.0303
    10) Attr7   0.0292
    11) Attr46  0.0251
    12) Attr25  0.0228
    13) Attr34  0.0212
    14) Attr41  0.0201
    15) Attr9   0.0185

The value in decimal is the relative importance for the respective feature. We can also plot this result to have a better overview. Below code shows one way of doing it.

```python
# Get cumsum of the n most important features
feat_imp = np.sort(importances)[::-1]
sum_feat_imp = np.cumsum(feat_imp)[:n]
```

```python
# Plot Feature Importance (both cumul., individual)
plt.figure(figsize=(12, 8))
plt.bar(range(n), importances[indices[:n]], align='center')
plt.xticks(range(n), feat_labels[indices[:n]], rotation=90)
plt.xlim([-1, n])
plt.xlabel('Feature')
plt.ylabel('Rel. Feature Importance')
plt.step(range(n), sum_feat_imp, where='mid',
         label='Cumulative importance')
plt.tight_layout();
```

Executing the code will rank the different features according to their relative importance. The definition of each `AttrXX` we would have to [look up in the data description](https://archive.ics.uci.edu/ml/datasets/Polish+companies+bankruptcy+data). Note that the feature importance values are normalized such that they sum up to 1.

Feature selection in the way shown in the preceding code snippets will not work in combination with a `pipeline` object. However, Scikit-learn has implemented such a function that could be used in a preprocessing step. Its name is `SelectFromModel` and details can be found [here](http://scikit-learn.org/stable/modules/feature_selection.html#feature-selection-using-selectfrommodel). Instead of selecting the top $n$ features you define a threshold, which selects those features whose importance is greater or equal to said threshold (e.g. mean, median etc.). For reference, below it is shown how the function is applied inside a pipeline.

```python
from sklearn.feature_selection import SelectFromModel

pipe = Pipeline([('feature_selection', SelectFromModel(RandomForestClassifier(), threshold='median')),
                 ('scaler', StandardScaler()),
                 ('classification', SVC())])
pipe.fit(X_train_bal, y_train_bal).score(X_test_bal, y_test_bal)
```

    0.78658536585365857

### Principal Component Analysis

In the previous section you learned an approach for reducing the dimensionality of a data set through feature selection. An alternative to feature selection is feature extraction, of which Principal Component Analysis (PCA) is the best known and most popular approach. It is an unsupervised method that aims to summarize the information content of a data set by transforming it onto a new feature subspace of lower dimensionality than the original one. With the rise of big data, this is a field that is gaining importance by the day. PCA is widely used in a variety of field - e.g. in finance to de-noise signals in stock market trading, create factor models, for feature selection in bankruptcy prediction, dimensionality reduction of high frequency data etc.. Unfortunately, the scope of this course does not allow us to discuss PCA in great detail. Nevertheless the fundamentals shall be addressed here briefly so that the reader has a good understanding of how PCA helps in reducing dimensionality.

To build an intuition for PCA we quote the excellent James et al. (2013, p. 375): *"PCA finds a low-dimensional representation of a dataset that contains as much as possible of the **variation**. The idea is that each of the $n$ observations lives in $p$-dimensional space, but not all of these dimensions are equally interesting. PCA seeks a small number of dimensions that are as interesting as possible, where the concept of interesting is measured by the amount that the observation vary along each dimension. Each of the dimensions found by PCA is a linear combination of the $p$ features."* Since each principal component is required to be orthogonal to all other principal components, we basically take correlated original variables (features) and replace them with a small set of principal components that capture their joint variation.

Below figures aim at visualizing the idea of principal components. In both figures we see the same two-dimensional dataset. PCA searches for the principal axis along which the data varies most. These principal axis measure the variance of the data when projected onto that axis. The two vectors (arrows) in the left plot visualize this. Notice that given an $[n \times p]$ feature matrix $\mathbf{X}$ there are at most $\min(n-1, p)$ principal components. The figure on the right-hand side displays the projection of the data points projected onto the first principal axis. In this way we have reduced the dimensionality from $\mathbf{R}^2$ to $\mathbf{R}^1$. In practice, PCA is of course primarily used for datasets with $p$ large and the selected number of principal components $q$ is usually much smaller than the dimension of the original dataset ($q << p)$.

The first principal component is the direction in space along which (orthogonal) projections have the largest variance. The second principal component is the direction which maximizes variance among all directions while being orthogonal to the first. The $k^{\text{th}}$ component is the variance-maximizing direction orthogonal to the previous $k-1$ components.

How do we express this in mathematical terms? Let $\mathbf{X}$ be an $n \times p$ dataset and let it be centered (i.e. each column mean is zero; notice that standardization is very important in PCA). The $p \times p$ variance-covariance matrix $\mathbf{C}$ is then equal to $\mathbf{C} = \frac{1}{n} \mathbf{X}^T \mathbf{X}$. Additionally, let $\mathbf{\phi}$ be a unit $p$-dimensional vector, i.e. $\phi \in \mathbb{R}^p$ and let $\sum_{i=1}^p \phi_{i1}^2 = \mathbf{\phi}^T \mathbf{\phi} = 1$.

The projections of the individual data points onto the principal axis are given by the linear combination of the form

\begin{equation}
Z_{i} = \phi_{1i} X_{1} + \phi_{2i} X_{2} + \ldots + \phi_{pi} X_{p}.
\end{equation}

In matrix notation we write

\begin{equation}
\mathbf{Z} = \mathbf{X \phi}
\end{equation}

Since each column vector $X_i$ is standardized, i.e. $\frac{1}{n} \sum_{i=1}^n x_{ip} = 0$, the average of $Z_i$ (the column vector for feature $i$) will be zero as well. With that, the variance of $\mathbf{Z}$ is

\begin{align}
\text{Var}(\mathbf{Z}) &=  \frac{1}{n} (\mathbf{X \phi})^T (\mathbf{X \phi}) \\
  &= \frac{1}{n} \mathbf{\phi}^T \mathbf{X}^T \mathbf{X \phi} \\
  &= \mathbf{\phi}^T \frac{\mathbf{X}^T \mathbf{X}}{n} \mathbf{\phi} \\
  &= \mathbf{\phi}^T \mathbf{C} \mathbf{\phi}
\end{align}

Note that it is common standard to use the population estimation of variance (division by $n$) instead of the sample variance (division by $n-1$).

Now, PCA seeks to solve a sequence of optimization problems:

\begin{equation}
\begin{aligned}
& \underset{\mathbf{\phi}}{\text{maximize}} & & \text{Var}(\mathbf{Z})\\
& \text{subject to} & & \mathbf{\phi}^T \mathbf{\phi}=1, \quad \phi \in \mathbb{R}^p \\
&&& \mathbf{Z}^T \mathbf{Z} = \mathbf{ZZ}^T = \mathbf{I}.
\end{aligned}
\end{equation}

Looking at the above term it should be clear why we haver restricted vector $\mathbf{\phi}$ to be a unit vector. If not, we could simply increase $\mathbf{\phi}$ - which is not what we want. This problem can be solved with Lagrange and via an eigen decomposition (a standard technique in linear algebra). The details of which are explained in the appendix of the script.

How we apply PCA within a pipeline workflow we have shown above. A more general setup is shown in below code snippet. We again make use of the polish bankruptcy set introduced above.

```python
from sklearn.decomposition import PCA

# Define no. of PC
q = 10

# Create PCA object and fit to find
# first q principal components
pca = PCA(n_components=q)
pca.fit(X_train_bal)
pca
```

    PCA(copy=True, iterated_power='auto', n_components=10, random_state=None,
      svd_solver='auto', tol=0.0, whiten=False)

To close, one last code snippet is provided. Running it will visualize the cumulative explained variance ratio as a function of the number of components. (Mathematically, the explained variance ratio is the ratio of the eigenvalue of principal component $i$ to the sum of the eigenvalues, $\frac{\lambda_i}{\sum_{i}^p \lambda_i}$. See the appendix in the script to better understand the meaning of eigenvalues in this context.) In practice, this might be helpful in deciding on the number of principal components $q$ to use.

```python
# Run PCA for all possible PCs
pca = PCA().fit(X_train_bal)

# Define max no. of PC
q = X_train_bal.shape[1]

# Get cumsum of the PC 1-q
expl_var = pca.explained_variance_ratio_
sum_expl_var = np.cumsum(expl_var)[:q]
```

```python
# Plot Feature Importance (both cumul., individual)
plt.figure(figsize=(12, 6))
plt.bar(range(1, q + 1), expl_var, align='center')
plt.xticks(range(1, q + 1, 5))
plt.xlim([0, q + 1])
plt.xlabel('Principal Components')
plt.ylabel('Explained Variance Ratio')
plt.step(range(1, 1 + q), sum_expl_var, where='mid')
plt.tight_layout();
```

This shows us that the first 5 principal components explain basically all variation in the data. Therefore we could focus to work with only these.

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Burges, Christopher J.C., 1998, A tutorial on support vector machines for pattern recognition, Data mining and knowledge discovery 2.2, 121-167.
* Friedman, Jerome, Trevor Hastie, and Robert Tibshirani, 2001, *The Elements of Statistical Learning* (Springer, New York, NY).
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Raschka, Sebastian, 2015, *Python Machine Learning* (Packt Publishing Ltd., Birmingham, UK).
* Shalizi, Cosma Rohilla, 2017, Advanced Data Analysis from an Elementary Point of View from website, http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ADAfaEPoV.pdf, 08/24/17.
* VanderPlas, Jake, 2016, *Python Data Science Handbook* (O'Reilly Media, Sebastopol, CA).
* Vapnik, Vladimir N., 2013, *The Nature of Statistical Learning* (Springer, New York, NY).


# Final project
The Allen–Cahn equation (after John W. Cahn and Sam Allen) is a reaction–diffusion equation of mathematical physics which describes the process of phase separation in multi-component alloy systems, including order-disorder transitions.

The equation describes the time evolution of a scalar-valued state variable $\eta$  on a domain $\Omega=[0,1]$  during a time interval $[0,T]$, and is given (in one dimension) by:

$$
\frac{\partial \eta}{\partial t} - \varepsilon^2 \eta'' + f'(\eta) = 0, \qquad \eta'(0, t) = \eta'(1, t) = 0,\qquad\eta(x,0) = \eta_0(x)
$$

where $f$ is a double-well potential, $\eta_0$ is the initial condition, and $\varepsilon$ is the characteristic width of the phase transition.

This equation is the L2 gradient flow of the Ginzburg–Landau free energy functional, and it is closely related to the Cahn–Hilliard equation.

A typical example of double well potential is given by the following function

$$
f(\eta) = \eta^2(\eta-1)^2
$$

which has two minima in $0$ and $1$ (the two wells, where its value is zero), one local maximum in $0.5$, and it is always greater or equal than zero.

The two minima above behave like "attractors" for the phase $\eta$. Think of a solid-liquid phase transition (say water+ice) occupying the region $[0,1]$. When $\eta = 0$, then the material is liquid, while when $\eta = 1$ the material is solid (or viceversa).

Any other value for $\eta$ is *unstable*, and the equation will pull that region towards either $0$ or $1$.

Discretisation of this problem can be done by finite difference in time. For example, a fully explicity discretisation in time would lead to the following algorithm.

We split the interval $[0,T]$ in `n_steps` intervals, of dimension `dt = T/n_steps`. Given the solution at time `t[k] = k*dt`, it i possible to compute the next solution at time `t[k+1]` as

$$
\eta_{k+1} = \eta_{k} + \Delta t \varepsilon^2 \eta_k'' - \Delta t f'(\eta_k)
$$

Such a solution will not be stable. A possible remedy that improves the stability of the problem, is to treat the linear term $\Delta t \varepsilon^2 \eta_k''$ implicitly, and keep the term $-f'(\eta_k)$ explicit, that is:

$$
\eta_{k+1} - \Delta t \varepsilon^2 \eta_k'' = \eta_{k} - \Delta t f'(\eta_k)
$$

Grouping together the terms on the right hand side, this problem is identical to the one we solved in the python notebook number 9, with the exception of the constant $\Delta t \varepsilon^2$ in front the stiffness matrix.

In particular, given a set of basis functions $v_i$, representing $\eta = \eta^j v_j$ (sum is implied), we can solve the problem using finite elements by computing

$$
\big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big) \eta^j_{k+1} = \big((v_i, v_j) \eta^j_{k} - \Delta t (v_i, f'(\eta_k)\big)
$$
where a sum is implied over $j$ on both the left hand side and the right hand side. Let us remark that while writing this last version of the equation we moved from a forward Euler scheme to a backward Euler scheme for the second spatial derivative term: that is, we used $\eta^j_{k+1}$ instead of $\eta^j_{k}$.

This results in a linear system

$$
A x = b
$$

where

$$
A_{ij} = M_{ij}+ \Delta t \varepsilon^2 K_{ij} = \big((v_i, v_j) + \Delta t \varepsilon^2  (v_i', v_j')\big)
$$

and

$$
b_i = M_{ij} \big(\eta_k^j - \Delta t f'(\eta_k^j)\big)
$$

where we simplified the integration on the right hand side, by computing the integral of the interpolation of $f'(\eta)$.

## Step 1

Write a finite element solver, to solve one step of the problem above, given the solution at the previous time step, using the same techniques used in notebook number 9.

In particular:

1. Write a function that takes in input a vector representing $\eta$, an returns a vector containing $f'(\eta)$. Call this function `F`.

2. Write a function that takes in input a vector of support points of dimension `ndofs` and the degree `degree` of the polynomial basis, and returns a list of basis functions (piecewise polynomial objects of type `PPoly`) of dimension `ndofs`, representing the interpolatory spline basis of degree `degree`

3. Write a function that, given a piecewise polynomial object of type `PPoly` and a number `n_gauss_quadrature_points`, computes the vector of global_quadrature_points and global_quadrature_weights, that contains replicas of a Gauss quadrature formula with `n_gauss_quadrature_points` on each of the intervals defined by `unique(PPoly.x)`

4. Write a function that, given the basis and the quadrature points and weights, returns the two matrices $M$ and $K$

## Step 2

Solve the Allen-Cahan equation on the interval $[0,1]$, from time $t=0$ and time $t=1$, given a time step `dt`, a number of degrees of freedom `ndofs`, and a polynomial degree `k`.

1. Write a function that takes the initial value of $\eta_0$ as a function, eps, dt, ndofs, and degree, and returns a matrix of dimension `(int(T/dt), ndofs)`  containing all the coefficients $\eta_k^i$ representing the solution, and the set of basis functions used to compute the solution

2. Write a function that takes all the solutions `eta`, the basis functions, a stride number `s`, and a resolution `res`, and plots on a single plot the solutions $\eta_0$, $\eta_s$, $\eta_{2s}$, computed on `res` equispaced points between zero and one

## Step 3

Solve the problem for all combinations of

1. eps = [01, .001]

2. ndofs = [16, 32, 64, 128]

3. degree = [1, 2, 3]

3. dt = [.25, .125, .0625, .03125, .015625]

with $\eta_0 = \sin(2 \pi x)+1$.

Plot the final solution at $t=1$ in all cases. What do you observe? What happens when you increase ndofs and keep dt constant?

## Step 4 (Optional)

Instead of solving the problem explicitly, solve it implicitly, by using backward euler method also for the non linear term. This requires the solution of a Nonlinear problem at every step. Use scipy and numpy methods to solve the non linear iteration.

```python
%pylab inline
import sympy as sym
import scipy
from scipy.interpolate import *
from scipy.integrate import *
```

    Populating the interactive namespace from numpy and matplotlib

```python
n = 1025 # number of sample points for evaluation
m = 16 # number of support points (i.e. dimension of vector space for approximation)
x = linspace(0,1, n) # equispaced points in interval for evaluation
q = linspace(0,1, m)
degree = 1
eta_init_func = lambda x: sin( 2 * pi * x ) + 1

def f(eta):
    '''
    Input : * 1D ndarray, points in interval where to evaluate standard double well potential

    Output: * 1D ndarray, evaluation of standard double well potential at inputs
    '''
    return eta**2 * (eta-1.)**2

def ref_sol(x, f):
    return 0.5 * sign(f(x)-0.5) + 0.5
```

```python
_ = plot(x, eta_init_func(x))
```

```python
_ = plot(x,f(x))
```

```python
# Step 1.1

def F(eta):
    '''
    Input : * 1D ndarray, points in interval where to evaluate derivative of standard double well potential

    Output: * 1D ndarray, evaluation of derivative of standard double well potential at inputs
    '''
    return 2. * eta * ( 2. * eta**2 - 3. * eta +1. )
```

```python
_ = plot(x,F(x))
```

```python
# Step 1.2

def compute_basis_functions(support_points, degree):
    '''
    Input : * 1D ndarray support_points, points in interval giving support points for constructing
                                         basis of piecewise polynomial objects.
            * int degree, specifies degree of the spline fit on each sub-interval.

    Output: * list basis, list of piecewise polynomials (objects of class scipy.interpolate.PPoly)
                          with basis of space of polynomials of degree at most support_points.shape-1
                          contructed by interpolating splines of input degree.
            * list dbasis, list of piecewise polynomials (objects of class scipy.interpolate.PPoly)
                           with derivatives of basis functions.
    '''
    basis = []
    M = support_points.shape[0]
    for i in range(M):
        c = zeros(M)
        c[i] = 1
        bi = PPoly.from_spline(splrep(support_points,c,k=degree))
        basis.append(bi)
    return basis
```

```python
B = compute_basis_functions(q, degree)
```

```python
def evaluate_list_functions(x, list_functions):
    N = x.shape[0]9
    M = len(list_functions)
    E = zeros((N, M))
    for i in range(M):
        E[:,i] = list_functions[i](x)
    return E
```

```python
_ = plot(x, evaluate_list_functions(x,B))
```

```python
_ = plot(x, evaluate_list_functions(x, [b_i.derivative(1) for b_i in B]))
```

```python
# Step 1.3

def compute_global_quadrature(basis, n_gauss_quadrature_points):
    '''
    Input : * list basis, list of objects of class scipy.interpolate.PPoly defining same piecewise subdivision,
                          for instance as given by piecewise polynomial basis.
            * int n_gauss_quadrature_points, number of sample points and weights for each subinterval
                                             (d+1 if considering piecewise interpolation with degree d splines).

    Output: * global_quadrature, 1D ndarray containing sample points for exact integration on interval [0,1]
            * global_weights, 1D ndarray containing weigths for exact integration over [0,1]
    '''

    # extract interval of piecewise subdivision
    intervals = unique(basis[0].x)

    # compute quadrature points and weights and rescale for interval [0,1]
    qp, w = numpy.polynomial.legendre.leggauss(n_gauss_quadrature_points)
    qp = (qp+1)/2.
    w /= 2.

    # replicate points and weights in all the intervals of piecewise subdivision
    h = diff(intervals)
    global_quadrature = array([intervals[i]+h[i]*qp for i in range(len(h))]).reshape((-1,))
    global_weights = array([w*h[i] for i in range(len(h))]).reshape((-1,))

    return global_quadrature, global_weights
```

```python
# Step 1.4

def compute_system_matrices(basis, global_quadrature, global_weights):
    '''
    Input :

    Output:
    '''
    dbasis = [b_i.derivative(1) for b_i in basis]
    Bq = array([b_i(global_quadrature) for b_i in basis]).T
    dBq = array([db_i(global_quadrature) for db_i in dbasis]).T
    M = einsum('ki, k, kj', Bq, global_weights, Bq)
    K = einsum('ki, k, kj', dBq, global_weights, dBq)

    return M, K
```

```python
# Step 2.1

def solve_allen_cahan(eta_0_function, eps, dt, ndofs, degree):
    ## prepare matrix for results
    T = 1.
    n_times = int(T/dt)
    eta = zeros((n_times+1, ndofs))

    # calculate time independent parts of the system
    support_points = linspace(0,1, ndofs)
    basis = compute_basis_functions(support_points, degree)
    Q, W = compute_global_quadrature(basis, degree + 1)
    M, K = compute_system_matrices(basis, Q, W)
    A = M + dt * eps**2 * K

    eta[0,:] = eta_0_function(support_points)
    for i in range(n_times):
        rhs = M.dot(eta[i,:] - dt * F(eta[i,:]))
        eta[i+1,:] = linalg.solve(A, rhs)

    return eta, basis
```

```python
# Step 2.2

def plot_solution(eta, basis, stride, resolution):
    x = linspace(0,1,resolution)
    plot(x, (evaluate_list_functions(x,basis).dot(eta.T))[:,::stride])
    plot(x, ref_sol(x,eta_init_func), color = 'r')
```

```python
eta, b = solve_allen_cahan(eta_init_func, 0.001, 0.0625/4, 64, 3)
```

```python
plot_solution(eta, b, 64, 1025)
```

```python
x[eta_init_func(x)>0.5].shape
```

    (684,)

```python

```


***Welcome to Cowcooloose***,

Here we will learn how cows are cool and often loosely correlated to math.
```
                                       /;    ;\
                                   __  \____//
                                  /{_\_/   `'\____
                                  \___ ---(=)--(=)--}
       _____________________________/          :--'
   ,-,'`@@@@@@@@       @@@@@@         \_    `__\
  ;:(  @@@@@@@@@        @@@             \___(o'o)
  :: )  @@@@          @@@@@@        ,'@@(  `===='
  :: : @@@@@:          @@@@         `@@@:
  :: \  @@@@@:       @@@@@@@)    (  '@@@'
  ;; /\      /`,    @@@@@@@@@\\   :@@@@@)
  ::/  )    {_----------------:  :~`, ;
 ;;'`; :   )                  :  / `; ;
;;;; : :   ;                  :  ;  ; :
`'`' / :  :                   :  :  : :
    )_ \__;      ";"          :_ ;  \_\       `,','
    :__\  \    * `,'*         \  \  :  \   *  8`;'*  *
        `^'     \ :/           `^'  `-^-'   \v/ :  \/   -Bill Ames-
```

A cow walks 3.2 miles in 1.5 hours, hungry for the patch of grass ahead
What is the speed of the cow in "mi/h"?

Cows like imperial units btw... Weirdos...

```python
''' since dist = 3.2, and time = 1.5, and speed = distance / time '''

dist = 3.2
time = 1.5

speed = dist / time

print('The cow walks at', speed, 'mi per hour')
```

    The cow walks at 2.1333333333333333 mi per hour

----
----
Now, what if as the cow walked, she *accelerated* (she's in a rush you see).

If she sped up 1.2 mi per hour per hour, how fast is she going in 2 hrs?

```python
accel = 1.2  # mi/hr per hr
time  = 2.0

speed = speed + accel * time  # mi/hr + (mi/hr)/hr * hr

print('The cow walks at', speed, 'mi per hour after', time, 'hours')
```

    The cow walks at 4.533333333333333 mi per hour after 2.0 hours

Now, more realistically, the cow will *oscillate* in speed...

Let's use sine ```y = sin(x)``` to model how she changes speed...

**So, let's try**    ```speed = 2 * sin(time / 2) + 2```

**IMPORTANT** Change of units for time, let time be in MINUTES

*... What does that look like on a graph?*

```python
import sympy
import numpy as np
import matplotlib.pyplot as plt

time = sympy.symbols('time')
speed = 2 * sympy.sin( (time / 2)*(time / 60)  ) + 2

# lamdify() is a great function.
# It turns sympy "symbolic" functions into real ones that can be USED!
speed_numpy = sympy.lambdify( time,  # speed(time)
                              speed, # speed function (symbolic)
                  modules=['numpy']  # use numpy's functions to make it real
                )
```

```python
time_walking = 60.0 # spend 60 minutes walking
times = np.linspace(0,  time_walking, 200) # make 100 points between 0 and 60 min
speeds = speed_numpy(times) # make 100 speeds at each time in times

plt.plot(times, speeds)
plt.title("speed = 2 * sympy.sin( (time / 2)*(time / 60)  ) + 2")
plt.xlabel("Time [minutes]")
plt.ylabel("Cow Speed [mi/h]")
plt.show()

#
```

A trickier one now...

**How FAR does she walk in 60 minutes at this
alternating speed?**

*This is impossible to get perfect without calculus btw...*

----
----

**To do this, basically we do this:**

Figure out the **Integral of Speed between time = 0 and time = 60**

- *What does that mean?*

In a nutshell it means to do this:

- for a ton of tiny slices of time (like really small, seriously):
    get the speed value at the start of that time slice

- take the `speed * the width of the time slice`
    (essentially get the area of the rectangle...
    (see whiteboard explanation for details))

- add that `speed*timewidth` to a running total of areas

- at the end, the **total area gathered** will be the amount of **distance covered in that time**...

**Isn't that kind of crazy?**

----

None of that actually matters, in practice, though.

All you have to do is this:

```python
# Remember, we already defined speed and time up above when we plotted
distance = sympy.integrate(speed, (time, 0, time_walking) )

# Evaluate result into a float (decimal number)
distance_walked = distance.evalf()
```

```python
print('The cow walks a grand total of', distance_walked, 'miles in',
                                        time_walking, 'minutes')

print("That's a fast cow!")

print("The cow's distance function is:", sympy.integrate(speed, time))
```

    The cow walks a grand total of 347.273352127154 miles in 60.0 minutes
    That's a fast cow!
    The cow's distance function is: 2*time + 3*sqrt(15)*sqrt(pi)*fresnels(sqrt(15)*time/(30*sqrt(pi)))*gamma(3/4)/gamma(7/4)

```python

```


# Newton's Method

## imports
For this project it will be useful to import the numpy and matplotlib libraries.

```python
import numpy as np
from matplotlib import pyplot as plt
```

## Deduction and implementation of the algorithm

Let $f$ be a function such that: \begin{cases}
f \in C^2[a, b] \\
f(\lambda) = 0 \text{ , } \lambda \in [a, b] \text{ for some $ a, b \in Dom(f) $.}
\end{cases}

Considering the first degree Taylor polynomial centered at some $ x_0 \in [a, b] $ for  $f$, we have:
\begin{equation}
f(x) = f(x_0) + (x - x_0)f'(x_0) + \frac{(x - x_0)^2}{2}f''(\xi) \text{ , with  } \xi = \xi(x_0)
\end{equation}

If we let $x_0$ be an approximation for $\lambda$ such that $f'(x_0) \neq 0 $ then:
\begin{equation}
f(\lambda) = f(x_0) + (\lambda - x_0)f'(x_0) + \frac{(\lambda - x_0)^2}{2}f''(\xi) \\
\Rightarrow 0 = f(x_0) + (\lambda - x_0)f'(x_0) + \frac{(\lambda - x_0)^2}{2}f''(\xi) \\
\Rightarrow 0 \approx f(x_0) + (\lambda - x_0)f'(x_0) \\
\Rightarrow -(\lambda - x_0)f'(x_0) \approx f(x_0) \\
\Rightarrow (\lambda - x_0) \approx -\frac{f(x_0)}{f'(x_0)} \\
\therefore \lambda \approx x_0 - \frac{f(x_0)}{f'(x_0)}
\end{equation}

That is, $\exists x_1 \in [a, b] $ such that $ x_1 = x_0 - \frac{f(x_0)}{f'(x_0)} $ is a better approximation for $\lambda$

Appling the same method repeatedly ($n$ times), we get $\lambda \approx x_n = x_{n-1} - \frac{f(x_{n-1})}{f'(x_{n-1})}$.

Assuming the convergence of the sequence $ (x_n)_{n=0}^{\infty} $, we can stop the $n$ iterations once $(x_n - x_{n-1}) < \epsilon $ , for some chosen precision $\epsilon$.

```python
def NewtonMethod(f, f_prime, x_0, a, b, epsilon=10**-4, max_it=100, ylim=10):
    """Finds the root of the real function <f> which is assumed C^2 in some interval closed interval between <a> and <b> containing <x_0> and the root.
    Also prints a simple table of the values for each iteration and plots all the tangents generated by the method.

    Args:
        f : function which root is desired to approximate
        f_prime : derivative of <f>
        x_0 : initial approximation
        [a, b]: endpoints of the interval containing the root
        epsilon : precision
        max_it : maximum number of iterations
        ylim: Oy axis limits in the plot

    Returns:
        x_1 : numerical approximation for the root
    """

    # iterations table
    print('%7s %6s %2s %12s' % ('Iteration', 'x', '', 'f(x)'))
    print('%4d %14f %2s %10.15f' % (0, x_0, '', f(x_0)))

    # plotting the iterations
    t = np.linspace(a, b, 301) # 301 points is an arbitrarily large and odd number (this way the plot will be "smooth" and the division by zero will be considered)
    with np.errstate(divide='ignore'):  # ignores (in the graph) possible division problems
        y = f(t)
    plt.plot(t, y)
    plt.plot(t, 0*t, 'k')  # Ox axis
    plt.xlim(a, b)
    plt.ylim(-ylim, ylim)
    plt.grid()

    # algorithm
    if abs(f(x_0)) < epsilon:
        return x_0  # if the initial approximation is good enough, there's no need for further calculations

    for k in range(max_it):
        if ( f_prime(x_0) != 0 ):
            x_1 = x_0 - ( f(x_0) / f_prime(x_0) )

            tangent = ( f_prime(x_0) * (t - x_0) ) + f(x_0)
            plt.plot(t, tangent)
            print('%4d %14.7f %2s %10.15f' % (k+1, x_1, '', f(x_1)))
        else:
            print("\n f'(x_%d) = 0 , therefore the method does not converge" % k)
            plt.show()
            return None

        if abs(f(x_1)) < epsilon:
            return x_1
        x_0  = x_1

    return x_1
```

## Examples

### #1

The Newton's Method can be easily adapted to solve equations with real roots.

Let's look at the equation:
$ \frac{1}{x} = 1+ x^3 $

```python
x = np.linspace(-10, 10, 301)  # 301 points is an arbitrarily large and odd number (this way the plot will be "smooth" and the division by zero will be considered)

y_a1 = 1 + x**3

with np.errstate(divide='ignore'):  # ignoring the undefined division
    y_a2 = x**-1

plt.plot(x, y_a1, 'b', label=r'$y = 1 + x^{3}$')
plt.plot(x, y_a2, 'r', label=r'$y = \frac{1}{x}$')
plt.legend()
plt.xlim(-10, 10)
plt.ylim(-10, 10)
plt.grid()
```

Since there are two intersections in the graphs of the functions, the equation has two real roots.

For simplicity, let's take a closer look exclusively at the positive one.

```python
plt.plot(x, y_a1, 'b', label=r'$y = 1 + x^{3}$')
plt.plot(x, y_a2, 'r', label=r'$y = \frac{1}{x}$')
plt.legend()
plt.xlim(-1, 3)
plt.ylim(0, 3)
plt.grid()
```

From the new plot we can let $[a, b] = [0, 1.5]$

But we should notice that the Newton's Method is applicable to functions, therefore we should write the equation as:

$ y(x) = 1 + x^3 - \frac{1}{x} $ , for some $x$ such that $y = 0$.

```python
def a(x):
    return 1 + x**3 - x**-1

def a_prime(x):
    return 3*(x**2) + x**-2
```

```python
y_a = y_a1 - y_a2
plt.plot(x, y_a, 'c', label=r'$y(x) = 1 + x^{3} - \frac{1}{x}$')
plt.plot(x, 0*x, 'k')
plt.legend()
plt.xlim(0, 1.5)
plt.ylim(-10, 10)
plt.grid()
```

```python
root = NewtonMethod(a, a_prime, 0.7, 0, 1.5, epsilon=10**-5)
print(f"\nThe intersection between the curves (1 + x^3) and (1/x) is, approximately, at x = %.5f." % root)
```

### #2

Another application of the Newton's Method is to find the reciprocal of number.

$ \pi^{-1} $ , for example, is the zero of the function: $ f(x) = \pi - \frac{1}{x} $

```python
def b(x):
    return np.pi - x**-1

def b_prime(x):
    return x**-2
```

```python
x = np.linspace(-10, 10, 301) # 301 points is an arbitrarily large and odd number (this way the plot will be "smooth" and the division by zero will be considered)

with np.errstate(divide='ignore'):  # ignoring the undefined division
    y_b = np.pi - x**-1

plt.plot(x, y_b, label=r'$f(x) = \pi - \frac{1}{x}$')
plt.plot(x, 0*x, 'k')
plt.legend()
plt.xlim(-10, 10)
plt.ylim(-10, 10)
plt.grid()
```

```python
# zooming in
plt.plot(x, y_b, label=r'$f(x) = \pi - \frac{1}{x}$')
plt.plot(x, 0*x, 'k')
plt.legend()
plt.xlim(-1, 2)
plt.ylim(-2.5, 2.5)
plt.grid()
```

```python
# Let's try to use an initial approximation of x_0 = 0.5

NewtonMethod(b, b_prime, .5, -1, 2, epsilon=10**-7, ylim=2.5)
```

```python
# zooming in
inverse_05 = NewtonMethod(b, b_prime, .5, .15, .45, epsilon=10**-7, ylim=.5)
print(f"\n If the initial approximation is x_0 = 0.5, the reciprocal of pi is approximately equal to %.7f." % inverse_05)
```

```python
# Now, what happens if we try to use x_0 = 0.7 ?

NewtonMethod(b, b_prime, .7, -10, 10, epsilon=10**-7, max_it=15)
```

In this example we could experience the sensibility of the Method regarding the initial approximation. That is, $x_0$ must be somewhat good as an approximation for the method to return a reasonable value for the root.

In this case, even though close, $x_0 = 0.5$ and $x_0=0.7$ resulted in completely different outcomes. Using $x_0 = 0.5$ the method was successful. However, with $x_0 = 0.7$, the method diverged.


# Expanding the robot

Our initial simplification had the robot a point - well, the ropes met a point. The real robot has dimensions, and the ropes are tied to four corners of the robot. Let's assume:

- a box, with the ropes tied to the upper face, and the claw at the centroid of the lower face
- the box dimensions are $2b \times 2d \times h$
  - the factor of 2 is for convenience in the equations: instead of $x - \frac{b'}{2}$, we can use $x - b$
- the claw is at $(x, y, z)$.

Then, our original equations for $l_i$ become:

$$
\begin{align}
l_1^2 &=& (0 - (x - b))^2 + (0 - (y - d))^2 + (H - (z + h))^2 &=& (x - b)^2 + (y - d)^2 + (H - h - z)^2 \\
l_2^2 &=& (0 - (x - b))^2 + (D - (y + d))^2 + (H - (z + h))^2 &=& (x - b)^2 + (D - d - y)^2 + (H - h - z)^2 \\
l_3^2 &=& (B - (x + b))^2 + (D - (y + d))^2 + (H - (z + h))^2 &=& (B - b - x)^2 + (D - d - y)^2 + (H - h - z)^2 \\
l_4^2 &=& (B - (x + b))^2 + (0 - (y - d))^2 + (H - (z + h))^2 &=& (B - b - x)^2 + (y - d)^2 + (H - h - z)^2
\end{align}
$$

As before, we can subtract pairs of equations to eliminate two of $x, y, z$ and get the remaining one:

$$
\begin{align}
l_1^2 - l_2^2 &=& (y - d)^2 - (D - d - y)^2 \\
 &=& (y - d + D - d - y)(y - d - D + d + y)\\
 &=& (D - 2d)(D + 2y) \\
\therefore y &=& \frac{1}{2}\left(\frac{l_1^2 - l_2^2}{D - 2d} - D\right)
\end{align}
$$

Similarly, for $x$:

$$
\begin{align}
l_2^2 - l_3^2 &=& (x - b)^2 - (B - b - x)^2 \\
 &=& (x - b + B - b - x)(x - b - B + b + x)\\
 &=& (B - 2b)(B + 2x) \\
\therefore x &=& \frac{1}{2}\left(\frac{l_2^2 - l_3^2}{B - 2b} - B\right)
\end{align}
$$

While the equations for $x$ and $y$ have changed a bit, the constraint on the rope lengths remains the same:

$$
l_1^2 - l_2^2 = l_4^2 - l_3^2
$$

```python

```


# Eigenvalue and eigenvectors calculation

$$
A\mathbf{x} = \lambda \mathbf{x}
$$

### Power method (vector iteration)
- find the largest eigenvalue $\lambda_{max}$
\begin{align}
\mathbf{q}_k & = \frac{\mathbf{z}_{k-1}}{\|\mathbf{z}_{k-1}\|_2}\\
\mathbf{z}_k & = A\mathbf{q}_{k}\\
\lambda_{max}^k & = \mathbf{q}^T_k \mathbf{z}_k
\end{align}

```python
%matplotlib inline
from numpy import *
from matplotlib.pyplot import *
import numpy.linalg
import scipy.linalg
```

```python
n = 9
h = 1./(n-1)
x=linspace(0,1,n)

# construct matrix
a = -ones((n-1,))
b = 2*ones((n,))
A = (diag(a, -1) + diag(b, 0) + diag(a, +1))
A /= h**2

#print A
```

```python
z0 = ones_like(x) # starting

def PM(A, z0, tol=1e-5, nmax=500):
    q = z0/numpy.linalg.norm(z0,2)
    it = 0
    err = tol + 1.
    while (it < nmax and err > tol):
        z = dot(A,q)
        l = dot(q.T,z)
        err = numpy.linalg.norm(z-l*q,2)
        q = z/numpy.linalg.norm(z,2)

        it += 1
    print("error =", err, "iterations =", it)
    print("lambda_max =", l)
    return l,q

l,x = PM(A,z0)
l_np, x_np = numpy.linalg.eig(A)

print("numpy")
print(l_np)
```

    error = 8.456086478475517e-06 iterations = 82
    lambda_max = 249.73523408577807
    numpy
    [249.73523409 231.55417528 203.23651229 167.55417528 128.
       6.26476591  24.44582472  88.44582472  52.76348771]

### Inverse power method
- find the eigenvalue $\lambda$ **closest** to $\mu$
\begin{align}
M & = A-\mu I\\
M & = LU \\
& \\
M\mathbf{x}_k &= \mathbf q_{k-1}\\
\mathbf{q}_k & = \frac{\mathbf{x}_k}{\|\mathbf{x}_k\|_2}\\
\mathbf{z}_k & = A\mathbf{q}_{k}\\
\lambda^k & = \mathbf{q}^T_k \mathbf{z}_k
\end{align}

```python
def IPM(A, x0, mu, tol=1e-5, nmax=500):
    M = A - mu*eye(len(A))
    P,L,U = scipy.linalg.lu(M)
    err = tol + 1.
    it = 0
    q = x0/numpy.linalg.norm(x0,2)
    while (it < nmax and err > tol):
        y = scipy.linalg.solve_triangular(L, dot(P.T,q), lower=True)
        x = scipy.linalg.solve_triangular(U, y)
        q = x/numpy.linalg.norm(x,2)
        z = dot(A,q)
        l = dot(q.T,z)
        err = numpy.linalg.norm(z-l*q,2)
        it += 1
    print("error =", err, "iterations =", it)
    print("lambda =", l)
    return l,q

l,x = IPM(A,z0,6.)
```

    error = 2.6310164587523326e-06 iterations = 3
    lambda = 6.2647659142204954


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
alpha = symbols('alpha')
beta = symbols('beta')
eq = Eq(diff(f(t), t),alpha * f(t) + beta * (f(t))**2)

dsolve(eq)
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

#### Get general equation for t = 0

```python
gen = dsolve(eq).subs(t, 0)
```

#### Solution for C1 at t_0

```python
at_0 = gen.subs(t, 0)
solutions = solve(Eq(at_0, p_0), C1)

solutions[0]
```

#### Substitute 1950 for p_0

```python
solutions[0].subs(p_0, 1950)
```


# Tutorial

We will here use a computer to gain some evidence to help tackle the following
problem.

```{admonition} Problem

Consider the following polynomial:

$$
    p(n) = n ^ 2 + n + 41
$$

1. Verify that $p(n)$ is prime for $n\in \mathbb{Z}$ up until $n=20$.
2. What is the smallest value of $n$ for which $p(n)$ is no longer prime?

```

We will start by defining a function for $p(n)$:

```python
def p(n):
    """
    Return the value of n ^ 2 + n + 41 for a given value of n.
    """
    return n ** 2 + n + 41
```

We will use `sympy` to check if a number is prime.

```python
import sympy as sym

sym.isprime(3)
```

    True

```python
sym.isprime(4)
```

    False

Now to answer the first question we will use a list comprehension to create a
list of boolean variables that confirm if $p(n)$ is prime.

```{tip}
This is similar to what we did in {ref}`probability`.
```

```python
checks = [sym.isprime(p(n)) for n in range(21)]
checks
```

    [True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True,
     True]

We can use the `all` tool to check if all the boolean values are true:

```python
all(checks)
```

    True

```{attention}
Using list comprehensions is a mathematical way of repeating code but at times
it might prove useful to repeat code in a different way using a standard `for`
statement.
```

In that case we can essentially repeat the previous exercise using:

```python
checks = []
for n in range(21):
    value = p(n)
    is_prime = sym.isprime(value)
    checks.append(is_prime)
all(checks)
```

    True

The main difference between the two approaches is that we can include multiple
lines of indented code to be repeated for every value of `n` in `range(21)`.

```{attention}
A `for` loop or a list comprehension should be used when we know how many
repetitions we want to make.
```

To answer the second question we will repeat the code until the value of $p(n)$
is no longer prime.

```python
n = 0
while sym.isprime(p(n)):
    n += 1
n
```

    40

```{attention}
A `while` loop should be used when we do not know how many times a repetition
should be made **but** we know under what conditions is should be made
```

Indeed for that value of $n$ we have:

```python
p(n)
```

    1681

and

```python
sym.isprime(p(n))
```

    False

`sympy` can also factor the number for us:

```python
sym.factorint(p(n))
```

    {41: 2}

Indeed:

```python
41 ** 2
```

    1681