## Classical Mechanics - Week 7

### Last Week:
- Simulated planetary motion
- Saw the limitations of Euler's Method
- Gained experience with the Velocity Verlet Method

### This Week:
- Introduce the SymPy package
- Visualize Potential Energy surfaces
- Explore packages in Python

# Why use packages, libraries, and functions in coding?
Another great question!

**Simply put:** We could hard code every little algorithm into our program and retype them every time we need them, OR we could call upon the functions from packages and libraries to do these tedious calculations and reduce the possibility of error.

We have done this with numpy.linalg.norm() to calculate the magnitude of vectors.

We will be introducing a new package call SymPy, a very useful [symbolic mathematics](https://en.wikipedia.org/wiki/Computer_algebra) library.

```python
# Let's import packages, as usual
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
sym.init_printing(use_unicode=True)
```

Let's analyze a simple projectile motion again, but this time using SymPy.

Assume we have the following equation to express our trajectory in the $x-y$ coordinates:

$y = y_0 - (\beta -x)^2$, where $y_0$ and $\beta$ are constants.

```python
# First we must declare and define our variables. Examine the syntax in this cell then run it. Notice that ordering matters
x, y0, beta = sym.symbols('x, y_0, beta')
```

```python
# Next we will define our function
y = y0 - (beta - x)**2
y # This line outputs a visualization of our equation y(x) below
```

```python
# Now we will set our constants, but leave x alone so we can plot as a function of x
y1 = sym.simplify(y.subs({y0:10, beta:1}))
y1 # This line outputs a visualization of our equation y1(x) below
```

```python
# Run this cell. What happens as you play around with the constants?
sym.plot(y1,(x,0,5), title='Height vs Distance',xlabel='x',ylabel='y')
```

## Q1.) How would you compare plotting with sympy versus what we have done so far? Which method do you prefer?

✅ Double click this cell, erase its content, and put your answer to the above question here.

#### Using what we just learned, please set up the following equation using sympy where $U(\vec{r})$ is our potential energy:

$U(\vec{r}) = -e^{-(x^2+y^2)}$

```python
 = sym.symbols() ## Set up our variables here. What should be on the left-hand and right-hand side?
U = -sym.exp() ## What should go in the exp? Notice that using SymPy we need to use the SymPy function for exp
U
```

### We have two ways in which we can graph this:

Either perform the substitution $r^2 = x^2+y^2$ in order to plot in a 2D space ($U(r)$) or we could plot in a 3D space keeping $x$ and $y$ in our equation (U(x,y)).

## Q2.) What do you think are the benefits/draw-backs of these two methods of analyzing our equation? Which do you prefer?

✅ Double click this cell, erase its content, and put your answer to the above question here.

#### Now let's graph the potential
For now we will use sympy to perform both a 2D and 3D plot. Let's do the 2D version first using what we just learned.

```python
r = sym.symbols('r') # Creating our variables
U2D = -sym.exp() # Finish the equation using our replacement for x^2 + y^2
sym.plot(U2D,title='Potential vs Distance',xlabel='Distance (r)',ylabel='Potential (U)')
```

## Q3.) What can you learn from this 2D graph?

✅ Double click this cell, erase its content, and put your answer to the above question here.

The cell below imports a function from the sympy package that allows us to graph in 3D. Using the "plot3d" call, make a 3D plot of our originally initalized equation.

For the 3D plot, try setting the x and y plot limits as close as you can to the origin while having a clear picture of what is happening at the origin. For example x and y could range from (-4,4)

```python
# The below import will allow us to plot in 3D with sympy
# Define "U" as your potential
from sympy.plotting import plot3d
```

```python
# Once you have your potential function set up, execute this cell to obtain a graph
# Play around with the x and y scale to get a better view of the potential curve
plot3d(U,(x,-10,10),(y,-10,10))
```

## Q4.) What can you learn from this 3D graph? (Feel free to make the graph's x and y-range smaller to observe the differences)

✅ Double click this cell, erase its content, and put your answer to the above question here.

##### Let's get some more in-depth coding experience:
Try to graph this last potential energy function using SymPy or with Numpy (there is a 3d plotting example back in Week 2's notebook).

$U(r) = 3.2\cdot e^{-(0.5x^2+0.25y^2)}$

```python
## Set up and graph the potential here

```

## Q5.) How would you describe the new potential?

✅ Double click this cell, erase its content, and put your answer to the above question here.

### Try this:
Center the new potential at (1,1) instead of (0,0).  (That is, move the peak of the graph from (0,0) to (1,1).)

```python
## Plot the adjustment here
```

## Q6.) How did you move the peak of the graph?

✅ Double click this cell, erase its content, and put your answer to the above question here.

# Notebook Wrap-up.
Run the cell below and copy-paste your answers into their corresponding cells.

```python
from IPython.display import HTML
HTML(
"""

"""
)
```

# Congratulations! Another week, another Notebook.

As we can see, there are many tools we can use to model and analyze different problems in Physics on top of Numerical methods. Libraries and packages are such tools that have been developed by scientists to work on different topics, each package specific to a different application. But this is just food for thought. Although we use some basic package functions, we won't be using advanced scientific packages to do simulations and calculations in this class.

```python

```


# Métricas para avaliação de modelos

A escolha da métrica a ser otimizada depende da aplicação!

#### Métricas para regressão

MSE (mean squared error)

$
\begin{align}
\frac{1}{n} \sum_{i=1}^{n}(y_i - \hat{y}_i)^2
\end{align}
$

MAE (mean absolute error)

$
\begin{align}
\frac{1}{n} \sum_{i=1}^{n}(y_i - \hat{y}_i)
\end{align}
$

#### Métricas para classificação

Sejam:

$
\begin{align}
TP = true \ positives \\
TN = true \ negatives \\
FP = false \ positives \ (erro \ tipo \ I)\\
FN = false \ negatives \ (erro \ tipo \ II)
\end{align}
$

Precisão

$
\begin{align}
\frac{TP}{TP+FP}
\end{align}
$

Recall / sensibilidade

$
\begin{align}
\frac{TP}{TP+FN}
\end{align}
$

Acurácia

$
\begin{align}
\frac{TP+TN}{TP+TN+FP+FN}
\end{align}
$

F1 score: média harmônica de precisão e recall

$
\begin{align}
2 \cdot \frac{precision \cdot recall}{precision + recall}
\end{align}
$

## Validação cruzada

Consiste em particionar o dataset em k subconjuntos e treinar o modelo k vezes, usando um subconjunto diferente para validação por vez e os k-1 restantes para treino.

Vamos usar uma função auxiliar que recebe uma instância de um modelo, a quantidade de partições $k$, a matriz de *features* $X$, o vetor de *targets* $y$ e uma função que computa a métrica desejada.

Experimente variar $k$ e a função de métrica nos exemplos abaixo: classificação de flores e regressão de valores de casas.

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston, load_iris
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import accuracy_score, recall_score, f1_score, mean_absolute_error, mean_squared_error
from sklearn.model_selection import KFold

def train_model(model, k, X, y, score_func):
    kfolds = KFold(k)
    scores = []
    for train_idx, test_idx in kfolds.split(X, y):
        X_train, y_train = X[train_idx], y[train_idx]
        X_test, y_test = X[test_idx], y[test_idx]
        model = model.fit(X_train, y_train)
        y_hat = model.predict(X_test)
        scores.append(score_func(y_test, y_hat))
    print('scores: ', np.round(scores,4), '\nmean:    ', np.round(np.mean(scores),4))
```

#### Classificação de flores

Dataset Iris.

```python
data_iris = load_iris()
df_iris = pd.DataFrame(data_iris.data, columns=data_iris.feature_names)

X_iris = df_iris.values
y_iris = data_iris.target
cols_iris = data_iris.feature_names

df_iris.head()
```

      sepal length (cm)
      sepal width (cm)
      petal length (cm)
      petal width (cm)

      0
      5.1
      3.5
      1.4
      0.2

      1
      4.9
      3.0
      1.4
      0.2

      2
      4.7
      3.2
      1.3
      0.2

      3
      4.6
      3.1
      1.5
      0.2

      4
      5.0
      3.6
      1.4
      0.2

Podemos visualizar as *features* duas a duas, como no *snippet* abaixo. Experimente variar as *features* (i.e. `X_iris[:,0]` e `X_iris[:,1]`). Não se esqueça de também trocar as labels do gráfico.

```python
plt.scatter(X_iris[:,0], X_iris[:,1], c=y_iris, alpha=0.7)
plt.xlabel(cols_iris[0])
plt.ylabel(cols_iris[1])
```

```python
lr_iris = LogisticRegression(solver='lbfgs', max_iter=200, multi_class='multinomial')
train_model(lr_iris, 5, X_iris, y_iris, accuracy_score)
```

    scores:  [1.     1.     0.8667 0.9333 0.8333]
    mean:     0.9267

#### Regressão de valores de casas

Dataset *Boston Housing*.

```python
data_boston = load_boston()
df_boston = pd.DataFrame(data_boston.data, columns=data_boston.feature_names)

X_boston = df_boston.values
y_boston = data_boston.target
cols_boston = data_boston.feature_names

df_boston.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

```python
plt.scatter(X_boston[:,0], X_boston[:,1], alpha=0.7)
plt.xlabel(cols_boston[0])
plt.ylabel(cols_boston[1])
```

```python
lr_boston = LinearRegression()
train_model(lr_boston, 10, X_boston, y_boston, mean_absolute_error)
```

    scores:  [2.2069 2.8968 2.7867 4.5985 4.1099 3.5647 2.6697 9.6564 5.0227 2.5373]
    mean:     4.0049


# First Order Initial Value Problem

The more general form of a first order Ordinary Differential Equation is:
\begin{equation}
\label{general ODE}
y^{'}=f(t,y).
\end{equation}
This can be solved analytically by integrating both sides but this is not straight forward for most problems.
Numerical methods can be used to approximate the solution at discrete points.

## Euler method

The simplest one step numerical method is the Euler Method named after the most prolific of mathematicians [Leonhard Euler](https://en.wikipedia.org/wiki/Leonhard_Euler) (15 April 1707 – 18 September 1783) .

The general Euler formula to the first order equation
$$ y^{'} = f(t,y) $$
approximates the derivative at time point $t_i$
$$y^{'}(t_i) \approx \frac{w_{i+1}-w_i}{t_{i+1}-t_{i}} $$
where $w_i$ is the approximate solution of $y$ at time $t_i$.
This substitution changes the differential equation  into a __difference__ equation of the form
$$
\frac{w_{i+1}-w_i}{t_{i+1}-t_{i}}=f(t_i,w_i) $$
Assuming uniform stepsize $t_{i+1}-t_{i}$ is replaced by $h$, re-arranging the equation gives
$$ w_{i+1}=w_i+hf(t_i,w_i),$$
 This can be read as the future $w_{i+1}$ can be approximated by the present $w_i$ and the addition of the input to the system $f(t,y)$ times the time step.

```python
## Library
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

## Population growth

The general form of the population growth differential equation is:
$$ y^{'}=\epsilon y $$
where $\epsilon$ is the growth rate. The initial population at time $a$ is
$$ y(a)=A $$
$$ a\leq t \leq b. $$
Integrating gives  the general analytic (exact) solution:
$$ y=Ae^{\epsilon x}. $$
We will use this equation to illustrate the application of the Euler method.

## Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
$$ h=\frac{b-a}{N}.$$
Here the interval is $0\leq t \leq 2$
$$ h=\frac{2-0}{20}=0.1.$$
This gives the 21 discrete points:
$$ t_0=0, \ t_1=0.1, \ ... t_{20}=2. $$
This is generalised to
$$ t_i=0+i0.1, \ \ \ i=0,1,...,20.$$
The plot below shows the discrete time steps.

```python
### Setting up time
t_end=2.0
t_start=0
N=20
h=(t_end-t_start)/(N)
time=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(time,0*time,'o:',color='red')
plt.xlim((0,2))
plt.title('Illustration of discrete time points for h=%s'%(h))
```

## Initial Condition
To get a specify solution to a first order initial value problem, an __initial condition__ is required.

For our population problem the intial condition is:
$$y(0)=10$$.
This gives the analytic solution
$$y=10e^{\epsilon t}$$.
### Growth rate
Let the growth rate $$\epsilon=0.5$$ giving the analytic solution.
$$y=10e^{0.5 t}$$.
The plot below shows the exact solution on the discrete time steps.

```python
## Analytic Solution y
y=10*np.exp(0.5*time)

fig = plt.figure(figsize=(10,4))
plt.plot(time,y,'o:',color='black')
plt.xlim((0,2))
plt.xlabel('time')
plt.ylabel('y')
plt.title('Analytic (Exact) solution')
```

## Numerical approximation of Population growth
The differential equation is transformed using the Euler method into a difference equation of the form
      $$ w_{i+1}=w_{i}+h \epsilon w_i. $$
This approximates a series of of values $w_0, \ w_1, \ ..., w_{N}$.
For the specific example of the population equation the difference equation is
      $$ w_{i+1}=w_{i}+h 0.5 w_i. $$
where $w_0=10$. From this initial condition the series is approximated.
The plot below shows the exact solution $y$ in black circles and Euler approximation $w$ in blue squares.

```python
w=np.zeros(N+1)
w[0]=10
for i in range (0,N):
    w[i+1]=w[i]+h*(0.5)*w[i]

fig = plt.figure(figsize=(10,4))
plt.plot(time,y,'o:',color='black',label='exact')
plt.plot(time,w,'s:',color='blue',label='Euler')
plt.xlim((0,2))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Analytic and Euler solution')
```

## Error
With a numerical solution there are two types of error:
* local truncation error at one time step;
* global error which is the propagation of local error.

### Derivation of  Euler Local truncation error
The left hand side of a initial value problem $\frac{dy}{dt}$ is approximated by __Taylors theorem__ expand about a point $t_0$ giving:
\begin{equation}y(t_1) = y(t_0)+(t_1-t_0)y^{'}(t_0) + \frac{(t_1-t_0)^2}{2!}y^{''}(\xi), \ \ \ \ \ \ \xi \in [t_0,t_1]. \end{equation}
Rearranging and letting $h=t_1-t_0$ the equation becomes
$$y^{'}(t_0)=\frac{y(t_1)-y(t_0)}{h}-\frac{h}{2}y^{''}(\xi). $$
From this the local truncation error is
$$\tau y^{'}(t_0)\leq \frac{h}{2}M  $$
where $y^{''}(t) \leq M $.
#### Derivation of  Euler Local truncation error for the Population Growth
In most cases $y$ is unknown but in our example problem there is an exact solution which can be used to estimate the local truncation
$$y'(t)=5e^{0.5 t}$$
$$y''(t)=2.5e^{0.5 t}$$
From this a maximum upper limit can be calculated for $y^{''} $ on the interval $[t_0,t_1]=[0,0.1]$
$$y''(0.1)=2.5e^{0.1\times 0.5}=2.63=M$$
$$\tau=\frac{h}{2}2.63=0.1315 $$
The plot below shows the exact local truncation error $|y-w|$ (red triangle) and the upper limit of the Truncation error (black v) for the first two time points $t_0$ and $t_1$.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time[0:2],np.abs(w[0:2]-y[0:2]),'^:'
         ,color='red',label='Error |y-w|')
plt.plot(time[0:2],0.1*2.63/2*np.ones(2),'v:'
         ,color='black',label='Upper Local Truncation')
plt.xlim((0,.15))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Local Truncation Error')
```

## Global Error
The error does not stay constant accross the time this is illustrated in the figure below for the population growth equation. The actual error (red triangles) increases over time while the local truncation error (black v) remains constant.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time,np.abs(w-y),'^:'
         ,color='red',label='Error |y-w|')
plt.plot(time,0.1*2.63/2*np.ones(N+1),'v:'
         ,color='black',label='Upper Local Truncation')
plt.xlim((0,2))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Why Local Truncation does not extend to global')
```

## Theorems
To theorem below proves an upper limit of the global truncation error.
### Euler Global Error
__Theorem Global Error__

Suppose $f$ is continuous and satisfies a Lipschitz Condition with constant
L on $D=\{(t,y)|a\leq t \leq b, -\infty < y < \infty \}$ and that a constant M
exists with the property that
$$ |y^{''}(t)|\leq M. $$
Let $y(t)$ denote the unique solution of the Initial Value Problem
$$ y^{'}=f(t,y) \ \ \ a\leq t \leq b \ \ \ y(a)=\alpha $$
and $w_0,w_1,...,w_N$ be the approx generated by the Euler method for some
positive integer N.  Then for $i=0,1,...,N$
$$ |y(t_i)-w_i| \leq \frac{Mh}{2L}|e^{L(t_i-a)}-1|. $$

### Theorems about Ordinary Differential Equations
__Definition__

A function $f(t,y)$ is said to satisfy a __Lipschitz Condition__ in the variable $y$ on
the set $D \subset R^2$ if a constant $L>0$ exist with the property that
$$ |f(t,y_1)-f(t,y_2)| < L|y_1-y_2| $$
whenever $(t,y_1),(t,y_2) \in D$.  The constant L is call the Lipschitz Condition
of $f$.

__Theorem__
Suppose $f(t,y)$ is defined on a convex set $D \subset R^2$. If a constant
$L>0$ exists with
$$ \left|\frac{\partial f(t,y)}{\partial y}\right|\leq L $$
then $f$ satisfies a Lipschitz Condition an $D$ in the variable $y$ with
Lipschitz constant L.

### Global truncation error for the population equation
For the population equation specific values $L$ and $M$ can be calculated.

In this case $f(t,y)=\epsilon y$ is continuous and satisfies a Lipschitz Condition with constant
$$ \left|\frac{\partial f(t,y)}{\partial y}\right|\leq L $$
$$ \left|\frac{\partial \epsilon y}{\partial y}\right|\leq \epsilon=0.5=L $$

on $D=\{(t,y)|0\leq t \leq 2, 10 < y < 30 \}$ and that a constant $M$
exists with the property that
$$ |y^{''}(t)|\leq M. $$
$$ |y^{''}(t)|=2.5e^{0.5\times 2} \leq 2.5 e=6.8. $$
Let $y(t)$ denote the unique solution of the Initial Value Problem
$$ y^{'}=0.5 y \ \ \ 0\leq t \leq 10 \ \ \ y(0)=10 $$
and $w_0,w_1,...,w_N$ be the approx generated by the Euler method for some
positive integer N.  Then for $i=0,1,...,N$
$$ |y(t_i)-w_i| \leq \frac{6.8 h}{2\times 0.5}|e^{0.5(t_i-0)}-1| $$

The figure below shows the exact error $y-w$ in red triangles and the upper global error in black x's.

```python
fig = plt.figure(figsize=(10,4))
plt.plot(time,np.abs(w-y),'^:'
         ,color='red',label='Error |y-w|')
plt.plot(time,0.1*6.8*(np.exp(0.5*time)-1),'x:'
         ,color='black',label='Upper Global Truncation')
plt.xlim((0,2))
plt.xlabel('time')
plt.legend(loc='best')
plt.title('Global Truncation Error')
```

```python

```

Chapter 01 - Euler Methods/.ipynb_checkpoints/01_Euler_method_with_Theorems_nonlinear_Growth_function-checkpoint.ipynb


### 나이브베이지 가정
- 모든 차원의 개별 독립변수가 서로 조건부독립이라는 가정을 사용한다.
- 벡터 x의 결합확률분포함수는 개별 스칼라 원소$x_d$의 확률분포함수의 곱
$$
P(x_1, \ldots, x_D \mid y = k) = \prod_{d=1}^D P(x_d \mid y = k)
$$
- 베이즈정리를 사용하여 조건부확률을 계산

$$
\begin{align}
P(y = k \mid x)
&= \dfrac{ P(x_1, \ldots, x_D \mid y = k) P(y = k) }{P(x)} \\
&= \dfrac{ \left( \prod_{d=1}^D P(x_{d} \mid y = k) \right) P(y = k) }{P(x)}
\end{align}
$$

### 정규분포 나이브베이즈 모형(`GaussianNB`)
* `theta_`: 정규분포의 기댓값 $\mu$
* `sigma_`: 정규분포의 분산 $\sigma^2$

```python
import scipy as sp
np.random.seed(0)
rv0 = sp.stats.multivariate_normal([-2, -2], [[1, 0.9], [0.9, 2]])
rv1 = sp.stats.multivariate_normal([2, 2], [[1.2, -0.8], [-0.8, 2]])
x0 = rv0.rvs(40)
x1 = rv1.rvs(60)
X = np.vstack([x0,x1])
y = np.hstack([np.zeros(40), np.ones(60)])

```

```python
from sklearn.naive_bayes import GaussianNB
model_norm = GaussianNB().fit(X,y)

```

```python
model_norm.classes_
#클래스 2개
```

    array([0., 1.])

```python
model_norm.class_count_
# 데이터 각각 40개, 60개
```

    array([40., 60.])

```python
model_norm.class_prior_
#y=0일 확률(사전확률)== 0.4, y=1일 확률==0.6
```

    array([0.4, 0.6])

```python
model_norm.theta_[0], model_norm.sigma_[0]
# y-0 일때의 기대값. 분포
```

    (array([-1.96197643, -2.00597903]), array([1.02398854, 2.31390497]))

```python
model_norm.theta_[1], model_norm.sigma_[1]
# y=1 일때의 기대값. 분포
```

    (array([2.19130701, 2.12626716]), array([1.25429371, 1.93742544]))

### X_new =[0,0]이라면 y는 무엇일지 예측하기

```python
x_new = [0,0]
model_norm.predict_proba([x_new])
# y=0일 확률 0.48, y=1일확률 0.51
```

    array([[0.48475244, 0.51524756]])

연습 문제 1
붓꽃 분류문제를 가우시안 나이브베이즈 모형을 사용하여 풀어보자.

(1) 각각의 종이 선택될 사전확률을 구하라.

(2) 각각의 종에 대해 꽃받침의 길이, 꽃받침의 폭, 꽃잎의 길이, 꽃잎의 폭의 평균과 분산을 구하라.

(3) 학습용 데이터를 사용하여 분류문제를 풀고 다음을 계산하라.

분류결과표
분류보고서
ROC커브
AUC

```python
from sklearn.datasets import load_iris
from sklearn.metrics import classification_report
from sklearn.naive_bayes import GaussianNB

iris = load_iris()
X = iris.data
y = iris.target

model_norm = GaussianNB().fit(X,y)
model_norm.class_prior_ #각 클래스의 사전확률
```

    array([0.33333333, 0.33333333, 0.33333333])

```python
model_norm.theta_[0], model_norm.sigma_[0]
```


## Linear independence

```python
import numpy as np
from sympy.solvers import solve
from sympy import Symbol

x = Symbol('x')
y = Symbol('y')
z = Symbol('z')
```

The set of vectors are called linearly independent because each of the vectors in the set {V0, V1, …, Vn−1} cannot be written as a combination of the others in the set.

### Linear Independent Arrays

```python
A = np.array([1,1,1])
B = np.array([0,1,1])
C = np.array([0,0,1])
Z = np.array([0,0,0])
```

```python
np.array_equal(
    Z,
    0*A + 0*B + 0*C
)
```

    True

```python
solve(x*A + y*B + z*C)
```

    {z: 0, y: 0, x: 0}

### Linear Dependent Arrays

```python
A = np.array([1,1,1])
B = np.array([0,0,1])
C = np.array([1,1,0])
```

```python
1*A + -1*B + -1*C
```

    array([0, 0, 0])

```python
solve(x*A + y*B + z*C)
```

    {y: z, x: -z}

```python
A = np.array([1,2,3])
B = np.array([1,-4,-4])
C = np.array([3,0,2])
```

```python
2*A + 1*B + -C
```

    array([0, 0, 0])

```python
solve(x*A + y*B + z*C)
```

    {y: -z, x: -2*z}


# Variational Inference
For many models of practical interest, evaluating the posterior distribution will be intractable either because of the high dimensionality of the latent space or a complex form of the posterior distribution. For such cases, approximation schemes are handy in deriving a good lower or upper bound on the posterior distribution. The approximation schemes fall into 2 categories - deterministic and stochastic. Stochastic methods involve sampling while deterministic methods are based on analytical approximations to the posterior. In this section, we review variational approximations to the posterior.

Suppose we have a fully Bayesian model with a set of $N$ observed variables $X = \{x_1,x_2,\ldots,x_N \}$ and latent variables $Z = \{z_1,z_2,\ldots,z_N \}$. Assume our model specifies the joint distribution $p(X,Z)$ and our goal is to find an approximation for the conditional $p(Z|X)$ which enables us to make predictions on $X$. We can express the log marginal likelihood of the data as: $ \log {p(X)} = \mathcal{L}(q) + KL(q||p) $ where

$$ \begin{align} \mathcal{L}(q) &= \int {q(Z) \log{\frac{p(X,Z)}{q(Z)}}dZ} \\
KL(q||p) &= \int{q(Z) \log{\frac{p(Z|X)}{q(Z)}}dZ}\end{align}$$

The variational inference proceeds by maximizing the lower bound $\mathcal{L}(q)$ with respect to the distribution $q(Z)$ which is equivalent to minimizing the KL divergence. The KL divergence is minimum (equals 0) when $q(Z) = p(Z|X)$. When the posterior $p(Z|X)$ is intractable, we use a restricted family of distributions for $q(Z)$ and then seek the member of the family that minimizes the KL divergence.

## Factorized Distributions
We restrict the family of distributions that factorize the joint distribution over the latent variables i.e. $q(Z) = \prod_{i=1}^M q_i(Z_i)$
where $M$ denotes the number of disjoint groups of latent variables.

$$ \begin{align} \mathcal{L}(q) &= \int{q(Z) \log{\frac{p(X,Z)}{q(Z)}}dZ} \\
&= \int{\prod_{i=1}^M q_i(Z) \Big( \log{p(X,Z)} - \log{q(Z)} \Big) dZ } \\
&= \int{q_j \Bigg(\int{\log{p(X,Z)} \prod_{i\ne j}^M q_i dZ_i}\Bigg) dZ_j} - \int{q_j \log{q_j} dZ_j } + \text{const.} \\
&= \int{q_j \log \tilde{p}(X,Z_j) dZ_j} - \int{q_j \log{q_j} dZ_j } + \text{const.} \end{align}$$
where $\tilde{p}(X,Z_j) = \mathbb{E}_{i\ne j}\left[ \log p(X,Z)\right] + \text{const.}$ representing the expection with respect to the q distributions over all variables $z_i$ for $i \ne j$.

Thus the lower bound $\mathcal{L}(q_j)$ can be maximized by minimizing the KL divergence between $q_j(Z_j)$ and $\tilde{p}(X,Z_j)$. Thus we obtain $\log{q_j^*(Z_j)} = \mathbb{E}_{i\ne j}\left[ \log p(X,Z)\right] + \text{const.}$

We determine the optimal solution over all the $z_j$ by initializing all the factors $q_j(Z_j)$ and then cycling through the factors replacing each in turn with a revised estimate given the right hand side of the above equation. We can ignore the constant term as it serves to normalize the distribution. Convergence is guaranteed because the bound is convex with respect to each of the factors $q_j(Z_j)$.

## Example

This is an example to demonstrate the variational approximation  and adapted from Figure 4.14 in [1]. This is also the same example used to demonstrate the [Laplace approximation](https://chandrusuresh.github.io/MyNotes/files/DensityEstimation/LaplaceApproximation.html).

Suppose $p(z) \propto \sigma(20z+4) \exp{\left(\frac{-z^2}{2}\right)}$ where $\sigma(\cdot)$ is the sigmoid function. This form is very common in classification problems and serves as a good practical example.

This is a distribution on a scalar random variable $z$ and we seek to approximate it with a Gaussian. The parameters for Gaussian distributions will be determined by variational approximation. To initialize, we introduce conjugate prior distributions for the mean $\mu$ and precision $\tau$ given by:

$$ \begin{align} p(\tau) &= \text{Gam}(\tau|a_0,b_0) \\
p(\mu|\tau)  &= \mathcal{N}(\mu|\mu_0,(\lambda_0 \tau)^{-1})\end{align}$$

We assume the posterior $q(\mu,\tau)$ factorizes as: $q(\mu,\tau) = q_{\mu}(\mu) \cdot q_{\tau}(\tau)$.

$$ \begin{align} \mathcal{L}(q) &= \int{q(Z) \ln\left\{\frac{p(X,Z)}{q(Z)}\right\}dZ} \\
&= \int{q_{\mu} q_{\tau} \ln\left\{\frac{p(X|\mu,\tau) \cdot p(\mu|\tau) \cdot p(\tau)}{q_{\mu} q_{\tau}}\right\}dZ}  \end{align}$$

### Optimal distribution for $\mu$
$$\begin{align}\Rightarrow \mathcal{L}(q_{\mu}) &= \int{q_{\mu} \int{q_{\tau} \log{\left[p(X|\mu,\tau) \cdot p(\mu|\tau) \cdot p(\tau)\right]}d\tau } d\mu} - \int q_{\mu} \log{q_{\mu}} d\mu - \int q_{\tau} \log{q_{\tau}} d\tau\\
&= \int{q_{\mu} \int{q_{\tau} \log{\left[p(X|\mu,\tau) \cdot p(\mu|\tau) \right] }d\tau} d\mu} - \int q_{\mu} \log{q_{\mu}} d\mu - \int q_{\tau} \log{q_{\tau}} d\tau + \int{q_{\mu} q_{\tau} \log{p(\tau)}}d\tau d\mu\\
&= \int{q_{\mu} \int{q_{\tau} \log{\left[p(X|\mu,\tau) \cdot p(\mu|\tau) \right] }d\tau} d\mu} - \int q_{\mu} \log{q_{\mu}} d\mu + \text{const.}\end{align}$$

The lower bound can be maximized by maximizing the right hand side. The right hand side is also the negative KL divergence $ KL(q_{\mu}||\mathbb{E}_{\tau} \left[p(X|\mu,\tau) \cdot p(\mu|\tau)\right])$

The factorized distribution assumption above implies that $p(\mu|\tau) = p(\mu)$, we can further simplify the above expression as follows:

$$ \log q^*_{\mu} = \mathbb{E}_{\tau} \log \left[p(X|\mu,\tau) \cdot q(\mu) \right]) + \text{const.}  $$
The optimum factors can be obtained as follows:

$$ \begin{align} \log{q_{\mu}^*(\mu)} &= \mathbb{E}_{\tau} \left[-\frac{\tau}{2} \sum_{n=1}^N (x_n-\mu)^2 - \frac{\lambda_0 \tau}{2} (\mu-\mu_0)^2  \right] + \text{const.} \\
&= -\frac{\mathbb{E}(\tau)}{2} \left[ \sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu-\mu_0)^2\right] + \text{const.} \end{align}$$

Notice that the expression is quadratic and therefore parameters of a Gaussian distribution can be determined by completing the square.

$$ \begin{align} \mu_N &= \frac{\lambda_0 \mu_0 + N\bar{x}}{N + \lambda_0}\\
\lambda_N &= (N+\lambda_0) \mathbb{E}(\tau)\end{align}$$

### Optimal distribution for $\tau$
$$\begin{align}\Rightarrow \mathcal{L}(q_{\tau}) &= \int{q_{\tau} \int{q_{\mu} \log{\left[p(X|\mu,\tau) \cdot p(\mu|\tau) \cdot p(\tau)\right]}d\mu } d\tau} - \int q_{\mu} \log{q_{\mu}} d\mu - \int q_{\tau} \log{q_{\tau}} d\tau\\
&= \int{q_{\tau} \int{q_{\mu} \log{\left[p(X|\mu,\tau) \cdot p(\mu|\tau) \cdot p(\tau) \right] }d\mu} d\tau} - \int q_{\tau} \log{q_{\tau}} d\tau - \int q_{\mu} \log{q_{\mu}} d\mu + \int{q_{\mu} q_{\tau} \log{p(\mu)}}d\tau d\mu\\
&= \int{q_{\mu} \int{q_{\tau} \log{\left[p(X|\mu,\tau) \cdot p(\mu|\tau) \cdot p(\tau) \right] }d\mu} d\tau} - \int q_{\tau} \log{q_{\tau}} d\tau + \text{const.}\end{align}$$

The lower bound can be maximized by maximizing the right hand side. The right hand side is also the negative KL divergence $ KL(q_{\tau}||\mathbb{E}_{\mu} \left[p(X|\mu,\tau) \cdot p(\mu|\tau) \cdot p(\tau)\right])$

From the factorized distribution assumption above, we can further simplify the above expression as follows:

$$ \log q^*_{\tau} = \mathbb{E}_{\mu} \log \left[p(X|\mu,\tau) \cdot q(\mu) \cdot q(\tau)\right]) + \text{const.} $$

The optimum factors can be obtained as follows:

$$ \begin{align} \log{q_{\tau}^*} &= \mathbb{E}_{\mu} \left[\frac{N}{2}\log{\tau} -\frac{\tau}{2} \sum_{n=1}^N (x_n-\mu)^2 - \frac{\lambda_0 \tau}{2} (\mu-\mu_0)^2  + (a_0-1)\log{\tau} - b \tau\right] + \text{const.} \\
&= \frac{N}{2}\log{\tau} + (a_0-1)\log{\tau} - b \tau - \frac{\tau}{2} \mathbb{E}_{\mu}\left[ \sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu-\mu_0)^2\right] + \text{const.} \end{align}$$

This implies that $q_{\tau}$ is a Gamma distribution with parameters:

$$ \begin{align} a_N &= a_0 + \frac{N}{2}\\
b_N &= b_0 + \frac{1}{2}\mathbb{E}_{\mu}\left[ \sum_{n=1}^N (x_n-\mu)^2 + \lambda_0(\mu-\mu_0)^2\right] \\
&= b_0 + \frac{\lambda_0 \mu_0^2}{2} + \frac{1}{2} \left[ \sum_{n=1}^N{x_n^2} - 2 (N\bar{x}+\lambda_0\mu_0) \mathbb{E}(\mu)+(N+\lambda_0)\mathbb{E}(\mu^2) \right]\end{align}$$

We iterate on computing optimal values of $\mu$ and $\tau$ till convergence to get the optimal parameters for the posterior distribution.

## Back to the Example

```python
import numpy as np
from scipy.integrate import trapz,cumtrapz
from scipy.stats import norm
import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline

from numpy.random import rand

def sigmoid(x):
    den = 1.0+np.exp(-x)
    return 1.0/den

def p_z(z):
    p = np.exp(-np.power(z,2)/2)*sigmoid(20*z+4)
    sum_p = trapz(p,z) ## normalize for plotting
    return p,p/sum_p

## Laplace Approximation
def findMode(z_init,max_iter = 25,tol = 1E-6):
    iter = 0
    z_next = np.finfo('d').max
    z_cur = z_init
    while (iter < max_iter and np.abs(z_next-z_cur) > tol):
        if iter > 0:
            z_cur = z_next
        y     = z_cur - 20*(1-sigmoid(20*z_cur+4))
        der_y = 1 + 400*sigmoid(20*z_cur+4)*(1-sigmoid(20*z_cur+4))
        z_next = z_cur - y/der_y
        iter = iter+1
    return z_next

def getHessian(z):
    sig_x = sigmoid(20*z+4)
    return 400*sig_x*(1-sig_x) + 1

## Variational Approximation
def getMu(X,E_tau,mu0,lambda0):
    N = float(len(X))
    mu_N = (lambda0*mu0 + np.sum(X))/(N+lambda0)
    lambda_N = (N+lambda0)*E_tau
    return mu_N,lambda_N

def getTau(X,mu_N,lambda_N,mu0,lambda0,a0,b0):
    N = float(len(X))
    a_N = a0 + N/2
    b_N = b0 + 0.5*lambda0*np.power(mu0,2) + 0.5*(np.sum(np.power(X,2)) - 2*mu_N*(N+lambda0)*np.mean(X))
    if lambda_N != 0:
        b_N = b_N + 0.5*(N+lambda0)*(1/lambda_N + np.power(mu_N,2))
    return a_N,b_N

def VariationalApproximation(X,mu0,lambda0,a0,b0,max_iter=25,tol=1E-6):
    count = 0
    E_tau = 0
    if b0 != 0:
        E_tau = a0/b0
    mu1 = mu0
    lambda1 = lambda0
    a1 = a0
    b1 = b0
    post_prob = []
    while count < max_iter:
        count = count+1
        mu_N,lambda_N = getMu(X,E_tau,mu0,lambda0)
        a_N,b_N = getTau(X,mu_N,lambda_N,mu0,lambda0,a0,b0)
        max_del = np.max(np.array([np.abs(mu_N-mu1),np.abs(lambda_N-lambda1),np.abs(a_N-a1),np.abs(b_N-b1)]))
#         print(count,max_del)
        if max_del <= tol:
            print("Converged after Iteration:"+str(count))
            break
        mu1 = mu_N
        lambda1 = lambda_N
        a1 = a_N
        b1 = b_N
        E_tau = a1/b1
    return mu_N,lambda_N,a_N,b_N

def sampleData(z,pzn,N):
    x = np.zeros((N,))
    cdf = cumtrapz(pzn,z,initial=0)
    for i in range(N):
        rnd = rand()
        dif = np.abs(cdf-rnd)
        idx = np.where(dif == np.min(dif))[0]
        x[i] = z[idx]
    return x
```

```python
z = np.linspace(-10,10,10000)
pz,pzn = p_z(z)

## Get Laplace distribution
z0 = findMode(0)
A = getHessian(z0)
z0_idx = np.where(np.abs(z-z0) == np.min(np.abs(z-z0)))[0]
p_z0 = pzn[z0_idx]

## Get approx Gaussian distribution
q_z_laplace = norm.pdf(z, z0, 1/np.sqrt(A))
```

```python
z_sample = sampleData(z,pzn,10000)
mu0=0
lambda0 = 0
a0=0
b0=0
mu_N,lambda_N,a_N,b_N = VariationalApproximation(z_sample,mu0,lambda0,a0,b0)
print('mu_opt     = ',mu_N)
print('lambda_opt = ',lambda_N)
print('a_opt      = ',a_N)
print('b_opt      = ',b_N)
q_z_va = norm.pdf(z, mu_N, np.sqrt(b_N/a_N))
```

    Converged after Iteration:6
    mu_opt     =  0.6656897689768977
    lambda_opt =  24425.335351142647
    a_opt      =  5000.0
    b_opt      =  2047.0548011395454

```python
fig,ax = plt.subplots(1,1,figsize=(5,3))
ax.cla()
ax.plot(z,pzn,color="orange")
ax.fill_between(z,pzn, 0,
                 facecolor="orange", # The fill color
                 color='orange',       # The outline color
                 alpha=0.2)          # Transparency of the fill
#ax.axvline(x=z0)#,ylim=0,ymax=0.7)
ax.vlines(z0, ymin=0, ymax=p_z0,linestyles='dotted')
ax.plot(z,q_z_laplace,'r')
ax.plot(z,q_z_va,'g')
ax.set_xlim([-2,4]);
ax.set_ylim([0,0.8]);
ax.set_yticks([0,0.2,0.4,0.6,0.8]);
ax.legend(['Original','Laplace','Variational'])
ax.set_title('Laplace & Variational approximations');
```

## References
1. Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.


# [Goldbach’s other conjecture](https://projecteuler.net/problem=46)
<blockquote>
It was proposed by Christian Goldbach that every odd composite number can be written as the sum of a prime and twice a square.

$
\begin{align}
9 & = 7 + 2 \times 1^2 \\
15 & = 7 + 2 \times 2^2 \\
21 & = 3 + 2 \times 3^2 \\
25 & = 7 + 2 \times 3^2 \\
27 & = 19 + 2 \times 2^2 \\
33 & = 31 + 2 \times 1^2
\end{align}
$

It turns out that the conjecture was false.

What is the smallest odd composite that cannot be written as the sum of a prime and twice a square?
</blockquote>

Starting with $9$, check each odd composite number $n$ to see if it is the sum of a prime and twice a square. Only check primes less than $n$ and integers not greater than $\sqrt{n / 2}$. If $n$ is not such a sum, we are done.

```julia
using Primes

f(n) = any(n == p + 2 * b^2 for p in primes(n), b in 1:isqrt(n÷2))
function g(n)
    n += 2
    while isprime(n) n += 2; end
    n
end

n = 9
while f(n)
    n = g(n)
end
n
```

    5777


# Combination - Passwords & Students

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Question A

> If passwords can consist of **6** *letters*, find the probability that a randomly chosen password will *not* have any *repeated* letters.

\begin{equation}
P = \frac{26 \times 25 \times \cdots \times 21}{26^{6}}
\end{equation}

```R
norpt <- 0
count <- 100000
for (i in c(1:count)) {
    pwd <- sample(x=letters, size=6, replace=TRUE)
    if (length(unique(pwd)) == 6) {
        norpt <- norpt + 1
    }
}

norpt / count
```

0.53777

## Question B

> How many ways can you get a sample of **6** letters *without* *repeated* letters, if the order does *not* matter?

\begin{equation}
^{26}C_6 = \frac{26!}{6! \times 20!}
\end{equation}

```R
choose(n=26, k=6)
```

230230

## Question C

> Use the `sample` command to simulate tossing a coin **10000** times. You can use a `for` loop and record the result of each toss. Then you can the use `table` command to find how often you got *heads* or *tails*.

```R
res <- sample(x=c("Head", "Tail"), size=10000, replace=TRUE)

prop.table(table(res))
```

    res
      Head   Tail
    0.5015 0.4985

## Question D

> If a class contains **60** *females* and **40** *males* and you choose a random sample of **5** students from the class, what is the probability of getting **5** *females*?

\begin{equation}
P = \frac{^{60}C_5}{^{100}C_{5}}
\end{equation}

```R
choose(n=60, k=5) / choose(n=100, k=5)
```

0.0725420627482483

Check the *Hypergeometric Distribution*.

```R
dhyper(x=c(0:5), m=60, n=40, k=5)
```

0.00873993458676817

0.072832788223068

0.232277540819514

0.354528878092943

0.259078795529458

0.0725420627482483

## Question E

> Use the `sample` command to simulate the situation in *Question D*. Repeat the sample **10000** times. How often do you get **5** *females*?

```R
students <- c(rep("Male", 40), rep("Female", 60))
noman <- 0
count <- 10000
for (i in c(1:count)) {
    group <- sample(x=students, size=5)
    if (group[1] == "Female" && length(unique(group)) == 1) {
        noman <- noman + 1
    }
}

noman / count
```

0.0771

exercises/Combination - Passwords & Students.ipynb


# 2021-11-03 Software discussion

## Last time

* Running multigrid with PETSc
* Projects
* GitHub codesearch, JOSS search
* Decide what's next: **Finite Element** or Finite Volume

## Today

* Project discussion
* Weak forms

# Integration by parts

## One dimension

> it's the product rule backwards

\begin{align} \int_a^b d(uv) &= \int_a^b u dv + \int_a^b v du \\
(uv)_a^b &= \int_a^b u dv + \int_a^b v du \\
\int_a^b u dv &= (uv)_a^b - \int_a^b v du
\end{align}

> you can move the derivative to the other term; it'll cost you a minus sign and a boundary term

## Multiple dimensions

\begin{align}
\int_\Omega v \nabla\cdot \mathbf f = -\int_\Omega \nabla v \cdot \mathbf f + \int_{\partial \Omega} v \mathbf f \cdot \mathbf n
\end{align}

## Strong form

$$ -\nabla\cdot(\kappa \nabla u) = 0 $$

## Weak form
* multiply by a test function and integrate by parts

\begin{align} -\int_\Omega v \nabla\cdot(\kappa \nabla u) = 0, \forall v \\
\int_\Omega \nabla v \cdot \kappa \nabla u - \int_{\partial\Omega} v \underbrace{\kappa \nabla u \cdot \mathbf n}_{\text{boundary condition}} = 0
\end{align}


```python
import sympy
sympy.init_printing(use_latex=True)
```

If you ever don't feel like taking derivatives, you can use a Python library called `sympy` to do the dirty work.

When we have a $g$ function like this:

$$
g = \begin{bmatrix}
u_{\phi} \\
\dot{y} - \sin(\phi) \Delta t \\
y + \dot{y} \Delta t
\end{bmatrix}
$$

and a state vector like this:

$$
x = \begin{bmatrix}
\phi \\
\dot{y} \\
y
\end{bmatrix}
$$

(Note that I'm writing $\phi$ here instead of $x_{\phi}$. Like wise with $\dot{y}$ and $y$)

we can use sympy to calculate $g'$ as follows:

```python
# 1. define sympy symbols
u_phi, phi, y_dot, y, dt = sympy.symbols(
'u_phi, phi, y_dot, y, dt')

# 2. define the state variable
x = sympy.Matrix([
    phi,
    y_dot,
    y])

# 3. define state transition function
g = sympy.Matrix([
    u_phi,
    y_dot - sympy.sin(phi) * dt,
    y + y_dot * dt
])

# 4. take jacobian of g with respect to x
g.jacobian(x)
```

$$\left[\begin{matrix}0 & 0 & 0\\- dt \cos{\left (\phi \right )} & 1 & 0\\0 & dt & 1\end{matrix}\right]$$

You'd still need to implement this matrix in code, but at least the derivatives have been taken care of!


### Sum of Exponentially Distributed Random Variables

Let $\{x_i\}$ be a collection of $n$ independent and identically distributed random variables with exponential distribution:

\begin{equation}
X_i \sim \lambda e^{-\lambda x}\Theta(x)
\end{equation}

Where $\Theta(\cdot)$ is the Heaviside unit step function. Define a new random variable, $S$, where $S = \sum^n_i X_i$.

This can be done by writing down the joint probability distribution $p(S,X_1,...,X_n)$ using the Dirac Delta function and then obtain $p(S)$ through marginalization. The first of the $n$ integrals is easy because of the Delta function:

\begin{equation}
\begin{array}{ll}
p(S) &= \int dX^n \ \underbrace{\prod_i^n \left[ \lambda e^{-\lambda X_i}\Theta(X_i) \right]}_{p(X_1,...,X_n)}\ \underbrace{\delta(S-\sum^n_i X_i)}_{P(S|X_1,...,X_n)}\\
&= \lambda^n \int dX^{n-1} e^{-\lambda (\sum_i^{n-1}X_i)} e^{-\lambda(s-\sum_i^{n-1}X_i)}\left[\prod_i^{n-1}\Theta(X_i)\right]\Theta(s-\sum_i^{n-1}X_i)
\end{array}
\end{equation}

Note that the sums in the exponential will cancel in a way that allows it to be removed from the integral. What remains to solve is an integral over a product of Heaviside functions.

\begin{equation}
\begin{array}{ll}
p(S) &= \lambda^n e^{-\lambda s} \int \mathrm{d}X^{n-1} \left[\prod_i^{n-1}\Theta(X_i)\right]\Theta(s-\sum_i^{n-1}X_i)
\end{array}
\end{equation}

The integral will be a product of the lengths of $n-1$ coordinate intervals where the Heaviside functions have support.

\begin{equation}
\begin{array}{ll}
&\ \int \mathrm{d}X^{n-1} \underbrace{\left[\prod_i^{n-1}\Theta(X_i)\right]\Theta(s-\sum_i^{n-1}X_i)}_{I_{n-1}}\\
&= \int\mathrm{d}X^{n-2} \left[\left[\prod_i^{n-2}\Theta(X_i)\right]X_{n-1}\right]_{(0,\infty)\cap(-\infty,s-\sum_i^{n-2}X_i)}\\
&= \int\mathrm{d}X^{n-2} \underbrace{\left[\prod_i^{n-2}\Theta(X_i)\right]\Theta(s-\sum_i^{n-2}X_i)\left(s-\sum_i^{n-2}X_i\right)}_{I_{n-2}}\\
&= \int\mathrm{d}X^{n-3} \underbrace{\left[\prod_i^{n-3}\Theta(X_i)\right]\Theta(s-\sum_i^{n-3}X_i)\frac{1}{2}\left(s-\sum_i^{n-3}X_i\right)^2}_{I_{n-3}}\\
&\vdots\\
&= \int\mathrm{d}X^{n-m} \underbrace{\left[\prod_i^{n-m}\Theta(X_i)\right]\Theta(s-\sum_i^{n-m}X_i)\frac{1}{(m-1)!}\left(s-\sum_i^{n-m}X_i\right)^{m-1}}_{I_{n-m}}\\
&\vdots\\
&=\frac{1}{(n-1)!}s^{n-1}\Theta(s)
\end{array}
\end{equation}

Where, in the third row, the length of the interval $(0,\infty)\cap(-\infty,s-\sum_i^{n-2}X_i)$ is expressed as $\Theta(s-\sum_i^{n-2}X_i)\left(s-\sum_i^{n-2}X_i\right)$.

The expression for the probability of seeing $n$ independent and identically distributed, exponentially distribued random variables with parameter $\lambda$, add up to a sum $S$, is then:

\begin{equation}
p(S) = \frac{s^{n-1}\lambda^n}{(n-1)!} e^{-\lambda s}\Theta(s)
\end{equation}

For $n=1$, this reduces to the expression of a single exponentially distributed random variable.

```python
import numpy as np
import matplotlib.pyplot as plt

def factorial(x):
    if x == 0:
        return 1
    else:
        res = 1
        for i in range(1,x+1):
            res *= i
        return res

def p_S(s,n):
    """
    Analytical solution for the sum of n exponentially distributed random variables
    """
    if s < 0:
        return 0
    else:
        return (s**(n-1)*lam**n*np.exp(-lam*s))/factorial(n-1)

lam = 1
ss = np.linspace(0,50,100)
nn = [1,5,10,20]

"""
Simulated data
"""
N = 1000
data = np.random.exponential(scale=lam,size=(max(nn),N))

"""
Plotting
"""
plt.figure(figsize=(12,5))
for n in nn:
    x = [p_S(s,n) for s in ss]
    plt.plot(ss,x,linewidth=3)
    res = np.sum(data[:n],axis=0)
    plt.hist(res,density=True,bins=ss)
    plt.text(1.2*n,max(x),'n='+str(n),Fontsize=12)

_ = plt.title('Simulated vs. Analytical')
```

Probability - Sum of Exponential Random Variables.ipynb


#Introduction
The goal of this notebook is to solve two examples of linear regression. The first with only one feature and the second with more than one feature, a simple example of a multidimensional case with two features. The notebook follows exercises proposed in the Standford Coursera course by Andrew Ng, but I comment, with my explanations, on many points that I think are relevant for the understanding of the concepts as well as the Python implementation.

The cost function for this model, namely the linear regression (a special case of the more general class of Generalized linear models) will be given by the mean squared error function
$$J(\theta) = \frac{1}{2} \sum_{i=1}^m \left( h^{(i)} - y^{(i)} \right)^2 = \frac{1}{2} \sum_{i=1}^m J(\theta;x^{(i)}) ,$$
where $m$ is the total number of training examples and $h$ is the hypothesis function considered to be the linear combination $ h= h(\theta; x^{(i)}) = \theta^t x^{(i)} $ with $\theta$ the column vector representing the weights and bias of the model and $x^{(i)}$ the i-th column vector with the features of the model. In particular, for a total of $n$-features, $x^{(i)} \in \mathbb{R}^{n+1}$, because we include an extra element $x_0=1$ that is associated with the bias $\theta_0$. The main purpose of the cost or loss function is to be minimized with respect to a parameter $\theta$ in order to make sure that the hypothesis converges to the values of the targets $y^{(i)}$, which are single scalar values. A column vector $y \in \mathbb{R}^{m}$ can accommodate all the $m$ target values from each training example $(x^{(i)}, y^{(i)} )$. Note that the notation $J(\theta; x)$ tries to distinguish the role of $\theta$ from the role of $x$: while $x$ are fixed for each training example, $\theta$ are the coefficients to be learned by the model. Finally, capital $X$ will denote the design matrix, which is made of rows $(x^t)^{(i)}$. Thus, $X \in \mathbb{R}^{(m, n+1)}$ and $\theta \in  \mathbb{R}^{(n+1, 1)}$  is a matrix.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#line magic function for plot inside notebook
%matplotlib inline
import seaborn as sns
#from numpy.core.function_base import linspace

```

# Single variable Linear Regression
"In this part of this exercise, you will implement linear regression with one
variable to predict profits for a food truck. Suppose you are the CEO of a
restaurant franchise and are considering different cities for opening a new
outlet. The chain already has trucks in various cities and you have data for
profits and populations from the cities.

You would like to use this data to help you select which city to expand
to next.The dataset for our linear regression problem is as follows: The first column is the population of a city and the second column is
the profit of a food truck in that city. A negative value for profit indicates a
loss.

For this dataset, you can use a scatter plot to visualize the data, since it has only two properties to plot (profit and population). (Many
other problems that you will encounter in real life are multi-dimensional and
can’t be plotted on a 2-d plot.)" (extract from Stanford-ML-ex01- Andrew Ng)

```python
path = '/content/drive/MyDrive/Machine_Learning_DS/Material/ML_Standford/ex1-octave/ex1data1.txt'
data = pd.read_csv(path, header=None, names=['Population', 'Profit'])

m = len(data)  #number of training values
x = np.array(data['Population'])   #capital X will be with x0s included...
y = np.array(data['Profit'])

data.head()
```

      Population
      Profit

      0
      6.1101
      17.5920

      1
      5.5277
      9.1302

      2
      8.5186
      13.6620

      3
      7.0032
      11.8540

      4
      5.8598
      6.8233

      <button class="colab-df-convert" onclick="convertToInteractive('df-c0485f5c-23a3-42ba-accb-155a1d5d69ea')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
print(len(data.columns))
print(len(data.index))
```

    2
    97

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 97 entries, 0 to 96
    Data columns (total 2 columns):
     #   Column      Non-Null Count  Dtype
    ---  ------      --------------  -----
     0   Population  97 non-null     float64
     1   Profit      97 non-null     float64
    dtypes: float64(2)
    memory usage: 1.6 KB

```python
x.shape

```

    (97,)

```python
x = np.array(data['Population']).reshape(m,1)
y = np.array(data['Profit']).reshape(m,1)

x.shape

```

    (97, 1)

#To visualize the data
Note that 'bx' stands for blue crosses (option) , a marker

```python
plt.plot(data['Population'], data['Profit'], 'rx'  )   #plt.plot(x, y, 'rx'  )
#plt.figure() # not necessary
plt.xlabel('population in 10k')
plt.ylabel('profit in $10k')
plt.title('Population Vs Profit')
```

#Cost function

Given my hypothesis function, consider the cost function
to be the quadratic cost function, which is a convex function
and works well with gradient descent.
Vectorization is the best way to go because it avoids using loops...

Recall:  $J(\theta) =  \frac{1}{2m} (X \theta - y)^T (X \theta - y)$
where X includes the $x_0=1$ related to the bias $\theta_0$

Obs: numpy.append(arr, values, axis=None)
Append values to the end of an array.

```python
X = np.append(np.ones((m,1)), x,  axis=1 )  #0 would be along the lines. axis=1 will add x as the next column after a column of ones
X.shape
```

    (97, 2)

```python
def cost_function(X, y, theta):
    """Calculate the cost function for the linear regression of a single variable """
    global m    #or define m again here locally
    return 1/(2*m) * ((np.dot(X,theta)) - y).T @ ( np.dot(X,theta) - y)
   # return 1/(2*m) * ((X @ theta) - y).T @ ((X @ theta) - y) #alternatively: sum(np.square( (X@ theta) - y  ))
```

```python
theta_initial = np.zeros( (2,1) )
print(cost_function(X, y, theta_initial))
```

    [[32.07273388]]

```python
#NOTE these
print ((np.dot(X, theta_initial)).shape)
print ((np.dot(X, theta_initial).T).shape)

(X @ theta_initial).shape
```

    (97, 1)
    (1, 97)

    (97, 1)

#Gradient descent
As you want to minimize a cost function, make sure you understand what you are trying to optimize and what is being updated. Keep in mind that the cost $J(\theta)$ is parameterized by the vector $\theta$, not $X$ and $y$. That is, we minimize the value of $J(\theta)$ by changing the values of the matrix $\theta$

```python
def gradientdescent(X, y, theta, alpha, itera):
  """Gradient to update weights theta. It returns theta and the cost function at each iteration step """

  m = len(y) #number of training examples
  J_history = np.zeros((itera,1))
  for i in range(itera):
    theta = theta - (alpha/m) * (X.T @ (np.dot(X,theta) - y )) #vectorized
    J_history[i] = cost_function(X, y, theta)  #store J value after updating theta and repeat process

    #alternative: theta[0] = theta[0] - ...(np.sum(X@theta - y) * 1 ) #since x0 =1
    # theta[1] = theta[1] - ....(np.sum(X@theta - y).T @ X[:,1:])
    #note that:  X[:,1:] is (97,1)  and  X[:,1] is (97,)
    #

  return theta, J_history

theta = np.zeros( (2,1) )
alpha = 0.02
itera = 1500  # TRY 3500 and see better value

theta, J_history = gradientdescent(X, y, theta, alpha, itera)
print(theta)
z = pd.DataFrame(J_history, columns=['J values'])
z.head()

```

    [[-3.87813769]
     [ 1.19126119]]

      J values

      0
      16.769642

      1
      10.407580

      2
      7.759183

      3
      6.653288

      4
      6.188106

      <button class="colab-df-convert" onclick="convertToInteractive('df-16fd202c-9e4b-476b-b60d-6f06f8bff899')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

#Plotting results and making a prediction

Your final values for θ will also be used to make predictions on profits in
areas of 35,000 and 70,000 people.

```python
predict1 = np.array([1, 3.5]).dot(theta)*10000 #data is given in 10k people and 10k money
predict2 = np.dot(np.array([1, 7]), theta)*10000 #same for 7 = 70k
print(predict1)
print(predict2)
```

    [2912.76490368]
    [44606.90671602]

```python
plt.figure()
plt.plot(X[:,1], y, 'rx', label='training' )
plt.plot(X[:,1], np.dot(X,theta), label='prediction' )
plt.xlabel('population in 10k')
plt.ylabel('profit in $10k')
plt.title('Result')
plt.legend()
#plt.show()
```

```python
#The cost function against the number of iterations below
plt.plot(J_history, label='cost')
plt.xlabel('number of iterations')
plt.ylabel('J($ \theta $)')
plt.title('Cost function decay')
plt.legend()
```

```python
#now contour plot (surface levels) and surface plot

theta0vals = np.linspace(-10,10,100)
theta1vals = np.linspace(-1,4,100)

J_values =  np.zeros((len(theta0vals), len(theta1vals)))

for i in range(len(theta0vals)):
  for j in range(len(theta1vals)):
    #t = np.array([ [theta0vals[i]], [theta1vals[j]] ])
     t = [[theta0vals[i]],[theta1vals[j]]]
     J_values[i,j] = cost_function(X,y,t)

J_values.shape
#J_values
```

    (100, 100)

```python
# J_values.T (transpose) is required for the way contour plot works. why?
scaling = np.logspace(-2,3,20)
plt.contour(theta0vals, theta1vals, J_values.T, levels = np.logspace(np.log10(0.01),np.log10(1000), 20))
plt.plot(theta[0], theta[1], 'r+')
plt.title('A contour plot for bias theta')
```

```python
#alternatively

J_matrix = np.array([np.array([cost_function(X,y, [[theta0vals[i]],[theta1vals[j]]]).reshape(1)
                                                                for i in range(len(theta0vals))]).reshape(100)
                                                                      for j in range(len(theta1vals))])

#now the matrix needs no transposing this time.. why?
plt.contour(theta0vals, theta1vals, J_matrix, levels = np.logspace(np.log10(0.01),np.log10(1000), 20)  )
plt.show()
print(J_matrix.shape)
J_matrix
```

```python
#Note the role of first and second components i,j in regard to row and colum
A = [1,2,3,4]
B= [1,2,3]
M = np.array([[(a,b) for a in A] for b in B])
print(M.shape)
M.tolist() #removes third dimension. 3 rows(b) and 4 columns(a)
```

    (3, 4, 2)

    [[[1, 1], [2, 1], [3, 1], [4, 1]],
     [[1, 2], [2, 2], [3, 2], [4, 2]],
     [[1, 3], [2, 3], [3, 3], [4, 3]]]

```python
#Another way using numpy meshgrid
theta0vals = np.linspace(-10,10,100)
theta1vals = np.linspace(-1,4,100)

def f(theta0vals, theta1vals):

  Z = np.zeros((len(theta0vals), len(theta1vals)))

  for (i, value0) in enumerate(U):
    for (j, value1 ) in enumerate(V):
      t = [[value0[i]],[value1[j]]]
      Z[i,j] = 1/(2*m) * ((np.dot(X,t)) - y).T @ ( np.dot(X,t) - y)
  return Z.T
#Take transpose as contour plot flips directions

U, V = np.meshgrid(theta0vals, theta1vals)

plt.contour(U,V, f(U,V), levels = np.logspace(np.log10(0.01),np.log10(1000), 20))
plt.plot(theta[0], theta[1], 'r+')
plt.title('A contour plot for bias theta')
plt.show()

```

```python

```

#Basic Linear Algebra/Statistics
For one feature, the relation between the parameter and the intercept is straightforward and can be easily checked against the covariance matrix and mean values.

```python
#The linear relation, measured by correlation, between the the feature and the target is clear
#in the following picture
sns.heatmap(data.corr(), annot=True)
```

```python
data.cov()
```

      Population
      Profit

      Population
      14.975999
      17.86687

      Profit
      17.866870
      30.36299

      <button class="colab-df-convert" onclick="convertToInteractive('df-9f29200f-f589-49ce-964f-f248227cdce7')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
data.Population.var()
```

    14.975998519375002

```python
#1 feature: theta1 = cov(x,y)/var(x)
expected_theta1 = 17.86687/ data.Population.var()
expected_theta1
```

    1.193033638250229

```python
#Thinking of vectors and subspaces, the first parameter is also given as
float((X[:,1]-X[:,1].mean()).T @ y)/( (X[:,1]-X[:,1].mean()).T   @ (X[:,1]-X[:,1].mean()))
```

    1.1930336441895941

```python
data.mean()
```

    Population    8.159800
    Profit        5.839135
    dtype: float64

```python
#1 feature: theta0 = mean(y) - theta1 * mean(x)
expected_theta0 = data.Profit.mean() - (17.86687/ data.Population.var())*(data.Population.mean())
expected_theta0

```

    -3.8957808298478307

```python
data.Population.std() * data.Profit.std()
```

    21.324073135832304

```python
data.corr()*(data.Population.std() * data.Profit.std()) #covariance of x and y off-diagonal
```

      Population
      Profit

      Population
      21.324073
      17.866870

      Profit
      17.866870
      21.324073

      <button class="colab-df-convert" onclick="convertToInteractive('df-0bd195bb-1296-460a-9bf2-b2af9c2a55a4')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

#Linear Regression with multiple variables

"In this part, you will implement linear regression with multiple variables to
predict the prices of houses. Suppose you are selling your house and you
want to know what a good market price would be. One way to do this is to
first collect information on recent houses sold and make a model of housing
prices.
The file ex1data2.txt contains a training set of housing prices in Port-
land, Oregon. The first column is the size of the house (in square feet), the
second column is the number of bedrooms, and the third column is the price
of the house." (extract from Stanford-ML-ex01- Andrew Ng)

```python
path = '/content/drive/MyDrive/Machine_Learning_DS/Material/ML_Standford/ex1-octave/ex1data2.txt'
data = pd.read_csv(path, header=None, names=['Size', 'Bedrooms', 'Price'])

m = len(data['Price'])  #number of training values

#make sure to create a matrix
features = [ 'Size','Bedrooms']

x = np.array( [ data['Size'], data['Bedrooms'] ] )   #capital X will be with x0s included...
y = np.array(data['Price'])

m = len(data)

data.head(7)
```

      Size
      Bedrooms
      Price

      0
      2104
      3
      399900

      1
      1600
      3
      329900

      2
      2400
      3
      369000

      3
      1416
      2
      232000

      4
      3000
      4
      539900

      5
      1985
      4
      299900

      6
      1534
      3
      314900

      <button class="colab-df-convert" onclick="convertToInteractive('df-86b5b2fb-22cd-4860-94de-880f4bdd5423')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 47 entries, 0 to 46
    Data columns (total 3 columns):
     #   Column    Non-Null Count  Dtype
    ---  ------    --------------  -----
     0   Size      47 non-null     int64
     1   Bedrooms  47 non-null     int64
     2   Price     47 non-null     int64
    dtypes: int64(3)
    memory usage: 1.2 KB

```python
data.describe()  #5-point description
```

      Size
      Bedrooms
      Price

      count
      47.000000
      47.000000
      47.000000

      mean
      2000.680851
      3.170213
      340412.659574

      std
      794.702354
      0.760982
      125039.899586

      min
      852.000000
      1.000000
      169900.000000

      25%
      1432.000000
      3.000000
      249900.000000

      50%
      1888.000000
      3.000000
      299900.000000

      75%
      2269.000000
      4.000000
      384450.000000

      max
      4478.000000
      5.000000
      699900.000000

      <button class="colab-df-convert" onclick="convertToInteractive('df-a7897103-7e79-4ba7-82ef-8f6cfa567475')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
#The examples with 5 bedrooms are scarce and will not give much information
#that is the same case for those with only 1 bedroom (check)
data[data['Bedrooms'] >=5 ]['Bedrooms'].count()
```

    1

```python
#The visible linear dependence of price and Size can be seen below
#note the behaviour of examples with 3 and 4 bedrooms
sns.pairplot(data, hue= 'Bedrooms' , diag_kind='kde')
```

```python
#the data size has 2 features , 1 target and 47 examples
print(data.shape)
print(x.shape)
print(y.shape)
```

    (47, 3)
    (2, 47)
    (47,)

```python
#but the shapes are not as we want at the moment
x = x.T   #transpose (ONCE!) to make examples along rows
(m,n) = x.shape
print(m,n)
```

    47 2

```python
x[:5]
```

    array([[2104,    3],
           [1600,    3],
           [2400,    3],
           [1416,    2],
           [3000,    4]])

```python
#Reshape such that we have column vectors and matrices
#x was corrected with transpose but make sure all formates are fixed as follows

x = x.reshape(m,n)
y = y.reshape(m,1)

print(x.shape)
print(y.shape)
```

    (47, 2)
    (47, 1)

```python
x[1]
```

    array([1600,    3])

```python
#Look at this plot below. The orders of magnitude of Size are larger than
#the orders of magnitude in Bedrooms. In fact, Sizes are of order 10^3 and Bedrooms only 10^0.
features = ['Size','Bedrooms']
data[features].plot()
```

```python

```

#Standard deviation and normalization
Before we normalize the features, note that Pandas standard deviation is different from Numpy std(). The reason is that Pandas will use the corrected version (Bessel's correction), where the mean value is a data estimation of the true mean value, while Numpy considers the uncorrected standard deviation. That is, Pandas assumes that the mean/expectation_value is calculated from a sample only. Numpy considers it is calculated from a population. These two concepts are different in statistics, because the sample is, generally, data coming from the population, a subset of all the data!

Specifically, let $N$ be the number of examples observed, these two quantities differ by

$$  \sigma_{\textrm{sample}} = \sqrt{ \left( \frac{1}{N-1} \sum_{i=1}^{N} (x_i - \bar{x})^2   \right) },$$

$$ \sigma_{\textrm{population}} =  \sqrt{ \left( \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2   \right) },$$
where we understand the true mean value as $\mu$ and the mean value $\bar{x}$ from the dataset (subset of a population) is a data estimation of the true mean. Note that $\sigma_{\textrm{sample}} \geq \sigma_{\textrm{population}}$.

If we tell Numpy to use "ddof=1", it will be equivalent to the Pandas default, namely, the standard deviation of a sample. Conversely, we would set "ddof=0" in Pandas to calculate the population standard deviation.

On a different note,  when we peform feature normalization, it is useful to store their values of mean() and std(), because we want to make prediction with unseen data and will need to normalize as well the unseen data.
Finally, the normalization is given by $ x \mapsto \left(\frac{x - \mu }{\sigma} \right) $, such that the range of the features is defined in terms of how much it deviates from the standard deviation $\sigma$. In the case of a Gaussian distribution, it is known ($68-95-99.7$ rule) that $95$% of the data lies within the range $(\mu \pm  2 \sigma)$ and $99.7$% within three standard deviations of the mean $(\mu \pm  3 \sigma)$. More generally, the Chebyshev's inequality stats that at least ($1 - \frac{1}{k^2}) \times 100$% of the data lies within $k$ standard deviations away from the mean. Hence, at least $75$% of the data is included in the range $\mu \pm  2 \sigma$. These observations can be seen below after we have performed feature normlatization.

"Implementation Note: When normalizing the features, it is important
to store the values used for normalization - the mean value and the stan-
dard deviation used for the computations. After learning the parameters
from the model, we often want to predict the prices of houses we have not
seen before. Given a new x value (living room area and number of bed-
rooms), we must first normalize x using the mean and standard deviation
that we had previously computed from the training set." (Stanford-ML-ex01- Andrew Ng)

```python
print( data['Size'].std(), data['Bedrooms'].std())
#slightly different
print(np.std(x, axis=0))

```

    794.7023535338897 0.7609818867800999
    [7.86202619e+02 7.52842809e-01]

```python
#More than a matter of visualization, which is important for interpretation,
#so distinct numbers will not be helpful in the process of convergence required
#in optimization problems. The so called 'feature normalization' is a way to deal with this problem.
#Basically, we will normalize the features using mean (expectation value) and standard deviation

def featureNormalize(x):
  """Given the features x, return the features normalized, mean and standard deviation, respectively. """
  x_norm = x

  mu = np.mean(x, axis=0)  #take elements along rows of same column (mean value of each column)
  sigma = x.std(axis=0, ddof=1)

#numpy broadcasting will make it easier to vectorize the code
  x_norm = (x - mu )/sigma     #divide is also broadcasted operation

  return x_norm, mu, sigma

```

```python
x_norma, mu, sigma = featureNormalize(x) #

x_norma[:3]
```

    array([[ 0.13000987, -0.22367519],
           [-0.50418984, -0.22367519],
           [ 0.50247636, -0.22367519]])

```python
print('expectation values are: ', mu)
print('standard deviations are: ', sigma)
```

    expectation values are:  [2000.68085106    3.17021277]
    standard deviations are:  [7.94702354e+02 7.60981887e-01]

Gradient descent is calculated again

```python
#Now that the features are normalized, we can include the bias by means of considering the extended X
#with a column of 1's for x0

X = np.append(np.ones((m,1)), x_norma, axis=1 )
X[:3]
```

    array([[ 1.        ,  0.13000987, -0.22367519],
           [ 1.        , -0.50418984, -0.22367519],
           [ 1.        ,  0.50247636, -0.22367519]])

```python
X.shape
```

    (47, 3)

```python
def gradientdescentMulti(X, y, theta, alpha, itera):
  """Gradient to update weights theta. It returns the thetas and the cost function at each iteration step """

  m = len(y) #number of training examples
  J_history = np.zeros((itera,1))
  for i in range(itera):
    theta = theta - (alpha/m) * (X.T @ (np.dot(X,theta) - y )) #vectorized
    J_history[i] = cost_function(X, y, theta)  #store J value after updating theta and repeat process

  return theta, J_history

theta = np.zeros( (3,1) )
alpha = 0.01
itera = 2000  #try 3000 for better value

theta, J_history = gradientdescentMulti(X, y, theta, alpha, itera)

print(theta)
z = pd.DataFrame(J_history, columns=['J values'])
z.head()
```

    [[340412.65894002]
     [110620.59436465]
     [ -6639.01835663]]

      J values

      0
      6.430075e+10

      1
      6.303679e+10

      2
      6.179911e+10

      3
      6.058714e+10

      4
      5.940033e+10

      <button class="colab-df-convert" onclick="convertToInteractive('df-22b1506d-cb73-4dd0-96b0-b1eff9cabf28')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

```python
#The cost function against the number of iterations below
#note the quick convergence for about from 0 to 200 iterations
plt.plot(J_history, label='cost')
plt.xlabel('number of iterations')
plt.ylabel('J(theta)')
plt.title('Cost function decay')
plt.legend()
```

As before, we are in a position to make predictions. What is the
estimated price of a 1650 ft^2 house with 3 bedrooms ?

```python
#As we normalized the features when fitting the model, we also need to
#normalize the features used for any prediction.
X_test = np.array([1, 1650, 3 ])

X_test_norma = np.array([1, (X_test[1] -mu[0])/sigma[0] , (X_test[2] - mu[1])/sigma[1] ])

price = np.dot(X_test_norma.T, theta)

print('The estimated price of a House with 1650 ft^2 and 3 bedrooms is: ' + str(price[0]) + '$')
```

    The estimated price of a House with 1650 ft^2 and 3 bedrooms is: 293083.73888660746$

#Exact solution (Normal equation)
For linear regression, an exact solution to the problem of finding the minimum of the cost function  can be found and it is called the normal equation.
As it involves the calculation of inverse matrices, the computational cost of the process grows with $\mathcal{O}(n^3)$, where $n$ is the number of examples.
That means the use of the normal equation shall be restricted to situations where the dataset is not very large, up to $10^3$ number of examples.
The solution is obtained by setting $\frac{\partial J}{\partial \theta} = 0$, as the quadratic function $J$ is convex and will have a global minimum rather than a maximum, which results in
\begin{equation}
\theta = \left( X^t X \right)^{-1} X^t y,
\end{equation}
which indeed is a $(n+1) \times 1$ matrix.
Note there is no need for using the learning rate $\alpha$ or iterative steps. The learning rate is a consequence of using approximations to a numerical problem and it is absent in the case of exact solutions.

```python
theta_min = np.linalg.inv(X.T @ X) @ X.T @ y

print(f'The weights that solve the problem exactly are \n {theta_min.flatten()}')
```

    The weights that solve the problem exactly are
     [340412.65957447 110631.05027885  -6649.47427082]

#Comparison with Scikit-learn

```python
#If we use our initial x without normalization
from sklearn.linear_model import LinearRegression

lr = LinearRegression(fit_intercept=True)
lr.fit(x, y)
```

    LinearRegression()

```python
x_test = np.array([[1650, 3]])
price_predicted = lr.predict(x_test)
price_predicted

#the prediction agress with what we found before
```

    array([[293081.4643349]])

```python
#But the values of the weights and bias are not the same. that is expected!
print('weights are: ',lr.coef_)
print('intercept is: ',lr.intercept_)
```

    weights are:  [[  139.21067402 -8738.01911233]]
    intercept is:  [89597.9095428]

```python
#Now, if we use our normalized features x_norma (before adding the bias)
lr2 = LinearRegression(fit_intercept=True)
lr2.fit(x_norma, y)

print('weights are: ',lr2.coef_)
print('intercept is: ',lr2.intercept_)

#same weights and bias we found before
```

    weights are:  [[110631.05027885  -6649.47427082]]
    intercept is:  [340412.65957447]

```python

```


# How

## How to create a symbolic numeric value

To create a symbolic numerical value use `sympy.S`.

````{tip}
```
sympy.S(a)
```
````

For example:

```python
import sympy

value = sympy.S(3)
value
```

$\displaystyle 3$

```{attention}
If we combine a symbolic value with a non symbolic value it will automatically
give a symbolic value:
```

```python
1 / value
```

$\displaystyle \frac{1}{3}$

## How to get the numerical value of a symbolic expression

We can get the numerical value of a symbolic value using `float` or `int`:

- `float` will give the numeric approximation in \\(\mathbb{R}\\)
  ````{tip}
  ```
  float(x)
  ```
  ````
- `int` will give the integer value
  ````{tip}
  ```
  int(x)
  ```
  ````

For example, let us create a symbolic numeric variable with value
\\(\frac{1}{5}\\):

```python
value = 1 / sympy.S(5)
value
```

$\displaystyle \frac{1}{5}$

To get the numerical value:

```python
float(value)
```

    0.2

If we wanted the integer value:

```python
int(value)
```

    0

```{attention}
This is not rounding to the nearest integer. It is returning the integer part.
```

## How to factor an expression

We use the `sympy.factor` tool to factor expressions.

````{tip}
```
sympy.factor(expression)
```
````

For example:

```python
x = sympy.Symbol("x")
sympy.factor(x ** 2 - 9)
```

$\displaystyle \left(x - 3\right) \left(x + 3\right)$

## How to expand an expression

We use the `sympy.expand` tool to expand expressions.

````{tip}
```
sympy.expand(expression)
```
````

For example:

```python
sympy.expand((x - 3) * (x + 3))
```

$\displaystyle x^{2} - 9$

## How to simplify an expression

We use the `sympy.simplify` tool to simplify an expression.

````{tip}
```
sympy.simplify(expression)
```
````

For example:

```python
sympy.simplify((x - 3) * (x + 3))
```

$\displaystyle x^{2} - 9$

```{attention}
This will not always give the expected (or any) result. At times it could be
more beneficial to use `sympy.expand` and/or `sympy.factor`.
```

## How to solve an equation

We use the `sympy.solveset` tool to solve an equation. It takes two values as
inputs. The first is either:

- An expression for which a root is to be found
- An equation

The second is the variable we want to solve for.

````{tip}
```
sympy.solveset(equation, variable)
```
````

Here is how we can use `sympy` to obtain the roots of the general quadratic:

\\[
a x ^ 2 + bx + c
\\]

```python
a = sympy.Symbol("a")
b = sympy.Symbol("b")
c = sympy.Symbol("c")
quadratic = a * x ** 2 + b * x + c
sympy.solveset(quadratic, x)
```

$\displaystyle \left\{- \frac{b}{2 a} - \frac{\sqrt{- 4 a c + b^{2}}}{2 a}, - \frac{b}{2 a} + \frac{\sqrt{- 4 a c + b^{2}}}{2 a}\right\}$

Here is how we would solve the same equation but not for \\(x\\) but for
\\(b\\):

```python
sympy.solveset(quadratic, b)
```

$\displaystyle \left\{- \frac{a x^{2} + c}{x}\right\}$

It is however clearer to specifically write the equation that we want to solve:

```python
equation = sympy.Eq(a * x ** 2 + b * x + c, 0)
sympy.solveset(equation, x)
```

$\displaystyle \left\{- \frac{b}{2 a} - \frac{\sqrt{- 4 a c + b^{2}}}{2 a}, - \frac{b}{2 a} + \frac{\sqrt{- 4 a c + b^{2}}}{2 a}\right\}$

## How to substitute a value in to an expression

Given a `sympy` expression it is possible to substitute values in to it using
the `.subs()` tool.

````{tip}
```
expression.subs({variable: value})
```
````

```{attention}
It is possible to pass multiple variables at a time.
```

For example we can substitute the values for \\(a, b, c\\) in to our quadratic:

```python
quadratic = a * x ** 2 + b * x + c
quadratic.subs({a: 1, b: sympy.S(7) / 8, c: 0})
```

$\displaystyle x^{2} + \frac{7 x}{8}$


# 微积分

SymPy支持微分和积分操作,也支持推导极限

```python
from sympy import init_printing
init_printing(use_unicode=True)
```

```python
from sympy import symbols
x, y, z = symbols('x y z')
```

## `diff()`微分(求导)

```python
from sympy import diff
```

```python
diff(x**3+x**2+x+1)
```

`diff(exp,var,level)`可以求多阶导数,需要指定变量和阶数

```python
diff(x**3+x**2+x+1,x,2)
```

同样的,也可以求偏导

```python
diff(x**3+x*y**2+x*y+1,x)
```

要创建未化简的的导数,需要使用导数类.它具有与diff相同的语法,但必须显式的指定是谁的微分

```python
from sympy import Derivative
```

```python
exp = diff(x**3+x**2+x+1)
exp
```

```python
Derivative(exp,x)
```

要推导导数类的实例,可以使用算式的`doit()`方法

```python
Derivative(exp,x).doit()
```

##  `integrate(exp,var)`积分

```python
from sympy import integrate
from sympy import cos
```

### 不定积分

```python
exp = cos(x)
exp
```

```python
integrate(exp, x)
```

### 定积分

定积分一般会有上下限,可以用元组(integration_variable, lower_limit, upper_limit)替换var,sympy.注意`oo`表示无穷大

```python
from sympy import oo,exp
```

$ \int _0^\infty e^{-x} dx$

```python
integrate(exp(-x), (x, 0, oo))
```

### 重积分

比如我们想求如下这个二重积分

$\int _{-\infty}^\infty \int _{-\infty}^\infty e^{-x^2-y^2} dx dy$

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

如果积分无法求得,那么它会返回原来的样子

```python
expr = integrate(x**x, x)
expr
```

就像微分一样,积分也有对应的类型Integral,定义方式也相似

```python
from sympy import Integral,log
```

```python
expr = Integral(log(x)**2, x)
expr
```

```python
expr.doit()
```

## 极限

SymPy可以使用limit函数计算极限。

```python
from sympy import limit,sin
```

```python
limit(sin(x)/x, x, 0)
```

```python
expr = x**2/exp(x)
expr.subs(x, oo)
```

$\displaystyle \text{NaN}$

```python
limit(expr, x, oo)
```

极限也有一个类,也和上面的微分积分差不多

```python
from sympy import Limit
```

```python
expr = Limit((cos(x) - 1)/x, x, 0)
expr
```

```python
expr.doit()
```

## 泰勒展开

泰勒公式是一个用函数在某点的信息描述其附近取值的公式.如果函数足够平滑的话,在已知函数在某一点的各阶导数值的情况之下,泰勒公式可以用这些导数值做系数构建一个多项式来近似函数在这一点的邻域中的值.泰勒公式还给出了这个多项式和实际的函数值之间的偏差.

SymPy可以计算围绕一个点的函数做泰勒级数展开.它使用算式的`.series(x, x0, n)`方法,就像之前在验证欧拉公式时我们做的那样

```python
expr = exp(sin(x))
expr
```

```python
expr.series(x, 0, 4)
```

```python
from sympy import symbols
x0 = symbols("x0")
```

```python
expr.series(x, x0, 4)
```

## 有限差分

到目前为止我们分别用分析导数和原始函数来研究表达式.
但是如果我们想要一个表达式来估计曲线的导数,如果我们缺少一个闭合形式表示,或者我们还不知道函数值那又怎么办呢?
一种方法是使用有限差分法.

SymPy中提供了接口`as_finite_difference()`可以在任何微分实例上来生成任意阶导数的近似:

```python
from sympy import Function,finite_diff_weights,apply_finite_diff
```

```python
f = Function('f')
dfdx =  Derivative(f(x))#.diff(x)
dfdx
```

```python
dfdx.as_finite_difference()
```

这里我们使用步长为1,等距离估计的最小点数来近似函数对x的一阶导数.我们可以使用任意的步长

```python
f = Function('f')
d2fdx2 = Derivative(f(x),x, 2)
d2fdx2
```

```python
h = symbols('h')
d2fdx2.as_finite_difference([-3*h,-h,2*h])
```

如果要评估权重,你可以手动计算

```python
finite_diff_weights(2, [-3, -1, 2], 0)[-1][-1]
```

注意,我们只需要从`finite_diff_weights`返回的最后一个子列表中取最后一个元素.这样做的原因是`finite_diff_weights`产生较低阶导数的权重,并使用较少的点.

如果使用`finite_diff_weights`直接看起来很复杂,并且觉得Derivative实例操作的`as_finite_difference`函数不够灵活,则可以使用`apply_finite_diff`,它接受order，x_list，y_list和x0作为参数

```python
x_list = [-3, 1, 2]
y_list = symbols('a b c')
apply_finite_diff(1, x_list, y_list, 0)
```


## Fourier methods

The Fourier transform (FT) for a well-behaved functions $f$ is defined as:

$$f(k) = \int e^{-ikx} f(x) ~dx$$

The inverse FT is then

$$f(x) = \frac{1}{2\pi} \int e^{ikx} f(k) ~dk$$

## Discrete Fourier transforms (DFTs)

If the function is periodic in real space, $f(x+L) = f(x)$, then the Fourier space is discrete with spacing $\frac{2\pi}{L}$. Moreover, if the real space is periodic as well as discrete with the spacing $h$, then the Fourier space is discrete as well as bounded.

$$f(x) = \sum e^{ikx} f(k) ~dk~~~~~~      \text{where } k = \Bigg[ -\frac{\pi}{h}, \frac{\pi}{h}\Bigg];~~ \text{with interval} \frac{2\pi}{L} $$

This is very much in line with crystallography with $ [ -\frac{\pi}{h}, \frac{\pi}{h} ]$ being the first Brillouin zone. So we see that there is a concept of the maximum wavenumber $ k_{max}=\frac{\pi}{h} $, we will get back to this later in the notes. Usually in computations we need to find FT of discrete function rather than of a well defined analytic function. Since the real space is discrete and periodic, the Fourier space is also discrete and periodic or bounded. Also, the Fourier space is continuous if the real space is unbounded. If the function is defined at $N$ points in real space and one wants to calculate the function at $N$ points in Fourier space, then **DFT** is defined as

$$f_k = \sum_{n=0}^{N-1} f_n ~ e^{-i\frac{2\pi~n~k}{N}}$$

while the inverse transform of this is

$$f_n = \frac1N \sum_{n=0}^{N-1} f_k ~ e^{~i\frac{2\pi~n~k}{N}}$$

To calculate each $f_n$ one needs $N$ computations and it has to be done $N$ times, i.e, the algorithm is simply $\mathcal{O}(N^2)$. This can be implemented numerically as a matrix multiplication, $f_k = M\cdot f_n$, where $M$ is a $N\times N$ matrix.

## Fast fourier tranforms (FFTs)

The discussion here is based on the Cooley-Tukey algorithm. FFTs improves on DFTs by exploiting their symmetries.

$$ \begin{align}
f_k &= \sum_{n=0}^{N-1} f_n  e^{-i~\frac{2\pi~k~n}{N}} \\
&= \sum_{n=0}^{N/2-1} f_{2n} e^{-i~\frac{2\pi~k~2n}{N}} &+ \sum_{n=0}^{N/2-1} f_{2n + 1} e^{-i~\frac{2\pi~k~(n+1)}{N}}\\
&= \sum_{n=0}^{N/2 - 1} f_{2n} e^{-i~\frac{2\pi k~n}{N/2}} &+ e^{-i\frac{2\pi k}{N}} \sum_{n=0}^{N/2 - 1} f_{2n + 1} e^{-i~\frac{2\pi~k~n~}{N/2}}\\
&=\vdots &\vdots
\end{align}$$

We can use the symmetry property, from the definition, $f_{N+k} = f_k$. Notice that, because of the tree structure, there are $\ln_2 N$ stages of the calculation. By applying the method of splitting the computation in two halves recursively, the complexity of the problem becomes  $\mathcal{O}(N \ln N)$  while the naive algorithm is $\mathcal{O}(N^2)$. Below we will look at a simple implementation of FFT.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def testFFT(x):
    ''' FFT in 1d  '''
    N = x.shape[0]
    if N % 2 > 0:
        raise Exception('x must have even size')
    elif N <= 16:
        ''' this is the naive implementation using matrix multiplication'''
        n = np.arange(N)
        k = n.reshape((N, 1))
        M = np.exp(-2j * np.pi * k * n / N)
        return np.dot(M, x)
    else:
        M1 = testFFT(x[::2])
        M2 = testFFT(x[1::2])
        fac = np.exp(-2j*np.pi*np.arange(N)/N)
        return np.concatenate([M1+fac[:N/2]*M2, M1+fac[N/2:]*M2])
```

```python
x = np.random.random(1024)
#np.allclose(np.fft.fftn(x), testFFT(x))
```

```python
# TRANSLATION by e^{ikr}!
L, N = 64, 128
ll =  np.linspace(0, L, N)
x, y = np.meshgrid(ll, ll)

# Fourier grid.
kx = 2 * np.pi / L * np.concatenate((np.arange(0, N/2+1,1),np.arange(-N/2+1, 0, 1)))  # k = (2\pi)/L
ky = 2 * np.pi / L * np.concatenate((np.arange(0, N/2+1,1),np.arange(-N/2+1, 0, 1)))
kx, ky = np.meshgrid(kx, ky)

def plotFirst(x, y, sig, n_):
    sp =  f.add_subplot(1, 3, n_ )
    plt.pcolormesh(x, y, np.real(sig), cmap=plt.cm.gist_heat_r)
    plt.axis('off');

f = plt.figure(figsize=(20, 5), dpi=80);
rr = np.sqrt( (x - L/2)*(x - L/2) + (y - L/2)*(y - L/2) )
sig = np.fft.fftn(np.exp(-0.1*rr))

xx = ([-L/4, L/2, -L/2,])
yy = ([-L/8, -L/9, L/2])
for i in range(3):
    kdotr = kx*xx[i] + ky *yy[i]
    sig = sig*np.exp(-1j*kdotr)
    plotFirst(x, y, np.fft.ifftn(sig), i+1)
```

## More examples
### Sampling: Aliasing error

We saw that because of the smallest length scale, $h$, in the real space there is a corresponding largest wave-vector, $k_{max}$ in the Fourier space. The error is because of this $k_{max}$ and a signal which has $k>k_{max}$ can not be distinguished on this grid. In the given example, below, we see that if the real space has 10 points that one can not distinguish between $sin(2\pi x/L)$ and $sin(34 \pi x/L)$. In general, $sin(k_1 x)$ and $sin(k_2 x)$ can not be distinguished if $k_1 -k_2$ is a multiple of $\frac{2\pi}{h}$. This is a manifestation of the gem called the sampling theorem which is defined, as on wikipedia:

If a function x(t) contains no frequencies higher than B hertz, it is completely determined by giving its ordinates at a series of points spaced 1/(2B) seconds apart.

```python
L, N = 1, 16
x = np.arange(0, L, L/512)
xx = np.arange(0, L, L/N)

def ff(k, x):
    return sin(k*x)
```

```python
f = plt.figure(figsize=(17, 6), dpi=80);  pi=np.pi; sin=np.sin; cos=np.cos

plt.plot(x, ff(x, 2*pi), color="#A60628", linewidth=2);
plt.plot(x, ff(x, 34*pi), color="#348ABD", linewidth=2);
plt.plot(xx, ff(xx, 2*pi), 'o', color="#020e3e", markersize=8)
plt.xlabel('x', fontsize=15); plt.ylabel('y(x)', fontsize=15);
plt.title('Aliasing in sampling of $sin(2\pi x/L)$ and $sin(34 \pi x/L)$', fontsize=15);
```

 ### Differentiation

 In this section we will use the in-built FFT modules of numpy and then perform differentiation. Differentiation in Fourier space is trivial, $\mathbf{\nabla}_{\alpha}$ gets replaced by $ik_{\alpha}$. Steps involved are

* FFT the function to be differentiated.
* multiply by suitable numbers of $ik$
* IFFT on the resulting thing to get the differentiated function in real space.

```python
def f1(kk, x):
    return cos(kk*x)

```

```python
f = plt.figure(figsize=(10, 5), dpi=80);

L, N = 1, 32
kk   = 2*pi/L
x = np.arange(0, N)*(L/N)
k = np.concatenate(( np.arange(0, N/2+1,1), np.arange(-N/2+1, 0, 1) ))*(2*pi/L)

fk = np.fft.fft(f1(kk, x))
f1_kk = -k*k*fk
f1_xx = np.fft.ifft(f1_kk)

plt.plot(x, -f1(kk, x)*kk*kk, color="#348ABD", label = 'analytical', linewidth=2)
plt.plot(x, f1_xx, 'o', color="#A60628", label = 'numerical', markersize=6)
plt.legend(loc = 'best')
plt.xlabel('x', fontsize=15)
plt.title('derivatives using FFT', fontsize=15)
plt.xlim([0, max(x)]);
```

There are additional symmetry properties for a real function. Wavenumbers corresponding to $j$ and $N-j$ are same and hence we only need to consider wavenumber till N/2!

```python
# since we know that function is real, then we could save some time and memory by using real FFTs.
f = plt.figure(figsize=(10, 5), dpi=80);

L, N = 1, 64
x = np.arange(0, L, L/N)
kk   = 4*pi/L
k =  np.arange(0, N/2 + 1)*(2*pi/L)

# k[2] = 0 # all the pattern is dead at a particular k mode which coooresponds to the signal!!

fk = np.fft.rfft(f1(kk, x))
fk = -k*k*fk
fx = np.fft.irfft(fk)

plt.plot(x, -kk*kk*f1(kk, x), color="#348ABD", label = 'analytical', linewidth=2)
plt.plot(x, fx, 'o', color="#A60628", label = 'numerical')
plt.legend(loc = 'best')
plt.xlabel('x', fontsize=15)
plt.title('derivatives using FFT', fontsize=15)
plt.xlim([0, max(x)]);
```

```python

```


# Predefined Metrics in Symbolic Module

### Importing some of the predefined tensors. All the metrics are comprehensively listed in EinsteinPy documentation.

```python
from einsteinpy.symbolic.predefined import Schwarzschild, DeSitter, AntiDeSitter, Minkowski, find
from einsteinpy.symbolic import RicciTensor, RicciScalar
import sympy
from sympy import simplify

sympy.init_printing()  # for pretty printing
```

### Printing the metrics for visualization
All the functions return instances of :py:class:`~einsteinpy.symbolic.metric.MetricTensor`

```python
sch = Schwarzschild()
sch.tensor()
```

$\displaystyle \left[\begin{matrix}1 - \frac{r_{s}}{r} & 0 & 0 & 0\\0 & - \frac{1}{c^{2} \left(1 - \frac{r_{s}}{r}\right)} & 0 & 0\\0 & 0 & - \frac{r^{2}}{c^{2}} & 0\\0 & 0 & 0 & - \frac{r^{2} \sin^{2}{\left(\theta \right)}}{c^{2}}\end{matrix}\right]$

```python
Minkowski(c=1).tensor()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & 1.0 & 0 & 0\\0 & 0 & 1.0 & 0\\0 & 0 & 0 & 1.0\end{matrix}\right]$

```python
DeSitter().tensor()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & e^{\frac{2 x}{\alpha}} & 0 & 0\\0 & 0 & e^{\frac{2 x}{\alpha}} & 0\\0 & 0 & 0 & e^{\frac{2 x}{\alpha}}\end{matrix}\right]$

```python
AntiDeSitter().tensor()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & \cos^{2}{\left(t \right)} & 0 & 0\\0 & 0 & \cos^{2}{\left(t \right)} \sinh^{2}{\left(\chi \right)} & 0\\0 & 0 & 0 & \sin^{2}{\left(\theta \right)} \cos^{2}{\left(t \right)} \sinh^{2}{\left(\chi \right)}\end{matrix}\right]$

### Calculating the scalar (Ricci) curavtures
They should be constant for De-Sitter and Anti-De-Sitter spacetimes.

```python
scalar_curvature_de_sitter = RicciScalar.from_metric(DeSitter())
scalar_curvature_anti_de_sitter = RicciScalar.from_metric(AntiDeSitter())
```

```python
scalar_curvature_de_sitter.expr
```

```python
scalar_curvature_anti_de_sitter.expr
```

On simplifying the expression we got above, we indeed obtain a constant

```python
simplify(scalar_curvature_anti_de_sitter.expr)
```

### Searching for a predefined metric
find function returns a list of available functions

```python
find("sitter")
```

    ['AntiDeSitter', 'AntiDeSitterStatic', 'DeSitter']

docs/source/examples/Predefined Metrics in Symbolic Module.ipynb


#Snippets and Programs from Chapter 4: Algebra and Symbolic Math with SymPy

```python
%matplotlib inline
```

```python
#P96/97: Basic factorization and expansion
from sympy import Symbol, factor, expand
x = Symbol('x')
y = Symbol('y')
expr = x**2 - y**2
f = factor(expr)
print(f)
# Expand
print(expand(f))
```

```python
#P97: Factorizing and expanding a complicated identity
from sympy import Symbol, factor, expand
x = Symbol('x')
y = Symbol('y')
expr = x**3 + 3*x**2*y + 3*x*y**2 + y**3

print('Original expression: {0}'.format(expr))
factors = factor(expr)
print('Factors: {0}'.format(factors))

expanded = expand(factors)
print('Expansion: {0}'.format(expanded))

```

```python
#P97: Pretty printing
from sympy import Symbol, pprint, init_printing
x = Symbol('x')
expr = x*x + 2*x*y + y*y
pprint(expr)
# Reverse order lexicographical
init_printing(order='rev-lex')
expr = 1 + 2*x + 2*x**2
pprint(expr)
```

*Since we have initialized pretty printing above, it will be active for all the output below this.*

```python
#P99: Print a series

'''
Print the series:
x + x**2 + x**3 + ... + x**n
    ____  _____         ____
      2     3             n
'''
from sympy import Symbol, pprint, init_printing
def print_series(n):
    # initialize printing system with
    # reverse order
    init_printing(order='rev-lex')
    x = Symbol('x')
    series = x
    for i in range(2, n+1):
        series = series + (x**i)/i
    pprint(series)

if __name__ == '__main__':
    n = input('Enter the number of terms you want in the series: ')
    print_series(int(n))
```

```python
#P100: Substituting in values
from sympy import Symbol
x = Symbol('x')
y = Symbol('y')
expr = x*x + x*y + x*y + y*y
res = expr.subs({x:1, y:2})
res
```

```python
#P102: Print a series and also calculate its value at a certain point

'''
Print the series:

x + x**2 + x**3 + ... + x**n
    ____  _____         ____
      2     3             n

and calculate its value at a certain value of x.
'''

from sympy import Symbol, pprint, init_printing
def print_series(n, x_value):
    # initialize printing system with
    # reverse order
    init_printing(order='rev-lex')
    x = Symbol('x')
    series = x
    for i in range(2, n+1):
        series = series + (x**i)/i
    pprint(series)
    # evaluate the series at x_value
    series_value = series.subs({x:x_value})
    print('Value of the series at {0}: {1}'.format(x_value, series_value))

if __name__ == '__main__':
    n = input('Enter the number of terms you want in the series: ')
    x_value = input('Enter the value of x at which you want to evaluate the series: ')
    print_series(int(n), float(x_value))
```

```python
# P104: Expression multiplier

'''
Product of two expressions
'''

from sympy import expand, sympify
from sympy.core.sympify import SympifyError
def product(expr1, expr2):
    prod = expand(expr1*expr2)
    print(prod)

if __name__=='__main__':
    expr1 = input('Enter the first expression: ')
    expr2 = input('Enter the second expression: ')
    try:
        expr1 = sympify(expr1)
        expr2 = sympify(expr2)
    except SympifyError:
        print('Invalid input')
    else:
        product(expr1, expr2)

```

```python
#P105: Solving a linear equation
>>> from sympy import Symbol, solve
>>> x = Symbol('x')
>>> expr = x - 5 - 7
>>> solve(expr)
```

```python
#P106: Solving a quadratic equation
>>> from sympy import solve
>>> x = Symbol('x')
>>> expr = x**2 + 5*x + 4
>>> solve(expr, dict=True)
```

```python
#P106: Quadratic equation with imaginary roots
>>> from sympy import Symbol
>>> x=Symbol('x')
>>> expr = x**2 + x + 1
>>> solve(expr, dict=True)
```

```python
#P106/107: Solving for one variable in terms of others
>>> from sympy import Symbol, solve
>>> x = Symbol('x')
>>> a = Symbol('a')
>>> b = Symbol('b')
>>> c = Symbol('c')
>>> expr = a*x*x + b*x + c
>>> solve(expr, x, dict=True)
```

```python
#P107: Express s in terms of u, a, t
>>> from sympy import Symbol, solve, pprint
>>> s = Symbol('s')
>>> u = Symbol('u')
>>> t = Symbol('t')
>>> a = Symbol('a')
>>> expr = u*t + (1/2)*a*t*t - s
>>> t_expr = solve(expr,t, dict=True)
>>> t_expr
```

```python
#P108: Solve a system of Linear equations
>>> from sympy import Symbol
>>> x = Symbol('x')
>>> y = Symbol('y')
>>> expr1 = 2*x + 3*y - 6
>>> expr2 = 3*x + 2*y - 12
>>> solve((expr1, expr2), dict=True)
```

```python
#P109: Simple plot with SymPy
>>> from sympy.plotting import plot
>>> from sympy import Symbol
>>> x = Symbol('x')
>>> plot(2*x+3)
```

```python
#P110: Plot in SymPy with range of x as well as other attributes specified
>>> from sympy import plot, Symbol
>>> x = Symbol('x')
>>> plot(2*x + 3, (x, -5, 5), title='A Line', xlabel='x', ylabel='2x+3')
```

```python
#P112: Plot the graph of an input expression
'''
Plot the graph of an input expression
'''
from sympy import Symbol, sympify, solve
from sympy.plotting import plot

def plot_expression(expr):
    y = Symbol('y')
    solutions = solve(expr, y)
    expr_y = solutions[0]
    plot(expr_y)

if __name__=='__main__':
    expr = input('Enter your expression in terms of x and y: ')
    try:
        expr = sympify(expr)
    except SympifyError:
        print('Invalid input')
    else:
        plot_expression(expr)
```

```python
#P113: Plotting multiple functions
>>> from sympy.plotting import plot
>>> from sympy import Symbol
>>> x = Symbol('x')
>>> plot(2*x+3, 3*x+1)
```

```python
#P114: Plot of the two lines drawn in a different color
>>> from sympy.plotting import plot
>>> from sympy import Symbol
>>> x = Symbol('x')
>>> p = plot(2*x+3, 3*x+1, legend=True, show=False)
>>> p[0].line_color = 'b'
>>> p[1].line_color = 'r'
>>> p.show()
```

```python
#P116: Example of summing a series
>>> from sympy import Symbol, summation, pprint
>>> x = Symbol('x')
>>> n = Symbol('n')
>>> s = summation(x**n/n, (n, 1, 5))
>>> s.subs({x:1.2})
```

    3.51206400000000

```python
#P117: Example of solving a polynomial inequality
>>> from sympy import Poly, Symbol, solve_poly_inequality
>>> x = Symbol('x')
>>> ineq_obj = -x**2 + 4 < 0
>>> lhs = ineq_obj.lhs
>>> p = Poly(lhs, x)
>>> rel = ineq_obj.rel_op
>>> solve_poly_inequality(p, rel)
```

    [(-oo, -2), (2, oo)]

```python
#P118: Example of solving a rational inequality
>>> from sympy import Symbol, Poly, solve_rational_inequalities
>>> x = Symbol('x')
>>> ineq_obj = ((x-1)/(x+2)) > 0
>>> lhs = ineq_obj.lhs
>>> numer, denom = lhs.as_numer_denom()
>>> p1 = Poly(numer)
>>> p2 = Poly(denom)
>>> rel = ineq_obj.rel_op
>>> solve_rational_inequalities([[((p1, p2), rel)]])
```

    (-oo, -2) U (1, oo)

```python
#P118: Solve a non-polynomial inequality
>>> from sympy import Symbol, solve, solve_univariate_inequality, sin
>>> x = Symbol('x')
>>> ineq_obj = sin(x) - 0.6 > 0
>>> solve_univariate_inequality(ineq_obj, x, relational=False)
```

    (0.643501108793284, 2.49809154479651)