# Finding the maximum value of 2d function
Given equation:
\begin{align}
f(x, y) = 2xy + 2x - x^2 -2y^2
\end{align}

### Implementation of needed functions:

```python
# Importing dependency functions and packages
from random import random
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d.axes3d import Axes3D, get_test_data
from matplotlib import cm
import numpy as np
%matplotlib notebook

# function
def f(x, y):
    return 2*x*y + 2*x - x**2 - 2*y**2

# random search algorithm
def random_search(f, n, xl, xu, yl, yu):
#     generate lists of random x and y values
    x_cand = [xl + (xu - xl)*random() for _ in range(n)]
    y_cand = [yl + (yu - yl)*random() for _ in range(n)]

#     calculate appropriate to x and y values function values
    poss_max = [f(x, y) for x, y in zip(x_cand, y_cand)]

#     finding index of maximum value (argmax)
    max_val = max(poss_max)
    max_indexes = [i for i, j in enumerate(poss_max) if j == max_val]

#     return maximum value of function and its parameters x, y
    return max_val, x_cand[max_indexes[0]], y_cand[max_indexes[0]]

# simple rms error function
def rms_error(x_ref, y_ref, x, y):
    return np.sqrt(((x_ref - x)/x_ref)**2 + ((y_ref - y)/y_ref)**2)
```

The problem is in guessing limits of parameters (x, y) for random number generation. The smaller limits are, the bigger accuracy we will get. Let's visualize function and judge from there.

```python
fig = plt.figure(figsize=plt.figaspect(0.5))
ax = fig.add_subplot(1, 1, 1, projection='3d')
x = np.arange(-100, 100, 5)
y = np.arange(-100, 100, 5)
x, y = np.meshgrid(x, y)
z = f(x, y)
surf = ax.plot_surface(x, y, z, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
fig.colorbar(surf, shrink=0.5, aspect=10)
plt.show()
```

    <IPython.core.display.Javascript object>

From above 3D plot it is seen that solution lies between -50 and 50 for both parameters x, y.

```python
f_ref, x_ref, y_ref = random_search(f, 10000000, -50, 50, -50, 50)
print('Using 10000000 random points and ranges between -50 and 50, it is found that maximum value of given function is:', f_ref)
```

    Using 10000000 random points and ranges between -50 and 50, it is found that maximum value of given function is: 1.999732917847197

The roots found using random search method using 10 million points and considered as reference roots and as ideal. This is done in order to determine root-mean-square error somehow and find dependency of error to number of random points. Since reference roots are found using 10 million random points, it is more accurate than roots found below for error plotting reasons.

```python
error_list = []
for n in np.logspace(1, 6, num=20):
    _, x, y = random_search(f, int(n), -50, 50, -50, 50)
    error_list += [rms_error(x_ref, y_ref, x, y)]
plt.semilogx(np.logspace(1, 6, num=20), error_list, '-')
plt.grid(which='both')
plt.xlabel('Number of random points')
plt.ylabel('RMS error of x and y roots')
plt.show()
```

    <IPython.core.display.Javascript object>

From error vs #of_points plot it is seen that line have linear dependency on number of random points (Exponential shape on log-scale). This confirms the fact that x and y values are generated uniformly and increased number of random points also increases accuracy linearly, distributing on 2d space equally.

```python

```


# Ens'IA - Session 2 - Intro to neural networks 1/2

Welcome the **second session** of Ens'IA.
Today, things are gonna get interesting !
We are gonna focus on **neural networks**
But first, (what a surprise), neural networks are make of neurons ! So what is a neuron ?

To make sure you understand what it is, we will go back in 1958 and give a look at **the perceptron**.

During the oral presentation, you should have seen that a perceptron have one or more inputs and a unique output.
The output is given by :
\begin{equation}
    s = \left\{
        \begin{array}{ll}
            1 & \mbox{if } \sum_{i=0}^{n} a_{i} \times w_{i} + b > 0 \\
            0 & \mbox{otherwise}
        \end{array}
    \right.
\end{equation}

Now, it's your time to *create your own perceptron*.

### Perceptron implementation

```
class Perceptron:

  """
  Build of a perceptron
  weights : List of weights
  bias : The bias.
  """
  def __init__(self,weights,bias):
    #TODO
    self.bias = bias
    self.weights = weights

  """
  Function called when you want to get the output from the input
  input : List of input values.
  """
  def forward(self,input):
    assert(len(input)==len(self.weights))
    #TODO
    sum = 0
    for inp,w in zip(input,self.weights):
      sum += inp*w
    if sum + self.bias > 0:
      return 1
    else:
      return 0
```

### Test

Now, let's **test** it !

```
#TODO
perceptron = Perceptron([1,1],-1)
assert(perceptron.forward([1,1])==1)
assert(perceptron.forward([1,0])==0)
assert(perceptron.forward([0,1])==0)
assert(perceptron.forward([0,0])==0)
```

If you have no message, then it must work!
Here we have created a perceptron with weights 1 and 1 and the bias is 2.
Do you notice any link between the inputs and the output? Maybe something you have seen in your processor architecture class...

### NAND implementation

Your next mission will be to create a perceptron that reproduces a NAND gate. It will have 2 inputs and a bias, but you have to find which values are the right ones ...

```
perceptron_nand = Perceptron([-2,-2],3)
assert(perceptron_nand.forward([1,1])==0)
assert(perceptron_nand.forward([1,0])==1)
assert(perceptron_nand.forward([0,1])==1)
assert(perceptron_nand.forward([0,0])==1)
#If you don't get any error messages when running this code, then you have found the right weights and bias!
```

### XOR implementation

And if you now try to find a perceptron that reproduces an XOR gate...?

```
#TODO
#Spoil : It's impossible :p
```


```python
from sympy import *
from sympy.stats import *
from sympy.utilities.lambdify import *
import math
```

```python
x, a, b = var("x a b")
f = (x-3)/2
i = integrate(f, (x, a, b))
i
```

$\displaystyle - \frac{a^{2}}{4} + \frac{3 a}{2} + \frac{b^{2}}{4} - \frac{3 b}{2}$

```python
i.subs({a: 3, b: 5}) # area da curva deve ser 1
```

$\displaystyle 1$

```python
i.subs({a: 3.3, b: 4}) # medindo a probabilidade
```

$\displaystyle 0.2275$

```python
mu = integrate(x*f, (x, a, b)) # media
mu
```

$\displaystyle - \frac{a^{3}}{6} + \frac{3 a^{2}}{4} + \frac{b^{3}}{6} - \frac{3 b^{2}}{4}$

```python
Var = integrate((x**2)*f) - mu**2 # varianca
Var
```

$\displaystyle \frac{x^{4}}{8} - \frac{x^{3}}{2} - \left(- \frac{a^{3}}{6} + \frac{3 a^{2}}{4} + \frac{b^{3}}{6} - \frac{3 b^{2}}{4}\right)^{2}$

```python
x = var('x')
f = Piecewise(((1/40)*(x-4), And(Ge(x, 8), Lt(x, 10))), (3/20, And(Ge(x, 10), Le(x, 15))), (0, True))
f
```

$\displaystyle \begin{cases} 0.025 x - 0.1 & \text{for}\: x \geq 8 \wedge x < 10 \\0.15 & \text{for}\: x \geq 10 \wedge x \leq 15 \\0 & \text{otherwise} \end{cases}$

```python
mu = integrate(x*f, (x, a, b))
mu.subs({ a: 8, b: 15})
```

$\displaystyle 11.6416666666667$

```python
E2 = integrate((x**2)*f, (x, a, b))
Var = E2 - (mu**2)
Var.subs({ a: 8, b: 15})
```

$\displaystyle 3.85493055555557$

```python
integrate(f, (x, a, b)).subs({ a: 9, b: 12 })
```

$\displaystyle 0.4375$

```python
integrate(f, (x, a, b)).subs({ a: 8, b: 14 }) / integrate(f, (x, a, b)).subs({ a: 8, b: oo })
```

$\displaystyle 0.850000000000001$


# BATEMAN’S EQUATIONS: CHAIN OF DECAYS OF 3 NUCLEAR SPECIES

## Import Libraries

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams; rcParams["figure.dpi"] = 300
from matplotlib.ticker import (AutoMinorLocator)
plt.style.use('seaborn-bright')
plt.rc('font', family='serif')
plt.rc('xtick', labelsize='x-small')
plt.rc('ytick', labelsize='x-small')
```

You will be examining properties of the Bateman equations that govern the decay of multiple nuclear species. While the problem is exactly solvable in iterative form we will consider it as a coupled system of equations, to determine the co-evolution of all species. It is well known that if there is only one nuclear species A the number of radioactive decays is proportional to the number of radioactive nuclei, $N_A$, i.e., the species evolves according to the ordinary differential equation (ODE).

$$\begin{equation}
\tag{1}
\begin{split}
\frac{dN_A}{dt} &= -\lambda_{A}N_A
\end{split}
\end{equation}$$

where $\lambda_A$ is related to the half-life of the species $A_{t_{1/2},A} = ln(2/\lambda_A)$

Now consider the case of a chain of two decays: one nucleus A decays into another B by one process, then B decays
into another C by a new process. The previous equation cannot be applied to the decay chain. Since A decays into B, and B decays into C, the activity of A adds to the total number of B nuclei. Therefore, the number of second generation nuclei B increases as a result of the decay of first generation A nuclei, and decreases as a result of its own decay into the third generation nuclei C, thus, the B species evolves as,

$$\begin{equation}
\tag{2}
\begin{split}
\frac{dN_B}{dt} &= -\lambda_{B}N_B + \lambda_{A}N_A
\end{split}
\end{equation}$$

We will now treat a more interesting possibility, where we have 3 radioactive species, the first generation A decays into second generation B and C. For example, $^{40}K$ has a 89.3% probability of decaying to $^{40}Ca$, and 10.7% to $^{40}Ar$. To make the problem more fun we will add the possibility that species B can, too, decay into species C. We will also consider that C decays into stable nuclei D. Based on the above considerations we have:

$$\begin{equation}
\tag{3}
\begin{split}
\frac{dN_A}{dt} &= -\lambda_{A}N_A \\
\frac{dN_B}{dt} &= -\lambda_{B}N_B + \lambda_{A,B}N_A \\
\frac{dN_C}{dt} &= -\lambda_{C}N_C + \lambda_{A,C}N_A + \lambda_{B}N_B \\
\frac{dN_D}{dt} &= \lambda_{C}N_C
\end{split}
\end{equation}$$

where $\lambda_{A,B} + \lambda_{A,C} = \lambda_{A}$, and $\lambda_{A,B}/\lambda_{A,C}$ must equal the ratio of the probability that A will decay into B to the probability that A will decay into C.

This system of equations has an integral (constant) of the motion,

$$\begin{equation}
\tag{4}
\begin{split}
N = N_A + N_B + N_C + N_D
\end{split}
\end{equation}$$

which is the total number of nuclei and is determined by the initial conditions.

The system (3) has multiple timescales involved, and hence it is not possible to completely non-dimensionalize it.
In addition, use N to introduce normalized species numbers $\tilde N_{i} = N_{i}/N$, i = A,B,C,D, so that the final equations in dimensionless time describe the evolution of the fraction of each species in an initial sample. This way the constant of the motion becomes,

$$\begin{equation}
\tag{5}
\begin{split}
 \tilde N_{A} + \tilde N_{B} + \tilde N_{C} + \tilde N_{D} = \tilde N = 1
\end{split}
\end{equation}$$

This constant $\tilde N$ will allow you to validate the quality of the numerical integration of Eq. (3).

Show in your term paper the derivation of the the normalized and dimensionless version of (3). To solve this normalized and dimensionless version of (3) you will need to specify initial conditions. Do the following:

### ***QUESTIONS***

**1)** Use RK4 to integrate numerically the dimensionless and normalized version the ODE (3) from t = 0 forward in time and for sufficiently large t until the populations of each species settles. You will need to be plotting $\tilde N_{i}$ vs $t$ to see if the solution is settling and to determine when to stop the integration. Run a couple of numerical experiments with different parameters to test the dynamics of the species populations.

Use your judgement as to how small a step size you need to solve this system accurately. If you cannot figure this out from pure thought, experiment with different step sizes and use $\delta \tilde N = |(\tilde N(t) - \tilde N(t=0))/\tilde N(t=0)|$ to determine this accuracy. If $\delta\tilde N$ is smaller than $10^{-3}$ for all integration times, then you have a decent accuracy.

**1)** Use RK4 to integrate numerically the dimensionless and normalized version the ODE (3) from t = 0 forward in time and for sufficiently large t until the populations of each species settles. You will need to be plotting $\tilde N_{i}$ vs $t$ to see if the solution is settling and to determine when to stop the integration. Run a couple of numerical experiments with different parameters to test the dynamics of the species populations.

## Tools for Numerical Integration

```python
# Define the RK4 Step (Taken from Class)
def RK4(RHS,y0,t,h,*P):
    """
    Implements a single step of a fourth-order, explicit Runge-Kutta scheme
    """
    thalf = t + 0.5*h
    k1 = h*RHS(y0, t, *P)
    k2 = h*RHS(y0+0.5*k1, thalf, *P)
    k3 = h*RHS(y0+0.5*k2, thalf, *P)
    k4 = h*RHS(y0+k3, t+h, *P)
    return y0 + (k1 + 2*k2 + 2*k3 + k4)/6

# Define the ODESolver (Taken from Class)
def odeSolve(t0, y0, tmax, h, RHS, method, *P):
    """
    ODE driver with constant step-size, allowing systems of ODE's
    """
    # make array of times and find length of array
    t = np.arange(t0,tmax+h,h)
    ntimes,  = t.shape
    # find out if we are solving a scalar ODE or a system of ODEs, and allocate space accordingly
    if type(y0) in [int, float]:  # check if primitive type -- means only one eqn
        neqn = 1
        y = np.zeros(ntimes)
    else:                         # otherwise assume a numpy array -- a system of more than one eqn
        neqn, = y0.shape
        y = np.zeros((ntimes, neqn))
    # set first element of solution to initial conditions (possibly a vector)
    y[0] = y0
    # march on...
    for i in range(0,ntimes-1):
        y[i+1] = method(RHS,y[i],t[i],h,*P)
    return t,y
```

## RHS for the 4 Coupled Autonomous Linear ODEs

```python
# Define the RHS
def nuclear_species_RHS(y,t,*P):
    lambda_B, lambda_AB, lambda_C, lambda_AC = P ## Unpack Parameters
    NA = y[0]
    NB = y[1]
    NC = y[2]
    ND = y[3]
    dNA_dt = -NA
    dNB_dt = -lambda_B*NB + lambda_AB*NA
    dNC_dt = -lambda_C*NC + lambda_AC*NA + lambda_B*NB
    dND_dt = lambda_C*NC
    array = np.array([dNA_dt, dNB_dt, dNC_dt, dND_dt])
    return array
```

### CASE 1: Species A decays very SLOWLY
***
First, consider the case where the A species decays very slowly and test whether it is possible to run out of B and C nuclei. For this experiment you will set $\lambda_{B}/\lambda_{A}$ = 5, $\lambda_{C}/\lambda_{A}$ = 10, $\lambda_{A,B}/\lambda_{A}$ = 0.85 and $\lambda_{A,C}/\lambda_{A}$ = 0.15. Consider initial conditions $\tilde N_{A}$ = 0.5, $\tilde N_{B}$ = 0.25, $\tilde N_{C}$ = 0.1, $\tilde N_{D}$ = 0.15.

Show plots of your solution for $\tilde N_{i}$ v/s t.

**QUESTION:** Does this evolution eliminate the species A and B completely?

```python
# Initial Conditions
t0 = 0.0
y0 = np.array([0.5, 0.25, 0.1, 0.15])
tmax = 10
h = 0.0001

# Parameters
lambda_B = 5.0
lambda_AB = 0.85
lambda_C = 10.0
lambda_AC = 0.15

# Solve the IVP
t,y = odeSolve(t0, y0, tmax, h, nuclear_species_RHS, RK4, lambda_B, lambda_AB, lambda_C, lambda_AC)

# dN Analysis
Nm = np.array([y[:,0],y[:,1],y[:,2],y[:,3]])
N = Nm.sum(axis=0)
Ne = N - 1
```

```python
# Plot Normalized Values v/s Dimensionless Time
f,a = plt.subplots()
a.plot(t,y[:,0],'r', label=r'$\tilde N_{A}$')
a.plot(t,y[:,1],'g', label=r'$\tilde N_{B}$')
a.plot(t,y[:,2],'b', label=r'$\tilde N_{C}$')
a.plot(t,y[:,3],'k', label=r'$\tilde N_{D}$')
a.set_xlabel(r'Dimensionless Time ($t$)')
a.set_ylabel(r'Normalized Values ($\tilde N_{i}$)')
a.set_title(r'$\tilde N_{i}$ v/s $t$: Species A decays VERY SLOWLY', fontweight='bold')
a.xaxis.set_minor_locator(AutoMinorLocator())
a.yaxis.set_minor_locator(AutoMinorLocator())
a.tick_params(which='minor', length=2.5, color='k')
a.legend()
a.grid()
plt.tight_layout()
plt.show()
```

```python
# Plot dN v/s Dimensionless Time
f,a = plt.subplots()
a.plot(t,Ne,'k')
a.set_xlabel(r'Dimensionless Time ($t$)')
a.set_ylabel(r'$\delta \tilde N$')
a.set_title(r'$\delta \tilde N$ v/s $t$: Species A decays VERY SLOWLY', fontweight='bold')
a.xaxis.set_minor_locator(AutoMinorLocator())
a.yaxis.set_minor_locator(AutoMinorLocator())
a.tick_params(which='minor', length=2.5, color='k')
a.grid(linestyle=':')
plt.tight_layout()
plt.show()
```

### CASE 2 : Species A decays very RAPIDLY
***
Second, consider the case where the A species decays very rapidly. For this experiment you will set $\lambda_{B}/\lambda_{A}$ = 0.05, $\lambda_{C}/\lambda_{A}$ = 0.1, $\lambda_{A,B}/\lambda_{A}$ = 0.85 and $\lambda_{A,C}/\lambda_{A}$ = 0.15. Consider initial conditions $\tilde N_{A}$ = 0.5, $\tilde N_{B}$ = 0.25, $\tilde N_{C}$ = 0.1, $\tilde N_{D}$ = 0.15.

Show plots of your solution for $\tilde N_{i}$ v/s t.

**QUESTION:** How is this evolution different from the previous one?

```python
# Initial Conditions
t0 = 0.0
y0 = np.array([0.5, 0.25, 0.1, 0.15])
tmax = 120
h = 0.0001

# Parameters
lambda_B = 0.05
lambda_AB = 0.85
lambda_C = 0.1
lambda_AC = 0.15

# Solve the IVP
t,y = odeSolve(t0, y0, tmax, h, nuclear_species_RHS, RK4, lambda_B, lambda_AB, lambda_C, lambda_AC)

# dN Analysis
Nm = np.array([y[:,0],y[:,1],y[:,2],y[:,3]])
N = Nm.sum(axis=0)
Ne = N - 1
```

```python
# Plot Normalized Values v/s Dimensionless Time
f,a = plt.subplots()
a.plot(t,y[:,0],'r', label=r'$\tilde N_{A}$')
a.plot(t,y[:,1],'g', label=r'$\tilde N_{B}$')
a.plot(t,y[:,2],'b', label=r'$\tilde N_{C}$')
a.plot(t,y[:,3],'k', label=r'$\tilde N_{D}$')
a.set_xlabel(r'Dimensionless Time ($t$)')
a.set_ylabel(r'Normalized Values ($\tilde N_{i}$)')
a.set_title(r'$\tilde N_{i}$ v/s $t$: Species A decays VERY RAPIDLY', fontweight='bold')
a.xaxis.set_minor_locator(AutoMinorLocator())
a.yaxis.set_minor_locator(AutoMinorLocator())
a.tick_params(which='minor', length=2.5, color='k')
a.legend()
plt.tight_layout()
plt.show()
```

```python
# Plot dN v/s Dimensionless Time
f,a = plt.subplots()
a.plot(t,Ne,'k')
a.set_xlabel(r'Dimensionless Time ($t$)')
a.set_ylabel(r'$\delta \tilde N$')
a.set_title(r'$\delta \tilde N$ v/s $t$: Species A decays VERY RAPIDLY', fontweight='bold')
a.xaxis.set_minor_locator(AutoMinorLocator())
a.yaxis.set_minor_locator(AutoMinorLocator())
a.tick_params(which='minor', length=2.5, color='k')
a.grid(linestyle=':')
plt.tight_layout()
plt.show()
```

```python
### CONVERGENCE ###

# Initial Conditions
t0 = 0.0
y0 = np.array([0.5, 0.25, 0.1, 0.15])
tmax = 120
h=np.array([1e2,1,0.75,0.6,0.5,0.3,0.1,0.05,0.01,0.005])

# Parameters
lambda_B = 0.05
lambda_AB = 0.85
lambda_C = 0.1
lambda_AC = 0.15

# Solve the IVP and dN Analysis
for i in range(len(h)):
    t,y = odeSolve(t0, y0, tmax, h[i], nuclear_species_RHS, RK4, lambda_B, lambda_AB, lambda_C, lambda_AC)
    Nm = np.array([y[:,0],y[:,1],y[:,2],y[:,3]])
    N = Nm.sum(axis=0)
    Ne = N - 1

f,a = plt.subplots()
a.plot(h[0],Ne[int(3.0/h[0])],'b.')
a.plot(h[1],Ne[int(3.0/h[1])],'b.')
a.plot(h[2],Ne[int(3.0/h[2])],'b.')
a.plot(h[3],Ne[int(3.0/h[3])],'b.')
a.plot(h[4],Ne[int(3.0/h[4])],'b.')
a.plot(h[5],Ne[int(3.0/h[5])],'b.')
a.plot(h[6],Ne[int(3.0/h[6])],'b.')
a.plot(h[7],Ne[int(3.0/h[7])],'b.')
a.plot(h[8],Ne[int(3.0/h[8])],'b.')
a.plot(h[9],Ne[int(3.0/h[9])],'b.')
a.set_xlabel('h')
a.set_ylabel(r'$\delta\~N(\~t=3)$')
a.set_title('Constant of Motion Error for Varying Step Sizes',fontsize=18)
a.set_xscale('log')
plt.show()
plt.tight_layout()
```

### CASE 3: Species B remains CONSTANT
***
Third, the system of equations has an "equilibrium" point for the B species, when $\lambda_{B} \tilde N_{B}/ \lambda_{A} = \lambda_{A,B} \tilde N_{b}/\lambda_{A}$, because then $\frac{d\tilde N_{B}}{dt} = 0$, which implies that the number of B species remain constant, and the B decay rate is balanced by the replenishment of B from the decay of A species. You can now study if this "equilibrium" is stable, by considering initial conditions that satisfy it. We will keep the same $\lambda_{A,B}/\lambda_{A}$ = 0.85 and $\lambda_{A,C}/\lambda_{A}$ = 0.15, and initial conditions $\tilde N_{A}$ = 0.5, $\tilde N_{B}$ = 0.25, $\tilde N_{C}$ = 0.1, $\tilde N_{D}$ = 0.15. The condition $\lambda_{B} \tilde N_{B}/ \lambda_{A} = \lambda_{A,B} \tilde N_{b}/\lambda_{A}$ implies $\lambda_{B}/\lambda_{A}$ = 1.7. And set again $\lambda_{C}/\lambda_{A}$ = 0.1.

Show plots of your solution for $\tilde N_{i}$ v/s t.

**QUESTION:** Does the population of B species remain constant?

```python
# Initial Conditions
t0 = 0.0
y0 = np.array([0.5, 0.25, 0.1, 0.15])
tmax = 60
h = 0.0001

# Parameters
lambda_B = 1.7
lambda_AB = 0.85
lambda_C = 0.1
lambda_AC = 0.15

# Solve the IVP
t,y = odeSolve(t0, y0, tmax, h, nuclear_species_RHS, RK4, lambda_B, lambda_AB, lambda_C, lambda_AC)

# dN Analysis
Nm = np.array([y[:,0],y[:,1],y[:,2],y[:,3]])
N = Nm.sum(axis=0)
Ne = N - 1
```

```python
# Plot Normalized Values v/s Dimensionless Time
f,a = plt.subplots()
a.plot(t,y[:,0],'r', label=r'$\tilde N_{A}$')
a.plot(t,y[:,1],'g', label=r'$\tilde N_{B}$')
a.plot(t,y[:,2],'b', label=r'$\tilde N_{C}$')
a.plot(t,y[:,3],'k', label=r'$\tilde N_{D}$')
a.set_xlabel(r'Dimensionless Time ($t$)')
a.set_ylabel(r'Normalized Values ($\tilde N_{i}$)')
a.set_title(r'$\tilde N_{i}$ v/s $t$: Species B remains CONSTANT', fontweight='bold')
a.xaxis.set_minor_locator(AutoMinorLocator())
a.yaxis.set_minor_locator(AutoMinorLocator())
a.tick_params(which='minor', length=2.5, color='k')
a.legend()
plt.tight_layout()
plt.show()
```

```python
# Plot dN v/s Dimensionless Time
f,a = plt.subplots()
a.plot(t,Ne,'k')
a.set_xlabel(r'Dimensionless Time ($t$)')
a.set_ylabel(r'$\delta \tilde N$')
a.set_title(r'$\delta \tilde N$ v/s $t$: Species B remains CONSTANT', fontweight='bold')
a.xaxis.set_minor_locator(AutoMinorLocator())
a.yaxis.set_minor_locator(AutoMinorLocator())
a.tick_params(which='minor', length=2.5, color='k')
a.grid(linestyle=':')
plt.tight_layout()
plt.show()
```

```python

```


GRADIENT DESCENT

Gradient descent is an optimization algorithm used to minimize some function by iteratively moving in the direction of steepest descent as defined by the negative of the gradient. In machine learning, we use gradient descent to update the parameters of our model. Parameters refer to coefficients in Linear Regression and weights in neural networks. (Source. Khan Academy). Gradient descent is best used when the parameters cannot be calculated analytically (e.g. using linear algebra) and must be searched for by an optimization algorithm.

# Task

This is the first an only project not related to the coffee data base. In this project, the idea is to implement the Gradient descent algorithm from scratch without using the code used in class. The algorithm is tested for a one-dimensional and two dimensional functions differentiable in a specific numerical range.

# One dimensional function

The function $ x^3-3x^2 + 5 $ is tested. This function is well behaved, with a minimum each time x = y. Furthermore, it has not got many different local minimum which can be problematic for gradient descent algorithm.

```python
from numpy import asarray
from numpy import arange
from numpy.random import rand
import matplotlib.pyplot as plt

```

```python
def objective(x):
    return x**3 - 3*x**2 + 5

# derivative of objective function
def derivative(x):
    return 3*x**2 - 6*x

# gradient descent algorithm
#def gradient_descent(objective, derivative, bounds, n_iter, step_size, initial_point, precision):
def gradient_descent(objective, derivative, step_size, initial_point, precision):
    # track all solutions
    solutions, scores = list(), list()
    gradient_new = initial_point
    gradient_old = 0
    counter=0

    while (gradient_new- gradient_old) > precision:
        counter=counter+1
        gradient_old= gradient_new
        gradient_new = gradient_old - (step_size * derivative(gradient_old))
        # evaluate candidate point
        solution_eval = objective(gradient_new)
        # store solution
        solutions.append(gradient_new)
        scores.append(solution_eval)
        # report progress
      #  print('Interaction %d f(%s) = %.5f' % (counter, round (gradient_new, 5), solution_eval))
    return solutions, scores, counter
```

### Implementing the gradient descent in the 2-d function

```python
# define range for input
bounds = asarray([[-1.0, 3.0]])
# define the total iterations
#n_iter = 15
# define the step size
step_size = 0.3
initial_point= 0.01
precision= 0.0001
# perform the gradient descent search
solutions, scores, counter = gradient_descent(objective, derivative, step_size, initial_point, precision)
# sample input range uniformly at 0.1 increments
inputs = arange(bounds[0,0], bounds[0,1]+0.1, 0.1)
# compute targets
results = objective(inputs)
plt.figure(figsize=[10,5])
# create a line plot of input vs result
plt.plot(inputs, results)
# plot the solutions found
plt.plot(solutions, scores, '.-', color='red')
plt.xlabel("w (weight)", fontsize = 15)
plt.ylabel("f(w)", fontsize = 15)
plt.title("Gradient descent", fontsize = 18)
plt.show()
# show the plot
```

### Step size sensitivity analysis
The efect of the step size for this particular function is analized as follow. We get the number of iteration for an step size from 0.1 to 0.8, with 0.01 increments. The inital point and precision is fixed. The result is presented in the following plot.

```python
step_size = np.arange(0.1, 0.8, 0.01)
initial_point= 0.01
precision= 0.0001
result = []
for i in (step_size):
# Gdetting the number of iteraction for each step
    solutions, scores, counter = gradient_descent(objective, derivative, i, initial_point, precision)
    result.append(counter)
plt.figure(figsize=[10,5])
plt.plot(step_size, result)
plt.xlabel("Increasing step", fontsize = 15)
plt.ylabel("Number of iterations", fontsize = 15)
plt.title("STEP SIZE ANALYSIS", fontsize = 18)
plt.show()

```

# Two dimensional Function
The function z= $x^2 + y^2 - 2xy$ is tested. This function is well behaved, with a minimum each time x = y. Furthermore, it has not got many different local minimums, which can be problematic for the gradient descent algorithm. I drew the 3D and contour plot for this function and subsequently calculated the gradient descent and the number of required interactions to satisfy a precision value.

```python
from numpy import exp,arange
from pylab import meshgrid,cm,imshow,contour,clabel,colorbar,axis,title,show
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

# the function I'm plotting
def z_func(x,y):
    return  x**2 + y**2 - 2*x*y

x = arange(-3.0,3.0,0.1)
y = arange(-3.0,3.0,0.1)

# grid of point
X,Y = meshgrid(x, y)
# evaluation of the function on the grid
Z = z_func(X, Y)

fig = plt.figure(figsize=[12,10])
ax = fig.gca(projection='3d')
surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1,
                      cmap=cm.RdBu,linewidth=0, antialiased=False)

ax.zaxis.set_major_locator(LinearLocator(8))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.title('3D PLOT $x^2+y^2-2xy$', fontsize = 18)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
plt.show()

```

###  Finding the result of interactions in the 3D function

```python
from sympy import *

x = Symbol('x')
y = Symbol('y')
z = Symbol('z')

f = x**2 + y**2 - 2*x*y
# First partial derivative with respect to x
fpx = f.diff(x)

# First partial derivative with respect to y
fpy = f.diff(y)

# Gradient
grad = [fpx,fpy]

# Data
theta = 10         #initial point x
theta1 = 3         #initial point y
step_size = 0.1
count = 0
precision = 0.00001
printData = True
maxIterations = 1000

while True:
    temptheta = theta - step_size*N(fpx.subs(x,theta).subs(y,theta1)).evalf()
    temptheta1 = theta1 - step_size*N(fpy.subs(y,theta1)).subs(x,theta).evalf()

    #If the number of iterations goes up too much, maybe theta is diverging. Stop, the function may not be convex
    count += 1
    if count > maxIterations:
        print("Adjust the step size and make sure that the function is convex")
        printData = False
        break

    # Verify the differece between the current value and presvious value is less than the precision, if yes, finish
    if abs(temptheta-theta) < precision and abs(temptheta1-theta1) < precision:
        break

    # Update
    theta = temptheta
    theta1 = temptheta1

if printData:
    print("The function "+str(f)+" converges to a minimum")
    print("Number of iterations:",count,sep=" ")
    print("Theta (x0) =",temptheta,sep=" ")
    print("Theta1 (y0) =",temptheta1,sep=" ")

```

    The function x**2 - 2*x*y + y**2 converges to a minimum
    Number of iterations: 25
    Theta (x0) = 6.50000995060081
    Theta1 (y0) = 6.49999004939919

### Result
The gradient descent finds a minimun starting at points 10 and 3, after 25 interactions, using a step size of 0.1. The minimal occurs in the point (6.5, 6.5) which can be verfiy in the 3D plot as well.


The goal of this notebok is to check the veracity of the following important information you may have seen on social networks

We will reproduce the previous equation, step by step as:
$$
\frac{\pi e^{\frac{A}{B}} - e^{i E}}{C D}
$$

```python
from sympy import *
init_printing(use_latex=True)
t = Symbol('t')
k = Symbol('k')
n = Symbol('n')
```

```python
A=Limit(2**(2*n) * factorial(n)**2 * log(7) / factorial(2*n) / sqrt(n), n, +oo)
A
```

```python
A.doit()
```

```python
B=Integral(exp(-t**2), (t,0,+oo))
B
```

```python
B.doit()
```

```python
C = Integral(3 / (t**6 + 1), (t,0,+oo))
C

```

```python
C.doit()
```

```python
D = Integral(exp(-pi*t**2), (t,-oo,+oo))
D
```

```python
D.doit()

```

```python
E=Sum( 8*pi/(4*k+1)/(4*k+3), (k,0,+oo))
E

```

```python
E.doit().simplify()
```

# Conclusion

```python
total=( pi*exp(A.doit()/B.doit()) - exp(I*(E).doit().simplify()) )/ C.doit() / D.doit()
total.simplify()
```

Which leads me to the conclusion the expression is false ! In fact, it seems that the sum expression shoud not have a $\pi$ in it, and that the both exponentials should be factorized by $\pi$.

The original web page with the mistake is http://www.brouty.fr/Maths/anniv.html.

```python
total2=( pi*(exp(A.doit()/B.doit()) - exp(I*(E/pi).doit().simplify()) )) / C.doit() / D.doit()
total2
```

notebooks/Checking Equations With SimPy.ipynb


# Elliptic PDE: Radially Symmetric Singular Solution
Copyright (C) 2010-2020 Luke Olson

Copyright (C) 2020 Andreas Kloeckner

MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

-----

Poisson Problem:
 Given open domain $\Omega \subset \mathbb{R}^2$
$$ -\nabla \cdot \nabla u = f(x)\quad \text{in}\, \Omega $$
$$ u = g(x)\quad \text{on}\, \partial\Omega $$
Let:
$$ f(x) = \delta(x) $$
$\delta(x)$ is the Dirac delta function. This problem describes a unit charge at the origin.

## Solution

Potential due to point charge:
$$ u(x,y) = -\frac{1}{2\pi}\ln(r) $$

 $r = \sqrt{x^2+y^2}$, the distance to the origin.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
import math

import sympy as sym
sym.init_printing()
```

## Set up Grid

```python
X = np.arange(-10, 10, 0.2)
Y = np.arange(-10, 10, 0.2)
X, Y = np.meshgrid(X, Y)
```

## Solution

```python
r = np.sqrt(X**2 + Y**2)
Z = -np.log(r)/(2*math.pi)
```

## Check Symbolically

```python
sx = sym.Symbol("x")
sy = sym.Symbol("y")
sr = sym.sqrt(sx**2 + sy**2)
ssol = sym.log(sr)

sym.simplify(sym.diff(ssol, sx, 2) + sym.diff(ssol, sy, 2))
```

## Plot

```python
fig = plt.figure(figsize=(8, 6))
ax = fig.add_subplot(111, projection='3d')
#ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm,
#                       linewidth=0, antialiased=False)
ax.plot_wireframe(X, Y, Z, linewidth=0.2)
#ax.set_zlim(-1.0, 1.0)
#plt.show()

```

Given $C\log(r)$ as the *free-space Green's function*, can we construct the solution to the PDE with a more general $f$?

demos/intro/Elliptic PDE Radially Symmetric Singular Solution.ipynb


```python
from sympy import *
from sympy.solvers.solveset import solveset
init_printing()
x, y, z = symbols('x,y,z')
```

## Solveset

La resolución de ecuaciones es una necesidad común y también un elemento común para algoritmos simbólicos más complicados.

Aquí presentamos la función `solveset`.

```python
solveset(x**2 - 4, x)
```

*Solveset* toma dos argumentos y un argumento opcional que especifica el dominio, una ecuación como $x^2 - 4$ y una variable sobre la cual queremos resolver, como $x$ y un argumento opcional *domain* que especifica la región en la que queremos resolver .

*Solveset* devuelve los valores de la variable, $x$, para los cuales la ecuación, $x^2 - 4$ es igual a 0.

### Ejercicio

¿Qué produciría el siguiente código? ¿Estás seguro?

```python
solveset(x**2 - 9 == 0, x)
```

## Soluciones infinitas

Una de las principales mejoras de `solveset` es que también admite una solución infinita.

```python
solveset(sin(x), x)
```

## Argumento *domain*

```python
solveset(exp(x) -1, x)
```

`solveset` por defecto resuelve todo en el dominio complejo. En el dominio complejo $\exp(x) = \cos(x) + i\sin(x)$ y la solución es básicamente igual a la solución de $\cos(x) = 1$. Si solo desea una solución real, puede especificar el dominio como `S.Reals`.

```python
solveset(exp(x) -1, x, domain=S.Reals)
```

`solveset` no siempre es capaz de resolver una ecuación dada, en tales casos devuelve un objeto` ConditionSet`. `ConditionSet` representa un conjunto que satisface una condición dada.

```python
solveset(exp(x) + cos(x) + 1, x, domain=S.Reals)
```

`solveset` tiene como objetivo devolver todas las soluciones de la ecuación. En los casos en que puede encontrar alguna solución pero no todas, devuelve una unión de las soluciones conocidas y `ConditionSet`.

```python
solveset((x - 1)*(exp(x) + cos(x) + 1), x, domain=S.Reals)
```

## Uso simbólico de `solveset`

Los resultados de `solveset` no necesitan ser numéricos, como `{-2, 2}`. Podemos usar solveset para realizar manipulaciones algebraicas. Por ejemplo, si conocemos una ecuación simple para el área de un rectángulo

    area = largo * ancho

podemos resolver esta ecuación para cualquiera de las variables. Por ejemplo, ¿cómo resolveríamos este sistema para `largo`, dado `area` y `ancho`?

```python
largo, ancho, area = symbols('largo, ancho, area')
solveset(area - largo*ancho, largo)
```

Ten en cuenta que nos hubiera gustado escribir

    solveset(area == largo * ancho, largo)

Pero el *gotcha* `==` nos muerde. En cambio, recordamos que `solveset` espera una expresión que sea igual a cero, por lo que reescribimos la ecuación

    area = largo * ancho

en la ecuación

    0 = largo * ancho - area

y eso es lo que le damos a `solveset`.

### Ejercicio

Calcula el radio de una esfera, dado el volumen. Recuerde, el volumen de una esfera de radio `r` está dado por

$$ V = \frac{4}{3}\pi r^3 $$

```python
# Resuelve para el radio de una esfera, dado el volumen
```

Probablemente obtendrás varias soluciones, eso está bien. La primera es probablemente la que quieres.

## Sustitución

A menudo queremos sustituir en una expresión por otra. Para esto, usamos el método de `subs`

```python
x**2
```

```python
# Sustituye x por y
(x**2).subs({x: y})
```

### Ejercicio

Sustituye $x$ por $\sin(x)$ en la ecuación $x^2 + 2\cdot x + 1$

```python
# Sustituye x por sin(x)

```

## Subs + Solveset

Podemos usar `subs` y `solveset` juntos para conectar la solución de una ecuación a otra

```python
# Resuelve para el largo de un rectangulo dada el area y el ancho

soln = list(solveset(area - largo*ancho, largo))[0]
soln
```

```python
# Define el perimetro de un rectangulo en terminos del largo y ancho

perimetro = 2*(largo + ancho)
```

```python
# Sustituye la solucion para el largo en la expresion para el perimetro

perimetro.subs({largo: soln})
```

### Ejercicio

En la última sección resolviste para el radio de una esfera dado su volumen

```python
V, r = symbols('V,r', positive=True)
4*pi/3 * r**3
```

```python
list(solveset(V - 4*pi/3 * r**3, r))[0]
```

Ahora vamos a calcular el área de una esfera en términos del volumen. Recuerde que el área de una esfera está dada por

$$ 4 \pi r^2 $$

```python
(?).subs(?)
```

¿La expresión se ve bien? ¿Cómo esperas que el área escale con respecto al volumen? ¿Cuál es el exponente en $V$?

## Trazado de gráficas

*Sympy* puede graficar expresiones fácilmente usando la función `plot`. Para esto, la biblioteca por defecto es  *matplotlib*.

```python
import matplotlib.pyplot as plt
```

```python
plot(x**2)
```

### Ejercicio

En el último ejercicio, obtuviste una relación entre el volumen de una esfera y su área. Grafica esta relación usando `plot`.

```python
plot(?)
```

## Mínimas dependencias

*SymPy* intenta ser un proyecto con pocas dependencia. Nuestra base de usuarios es muy amplia. Algunos aspectos entretenidos resultan. Por ejemplo, `textplot`.

```python
textplot(x**2, -3, 3)
```

### Ejercicio

Juega con `textplot` y disfruta :)


# Logistic Regression Single Neuron

In this notebook, we briefly introduce the logistic regression single neuron model, then apply it on the CVDs dataset.

---

## Introduce

[Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) is a statistical model that in its basic form uses a logistic function to model a binary dependent variable. In regression analysis, logistic regression is estimating the parameters of a logistic model.

Logistic regression is used in various fields, including machine learning, most medical fields, and social sciences.

---

## Algorithm

Instead of creating a single neuron model for predicting a class deterministic label, we will next build a single neuron model that predicts a *class probability*.

### Sigmoid activation function

As before, we must first decide on an activation function before deciding on a cost/ loss function. For this purpose, we choose the *sigmoid* activation function:

$$
\sigma(z) = \frac{1}{1 + e^{-z}}
$$

This differentiable function has a range in $(0, 1)$, so it would seem suitable for a possible function to turn the pre-activation value into a value representing a probability. Moreover, the sigmoid function (sometimes called the *logistic function*) has a smooth "S"-shape that is perfect for probabilities values transitioning, either growing or shrinking, as the input feature changes.

## Cross Entropy loss function

Suppose that we have two target values, 0 and 1. Now we are wishing to predict that *probability of each of these labels given a single feature measurement*. Thus, we encounter the conditional probability function:

$$
P\Big(y^{(i)}\mid x^{(i)}\Big)=\begin{cases}
          \hat{y}^{(i)}, \quad &  y^{(i)} = 1 \\
          1-\hat{y}^{(i)}, \quad & y^{(i)} = 0 \\
     \end{cases}
$$

Notice that this conditional probability depends on the value of $\hat{y}^{(i)}$, which in-turn depends on the values of our weight and bias. Moreover, we wish to *maximize* this probability over all training examples since this quantity is largest when our predicted probabilities are close approximations to the true 0-1 labels. Thus, we seek to solve the following maximization problem:

$$
\max_{\mathbf{w}, b} \sum_{i=1}^{N}P\Big(y^{(i)}\mid x^{(i)}\Big).
$$

Before considering this optimization problem, we next recall the famous Bernoulli formula for binary probabilities:
$$
P\Big(y^{(i)}\mid x^{(i)}\Big) = [\hat{y}^{(i)}]^{y}[1 - \hat{y}^{(i)}]^{(1-y)}
$$

Taking the logorithm on both sides of this equation yields (dropping the index notation to avoid messy equations):
$$
\begin{align}
\log P\Big(y^{(i)}\mid x^{(i)}\Big)&= \log \hat{y}^{y}(1 - \hat{y})^{(1-y)}\\
&= y\log \hat{y} + (1-y) \log (1 - \hat{y})\\
\end{align}
$$

Since the logorithmic function is an *increasing function*, maximimizing $P\Big(y^{(i)}\mid x^{(i)}\Big)$ is equivalent to maximizing $\log P\Big(y^{(i)}\mid x^{(i)}\Big)$. Equivalently, we could also considering minimizing this function. Thus, we arrive at our single neuron coss/loss function for a single entry of data, which implies a full loss function.

### Binary Cross Entropy Loss Function:
$$
L(\mathbf{w}, b) = -\frac{1}{N} \sum_{i=1}^{N} \log P\Big(y^{(i)}\mid x^{(i)}\Big) = \frac{1}{N}\sum_{i=1}^{N}\Big[ -y^{(i)}\log \hat{y}^{(i)} - (1-y^{(i)}) \log (1 - \hat{y}^{(i)})\Big ]
$$

### Calculuting the Gradient of Binary Cross Entropy Loss Function
In order to optimize the logistic regression single neuron model with stochastic gradient descent, we first need understand how to calculate the gradient. As before, we will consider the cost function on a single instance of data:

$$
C(w_1, b; x^{(i)},y^{(i)}) = -y^{(i)}\log \hat{y}^{(i)} - (1-y^{(i)}) \log (1 - \hat{y}^{(i)})
$$

When considering this equation it is important to remember that $\hat{y}^{(i)}$ really is a composite function. More specifically, we note

$$
\hat{y}^{(i)} = \sigma(z) = \sigma(w_1x^{(i)} + b).
$$

Next we note the particularly nice closed form of the derivative of the sigmoid function.

$$
\sigma'(z) = \sigma(z)(1 - \sigma(z))
$$

With these two equations, we are now ready to compute the partial derivatives of $C(w_1, b; x_{1}^{(i)},y^{(i)})$ with respect to $w_1$ and $b$. Note that this cost function contains two pieces, namely $-y^{(i)}\log \hat{y}^{(i)}$ and $- (1-y^{(i)}) \log (1 - \hat{y}^{(i)})$. Since the derivative is a linear map, we may calculate $\partial C/ \partial w_1$ by calculating the the derivative of each piece of this equation and then add them together.

$$
\begin{split}
\frac{\partial}{\partial w_1}[-y^{(i)}\log \hat{y}^{(i)}] & = \frac{\partial}{\partial w_1}[-y^{(i)}\log \sigma(w_1 x^{(i)}+b)] \\
 & = - \frac{y^{(i)}}{\sigma(w_1 x^{(i)}+b)}\frac{\partial}{\partial w_1} [\sigma(w_1 x^{(i)}+b)] \\
 & = - \frac{y^{(i)}}{\sigma(w_1 x^{(i)}+b)}\sigma(w_1 x^{(i)}+b)(1 - \sigma(w_1 x^{(i)}+b))\frac{\partial}{\partial w_1}[w_1 x^{(i)}+b] \\
 & = - y^{(i)}(1 - \sigma(w_1 x^{(i)}+b))x^{(i)} \\
 & = - y^{(i)}(1 - \hat{y}^{(i)})x^{(i)}
\end{split}
$$

$$
\begin{split}
\frac{\partial}{\partial w_1}[-(1-y^{(i)}) \log (1 - \hat{y}^{(i)})] & = \frac{\partial}{\partial w_1}[-(1-y^{(i)})\log (1 - \sigma(w_1 x^{(i)}+b))] \\
 & = - \frac{(1 - y^{(i)})}{(1 - \sigma(w_1 x^{(i)}+b))}\frac{\partial}{\partial w_1} [1 - \sigma(w_1 x^{(i)}+b) ]\\
 & = -  \frac{(1 - y^{(i)})}{(1 - \sigma(w_1 x^{(i)}+b))} -\sigma(w_1 x^{(i)}+b)(1 - \sigma(w_1 x^{(i)}+b))\frac{\partial}{\partial w_1}[w_1 x^{(i)}+b] \\
 & = (1 - y^{(i)})\sigma(w_1 x^{(i)}+b))x^{(i)} \\
 & = (1 - y^{(i)})\hat{y}^{(i)}x^{(i)}
\end{split}
$$

Now that we have calculated the derivative with respect to $w_1$ for each part of the binary cross entropy loss function, we next sum these derivatives:

$$
\begin{split}
\frac{\partial C(w_1, b; x^{(i)},y^{(i)})}{\partial w_1} & = - y^{(i)}(1 - \hat{y}^{(i)})x^{(i)} + (1 - y^{(i)})\hat{y}^{(i)}x^{(i)} \\
 & = [- y^{(i)}(1 - \hat{y}^{(i)}) + (1 - y^{(i)})\hat{y}^{(i)}]x^{(i)} \\
 & = [- y^{(i)} + y^{(i)}\hat{y}^{(i)} + \hat{y}^{(i)} - y^{(i)}\hat{y}^{(i)}]x^{(i)} \\
 & = (\hat{y}^{(i)} - y^{(i)}) x^{(i)}

\end{split}
$$

A similar calculation also yields the partial derivative of our cost function with respect to the bias $b$:

$$
\frac{\partial C(w_1, b; x^{(i)},y^{(i)})}{\partial b} = (\hat{y}^{(i)} - y^{(i)})
$$

---

## Coding

We will import the ```SingleNeuron``` Class from [Modules](https://github.com/YulinLi98/Sample_Repo/blob/main/Supervised_Learning/Modules) and define the sigmoid activation function and cross entropy loss function.

```python
# Import the libraries
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report

# Set theme for plotting
sns.set_theme()

import sys
sys.path.append('..')

# Import SingleNeuron Class
from Modules.single_neuron import SingleNeuron

# Define sigmoid activation function
def sigmoid(z):
    return 1.0/(1.0 + np.exp(-z))

# Define MSE cost function
def cross_entropy_loss(y_hat, y):
    return - y*np.log(y_hat) - (1 - y)*np.log(1 - y_hat)
```

We load the CVDs data set and preprocess the data.

```python
# Import the data
df = pd.read_csv("https://raw.githubusercontent.com/YulinLi98/Sample_Repo/main/heart.csv")

# Data Preprocessing by onehot encoding
df.Sex = df.Sex.replace({'M':1, 'F':0})
df.ExerciseAngina = df.ExerciseAngina.replace({'Y':1, 'N':0})

ChestPainType = pd.get_dummies(df.ChestPainType,drop_first=True)
RestingECG = pd.get_dummies(df.RestingECG,drop_first=True)
ST_Slope = pd.get_dummies(df.ST_Slope,drop_first=True)
df = pd.concat([df,ChestPainType, RestingECG, ST_Slope],axis=1)
df.drop(['ChestPainType', 'RestingECG', 'ST_Slope'],axis=1,inplace=True)

# Standardize the data
df.Age = preprocessing.scale(df.Age)
df.RestingBP = preprocessing.scale(df.RestingBP)
df.MaxHR = preprocessing.scale(df.MaxHR)
df.Cholesterol = preprocessing.scale(df.Cholesterol)
df.head()
```

      Age
      Sex
      RestingBP
      Cholesterol
      FastingBS
      MaxHR
      ExerciseAngina
      Oldpeak
      HeartDisease
      ATA
      NAP
      TA
      Normal
      ST
      Flat
      Up

      0
      -1.433140
      1
      0.410909
      0.825070
      0
      1.382928
      0
      0.0
      0
      1
      0
      0
      1
      0
      0
      1

      1
      -0.478484
      0
      1.491752
      -0.171961
      0
      0.754157
      0
      1.0
      1
      0
      1
      0
      1
      0
      1
      0

      2
      -1.751359
      1
      -0.129513
      0.770188
      0
      -1.525138
      0
      0.0
      0
      1
      0
      0
      0
      1
      0
      1

      3
      -0.584556
      0
      0.302825
      0.139040
      0
      -1.132156
      1
      1.5
      1
      0
      0
      0
      1
      0
      1
      0

      4
      0.051881
      1
      0.951331
      -0.034755
      0
      -0.581981
      0
      0.0
      0
      0
      1
      0
      1
      0
      0
      1

Then we extract the exploratory variables and response variable. We split the data into training set and testing set.

```python
X = df.drop('HeartDisease',axis=1).to_numpy()
y = df.HeartDisease

# Create a training set and a testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)
```

We instantiate an instance of logistic regression single neuron class with sigmoid function, cross entropy loss and SinleNeuron class. Then we fit the logit model on the training set.

```python
# Instantiate one instance of logistic regression single neuron class
np.random.seed(2)
node = SingleNeuron(sigmoid, cross_entropy_loss)

# Call the train method to train the weights and bias of the given instance
node.train(X_train, y_train, alpha=0.01, epochs=50)
```

    <Modules.single_neuron.SingleNeuron at 0x24721604250>

```python
node.plot_cost_function()
plt.show()
```

The cross entropy cost is decreasing over each epoch, meaning that our logit model is learning. Now let's make prediction on the testing set and check the confusion matrix and classification report.

```python
# Make predictions on testing data
y_pred = np.rint(node.predict(X_test))

# Calculate the confusion matrix
cf_matrix = confusion_matrix(y_test, y_pred)

# Print the confusion matrix
print(f"cf_matrix = {cf_matrix} \n")

# View the confusion matrix using the seaborn package
plt.figure(figsize = (10, 8))
ax = sns.heatmap(cf_matrix, annot=True, cmap='Blues', cbar=False)

ax.set_title('Seaborn Confusion Matrix with labels\n\n');
ax.set_xlabel('\nPredicted Values')
ax.set_ylabel('Actual Values ');

## Ticket labels - List must be in alphabetical order
ax.xaxis.set_ticklabels(['False','True'])
ax.yaxis.set_ticklabels(['False','True'])
plt.show()
```

```python
#Check performance of our model with classification report
print(classification_report(y_test, y_pred))
```

                  precision    recall  f1-score   support

               0       0.82      0.89      0.85       112
               1       0.92      0.87      0.89       164

        accuracy                           0.88       276
       macro avg       0.87      0.88      0.87       276
    weighted avg       0.88      0.88      0.88       276

The accuracy is 0.88, which is pretty high when compared with other classification models. Therefore, the logistic regression model performs well on our main data set. Now we will do an experiment by increasing the epoch number to 1_000.

```python
# increase the epochs number to 1000
np.random.seed(2)
node2 = SingleNeuron(sigmoid, cross_entropy_loss)
node2.train(X_train, y_train, alpha=0.01, epochs=1000)

node2.plot_cost_function()
plt.show()

# Make predictions on testing data
y_pred = np.rint(node2.predict(X_test))

# Calculate the confusion matrix
cf_matrix = confusion_matrix(y_test, y_pred)

# Print the confusion matrix
print(f"cf_matrix = {cf_matrix} \n")

# View the confusion matrix using the seaborn package
plt.figure(figsize = (10, 8))
ax = sns.heatmap(cf_matrix, annot=True, cmap='Blues', cbar=False)

ax.set_title('Seaborn Confusion Matrix with labels\n\n');
ax.set_xlabel('\nPredicted Values')
ax.set_ylabel('Actual Values ');

## Ticket labels - List must be in alphabetical order
ax.xaxis.set_ticklabels(['False','True'])
ax.yaxis.set_ticklabels(['False','True'])
plt.show()

#Check performance of our model with classification report
print(classification_report(y_test, y_pred))
```

Both the confusion matrix and classification report indicate that increasing the epoch number does not make significant improvement on the model performance. Now let's try performing logistic regression using scikit-learn.

```python
from sklearn.linear_model import LogisticRegression

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)
y_pred = log_reg.predict(X_test)

# Calculate the confusion matrix
cf_matrix = confusion_matrix(y_test, y_pred)

# Print the confusion matrix
print(f"cf_matrix = {cf_matrix} \n")

# View the confusion matrix using the seaborn package
plt.figure(figsize = (10, 8))
ax = sns.heatmap(cf_matrix, annot=True, cmap='Blues', cbar=False)

ax.set_title('Seaborn Confusion Matrix with labels\n\n');
ax.set_xlabel('\nPredicted Values')
ax.set_ylabel('Actual Values ');

## Ticket labels - List must be in alphabetical order
ax.xaxis.set_ticklabels(['False','True'])
ax.yaxis.set_ticklabels(['False','True'])
plt.show()

#Check performance of our model with classification report
print(classification_report(y_test, y_pred))
```

The results from ```LogisticRegression``` are the same as our logit single neuron model.

---

## Conclusion

From the result above, we can see that the logistic regression single neuron model performs well on the CVDs data set. Compared with the single neuron perceptron model, the logistic regression model has better performance. The main reason is that the logistic regression model can deal with non-linearly separable data.


# Week 2 - Crossmatching Catalogues
####

```python
import numpy as np
import time
from astropy.coordinates import SkyCoord
from astropy import units as u
```

### Convert from HMS & DMS notation to Decimal degrees

```python
def hms2dec(h,m,s):
  return 15*(h + m/60 + s/(60*60))

def dms2dec(h,m,s):
  return h*(1 + m/(abs(h)*60) + s/(abs(h)*60*60))

# The first example from the question
print(hms2dec(23, 12, 6))

# The second example from the question
print(dms2dec(22, 57, 18))

# The third example from the question
print(dms2dec(-66, 5, 5.1))
```

    348.025
    22.955000000000002
    -66.08475

####

### Haversine Formula for calculating Angular Distance
######
\begin{align}
d = 2 \arcsin \sqrt{ \sin^2 \frac{|\delta_1 - \delta_2|}{2} + \cos \delta_1 \cos \delta_2 \sin^2 \frac{|\alpha_1 - \alpha_2|}{2} }
\end{align}

```python
def angular_dist(r1,d1,r2,d2):
  r1, r2, d1, d2 = np.radians(r1), np.radians(r2), np.radians(d1), np.radians(d2)
  a = np.sin(np.abs(d1 - d2)/2)**2
  b = np.cos(d1)*np.cos(d2)*np.sin(np.abs(r1 - r2)/2)**2
  d = 2*np.arcsin(np.sqrt(a + b))
  return np.degrees(d)

# Run your function with the first example in the question.
print(angular_dist(21.07, 0.1, 21.15, 8.2))

# Run your function with the second example in the question
print(angular_dist(10.3, -3, 24.3, -29))
```

    8.100392318146504
    29.208498180546595

####

### Reading data from AT20G BSS and SuperCOSMOS catalogues

```python
def import_bss(path):
  data = np.loadtxt(path, usecols=range(1, 7))
  out = []
  for i, row in enumerate(data, 1):
    out.append((i, hms2dec(row[0], row[1], row[2]), dms2dec(row[3], row[4], row[5])))
  return out

def import_super(path):
  data = np.loadtxt(path, delimiter=',', skiprows=1, usecols=[0, 1])
  out = []
  for i, row in enumerate(data, 1):
    out.append((i, row[0], row[1]))
  return out

# Output of the import_bss and import_super functions
bss_cat = import_bss('Data 2/bss_truncated.dat')
super_cat = import_super('Data 2/super_truncated.csv')

print('Object ID  |  Right Ascension°  |  Declination°\n')
print(bss_cat)
print(super_cat)
```

    Object ID  |  Right Ascension°  |  Declination°

    [(1, 1.1485416666666666, -47.60530555555555), (2, 2.6496666666666666, -30.463416666666664), (3, 2.7552916666666665, -26.209194444444442)]
    [(1, 1.0583407, -52.9162402), (2, 2.6084425, -41.5005753), (3, 2.7302499, -27.706955)]

####

### Finding closest neighbour for a target source (RA°, Dec°) from a catalogue

```python
def find_closest(data, RA1, Dec1):
  ind = 0
  closest = angular_dist(RA1, Dec1, data[0][1], data[0][2])
  for i, row in enumerate(data, 0):
    test = angular_dist(RA1, Dec1, row[1], row[2])
    if test < closest:
      ind = i
      closest = test
  return (data[ind][0], closest)

cat = import_bss('Data 2/bss.dat')
print('ID  |  Angular Distance°\n')

# First example from the question
print(find_closest(cat, 175.3, -32.5))

# Second example in the question
print(find_closest(cat, 32.2, 40.7))

```

    ID  |  Angular Distance°

    (156, 3.7670580226469013)
    (26, 57.729135775621295)

####

## Crossmatching 2 catalogues within a given distance

```python
def crossmatch(cat1, cat2, dist):
  matches, no_matches = [], []
  for i, row in enumerate(cat1,1):
    test = find_closest(cat2, row[1], row[2])
    if  test[1] < dist:
      matches.append((i, test[0], test[1]))
    else:
      no_matches.append(i)
  return matches, no_matches

bss_cat = import_bss('Data 2/bss (2).dat')
super_cat = import_super('Data 2/super.csv')

# First example in the question
max_dist = 40/3600
matches, no_matches = crossmatch(bss_cat, super_cat, max_dist)
print('1st Object ID  |  2nd Object ID  |  Angular Distance°\n')
print(matches[:3])
print('Unmatched IDs from 1st Catalogue - ', no_matches[:3])
print('No. of Unmatched objects in 1st Catalogue = ', len(no_matches), '\n')

# Second example in the question
max_dist = 5/3600
matches, no_matches = crossmatch(bss_cat, super_cat, max_dist)
print(matches[:3])
print('Unmatched IDs from 1st Catalogue - ', no_matches[:3])
print('No. of Unmatched objects in 1st Catalogue = ', len(no_matches))

```

    1st Object ID  |  2nd Object ID  |  Angular Distance°

    [(1, 2, 0.00010988610939332616), (2, 4, 0.0007649845967220993), (3, 5, 0.00020863352870707666)]
    Unmatched IDs from 1st Catalogue -  [5, 6, 11]
    No. of Unmatched objects in 1st Catalogue =  9

    [(1, 2, 0.00010988610939332616), (2, 4, 0.0007649845967220993), (3, 5, 0.00020863352870707666)]
    Unmatched IDs from 1st Catalogue -  [5, 6, 11]
    No. of Unmatched objects in 1st Catalogue =  40

####

### Microoptimising the crossmatch

```python
def angular_dist(r1,d1,r2,d2):
  a = np.sin(np.abs(d1 - d2)/2)**2
  b = np.cos(d1)*np.cos(d2)*np.sin(np.abs(r1 - r2)/2)**2
  d = 2*np.arcsin(np.sqrt(a + b))
  return d

def find_closest(data, RA1, Dec1):
  ind = 0
  closest = angular_dist(RA1, Dec1, data[0][0], data[0][1])
  for i, row in enumerate(data, 0):
    test = angular_dist(RA1, Dec1, row[0], row[1])
    if test < closest:
      closest = test
      ind = i
  return (ind, closest)

def crossmatch(cat1, cat2, dist):
  start = time.perf_counter()
  matches, no_matches = [], []
  cat1 = np.radians(cat1)
  cat2 = np.radians(cat2)
  dist = np.radians(dist)
  for i, row in enumerate(cat1,0):
    test = find_closest(cat2, row[0], row[1])
    if  test[1] < dist:
      matches.append((i, test[0], np.degrees(test[1])))
    else:
      no_matches.append(i)
  seconds = time.perf_counter() - start
  return matches, no_matches, seconds

# The example in the question
cat1 = np.array([[180, 30], [45, 10], [300, -45]])
cat2 = np.array([[180, 32], [55, 10], [302, -44]])
matches, no_matches, time_taken = crossmatch(cat1, cat2, 5)
print('1st Object ID  |  2nd Object ID  |  Angular Distance°\n')
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken, '\n')

# A function to create a random catalogue of size n
def create_cat(n):
    ras = np.random.uniform(0, 360, size=(n, 1))
    decs = np.random.uniform(-90, 90, size=(n, 1))
    return np.hstack((ras, decs))

# Test your function on random inputs
np.random.seed(0)
cat1 = create_cat(10)
cat2 = create_cat(20)
matches, no_matches, time_taken = crossmatch(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken)
```

    1st Object ID  |  2nd Object ID  |  Angular Distance°

    matches: [(0, 0, 2.0000000000000027), (2, 2, 1.7420109046547023)]
    unmatched: [1]
    time taken: 0.0003623000000061438

    matches: []
    unmatched: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    time taken: 0.005214199999954872

####

### Vectorisation using NumPy

```python
def crossmatch_vect(cat1, cat2, max_radius):
  start = time.perf_counter()
  max_radius = np.radians(max_radius)

  matches, no_matches = [], []

  # Convert coordinates to radians
  cat1 = np.radians(cat1)
  cat2 = np.radians(cat2)
  ra2s = cat2[:,0]
  dec2s = cat2[:,1]

  for id1, (ra1, dec1) in enumerate(cat1):
    dists = angular_dist(ra1, dec1, ra2s, dec2s)
    min_id = np.argmin(dists)
    min_dist = dists[min_id]
    if min_dist > max_radius:
      no_matches.append(id1)
    else:
      matches.append((id1, min_id, np.degrees(min_dist)))

  time_taken = time.perf_counter() - start
  return matches, no_matches, time_taken

# The example in the question
ra1, dec1 = np.radians([180, 30])
cat2 = [[180, 32], [55, 10], [302, -44]]
cat2 = np.radians(cat2)
ra2s, dec2s = cat2[:,0], cat2[:,1]
dists = angular_dist(ra1, dec1, ra2s, dec2s)
print('Angular distance° - ', np.degrees(dists), '\n')

cat1 = np.array([[180, 30], [45, 10], [300, -45]])
cat2 = np.array([[180, 32], [55, 10], [302, -44]])
matches, no_matches, time_taken = crossmatch_vect(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken, '\n')

# Test your function on random inputs
cat1 = create_cat(10)     # Create a random catalogue of size 10
cat2 = create_cat(20)
matches, no_matches, time_taken = crossmatch_vect(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken)
```

    Angular distance° -  [  2.         113.72587199 132.64478705]

    matches: [(0, 0, 2.0000000000000027), (2, 2, 1.7420109046547023)]
    unmatched: [1]
    time taken: 0.00032420000025012996

    matches: []
    unmatched: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    time taken: 0.0007706999999754771

####

### Breaking out after maximum match radius : Searching within -90° < δ° < (δ + r)°

```python
def crossmatch(cat1, cat2, max_radius):
  start = time.perf_counter()
  max_radius = np.radians(max_radius)
  matches, no_matches = [], []

  cat1 = np.radians(cat1)
  cat2 = np.radians(cat2)
  order = np.argsort(cat2[:,1])
  cat2_ordered = cat2[order]

  for id1, (ra1, dec1) in enumerate(cat1):
    min_dist = np.inf
    min_id2 = None
    max_dec = dec1 + max_radius
    for id2, (ra2, dec2) in enumerate(cat2_ordered):
      if dec2 > max_dec:
        break
      dist = angular_dist(ra1, dec1, ra2, dec2)
      if dist < min_dist:
        min_id2 = order[id2]
        min_dist = dist
    if min_dist > max_radius:
      no_matches.append(id1)
    else:
      matches.append((id1, min_id2, np.degrees(min_dist)))

  time_taken = time.perf_counter() - start
  return matches, no_matches, time_taken

# The example in the question
cat1 = np.array([[180, 30], [45, 10], [300, -45]])
cat2 = np.array([[180, 32], [55, 10], [302, -44]])
matches, no_matches, time_taken = crossmatch(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken, '\n')

# Test your function on random inputs
np.random.seed(0)
cat1 = create_cat(10)
cat2 = create_cat(20)
matches, no_matches, time_taken = crossmatch(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken)
```

    matches: [(0, 0, 2.0000000000000027), (2, 2, 1.7420109046547023)]
    unmatched: [1]
    time taken: 0.015049199999793927

    matches: []
    unmatched: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    time taken: 0.006830000000263681

####

### Boxing Match : Searching within (δ - r)° < δ° < (δ + r)°

```python
def crossmatch_boxing(cat1, cat2, max_radius):
  start = time.perf_counter()
  max_radius = np.radians(max_radius)
  matches, no_matches = [], []

  cat1 = np.radians(cat1)
  cat2 = np.radians(cat2)
  order = np.argsort(cat2[:,1])
  cat2_ordered = cat2[order]

  for id1, (ra1, dec1) in enumerate(cat1):
    min_dist = np.inf
    min_id2 = None
    max_dec = dec1 + max_radius
    index = np.searchsorted(cat2_ordered[:,1], dec1 - max_radius, side='left')

    for id2, (ra2, dec2) in enumerate(cat2_ordered[index:,:]):
      if dec2 > max_dec:
        break
      dist = angular_dist(ra1, dec1, ra2, dec2)
      if dist < min_dist:
        min_id2 = order[index:][id2]
        min_dist = dist
    if min_dist > max_radius:
      no_matches.append(id1)
    else:
      matches.append((id1, min_id2, np.degrees(min_dist)))

  time_taken = time.perf_counter() - start
  return matches, no_matches, time_taken

# The example in the question
cat1 = np.array([[180, 30], [45, 10], [300, -45]])
cat2 = np.array([[180, 32], [55, 10], [302, -44]])
matches, no_matches, time_taken = crossmatch_boxing(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken, '\n')

# Test your function on random inputs
np.random.seed(0)
cat1 = create_cat(10)
cat2 = create_cat(20)
matches, no_matches, time_taken = crossmatch_boxing(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken)
```

    matches: [(0, 0, 2.0000000000000027), (2, 2, 1.7420109046547023)]
    unmatched: [1]
    time taken: 0.044778799999676266

    matches: []
    unmatched: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    time taken: 0.0005314999998518033

####

### Crossmatching with k-d Trees

```python
def crossmatch_kd(cat1, cat2, dist):
  start = time.perf_counter()
  sky_cat1 = SkyCoord(cat1*u.degree, frame='icrs')
  sky_cat2 = SkyCoord(cat2*u.degree, frame='icrs')
  closest_ids, closest_dists, closest_dists3d = sky_cat1.match_to_catalog_sky(sky_cat2)
  matches, no_matches = [], []
  for i, ele in enumerate(closest_dists.value):
    if  ele < dist:
      matches.append((i, closest_ids[i], ele))
    else:
      no_matches.append(i)
  seconds = time.perf_counter() - start
  return matches, no_matches, seconds

# The example in the question
cat1 = np.array([[180, 30], [45, 10], [300, -45]])
cat2 = np.array([[180, 32], [55, 10], [302, -44]])
matches, no_matches, time_taken = crossmatch_kd(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken, '\n')

# Test your function on random inputs
np.random.seed(0)
cat1 = create_cat(10)
cat2 = create_cat(20)
matches, no_matches, time_taken = crossmatch_kd(cat1, cat2, 5)
print('matches:', matches)
print('unmatched:', no_matches)
print('time taken:', time_taken)
```

    matches: [(0, 0, 2.0000000000000036), (2, 2, 1.7420109046547163)]
    unmatched: [1]
    time taken: 2.324950500000341

    matches: []
    unmatched: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    time taken: 0.005834800000229734


# Exercise 1 - The Mandelbrot Set

The mandelbrot set is a beautiful fractal ([Wikipedia](http://en.wikipedia.org/wiki/Mandelbrot_set)).
More precisely, it contains all numbers from the complex plane where the complex quadratic polynomial

\begin{align}
    z_{n+1} = z_{n}^2 + c
\end{align}

remains bounded. A complex number $c$ is part of the Mandelbrot set when starting with $z_0=0$ and applying the iteration repeatedly, the absolute value of $z_n$ remains smaller or equal than 2 regardless how large $n$ becomes.

Moreover, one can draw very beautiful images by looking at the *escape time* of a numerical computation:
We will numerically determine whether a complex number $c$ belongs to the set, and if not we will keep track how many iterations are needed until $|z_n| > 2$. Plotting the escape times of a sampled grid of complex numbers as a 2D image will yield the famous *Apfelmaennchen* you might have seen before.

#Task:#

* Write a function ``mandelbrot(relim, imlim, resteps, imsteps, maxiterations)`` that computes and returns the escape times for the mandelbrot set.
    * ``relim`` and ``imlim`` are tuples that define the boundary of the complex plane (e.g. ``relim=(-2,1)`` and ``imlim=(-1,1)``), for convenience ``imlim`` should contain real numbers although it represents the complex axis.
    * ``resteps`` and ``imsteps`` are integers and define the sampling resolution along each axis (e.g. 300 and 200 steps).
    * ``maxiterations`` is an integer defining the maximum number of iterations (e.g. 50).

    * The function should sample complex numbers from the plane as defined by the parameters ``relim, imlim, resteps, imsteps`` and repeatedly apply the quadratic polynomial from above until the absolute value of a complex number is larger than 2. In case a maximum number of iterations is reached (``maxiterations``) the number is believed to be part of the set.
    * The function should return a 2D array containing the number of iterations needed for every sampled complex number and two 1D arrays containing the sampled values along each axis. Escape times in the 2D array of exactly ``maxiterations`` indicate complex numbers that are believed to be part of the set.
* Make a 2D color plot of the escape times, you might want to look at matplotlib's [imshow](http://www.mathworks.de/de/help/images/ref/imshow.html) function.

###Hints how the function ``mandelbrot`` may work:###
* Create a 2D numpy array containing the escape times, in the beginning filled with zeros.

* Secondly, create two 1D arrays sampling each dimension of the complex plane.  You might want to take a look at the [np.linspace](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html) function. You can multiply one of the arrays with ``1j`` to create the imaginary axis. Later on you can calculate each starting complex number simply from summing individual elements of both arrays.

* The easy brute-force solution involves three nested for loops. Be aware that you can use python's ``break`` statement to leave a for loop and spare unnecessary computations.

* Iterate at most ``maxiterations`` times for any value in your complex plane and apply the quadratic polynomial from above. If the absolute value gets larger than 2 you can stop iterating and store the number of iterations you have needed in the 2D escape array. Thus, for complex numbers that are actually part of the Mandelbrot set, the 2D array of escape times should contain the value ``maxiterations``.

###Hints for optimization:###
* You can aim for an optimized version that uses vectorized computation. Here you should only need a single for loop.

* Create a second and third 2D array containing the starting complex numbers and the intermediate values, checkout the [meshgrid](http://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html) function to create these.

* Use the second 2D array to iteratively update all values in the third array at once (vectorized computation!). Mask all values that already escaped the boundary via boolean indexing to speed up the computation.

* Remember that boolean indexing creates copies, not views! So you need to find a smart way to only apply the polynomial
to values that haven't escaped without always keeping the full 2D array, otherwise internally numpy has to reiterate the full array every timestep. If you also create mesh grids of matrix indices you can reduce the 2D arrays each iteration to 1D arrays containing only the currently not escaped values. This is tricky!

* Now you can use finer resolutions than above!

# Exercise 2 - Numerical Integration of a Neuron Model

We are going to simulate our first neuron model with Euler integration ([Wikipedia](http://en.wikipedia.org/wiki/Euler_method)).
Basically, our neuron model consists of a single differential equation that describes the development of the membrane voltage over time. For now let's assume this equation is an arbitrary function $f$:
\begin{align}
    \frac{dV}{dt} = f(V)
\end{align}
To obtain the voltage as a function of time, i.e. $V(t)$, we have to solve the differential equation. Lucky for you, we are in the computer practical and not the analytical tutorial. We are going to solve it numerically, so there's no need for a complicated Ansatz :-)

As said before, we will use simple Euler integration. Accordingly, if we assume discretized time we can easily compute $V(t+1)$, the membrane voltage of the next time step, in case we now the previous voltage $V(t)$:
\begin{align}
V(t+1) = V(t) + f(V) * dt
\end{align}
with $dt$ the size of the discretized timesteps.

If we start with a chosen initial value of $V(0)$ we can iteratively solve the differential equation.

By this method we can simulate very complex neuron models. Let's simulate a rescaled version of the exponential integrate and fire neuron ([Scholarpedia](http://www.scholarpedia.org/article/Adaptive_exponential_integrate-and-fire_model)):
\begin{align}
    \frac{dV}{dt} = -V + \exp(V) + I
\end{align}

$I$ describes a fixed input current. We will simulate several neurons fed with different current values.
For some values of $I$ the membrane potential $V$ will rise to infinity, this corresponds to the upstroke of an action potential (you remember action potentials from a neurobio course, right?). However, our neuron model cannot recover from this upstroke by itself. For a smooth recovery we would need a second differential equation. However, we will keep our model simple and add a so called *reset rule*: whenever $V$ crosses a particular threshold $V(t)\geq V_t$ then we set it back to a reset value at the next timestep $V(t+1)=V_r$.

#Task#

* Write a function ``expIF_neuron(V, I, Vt, Vr, duration, dt)`` that simulates one or more exponetial integrate-and-fire neurons.
    * The parameter ``V`` can be a scalar or numpy array describing the initial conditions
    * The parameter ``I`` can be a scalar or numpy array describing the input currents
    * The parameter ``Vt`` is a scalar value defining the spiking threshold
    * The parameter ``Vr`` is a scalar defining the reset value after threshold crossing
    * The parameter ``duration`` gives the length of the simulation
    * The parameter ``dt`` describes the stepsize of the Euler integration
    * The function should return
        * A 2D array of voltage traces, i.e. the simulated development of the membrane potential
            * First dimension is the number of neurons, second dimension the voltage trace over time
            * First entries in second dimension should contain the initial values
        * A 1D array containing the discretized timesteps
    * In case ``V`` and ``I`` are arrays, the function should be vectorized, i.e. there should only be a single loop over all timesteps, but no loop over all neurons!

* Simulate 5 neurons at once (do NOT call the function 5 times!) with 5 different input currents $I\in\{-3.0, -2.0, -1.0, 0.0, 1.0\}$.
    * Choose the other parameters as
        * $V_r=-1.0$
        * $V_t=5.0$
        * $duration=10.0$
        * $dt = 0.01$
    * Set the initial $V$ values to $V_r$
* Plot all 5 voltage traces in a single plot, add a legend and label the axis.

###Hints###
* You can use the template provided below. This exercise can be solved in just a handful of lines ;-)
* To incorporate the reset rule you may try boolean indexing.

```
## The template: ##

def expIF_neuron(V=-1.0, I=0.0, Vt=5.0, Vr=-1.0, duration=10.0, dt=0.01):
    """Numerically integrates the expIF neuron membrane equation with the Euler-Method.

    The neurons obey a reset rule, when the membrane potential crosses `Vt`
    it is set back to `Vr`.

    :param V: array of initial membrane values (or a scalar)
    :param VT: spiking threshold (scalar)
    :param I: array of input currents (or scalar)
    :param duration: duration of experiment (scalar)
    :param dt: stepsize of Euler integration (scalar)

    :return:

        2D array of voltage time series, first dimension the neurons,
        second dimension the voltage trace.
        First entries contain the initial values.

        1D array of simulation times

    """

    steps = int(duration/dt) # Calculate the number of simulation steps

    if isinstance(V, np.ndarray): # V can be scalar or an array, we need to check first
        nneurons = len(V) # Infer the number of neurons from the length of the initial conditions
    else:
        nneurons = 1

    V_series = np.zeros((nneurons, steps+1)) # Array that will contain the voltage traces
    # 1st dim neurons, 2nd dim voltage traces
    # i.e. V[2,10] would return the voltage of neuron #2 at the 10th timestep!
    # Wee need steps+1 since the 0th entry should contain the initial conditions

    V_series[:, 0] = V # Set initial conditions

    times = np.zeros(steps+1) # Array of timesteps

    for step in range(1, steps+1): # Loop starting from step 1 (0th contains initial conditions)

        ############# Your code ##############

        # Manipulate V_series here to simulate the neuron model
        # Iteratively compute f(V(t)) = -V(t) + exp(V(t)) + I and V(t+1) = V(t) + f(V(t)) * dt
        # Do not introduce another for loop, try to think vectorized
        # Try using boolean indexing to implement the threshold crossing and voltage reset

        ######### End of your code ###########

        times[step] = times[step-1] + dt # You actually don't need the times explicitly, but returning them
        # will make plotting easier

    return V_series, times

```

```

```