# Lab 3
## Introduction
In this lab we will analyse population dynamics under the logisitic model with managed harvesting.

First import the modules we need.

```python
from plotly.figure_factory import create_quiver
from plotly import graph_objs as go
from numpy import meshgrid, arange, sqrt, linspace
from scipy.integrate import odeint
```

## Harvesting of fish
A population of fish in a lake, left to its own devices, is modelled by the logistic differential equation
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y),
\end{align}
where the population $y$ is in units of thousands of fish and time $t$ is measured in years.

First define a function for $\mathrm{d}y/\mathrm{d}x$ in terms of $y$ and $x$.

```python
def diff_eq(y, x):
    return 4 * y * (1 - y)
```

Next define a function that creates a Plotly Figure object that contains a slope field and, optionally, a few solutions to initial value problems.

It automates a few things we did in the last lab.

- `diff_eq` is the differential equation to be plotted
- `x` and `y` should be outputs from `meshgrid`.
- `args` is any additional arguments to `diff_eq` (we will use that below).
- `initial_values` is a list (or array) of starting $y$ values from which approximate solutions will start. The corresponding $x$ value is the minimum element of `x`.

Note that the numerical solutions will plotted for the whole range of $x$ values in `x`, so if they blow up you will probably get a warning and less-than-useful plot.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
```

The slope field below should hopefully give you some idea for the fish population dynamics.

Note that we use `arange` rather than `linspace` this week so that we can carefully control the increments between our grid points. `arange(0, 1.1, 0.25)` returns an array that starts with 0 and increments by 0.25 until it exceeds 1.1.

The plot also contains the solution curves for
$y(0) = 1$ and $y(0) = 0.4$. Edit the cell to also include the solution curve for $y(0)=1.4$.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, initial_values=(0.4, 1, 1.4))
fig.show('png')
```

### Equilibrium solutions
Looking back to our differential equation, $\mathrm{d}y/\mathrm{d}t = 0$ when $y(t) = 0$ or $y(t) = 1$. Looking at the slope field, we see that the equilibrium solution $y(t) = 1$ is stable (this is the carrying capacity here, corresponding to 1000 fish), whereas the equilibrium solution $y(t) = 0$ is unstable. Any non-zero initial population will eventually stabilise at 1000 fish.

### What will happen if harvesting is now commenced at a steady rate?
For the simplest harvesting model, assume that $H$ units (thousands) of fish are taken
continuously (smoothly) over the year, rather than at one instant each year.
Note that the units of $H$ are the same as those of $\mathrm{d}y/\mathrm{d}t$, thousands of fish per year, so we simply subtract $H$ from the RHS of our existing equation to give the DE with harvesting as
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y) - H.
\end{align}
Again, the (constant) equilibrium solutions are found by setting $\mathrm{d}y/\mathrm{d}t = 0$, giving from the quadratic formula (check this),
\begin{align}
y(t) = \frac{4\pm\sqrt{16-16H}}{8} = \frac{1\pm\sqrt{1-H}}{2}.
\end{align}
What happens after harvesting starts will depend on the equilibrium solutions, their
stability and the initial number of fish $y(0)$.

Start by redefining `diff_eq` to include the `H` parameter. Note that defining `diff_eq` again overides our original definition.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
```

Now set $H = 0.6$ and plot the slope field. This is done by setting `args=(0.6,)` when we call `create_slope_field`. This is exactly how you would pass additional arguments like this one to `odeint` if you were calling it directly.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,))
fig.show('png')
```

From the solutions to the quadratic equation above, the equilibrium solutions of the DE are found to be $y(t) \approx 0.184$ and $y(t) \approx 0.816$. The previous equilibrium solution with no harvesting at $y(t) = 0$ has moved up to $y(t) \approx 0.184$, while the previous equilibrium solution with no harvesting at $y(t) = 1$ has moved down to $y(t) \approx 0.816$.

From the slope field, we see that the equilibrium solution $y(t) \approx 0.184$ is unstable, whereas the equilibrium solution $y(t) \approx 0.816$ is stable. If the population ever falls below about 0.184, or 184 fish, it will then drop to 0. This is a new feature, introduced by harvesting.

In the cell below, use `create_slope_field` to experiment by plotting the solutions to the initial value problems $y(0)=0.183$ and $y(0)=0.25$. Extend the $x$ range of your slope field until the top line is close to equlibrium. Note that if you extend it too far you will break `odeint` (why?). You may also like to increase the increments in `arange` to make the plot clearer.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.25))
fig.show('png')
```

## Exercises

In this lab you will experiment with the population dynamics given by the logistic equation with harvesting that we started analysing in the lab.

This week the questions will be a combination of plots and written answers.

1. Assume that the harvest is 600 fish per year. **On the same figure,**
  a. plot the slope field,
  b. plot the equilibrium solutions that we found in above, and
  c. plot the solution curves for $y(0)=1$, $y(0)=0.3$, and $y(0)=0.15$.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.816, 1, 0.3, 0.15))
fig.show('png')
```

1.  d. In the cell below, describe the behaviour of the fish population for each of these five initial numbers of fish.

At the equilibrium points the fish population will stagnate and remain constant. At the starting point of 1 it will decline towards 0.816 and at 0.3 it will increase towards it. For 0.15 it will continue to decline until there are no fish left.

2. a. i. Assume that $H=0.8$. Plot the slope field and five solutions, one for each equilibrium solution and one for each region between, above, or below them. You can use the equation from the lab to calculate the equilibrium solutions.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.8,), initial_values=(0.276, 0.724, 1, 0.4, 0.2))
fig.show('png')
```

2. a. ii. In the cell below, describe the limiting behaviour of each line.

At the equilibrium points the fish population will stagnate and remain constant. At the starting point of 1 it will decline towards 0.724 and at 0.1 it will increase towards it. For 0.2 it will continue to decline until there are no fish left.

2. b. i. Assume that $H=1$. Plot the slope field and three solutions for the equilibrium solution and the regions above and below it.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(1,), initial_values=(0.5, 0.8, 0.3))
fig.show('png')
```

2. b. ii. Describe the limiting behaviour of each line.

all values trend towards 0.5.

2. c. i. Assume that  𝐻=1.2. Plot the slope field and two or three solutions.

```python
x, y = meshgrid(arange(0, 1.1, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(1.2,), initial_values=(0.5,))
fig.show('png')
```

2. c. iii. Describe the limiting behaviour of the lines.

regardless of input fish population tends towards extinction

3. Summarize what happens to the equilibrium solutions and their stability as $H$
is increased from 0 to beyond 1. Refer to your plots to support your answers.

the stable soultion, or the stable population number decreases and the unstable number increases until 1 where there is a single semi-stable soultion.

4. What is a reasonable strategy for sustainable fishing in this case?
Don’t forget to allow qualitatively for minor catastrophes, such as disease or temporary overfishing.

dont allow for fishing above 1000 fish to keep the population at a stable rate, assuming it stated above 500 fish.

```python

```


# Problem 3.6 Conditional Distribution

Show that
$$P_{C\mid E=2}\sim\mathcal{N}\left(\frac{8}{17},\frac{1}{17}\right)$$
for the SCM
\begin{align}
C:=&\mathcal{N}(0,1)\\
E:=&4C+\mathcal{N}(0,1)
\end{align}

## Proof

The SCM can be seen as a [multivariate normal distribution](https://en.wikipedia.org/wiki/Multivariate_normal_distribution)
$$
P\left(\begin{matrix}C\\E\end{matrix}\right)=
\mathcal{N}\left(\begin{bmatrix}\mu_C\\\mu_E\end{bmatrix},\Sigma\right)
$$
where $\mu_C=\mu_E=0$ because both distributions have an expecation of zero and $\Sigma$ is the covariance matrix which is
\begin{align}
\Sigma:=&\begin{bmatrix}\sigma^2_C&\rho\sigma_C\sigma_E\\\rho\sigma_C\sigma_E&\sigma_C^2\end{bmatrix}\\
=&\begin{bmatrix}1&14\\14&4^2+1\end{bmatrix}
\end{align}

For the bivariate case the conditional distribution is defined as
$${\displaystyle X_{1}\mid X_{2}=a\ \sim \ {\mathcal {N}}\left(\mu _{1}+{\frac {\sigma _{1}}{\sigma _{2}}}\rho (a-\mu _{2}),\,(1-\rho ^{2})\sigma _{1}^{2}\right).}$$

Applied to $C\mid E=2$ that yields $$\mathcal{N}\left(\frac{8}{17},\frac{1}{17}\right)$$


```
%pylab inline
import numpy as np
```

# Using finite differences to approximate derivatives

The material in this notebook is a supplement to chapter 1 of the text.

Suppose that you're given a table of values of some "secret" function $f(x)$, but you are not given the function itself.  For example:

```
x_values = (0,1,2,3,4)
y_values = (0,1,8,27,64)
```

In order to understand the function that generated these values, we could plot them:

```
plt.plot(x_values,y_values,'-o')
```

In the plot, the points have been connected by a line, and you might use the slope of this line if you wanted to estimate the derivative of the function.  For instance, to estimate the slope at $x=2$, you could use the slope of the line between $x=2$ and $x=3$:

```
plt.plot(x_values,y_values,'-o')
plt.hold(True)
plt.plot(x_values[2:4],y_values[2:4],'-r',linewidth=5)
```

$$\left. f'(x) \right|_{x=2} \approx \frac{f(3)-f(2)}{3-2} = 19.$$

Alternatively, you might use the slope of the line between $x=1$ and $x=2$:

```
plt.plot(x_values,y_values,'-o')
plt.hold(True)
plt.plot(x_values[1:3],y_values[1:3],'-r',linewidth=5)
```

$$\left. f'(x) \right|_{x=2} \approx \frac{f(2)-f(1)}{2-1} = 7.$$

Remember that the derivative is defined as the limit of a formula rather like our first approximation above (referred to as a *forward difference*):

$$ f'(x) = \lim_{h\to 0} \frac{f(x+h)-f(x)}{(x+h)-x} = \lim_{h\to 0} \frac{f(x+h)-f(x)}{h}.$$

Notice that the fraction in the definition is just the slope of the line connecting the values of $f$ at $x$ and $x+h$.  We could equally well define the derivative in terms of a *backward difference*, similar to our second formula above:

$$ f'(x) = \lim_{h\to 0} \frac{f(x)-f(x-h)}{x-(x-h)} = \lim_{h\to 0} \frac{f(x)-f(x-h)}{h}.$$

Yet another way of defining the derivative would be to use the slope of the line connecting points $x+h$ and $x-h$, leading to

$$ f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x-h)}{2h}.$$

Here the factor $2h$ in the denominator is just the length of the interval $(x-h,x+h)$.  Using this formula with our values gives

```
plt.plot(x_values,y_values,'-ok')
plt.hold(True)
plt.plot((x_values[1],x_values[3]),(y_values[1],y_values[3]),'-r',linewidth=5)
```

$$\left. f'(x) \right|_{x=2} \approx \frac{f(3)-f(1)}{3-1} = 13.$$

When we only know the values of a function at a finite set of points, we can't compute these limits, so it makes sense to use a finite value of $h$ in order to approximate the derivative.  This approach is known as the *finite difference method*.  Soon we will use it to solve differential equations, but first let's examine its effectiveness in approximating derivatives.

You may have guessed that the "secret" function that generated these values is $f(x)=x^3$.  Let's compare our three estimates of the derivative with the true derivative: $f'(2) = 3\cdot 2^2 = 12$.  Clearly the third formula (known as a *centered difference*) is the best approximation.  We can also see this by plotting the approximations and the tangent line (whose slope is that of the true derivative):

```
def f(x):
    return x**3

x = np.linspace(0,4)
tangent = 8 + 12*(x-2)
plot(x,f(x),'k',linewidth=2)
plt.plot(x_values[2:4],y_values[2:4],'-g',linewidth=3)
plt.plot(x_values[1:3],y_values[1:3],'-b',linewidth=3)
plt.plot((x_values[1],x_values[3]),(y_values[1],y_values[3]),'-r',linewidth=3)
plt.plot(x,tangent,'--k',linewidth=3)
plt.legend(['f(x)','Forward','Backward','Centered','Tangent'],loc='best')
plt.axis((0.5,3.5,0,40))
```

It's clear that the centered difference approximation (slope of the red line) is closest to the derivative (slope of the dotted line).  Would this be true if we picked another function $f(x)$, or if we used function values at different points?  Let's answer the second question with an experiment.

```
x = 2.
df = 12.

for h in (0.1, 0.05, 0.025):
    forward = (f(x+h)-f(x))/h
    forward_error = forward - df
    print forward_error
```

It seems clear from the results that reducing $h$ by a factor of two also reduces the error in the forward difference approximation by a factor of two.

**Now add backward and centered difference approximations to the code above and try to figure out what happens to their errors as $h$ decreases.  Make a table like Table 1.1 on page 5 of the textbook.**  Is the centered difference always the most accurate?  Do you think it will still be the most accurate if we continue reducing $h$?

**Now redefine the function $f(x)$ to be $x^2$ and generate the same table.  Do you see anything surprising?  Can you explain it?**  If not, make a guess.

## Estimating truncation errors

The error made by a finite difference approximation is called *truncation error*.  Why?  Well, we can estimate the error by expanding each function value in a *Taylor series*:

$$f(x+h) = f(x) + h f'(x) + \frac{1}{2}h^2 f''(x) + \frac{1}{6} h^3 f'''(x) + {\mathcal O}(h^4)$$

Here ${\mathcal O}(h^4)$ indicates that the rest of the terms in the series vanish at least as quickly as $h^4$ when $h\to 0$ (see Appendix A of the text).
Substituting this series in our forward difference formula gives
\begin{align}
\frac{f(x+h) - f(x)}{h} & = \frac{f(x) + h f'(x) + \frac{1}{2}h^2 f''(x) + \frac{1}{6} h^3 f'''(x) + {\mathcal O}(h^4) - f(x)}{h} \\
& = f'(x) + \frac{1}{2}h f''(x) + \frac{1}{6} h^2 f'''(x) + {\mathcal O}(h^3) \\
& = f'(x) + \frac{1}{2}h f''(x) + {\mathcal O}(h^2).
\end{align}

This analysis confirms our intuition that the forward difference approximates $f'(x)$, but it tells us much more.  Most importantly, it shows that the largest term in the error in this approximation is proportional to $h$.  That's why we saw that decreasing $h$ by a factor of two caused the error to decrease by the same amount.

Notice that if we truncated the Taylor series after the first term, we would get the forward difference formula exactly.  That is why the error is referred to as *truncation error* -- it's the error we get from truncating an infinite series.  The term $\frac{1}{2} h f''(x)$ is referred to as the *leading truncation error* because when $h$ is very small we expect that term to be much bigger than all the ones that come after it.

Since in our simple example we know the function $f(x)$, we could evaluate error terms in the series above to get a better approximation of the error.  But typically we won't know what the function is (if we did, why would we need finite differences?), so we'll be most interested in knowing what power of $h$ multiplies the leading error term.

**Write down the Taylor series for $f(x-h)$ about $x$.  Use that (and the series above) to work out the leading truncation error terms for the backward and centered difference formulas.**  Does your result for the centered formula allow you to explain the results you found before when applying it to $f(x)=x^2$?

## Polynomial interpolation and finite difference formulas

In sections 1.2-1.5 of the text, a method for finding finite difference formulas is given, based on Taylor series.  Here we will approach the same topic from a different route, using polynomial interpolation.  The idea is this: to approximate the derivative of a function whose point values are known, we first find a polynomial that interpolates those values, then evaluate the derivative of that polynomial.

To begin, suppose we are given three function values: $f(x_0-h), f(x_0), f(x_0+h)$, which we'll denote by $(f_1,f_2,f_3)$.  We wish to find a polynomial that passes through these three points.  As you may know, a set of $n$ values uniquely defines a polynomial of degree $n-1$, so we will look for a quadratic polynomial.  To make the computation simpler, we'll write it this way: $p(x) = a + b (x-x_0) + c (x-x_0)^2$.  We know that $p$ and $f$ must agree at the three given points, which means
\begin{align}
a + b(-h) + c (-h)^2  & = f_1 \\
a + b(0) + c(0)^2 & = f_2 \\
a + b (h) + c (h)^2 & = f_3 \\
\end{align}

or simply

\begin{align}
a - hb + h^2c  & = f_1 \\
a & = f_2 \\
a + hb + h^2c & = f_3 \\
\end{align}

We can rewrite this system of equations in matrix form:

\begin{align}
\begin{pmatrix}
1 & -h & h^2 \\
1 & 0   & 0 \\
1 & h & h^2 \\
\end{pmatrix}
\begin{pmatrix} a \\ b \\ c \end{pmatrix}
& = \begin{pmatrix} f_1 \\ f_2 \\ f_3 \end{pmatrix}
\end{align}

This is a linear system that we can solve for the coefficients $a,b,c$ in terms of $h$, and the values of $f$.  The result is

\begin{align}
a & = f_2, & b & = \frac{f_3-f_1}{2h}, & c & = \frac{f_1 -2 f_2 + f_3}{2h^2}.
\end{align}

Thus

\begin{align}
p(x) = f_2 + \frac{x-x_0}{2h}(f_3-f_1) + \frac{(x-x_0)^2}{2h^2}(f_1-2f_2+f_3).
\end{align}

Now suppose we wish to approximate the derivative of $f$.  We have an easy approach: use the derivative of $p$:

\begin{align}
p'(x) = \frac{f_3-f_1}{2h} + \frac{x-x_0}{h^2}(f_1 -2 f_2 + f_3).
\end{align}

In particular, if want to approximate the derivative at $x=x_0$, we get simply

$$f'(x) \approx p'(x_0) = \frac{f_3-f_1}{2h} = \frac{f(x_0+h) - f(x_0-h)}{2h}.$$

**Do you recognize this approximation?  Now use $p(x)$ to get an approximation for the second derivative of $f(x)$.  Write down the resulting formula.**

<font color='red'>
Derive a formula for $f''(x)$ based on the values $f(x), f(x-h), f(x-3h)$ by determining the interpolating polynomial and differentiating it.  How accurate is your formula?
</font>

**Can you think of a similar approach that would allow you to derive rules for numerical quadrature (integration)?**

```

```


## Higher Order Integration Schemes

The composite trapezoidal rule is functional, but not all that accurate.  At this point, the only method we have to improve accuracy is to divide the interval into smaller segments.  We can, however, use results for different step sizes $h$ to improve both results by eliminating the lowest order error.  To illustrate, consider the trapezoidal rule for a step $h_1=(b-a)$ and then $h_2=(b-a)/2$,

$$
\begin{align}
I_1 &=\int_a^b f(x) dx = \frac{h_1}{2}\left[f(a)+f(b)\right] - (b-a)\frac{h_1^2}{12} f''(\eta_1),\\
I_2 &=\int_a^b f(x) dx = \frac{h_2}{2}\left[f(a)+2f(a+h)+f(b)\right] - (b-a)\frac{h_2^2}{12} f''(\eta_2).
\end{align}
$$

The problem with attempting to use these two equations to eliminate the error term lies in the fact that $\eta_1$ and $\eta_2$ are not likely to be the same.  However, it is possible to show that, for a suitably smooth function, the error term can be replaced by a power series om even powers of $h$, namely

$$
E(h) = - (b-a)\frac{h^2}{12} f''(\eta_1) = \sum_{j=1}^n K_j h^{2j} + \mathcal{O}(h^{2n+2}),
$$

where the $K_j$ are constants, independent of the division $h$. Using this, and $h_2=h_1/2$ we can rewrite the two trapezoidal methods above as

$$
\begin{align}
I_1 &= \frac{h_1}{2}\left[f(a)+f(b)\right] + K_1 h_1^2 + K_2 h_1^4+ \mathcal{O}(h_1^6),\\
I_2 &= \frac{h_2}{2}\left[f(a)+2f(a+h_2)+f(b)\right] + K_1 h_2^2 + K_2 h_2^4 + \mathcal{O}(h_2^6),\\
&= \frac{h_1}{4}\left[f(a)+2f(a+h_2)+f(b)\right] + K_1 \frac{h_1^2}{4} + K_2 h_2^4+\mathcal{O}(h_1^6).
\end{align}
$$

Now note that

$$
\int_a^b f(x) dx=\frac{4 I_2 - I_1}{3} = \frac{h_2}{3}\left[f(a) + 4 f(a+h_2) + f(b) \right] - \frac{12}{3} K_2 h_2^4 + \mathcal{O}(h_2^6).
$$

The resulting numerical integration scheme is known as *Simpson's Rule* and the error is now $\mathcal{O}(h^4)$ rather than the $\mathcal{O}(h^2)$ for the (composite) trapezoidal rule.

This technique of eliminating error terms using results from two step sizes is generally referred to as Richardson's extrapolation.  We can also use this method to get rid of higher order error terms as well which brings us to *Romberg's Method*.

## Romberg's Method

We start by using the composite trapezoidal rule for $m_1=1,\, m_2=2,\, m_3=4,\,\cdots m_n=2^{n-1}$ subintervals with regularly spaced points $h_k=\frac{b-a}{m_k}$ apart. In particular, when we do this we don't want to recompute $f(x)$ any more than necessary so we need to write the trapezoidal rule for the different intervals in such a way that we use our previous work:

$$
\begin{align}
R_{1,1} &= \frac{h_1}{2}\left[f(a)+f(b)\right],\\
R_{2,1} &= \frac{h_2}{2}\left[f(a)+f(b) + 2 f(a+h_2)\right],\\
&= \frac{1}{2}\left[ \frac{h_1}{2}\left\{ f(a)+f(b)\right\} + 2h_2 f(a+h_2)\right],\\
&= \frac{1}{2}\left[ R_{1,1} + h_1 f(a+h_2)\right],\\
R_{3,1} &= \frac{h_3}{2}\left[f(a)+f(b) + 2\left\{f(a+h_3)+f(a+2 h_3) + f(a+3 h_r)\right\}\right],\\
&= \frac{1}{2}\left[ R_{2,1} + h_2 \left\{ f(a+h_3)+ f(a+3 h_r)\right\}\right],\\
&\vdots \\
R_{k,1} &= \frac{1}{2}\left[ R_{k-1,1}+h_{k-1} \sum_{i=1}^{2^{k-2}} f(a+(2i-1)h_k) \right].
\end{align}
$$

We now apply the extrapolation technique to refine our results using

$$
R_{k,2}=\frac{4 R_{k,1}-R_{k-1,1}}{3},
$$

on each successive pair of our previous results.

We then apply the extrapolation technique to successivly higher orders using

$$
R_{i,j} = \frac{4^{j-1}R_{i,j-1}-R_{i-1,j-1}}{4^{j-1}-1},\qquad j=2,3,\cdots,n.
$$

**Example**  Suppose we want to compute $\int_0^\pi \sin x\, dx$  (which in this case we know is $2$).  We construct the following tableau:

$$
\begin{align}
{\begin{array}{ccccccc}
    R_{1,1}    &         &         &         &         & &\\
    \downarrow & \rangle & R_{2,2} &         &         & &\\
    R_{2,1}    &         &         & \rangle & R_{3,3} & &\\
    \downarrow & \rangle & R_{3,2} &         &         & \rangle & R_{4,4}\\
    R_{3,1}    &         &         & \rangle & R_{4,3} & &\\
    \downarrow & \rangle & R_{4,2} &         &         & &\\
    R_{4,1}    &         &         &         &         &\\
  \end{array} }
\end{align}
$$

With actual numbers, this translates to

$$
\begin{align}
{\begin{array}{ccccccc}
    0    &         &         &         &         & &\\
    \downarrow & \rangle & 2.094... &         &         & &\\
    1.57079...    &         &         & \rangle & 1.9985... & &\\
    \downarrow & \rangle & 2.0045... &         &         & \rangle & 2.0000055...\\
    1.896...    &         &         & \rangle & 1.999983... & &\\
    \downarrow & \rangle & 2.00026... &         &         & &\\
    1.974...    &         &         &         &         &\\
  \end{array} }
\end{align}
$$

We see that even though the underlying composite trapezoidal rule results in the first column are not all that accurate, extrapolation of those results to the fourth column gives  six digits of accuracy.  It should also be clear that we can get an estimate for our (forward) error by comparing the difference between our best two results, in this case $R_{4,4}$ and $R_{4,3}$ where $|R_{4,4}-R_{4,3}|\sim 2\times 10^{-5}$.  Here, this is the error in $R_{4,3}$ and so an overestimate of the error for $R_{4,4}$.

While these are fairly easy to code up yourself, there are also scipy versions of all of these integration routines as illustrated below where we use the various schemes to integrate $\frac{1}{\sqrt{\pi}}e^{-x^2}$ from $0$ to $2$.  We start with the results of the cumulative trapezoidal and cumulative Simpson's rule with our interval divided into eight segments:

```python
import numpy as np

def f(x):
   return 1/np.sqrt(np.pi) * np.exp(-x**2)

x = np.linspace(0, 2, num=9, endpoint=True)
print("sample points: ",x)

y = f(x)
from scipy import integrate
I1 = integrate.trapezoid(y, x)
I2 = integrate.simpson(y, x)
print("trapezoidal = ",I1, ", Simpsons = ",I2)
```

    sample points:  [0.   0.25 0.5  0.75 1.   1.25 1.5  1.75 2.  ]
    trapezoidal =  0.49744809484415425 , Simpsons =  0.4976521729751664

The accuracy of these results is difficult to judge, other than by comparing them which gives us an estimate for the trapezoidal rule.  In contrast, the Romberg integration keeps adding a row to the tableau until the results converge:

```python
from scipy import integrate

normaldist = lambda x: 1/np.sqrt(np.pi) * np.exp(-x**2)
result = integrate.romberg(normaldist, 0, 2, show=True)
```

    Romberg integration of <function vectorize1.<locals>.vfunc at 0x0000020286FC9AF0> from [0, 2]

     Steps  StepSize   Results
         1  2.000000  0.574523
         2  1.000000  0.494815  0.468246
         4  0.500000  0.496836  0.497509  0.499460
         8  0.250000  0.497448  0.497652  0.497662  0.497633
        16  0.125000  0.497607  0.497661  0.497661  0.497661  0.497661
        32  0.062500  0.497648  0.497661  0.497661  0.497661  0.497661  0.497661
        64  0.031250  0.497658  0.497661  0.497661  0.497661  0.497661  0.497661  0.497661

    The final result is 0.4976611325094085 after 65 function evaluations.

Our previous results from the trapezoidal and Simpson's rule correspond to the first two results in the fourth row.

```python

```


dS/dt=-bSI+gI, dI/dt=bSI-gI (uso b para beta y g para gamma)

```python
from sympy import *
from sympy.abc import S,I,t,b,g
```

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
import pylab as pl
```

```python
#puntos criticos
P=-b*S*I+g*I
Q=b*S*I-g*I
#establecer P(S,I)=0 y Q(S,I)=0
Peqn=Eq(P,0)
Qeqn=Eq(Q,0)
print(solve((Peqn,Qeqn),S,I))
#matriz Jacobiana
J11=diff(P,S)
J12=diff(P,I)
J21=diff(Q,S)
J22=diff(Q,I)
J=Matrix([[J11,J12],[J21,J22]])
pprint(J)
```

    [(S, 0), (g/b, I)]
    ⎡-I⋅b  -S⋅b + g⎤
    ⎢              ⎥
    ⎣I⋅b   S⋅b - g ⎦

```python
#J en el punto critico
Jc1=J.subs([(S,S),(I,0)])
pprint(Jc1)
pprint(Jc1.eigenvals())
pprint(Jc1.eigenvects())
Jc2=J.subs([(S,g/b),(I,I)])
pprint(Jc2)
pprint(Jc2.eigenvals())
pprint(Jc2.eigenvects())
```

    ⎡0  -S⋅b + g⎤
    ⎢           ⎥
    ⎣0  S⋅b - g ⎦
    {0: 1, S⋅b - g: 1}
    ⎡⎛      ⎡⎡1⎤⎤⎞  ⎛            ⎡⎡-1⎤⎤⎞⎤
    ⎢⎜0, 1, ⎢⎢ ⎥⎥⎟, ⎜S⋅b - g, 1, ⎢⎢  ⎥⎥⎟⎥
    ⎣⎝      ⎣⎣0⎦⎦⎠  ⎝            ⎣⎣1 ⎦⎦⎠⎦
    ⎡-I⋅b  0⎤
    ⎢       ⎥
    ⎣I⋅b   0⎦
    {0: 1, -I⋅b: 1}
    ⎡⎛      ⎡⎡0⎤⎤⎞  ⎛         ⎡⎡-1⎤⎤⎞⎤
    ⎢⎜0, 1, ⎢⎢ ⎥⎥⎟, ⎜-I⋅b, 1, ⎢⎢  ⎥⎥⎟⎥
    ⎣⎝      ⎣⎣1⎦⎦⎠  ⎝         ⎣⎣1 ⎦⎦⎠⎦

Los puntos criticos son no hiperbolicos, por lo que no cumplen el teorema de Hartmann.

```python
b=1
g=1
def dx_dt(x,t):
    return [ -b*x[0]*x[1]+g*x[1] , b*x[0]*x[1]-g*x[1] ]
#trayectorias en tiempo hacia adelante
ts=np.linspace(0,10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#trayectorias en tiempo hacia atras
ts=np.linspace(0,-10,500)
ic=np.linspace(20000,100000,3)
for r in ic:
    for s in ic:
        x0=[r,s]
        xs=odeint(dx_dt,x0,ts)
        plt.plot(xs[:,0],xs[:,1],"-", color="orangered", lw=1.5)
#etiquetas de ejes y estilo de letra
plt.xlabel('S',fontsize=20)
plt.ylabel('I',fontsize=20)
plt.tick_params(labelsize=12)
plt.ticklabel_format(style="sci", scilimits=(0,0))
plt.xlim(0,100000)
plt.ylim(0,100000)
#campo vectorial
X,Y=np.mgrid[0:100000:15j,0:100000:15j]
u=-b*X*Y+g*Y
v=b*X*Y-g*Y
pl.quiver(X,Y,u,v,color='dimgray')
plt.savefig("SISinf.pdf",bbox_inches='tight')
plt.show()
```

Analsis de Bifurcaciones

El sistema tiene dos puntos criticos, (S,0) y (gamma/beta,I), el primer punto no depende de gamma ni beta, por lo que no cambia.


I assume Spong angles are $\theta_1$ and $\theta_2$.

```python
from __future__ import print_function, division
from sympy import *
init_printing(use_unicode=True)
```

```python
m1, l1, lc1, I1, theta1, theta1dot = symbols('m1 l1 l_{c1} I1 theta1 thetadot1')
m2, lc2, I2, theta2, theta2dot = symbols('m2 l_{c2} I2 theta2 thetadot2')
```

```python
M = Matrix([[m2*l1*(l1+2*lc2*cos(theta2))+I1+I2,
             m2*l1*lc2*cos(theta2)+I2],
            [m2*l1*lc2*cos(theta2)+I2,
             I2]])
```

```python
M
```

```python
C = Matrix([[-2*m2*l1*lc2*sin(theta2)*theta2dot,
             -m2*l1*lc2*sin(theta2)*theta2dot],
            [m2*l1*lc2*sin(theta2)*theta1dot,
             0]])
```

```python
C
```

```python
g = symbols('g')
G = g*Matrix([[-m1*lc1*sin(theta1)-m2*(l1*sin(theta1)+lc2*sin(theta1+theta2))],
              [-m2*lc2*sin(theta1+theta2)]])
```

```python
G
```

Now, let's change the notations to the one that Russ used [here](http://underactuated.csail.mit.edu/underactuated.html?chapter=acrobot) by having ${}^{\texttt{Russ}}I={}^{\texttt{Spong}}I+ml_c^2$ and ${}^{\texttt{Russ}}\theta_1={}^{\texttt{Spong}}\theta_1+\pi/2$

```python
q1, q1dot = symbols('q1 q1dot')
q2, q2dot = symbols('q2 q2dot')
```

```python
M.subs([(I1,I1+m1*lc1**2), (I2,I2+m2*lc2**2),
        (theta1,q1+pi/2), (theta2,q2)])
```

```python
C.subs([(I1,I1+m1*lc1**2), (I2,I2+m2*lc2**2),
        (theta1,q1+pi/2), (theta2,q2)])
```

```python
G.subs([(I1,I1+m1*lc1**2), (I2,I2+m2*lc2**2),
        (theta1,q1+pi/2), (theta2,q2)])
```


# 3.2 Linear Regression Models and Least Squares

We have an input vector $X^T=(X_1,...,X_p)$ and want to predict a real-valued output $Y$.The linear regression model has the form:

$$f(X) = B_0 + \sum_{j=1}^p {X_j\beta_j}$$

 Typically we have a set of training data $(x_1, y_1)...(x_N, y_n)$ from which to estimate the parameters $\beta$. The most popular estimation method is *least squares*, in which we pick $\beta$ to minimize the residual sum of squares, (3.2):
$$
\begin{align}
RSS(\beta)&=\sum_{i=1}^N(y_i-f(x_i))\\
&=\sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^p{x_{ij}\beta_j})^2
\end{align}
$$

How do we minimize (3.2)? We can write the (3.2) using matrix, (3.3):
$$RSS(\beta)=(\mathbf{y}-\mathbf{X}\beta)^T(\mathbf{y}-\mathbf{X}\beta)$$

Differentiating with respect to $\beta$ we obtain:
$$
\begin{align}
\frac{\partial{RSS}}{\partial\beta} = -2\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)
\end{align}
$$

Assuming that **X** has full column rank, and hence the second derivative is positive definite:
$$\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)=0$$

and the unique solution is:
$$\hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

The predicted value at an input vector $x_0$ are given by $\hat{f}(x_0)=(1:x_0)^T\hat{\beta}$:

$$\hat{y}=\mathbf{X}\hat{\beta}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

The matrix $\mathbf{H}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ is sometimes called the "hat" matrix.

**Geometrical representation of the least squares:** We denote the column vectors of **X** by $x_0, x_1, ..., x_p$. These vectors span a subspace of $\mathcal{R}^N$, also referred as the column space of **X**. We minimize $RSS(\beta)=||\mathbf{y}-\mathbf{X}\beta||^2$ by choosing $\hat{\beta}$ so that the residual vector $\mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to this subspace and the orthogonality is expressed by $\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)=0$. The hat matrix **H** is the projection matrix.

**Sampling properties of $\hat{\beta}$**: In order to pin down the sampling properties of $\hat{\beta}$, we assume that the observations $y_i$ are uncorrelated and have constant variance $\sigma^2$, and that the $x_i$ are fixed. The variance-covariance matrix is given by (3.8):

$$
\begin{align}
Var(\hat{\beta}) &= E\left[(\hat{\beta}-E(\hat{\beta}))(\hat{\beta}-E(\hat{\beta})^T)\right]\\
&= E\left[(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{\varepsilon}\mathbf{\varepsilon}^T\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\right]\\
&= \sigma^2(\mathbf{X}^T\mathbf{X})^{-1}
\end{align}
$$

One estimates the variance $\sigma^2$ by:
$$
\hat{\sigma}^2 = \frac{1}{N-p-1} \sum_{i=1}^N(y_i-\hat{y_i})^2
$$

The N-p-1 rather than N in the denominator makes $\hat{\sigma}^2$ an unbiased estimate of $\sigma^2$: $E(\hat{\sigma}^2)=\sigma^2$.

*Proof*:
$$
\begin{align}
\hat{\varepsilon} &= \mathbf{y} - \mathbf{\hat{y}}\\
&= \mathbf{X}\beta + \varepsilon - \mathbf{X}\hat{\beta}\\
&= \mathbf{X}\beta + \varepsilon - \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T(\mathbf{X}\beta + \varepsilon)\\
&= \varepsilon - \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon\\
&= (\mathbf{I}_n - \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T)\varepsilon\\
&= (\mathbf{I}_n - \mathbf{H})\varepsilon
\end{align}
$$

and we would like to find $Var(\hat{\varepsilon})=E(\hat{\varepsilon}^T\hat{\varepsilon})$:

$$
\begin{align}
E[\hat{\varepsilon}^T\hat{\varepsilon}]
&= E\left[\varepsilon^T(\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H})\varepsilon\right]\\
&= E\left[tr(\varepsilon^T(\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H})\varepsilon)\right]\\
&= E\left[tr(\varepsilon\varepsilon^T(\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H}))\right]\\
&= \sigma^2E\left[tr((\mathbf{I}_n - \mathbf{H})^T(\mathbf{I}_n - \mathbf{H}))\right]\\
&= \sigma^2E\left[tr(\mathbf{I}_n - \mathbf{H})\right]\\
&= \sigma^2E\left[tr(\mathbf{I}_n) - tr(\mathbf{I}_{p+1})\right]\\
&= \sigma^2(n-p-1)
\end{align}
$$

Note that, both $\mathbf{H}$ and $\mathbf{I_n}-\mathbf{H}$ are:

- Symmetry matrix, i.e $\mathbf{H}^T=\mathbf{H}$

- Idempotent matrix, i.e $\mathbf{H}^2=\mathbf{H}$

**Inferences about the parameters and the model:** We now assume that deviations of Y around its expectations and Gaussian. Hence (3.9):

$$
\begin{align}
Y &=E(Y|X_1,...,X_p)+\varepsilon\\
&= \beta_0 + \sum_{j=1}^P{X_j\beta_j} + \varepsilon
\end{align}
$$

where $\varepsilon \sim N(0, \sigma^2) $

Under (3.9), it is easy to show that (3.10):
$$
\hat{\beta} \sim N(\beta, (\mathbf{X}^T\mathbf{X})^{-1}\sigma^2)
$$

Also (3.11):
$$(N-p-1)\hat{\sigma}^2 \sim \sigma^2\chi_{N-p-1}^2$$

a chi-squared distribution with N-p-1 degrees of freedom and $\hat{\beta}$ and $\hat{\sigma}$ are statistically independent.

**Hypothesis test:** To test $H_0: \beta_j = 0$ we form the standardized coefficient or *Z-score*:
$$
z_j=\frac{\hat{B}_j}{\hat{\sigma}\sqrt{v_j}}
$$

where $v_j$ is the jth diagonal element of $(\mathbf{X}^T\mathbf{X})^{-1}$. Under the null hypothesis $z_j$ is distributed as $t_{N-p-1}$, and hence a large value of $z_j$ will lead to rejection. If $\hat{\sigma}$ is replaced by $\sigma$ then $z_j$ is a standard normal distribution. The difference between tail quantiles of a t-distribution and a standard normal become negligible as the sample size increases, see the Figure (3.3) below:

```python
# Figure 3.3
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm, t

fig = plt.figure(figsize = (12, 8))
axes = fig.add_subplot(1, 1, 1)

z = np.linspace(1.9, 3, 500)

normal_probabilities = 1 - norm.cdf(z) + norm.cdf(-z)
t_30_probabilities = 1 - t.cdf(z, 30) + t.cdf(-z, 30)
t_100_probabilities = 1 - t.cdf(z, 100) + t.cdf(-z, 100)

axes.plot(z, normal_probabilities, color='C0', label = 'normal')
axes.plot(z, t_30_probabilities, color='C1', label = '$t_{30}$')
axes.plot(z, t_100_probabilities, color='C2', label = '$t_{100}$')

xlim = axes.get_xlim()

for y in [0.01, 0.05]:
    axes.plot(xlim, [y, y], '--', color = 'gray',
              scalex = False, scaley = False)

    for index, probs in enumerate([normal_probabilities, t_30_probabilities,
                                   t_100_probabilities]):
        x = z[np.abs(probs - y).argmin()]
        axes.plot([x, x], [0, y], '--', color = f"C{index}",
                  scalex = False, scaley = False)

axes.legend()
axes.set_xlabel('Z')
axes.set_ylabel('Tail Probabilities')
plt.show()
```

**Test for the significance of groups of coefficients simultaneously**: We use the F-statistics (3.13):

$$F=\frac{(RSS_0-RSS_1) / (p_1 - p_0)}{RSS_1/(N-p_1-1)}$$

Where $RSS_1$ is for the bigger model with $p_1+1$ parameters and $RSS_0$ for the nested smaller model with $p_0+1$ parameters. Under the null hypothesis that the smaller model is correct, the F statistic will have a $F_{p_1-p_0,N-p_1-1}$ distribution. The $z_j$ in (3.13) is equivalent to the F statistic for dropping the single coefficient $\beta_j$ from the model.

Similarly, we can isolate $\beta_j$ in (3.10) to obtain $1-2\alpha$ confidence interval (3.14)
$$(\hat{\beta_j} - z^{(1-\alpha)}v_j^{\frac{1}{2}}\hat{\sigma}, \hat{\beta_j} + z^{(1-\alpha)}v_j^{\frac{1}{2}}\hat{\sigma})$$

In a similar fashion we can obtain an approximate confidence set for the entire parameter vector $\beta$ (3.15):
$$ C_{\beta} = \{{\beta|(\hat{\beta}-\beta)^T\mathbf{X}^T\mathbf{X}(\hat{\beta}-\beta)} \le \hat{\sigma}^2{\chi_{p+1}^2}^{(1-\alpha)} \}$$

where ${\chi_{l}^2}^{(1-\alpha)}$ is the $1-\alpha$ percentile of the chi-squared distribution of $l$ degrees of freedom.


## Regresión lineal

**Temas Selectos de Modelación Numérica**

Facultad de Ciencias, UNAM

Semestre 2021-2

En este notebook aprenderemos como hacer una regresión lineal por el método de mínimos cuadrados y por el método matricial.

No olvides resolver los ejercicios de tarea al final del notebook. Entrega tu solución en un notebook en la carpeta de Classroom con el nombre `apellido_nombre_tarea05.ipynb`.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## 1. Mínimos cuadrados

Ajuste de rectas de la forma $y=mx+b$ por mínimos cuadrados. La idea detrás de este método es que queremos encontrar la pendiente $m$ y la ordenada al origen $b$ que nos dan la recta que minimiza la suma de los cuadrados de las distancias entre los puntos (digamos, datos) y la recta ajustada:

Es decir, que cuando sumemos el cuadrado de todas las distancias de los puntos a la recta (líneas azules) el valor que obtengamos sea el más pequeño posible (este tipo de problemas se llaman problemas de optimización).

Se pueden ajustar todo tipo de funciones por este método, no sólo rectas, pero para el caso particular de la recta, se obtiene que la pendiente y la ordenada al origen de la recta que minimiza el cuadrado de las distancias se calcula como:

\begin{align}
\tag{1}
m =\frac{N \sum(x_iy_i) − \sum x_i\sum y_i}{N \sum(x_i^2) − (\sum x_i)^2} \label{eq1}\\
\end{align}

\begin{align}
b = \frac{\sum y_i − m \sum x_i}{N} \tag{2}
\end{align}

en donde $N$ es el número de mediciones o puntos, $x_i$, $y_i$ son las mediciones y las sumas ($\sum$) son sobre todas las mediciones.

**OJO**: Debido a que no es necesario graficar los datos para realizar un ajuste por mínimos cuadrados, se puede caer en errores graves como tratar de ajustar una recta a un conjunto de mediciones cuya relación no es lineal. Por eso **es muy importante graficar** los datos y asegurarse de que la relación entre las variables es lineal antes de aplicar el método de mínimos cuadrados.

Siguiendo las ecuaciones anteriores definamos una función `reg_lineal` que calcule a pendiente y ordenada al origen de la recta que mejor se ajusta a los "datos" usando el método de mínimos cuadrados:

```python
def reg_lineal(X,Y):
    '''Esta función calcula la pendiente y la ordenada al origen de la recta y=mx+b por mínimos
    cuadrados a partir de los vectores de mediciones X y Y.
    Input:
    X - arreglo de numpy 1D
    Y - arreglo de numpy 1D del mismo tamaño que X.
    Output:
    m, b : Escalares, la pendiente y ordenada al origen.
    '''
    N = len(X) # numero de valores del vector X
    sum_xy = np.sum(X*Y) # suma de todos los Xi*Yi
    sum_x = np.sum(X) # suma de todas las X
    sum_y = np.sum(Y) # suma de todas las Y
    sum_x2 = np.sum(X**2) # suma de todas las Xˆ2

    m = ((N*sum_xy) - (sum_x*sum_y)) / ((N*sum_x2) - (sum_x**2))
    b = (sum_y - (m*sum_x)) / N
    return(m, b)
```

Probemos nuestra función que calcula la regresión lineal. Para ello generamos un vector X y un vector f(X)=Y de la siguiente manera:

```python
X = np.linspace(1,10,10)
Y = 1 + 2*X + 1*np.random.randn(1) # f(x)=y=1+2x+d

plt.plot(X,Y,'o')
plt.xlabel('x')
plt.ylabel('y=f(x)')
plt.show()
```

Ahora podemos probar la función `reg_lineal` usando X y Y:

```python
m, b = reg_lineal(X,Y)
print('La pendiente m es %f y la ordenada b es %f' %(m,b))
Y2 = m*X+b

plt.plot(X,Y,'o', label='Y')
plt.plot(X,Y2,'-',label='regresión' )
plt.xlabel('x')
plt.ylabel('y=f(x)')
plt.legend()
plt.show()
```

## 2. Método matricial
(Nota: El material de esta sección fue tomado del blog [cmdlinetips](https://cmdlinetips.com/2020/03/linear-regression-using-matrix-multiplication-in-python-using-numpy/))

También podemos hacer regresiones lineales usando el método matricial. Recordemos que en una regresión lineal queremos ajustar nuestros datos, observaciones, etc. usando el modelo lineal $$y=\beta_0+\beta_1X+\epsilon$$ y estimar los parámetros del modelo $\beta_0$ y $\beta_1$ que son la ordenada al origen y la pendiente, respectivamente.

Podemos combinar las "variables predictivas", en este caso X, en una matriz que tiene un vector columna lleno de unos (lo que multiplica a $\beta_0$) y X (lo que multiplica a $\beta_1$):

```python
X_mat = np.vstack((np.ones(len(X)), X)).T # en este caso usamos la función vstack
                                          # y el método T (transponer) para obtener las dimensiones
                                          # adecuadas
```

Con un poco de álgebra lineal y el objetivo de minimizar el error cuadrático medio del sistema de ecuaciones lineales llegamos a que podemos calcular el valor de los parámetros $\hat{\beta}=(\beta_0, \beta_1)$ de la forma:

$$\hat{\beta}=(X^T. X)^{-1}. X^T. Y$$

Podemos implementar esta ecuación usando las funciones para la inversa de una matriz y multiplicación matricial del módulo de álgebra lineal de numpy `linalg`:

```python
beta = np.linalg.inv(X_mat.T.dot(X_mat)).dot(X_mat.T).dot(Y)

print('La pendiente beta_1 es %f y la ordenada beta_0 es %f' %(beta[1], beta[0]))
```

    La pendiente beta_1 es 2.000000 y la ordenada beta_0 es 2.756253

que son los mismos valores para la pendiente y ordenada al origen que encontramos usando la función `reg_lineal`. Ahora usemos estos parámetros para estimar los valores de Y:

```python
Y_mat = X_mat.dot(beta)

plt.plot(X,Y,'o', label='Y')
plt.plot(X,Y_mat,'-',label='regresión método matricial' )
plt.xlabel('x')
plt.ylabel('y=f(x)')
plt.legend()
plt.show()
```

## 3. Ejemplo usando ambos métodos

```python
X = np.linspace(1,10)
Y = 1 + 2*X + 3*X**2 + 20*np.random.randn(1)

# Método matricial - ahora tenemos otro término, Xˆ2
X_mat = np.vstack((np.ones(len(X)), X, X**2)).T
beta = np.linalg.inv(X_mat.T.dot(X_mat)).dot(X_mat.T).dot(Y)
Y_mat = X_mat.dot(beta)

# Función reg_lineal
m, b = reg_lineal(X,Y)
Y2 = m*X+b

plt.plot(X,Y,'o',label='datos')
plt.plot(X,Y_mat, label='Método matricial')
plt.plot(X,Y2, label='Mínimos cuadrados - recta')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend()
plt.show()
```

Ups, claramente ajustar los datos a una recta no era una buena idea. Por eso es importante hacer una inspección gráfica de lo que queremos ajustar.

### Ejercicios de tarea:

1. Desarrollar las expresiones para la ordenada al origen y la pendiente en una recta usando el método de mínimos cuadrados. (Entregar el desarrollo en un archivo aparte).

2. Programar una función que calcule la regresión de un polinomio de grado 3. En un notebook define la función y puébala usando el polinomio de grado 3 $y = 2x^3+5x^2-11x+7$.

```python

```


# Part 1: Linear Regression

```python
# Execute this code block to install dependencies when running on colab
try:
    import torch
except:
    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision
```

## Getting started

At its heart, PyTorch is just a library for manipulating tensors. We're going to start learning how to use
PyTorch by looking at how we can implement simple linear regression.

Code speaks better than words, so lets start by looking at a bit of pytorch code to generate some 2d data to regress:

```python
import torch

# Generate some data points on a straight line perturbed with Gaussian noise
N = 1000 # number of points
theta_true = torch.Tensor([[1.5], [2.0]]) # true parameters of the line

X = torch.rand(N, 2)
X[:, 1] = 1.0
y = X @ theta_true + 0.1 * torch.randn(N, 1) # Note that just like in numpy '@' represents matrix multiplication and A@B is equivalent to torch.mm(A, B)
```

The above code generates $(x,y)$ data according to $y = 1.5x + 2$, with the $x$'s chosen from a uniform distribution. The $y$'s are additionally purturbed by adding an amount $0.1z$, where $z\sim \mathcal{N}(0,1)$ is a sample from a standard normal distribution.

Note that we represent our $x$'s as a two-dimensional (row) vector with a 1 in the second element so that the offset can be rolled into the matrix multiplication for efficiency:

\begin{align}
    y &= \mathbf{X}\begin{bmatrix}
           1.5 \\
           2
         \end{bmatrix}
  \end{align}

Let's use `matplotlib` to draw a scatter so we can be sure of what our data looks like:

```python
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X[:,0].numpy(), y.numpy())
plt.show()
```

__Make sure you understand how the code above is generating data; feel free to change the parameters to see what effect they have.__

Now, lets consider the situation where we have been given the tensors $X$ and $y$ and wish to compute the regression parameters. Our model looks like $\mathbf{y} = \mathbf{X\theta}$, and we wish to recover the parameters $\theta$.

As the problem is both overcomplete (only two data pairs are required to find $\theta$), and the data is noisy, we can use the Moore-Penrose Pseudoinverse to find the least-squares solution to $\theta$: $\theta = \mathbf{X^+y}$. PyTorch has a built-in pseudoinverse method (`pinverse`) that can do all the work for us:

```python
# direct solution using moore-penrose pseudo inverse
X_inv = torch.pinverse(X)
theta_pinv = torch.mm(X_inv, y)
print(theta_pinv)
```

Running the above code should give you a solution vector for $\theta$ that is very similar to the true parameter vector (`theta_true`).

## Exercise: computing the pseudoinverse from the Singular Value Decomposition.

The standard way of computing the pseudoinverse is by using the Singular Value Decomposition (SVD). The SVD is defined as: $\mathbf{X} = \mathbf{U}\Sigma\mathbf{V}^\top$. The pseudoinverse is thus $\mathbf{X}^+ = \mathbf{V}\Sigma^{-1}\mathbf{U}^\top$ where $\Sigma^{-1}$ is a diagonal matrix in which the reciprocal of the corresponding non-zero elements in $\Sigma$ has been taken.

__Use the code block below to compute the parameter vector using the SVD directly rather than the through the `pinverse` method.__ You need to store your manually computed pseudoinverse in `X_inv_svd`. Useful methods will be `torch.svd()` to compute the SVD, `[Tensor].t()` to transpose a matrix and `torch.diag()` to form a diagonal matrix from a vector.

```python
# YOUR CODE HERE
raise NotImplementedError()

theta_pinv_svd = torch.mm(X_inv_svd, y)
print(theta_pinv_svd)
```

```python
assert(torch.all(torch.lt(torch.abs(torch.add(theta_pinv, -theta_pinv_svd)), 1e-6)))
```

## Gradient based Linear Regression

Fundamentally, with linear regression we are trying to find a solution vector, $theta$ that minimises $f(\theta) = 0.5\|\mathbf{X}\theta - \mathbf{y}\|_2^2$.

We've already seen how this can be minimised directly using the pseudoinverse, but it could also be minimised by using gradient descent: $\theta \gets \theta - \alpha f'(\theta)$. (_Interesting aside_: SVD (and thus the pseudoinverse) can also be solved using gradient methods - in fact this becomes the only practical way for really large matrices.).

__Use the following block to derive and write down the gradient, $f'(\theta)$, of $f(\theta)$__. Note that you can insert latex code by wrapping expressions in dollar symbols.

__Now complete the following code block to implement your gradient as pytorch code:__

```python
def linear_regression_loss_grad(theta, X, y):
    # theta, X and y have the same shape as used previously
    # YOUR CODE HERE
    raise NotImplementedError()
    return grad
```

```python
assert(linear_regression_loss_grad(torch.zeros(2,1), X, y).shape == (2,1))

```

Now we can plug that gradient function into a basic gradient descent solver and check that the solution is close to what we get with the pseudoinverse:

```python
alpha = 0.001
theta = torch.Tensor([[0], [0]])
for e in range(0, 200):
    gr = linear_regression_loss_grad(theta, X, y)
    theta -= alpha * gr

print(theta)
```

## Real data

Doing linear regression on synthetic data is a great way to understand how PyTorch works, but it isn't quite as satisfying as working with a real dataset. Let's now apply or understanding of computing linear regression parameters to a dataset of house prices in Boston.

We'll load the dataset using scikit-learn and perform some manipulations in the following code block:

```python
from sklearn.datasets import load_boston

X, y = tuple(torch.Tensor(z) for z in load_boston(True)) #convert to pytorch Tensors
X = X[:, [2,5]] # We're just going to use features 2 and 5, rather than using all of of them
X = torch.cat((X, torch.ones((X.shape[0], 1))), 1) # append a column of 1's to the X's
y = y.reshape(-1, 1) # reshape y into a column vector
print('X:', X.shape)
print('y:', y.shape)

# We're also going to break the data into a training set for computing the regression parameters
# and a test set to evaluate the predictive ability of those parameters
perm = torch.randperm(y.shape[0])
X_train = X[perm[0:253], :]
y_train = y[perm[0:253]]
X_test = X[perm[253:], :]
y_test = y[perm[253:]]
```

__Use the following code block to compute the regression parameters using the training data in the variable `theta` by solving using the pseudoinverse directly:__

```python
# compute the regression parameters in variable theta
# YOUR CODE HERE
raise NotImplementedError()
```

We can now print out the error achieved on the test set, as well as the parameter vector:

```python
assert(theta.shape == (3,1))

print("Theta: ", theta.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta, y_test))
```

Now let's try using gradient descent:

```python
alpha = 0.00001
theta_gd = torch.rand((X_train.shape[1], 1))
for e in range(0, 10000):
    gr = linear_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr

print("Gradient Descent Theta: ", theta_gd.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))
```

__Use the following block to note down any observations you can make about the choice of learning rate and number of iterations in the above code. What factors do you think influence the choice?__

```python
# YOUR CODE HERE
raise NotImplementedError()
```

Finally, just so we can visualise what our model has learned, we can plot the predicted house prices (from both the direct solution and from gradient descent) along with the true value for each of the houses in the test set (ordered by increasing true value):

```python
perm = torch.argsort(y_test, dim=0)
plt.plot(y_test[perm[:,0]].numpy(), '.', label='True Prices')
plt.plot((X_test[perm[:,0]] @ theta).numpy(), '.', label='Predicted (pinv)')
plt.plot((X_test[perm[:,0]] @ theta_gd).numpy(), '.', label='Predicted (G.D.)')
plt.xlabel('House Number')
plt.ylabel('House Price ($,000s)')
plt.legend()
plt.show()
```


# Scientific Computing: Introduction to Python

Dr. Ilian Iliev

Let's start with some examples of simple printing. Note that in Python 3 the brackets are required (not the case in Python 2). Quotes can be single or double.

```python
print('hello')
```

    hello

```python
print("hello")
```

    hello

## Using Python as a (quite powerful) calculator
You can print arithmetic expressions directly at the Python prompt and it will evaluate them and give you the result, just as a calculator will do. (Note: The pound sign (#) denotes the beginning of a comment – all characters between # and the end of the line are ignored
by the Python interpreter.). For example:

```python
5+9
```

    14

```python
3**50
```

    717897987691852588770249L

```python
2000/25  #division
```

    80

```python
2000//25 #integer division
```

    80

```python
2345677*25
```

    58641925

```python
36**0.5  #square root (Note: sqrt() is not in core Python!)
```

    6.0

Note that (unlike many other programming languages) Python can do arbitrary-length arithmetic (Note: When the number has more digits than normal arithmetic Python 2 appends 'L' at the end of the result, for 'Long'; Python 3 does not do that.).

```python
45%7   # modulus (remainder)
```

    3

# Note: Beware of integer division in Python 2!
Many people have been tripped by it. Try for example:

```python
print(3/2, 3/2.0, 3.0/2.0, float(3)/2, 3/float(2))
```

    (1, 1.5, 1.5, 1.5, 1.5)

The best solution is to make sure you always divide real numbers (floats). That is not an issue for Python 3, however.
# Exercise: Evaluate some more arithmetic expressions on your own.
Core Python supports only a few mathematical functions, shown with some examples below

```python
a=-7
```

```python
b={3,9,2,55}
```

```python
c=3.1415
```

```python
abs(a)  #absolute value
```

    7

```python
max(b)  # Largest element in a sequence
```

    55

```python
min(b)  # Smallest element in a sequence
```

    2

What about other mathematical functions like e.g. square root, logarithms, trigonometric functions, etc.? Those are not built into the core Python, but are available by loading the math module (we will also discuss other modules later).

There are three ways of accessing the functions available in a module:

```python
import math  # make the 'math' module available. This is the
             # RECOMMENDED method because it is economical on resources
             # and a good programming practice. This is the method used
             # by most programmers.
```

```python
print(math.log(math.sin(0.5))) # Then this is how you use it - each
                               # function gets 'math' prefix telling
                               # Python where to get this function from.
```

    -0.735166686385

You can also 'rename' modules when importing them, for brevity or other reasons:

```python
import math as m  # From now on module math has also this other name 'm'
```

```python
m.cosh(3)
```

    10.067661995777765

You can also import a module like this:

```python
from math import *   # import ALL math module functions
```
For small modules this could be OK, but for big ones with many functions like 'math' this is wasteful, and also a bad idea for other reasons.

Finally, you can also just import specific functions that you need, in which case you can use the functions without a module prefix:

```python
from math import log, sin
```

```python
print(log(sin(0.5)))
```

    -0.735166686385

The contents of a module could be printed as follows (for example to check if a certain function is available). This type of inquiry is usually called 'introspection':

```python
dir(math)
```

    ['__doc__',
     '__file__',
     '__name__',
     '__package__',
     'acos',
     'acosh',
     'asin',
     'asinh',
     'atan',
     'atan2',
     'atanh',
     'ceil',
     'copysign',
     'cos',
     'cosh',
     'degrees',
     'e',
     'erf',
     'erfc',
     'exp',
     'expm1',
     'fabs',
     'factorial',
     'floor',
     'fmod',
     'frexp',
     'fsum',
     'gamma',
     'hypot',
     'isinf',
     'isnan',
     'ldexp',
     'lgamma',
     'log',
     'log10',
     'log1p',
     'modf',
     'pi',
     'pow',
     'radians',
     'sin',
     'sinh',
     'sqrt',
     'tan',
     'tanh',
     'trunc']

Many of the above functions should be familiar to you. If you are not sure what a given function do or how to call it, you can invoke the Python help system, as follows:

```python
help(modf)
```

    Help on built-in function modf in module math:

    modf(...)
        modf(x)

        Return the fractional and integer parts of x.  Both results carry the sign
        of x and are floats.

Exercises:

1) The Universe is 13.6 billion years old, how many seconds old is it?

2) Compute 2 to the 266th power. This is (roughly) the number of atoms
in the Universe.

3) Import the math module and calculate:

\begin{equation}
\cos(\pi/4)
\\
\pi-4 atan(1)
\end{equation}

Do results conform with your expectations?

# Variables and important data types
In Python, all numbers (and everything else, including functions) are
OBJECTS (i.e. Python, like C++ is an object-oriented language). A variable can be used to store a certain value or object. There are several basic types of variables that we will need: integers, floats (real numbers), complex numbers, strings and combinations of these (lists, tuples and arrays).

Variables are typed dynamically, i.e. their type (which can be checked with 'type(var)') is set whenever the variable is assigned a value and can be changed by assigning another value:

```python
a=1  #integer
```

```python
type(a)
```

    int

```python
b=1.0 #float=real
```

```python
type(b)
```

    float

```python
c='1.0' #string
```

```python
type(c)
```

    str

```python
d=2+3.0j #complex number
```

```python
type(d)
```

    complex

A string is a sequence of characters enclosed in single or double quotes. Long strings continuing on several lines are defined using triple quotes.

Strings are concatenated with the plus (+) operator, whereas slicing (:) is used to extract a portion of the string:

```python
string1 = 'Press return to exit'
```

```python
string2 = 'the program'
```

```python
print(string1 + ' ' + string2) # Concatenation
```

    Press return to exit the program

```python
print(string1[0:12])  # Slicing
```

    Press return

Strings are immutable objects (i.e. its individual characters cannot be modified with an assignment statement; the whole string CAN be assigned a different value) with a fixed length.

An attempt to violate immutability will result in TypeError, as shown here:

```python
s='Press return to exit'
```

```python
s[0]  # Individual characters can be accessed
```

    'P'

```python
s[0]='T' # ... but they cannot be modified
```

```python
b=2; c=4 # b and c are of an integer type
```

```python
b=b*1.0; c=float(c) # two ways to convert them back into floats
```

```python
print(b,c) # Both b and c are floats now, the original values were destroyed
```

    (2.0, 4.0)

```python
d=int(b); print(d) # 'int' converts a number into an integer
```

    2

The function 'round' with a single argument rounds up a number to the nearest integer (up or down). You can also round a number with certain number of significant digits after the decimal point.

Strings also can be converted into numbers if they are made of numbers:

```python
a=5.78666; print(round(a)); print(round(a,3))
```

    6.0
    5.787

```python
s='357'; type(int(s))
```

    int

```python
print(int(s))
```

    357

# Exercises:

1) Create a string variable 'myname' that is initialised to
your full name - first, middle and last.

2) Using a slice operator, print your first name only.

3) Using a slice operator, print your last name only.

4) Using the slice and concatenation operators, print
your name in the form 'Last name, First name'.

5) Create a new string where your middle name is replaced by
your middle initial.

6) Use Python to check how long the 'myname' string is and
if it contains the letter 'a'.

7) What happens if you multiply a string and a number?

# Tuples and lists
A tuple is a sequence of arbitrary objects separated by commas and enclosed in parentheses. If the tuple contains a single object, a final comma is required; for example, 'x = (2,)'. Tuples support the same operations as strings and are also immutable. Here is an example where the tuple 'rec' contains another tuple '(6,23,68)':

```python
rec = ('Smith','John',(24,7,1988))   # This is a tuple
```

```python
lastName,firstName,birthdate = rec # Unpacking the tuple into its parts
```

```python
print(firstName)
```

    John

```python
birthYear=birthdate[2]; print(birthYear)
```

    1988

```python
name = rec[1] + ' ' + rec[0] # combine the two name strings
```

```python
print(name)
```

    John Smith

```python
print(rec[0:2]) # Slice the first 2 elements of the tuple 'rec'
```

    ('Smith', 'John')

### Important note: In Python sequences have zero offset, so that a[0] represents the first element of a, a[1] the second one, and so forth. Similarly, indices in Python always start from 0, so e.g. 2 is the third element, etc.
Lists are ordered sets of objects similar to tuples, but they are mutable, i.e. their elements and length can be changed. A list is identified by enclosing its elements in square brackets, [,]. The objects in a list could be of any type, e.g. real or integer numbers, strings, even other lists.

Below is a sampling of operations that can be performed on lists:

```python
a = [1.0,'two',3]  # Define a list
```

```python
a[0]  # First element of the list
```

    1.0

```python
a[1]  # Second element of the list
```

    'two'

```python
a[3]  # There is no such element, so this gives an error.
```

```python
a.append(33.0)  # Append another element (33.0) to the list
```

```python
print(a)
```

    [1.0, 'two', 3, 33.0]

```python
a.insert(0,22)  # Insert another element (22) at position 0
```

```python
print(a)
```

    [22, 1.0, 'two', 3, 33.0]

Question: What does a[-1] give? What about a[-3]?If a is a mutable object, such as a list, the assignment statement b=a
does not result in a new object b, but simply creates a new reference to a (i.e. now the same object has 2 names - a and b). Thus any changes made to b will be reflected in a (but not vice-versa). To create an independent copy of a list a, use the statement 'c = a[:]', as shown below:

```python
b=a  # 'b' becomes an alias of a
```

```python
b[0]=5  # change an element of b
```

```python
print(a) # a has changed
```

    [5, 1.0, 'two', 3, 33.0]

```python
c=a[:]  # 'c' is an independent copy of a
```

```python
c[0]=4  # change an element of c
```

```python
print(a) # a remains the same
```

    [5, 1.0, 'two', 3, 33.0]

There are various oprations that can be performed on sequences like lists. Some examples:

```python
print(a[1:3]) # slices a part of a (here the second and third elements)
```

    [1.0, 'two']

```python
len(a)        # how many elements does 'a' have?
```

    5

```python
'two' in a   # checks if a certain element can be found in the list
```

    True

```python
a[2]=7       # lets make the list wholly of numbers (strings cannot be compared to numbers!)
```

```python
max(a)       # what is the largest element of the sequence?
```

    33.0

```python
min(a)       # what is the smallest element in the sequence?
```

    1.0

```python
d=a+c        # concatenates the two sequences
```

```python
print(d)
```

    [5, 1.0, 7, 3, 33.0, 4, 1.0, 'two', 3, 33.0]

# Comparison operators
The comparison operators are:
< (less than),
> (greater than),
<= (less than or equal to),
>= (greater than or equal to),
== (equal to)
!= (not equal to).

They are quite useful for programming, as we shall see. The result is a logical expression ('True' or 'False'). These can be used to construct more complicated logical statements using also 'or' and 'and'.

Numbers of different type (integer, floating point, etc.) are converted to a common type before the comparison is made. Otherwise, objects of different type are considered to be unequal (e.g. 2 does not equal 2.0). Here are a few examples:

```python
a=5.7
```

```python
b=4.3
```

```python
print(a>b)
```

    True

```python
print(a==b)
```

    False

```python
c=3
```

```python
a>b and c<b
```

    True

# Exercises:

1) What is the result of the Boolean expression 'not 8>5'?

2) What is the result of the Boolean expression 'not(True and False)'? Is this what you expected?

3) Write a compound Boolean expression that returns True if the value    of the variable 'count' is between 1 and 10 inclusive.

# Conditionals
Conditionals are quite useful in construction of algorithms and are
an important program control structure. The 'if' construct

if condition:
    block

executes a block of statements (which MUST be indented since that
is how Python knows what is inside the 'if' and what is not!) if the
condition returns true. If the condition returns false, the block
is skipped. The if conditional can be followed by any number of
'elif' (short for “else if”) constructs

if condition:
    block1
elif condition:
    block2

which work in the same manner. The 'else' clause

else:
   block

can be used to define the block of statements that are to be executed if none of the if-elif clauses is true. The function below illustrates the use of the conditionals:

```python
def sign_of_a(a):
    if a < 0.0:
        sign = 'negative'
    elif a > 0.0:
        sign = 'positive'
    else:
        sign = 'zero'
    return sign

a = 1.5
print('a is ' + sign_of_a(a))
```

    a is positive

Here 'def' defines a function, to be discussed below (similar to Matlab
functions).
# Exercises:

1) If you have several nested 'if/else'  constructs, how does Python
know to which 'if' an 'else' belongs? Try to check your answer
using an example.

2) Write a construct that sets the value of a variable called
'grade' to the value 4 if a variable named 'score' is greater
than 70, 3 if 'score' is between 60 and 69, 2 if 'score' is
between 50 and 59, 1 if 'score' is between 40 and 49, and 0 otherwise.
This statement (roughly) converts British-style grades to American-style ones.

# Loops
One of the most typical tasks we use a computer for is to do a certain task multiple times (e.g. for different input values). Such repetitive tasks (at which computers are really good, since unlike people they do not get bored) are performed by loops. There are two basic types of loops - a 'while' loop and a 'for' loop.
## 'While' loops
The 'while' construct:

while condition:
    block

executes a block of (again, indented) statements if the condition is true. After execution of the block, the condition is evaluated again. If it is still true, the block is executed again. This process is continued until the condition becomes false. The 'else' clause

else:
    block

can be used to define the block of statements that are to be executed if the condition is false.

Here is an example that creates the list [1, 1/2, 1/3, . . .]:

```python
nMax=5
```

```python
n=1
```

```python
a=[]  # Create empty list
```

```python
while n < nMax:
    a.append(1.0/n) # Append element to list
    n = n + 1
    print(a)
```

    [1.0]
    [1.0, 0.5]
    [1.0, 0.5, 0.3333333333333333]
    [1.0, 0.5, 0.3333333333333333, 0.25]

Note the indentations above - they indicate what does or does not belong to the loop (since there is no explicit loop end) and are thus absolutely necessary. The IPython shell will do the indentations for you automatically (you indicate the end the loop by pressing 'Return' twice), but an external editor might not do it for you, in which case you have to do it yourself.
## The range() command

A special type of list is frequently required (often together with
for-loops, see below) and therefore a command exists to generate that
list: the 'range(stop)' command generates a list of integers starting
from 0 and going up to but NOT INCLUDING n. You can also specify
both the start and stop points, as well as (optionally) the step

range(start,stop[,step])

(see help for further details). Here are a few examples:

```python
for a in range(5):
    print(a)          # print a
```

    0
    1
    2
    3
    4

```python
for a in range(3,10,2):
    print(a)
```

    3
    5
    7
    9

# For loops
The 'for' loops are explicit in terms of how many times the loop statements will be executed, rather than conditional as the 'while' loops we discussed above:

for target in sequence:
    block of instructions

and repeat the operations in 'block' for all values in 'sequence'. Unlike many other programming languages (e.g. C, Fortran), the loops in Python need not be over an integer index (as were the 'range' examples above), but can be performed directly over the sequence elements, which need not even be numbers. Here is an example where we iterate over a list of strings and within it over the letters in each string:

```python
fruits = ['apple','banana','orange']
```

```python
for fruit in fruits:
    for letter in fruit:
         print(letter)
```

    a
    p
    p
    l
    e
    b
    a
    n
    a
    n
    a
    o
    r
    a
    n
    g
    e

Python provides further statements for controlling the flow of a program: break, continue and else.

The 'break' command issued inside a loop immediately ends the loop and moves execution to the statements immediately following that loop.

```python
x=0
```

```python
while True: # this will go on forever unless stopped
    x += 1  # add 1 to x and put the result back in x
    if not (x % 15 or x % 25):
        break
print(x,  'is divisible by both 15 and 25')
```

    (75, 'is divisible by both 15 and 25')

The 'continue' statement allows us to skip a portion of the statements in an iterative loop. If the interpreter encounters the continue statement, it immediately returns to the beginning of the loop to start the next iteration without executing the statements below 'continue'.

The following example compiles a list of all numbers between 1 and 99 that are divisible by 7:

```python
x = []  # Create an empty list
for i in range(1,100):
   if i%7!= 0: continue # If not divisible by 7, skip rest of loop
   x.append(i) # Append i to the list
print(x)
```

    [7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98]

Finally, a 'for' or 'while' loop may be followed by an 'else' block of statements, which is executed only if the loop finished 'normally', i.e. without using 'break'.
# Exercises:

1) Write a for loop which calculates the first 10 terms of the Madhava series:

\begin{equation}
\sqrt{12}\left(1-\frac1{3\times3}+\frac1{5\times3^2}-\frac1{7\times3^3}+...\right)
\end{equation}

(this is one way to calculate $\pi$ using series).

2) Use a 'while' loop to calculate

\begin{equation}
x_{k+1}=\frac12\left(x_k+\frac2{x_k}\right)
\end{equation}

with x(0)=1, until $abs(x(k+1)-\sqrt(2)) < 0.0001$. (As we will discuss later in the term, this is a way to calculate square roots using just addition and division).

3) The double factorial function, n!!, is the product of the positive odd integers up to and including n (which must itself be odd):

\begin{equation}
 n!!=\prod\limits_{i=1}^{(n+1)/2}(2i-1)=1\cdot3\cdot5\dots(n-2)\cdot n
\end{equation}

Write a routine to calculate n!! in Python.

4) A very elegant way to calculate the greatest common denominator of two numbers is the ver elegant Euclid agorithm:

In [7]: a,b=1071,462

In [8]: while b:

            a,b=b,a%b

In [9]: print a

21

Explain how it works.

# Reading input and writing output
The intrinsic Python 2.x function for accepting user input is

raw_input(prompt)

Note: In Python 3.x this function is renamed 'input()'. In the Python 3.x version of our textbook (Kyusalaas) this is incorrect and 'raw_input' is still being used.

The 'raw_input' function displays the prompt and then reads a line of input that is converted into a string. To convert the string back into a numerical value (if it is a number), you can use the function

eval(string)

```python
a = raw_input("Input a: ") # Ask for input
```

    Input a: 34

```python
print(a,type(a))  # Print a and its type
```

    ('34', <type 'str'>)

```python
b=eval(a)  # b is a number
```

```python
print(b,type(b))  # 'b' is a number (integer)
```

    (34, <type 'int'>)

As we saw earlier, output can be displayed with the print statement:

print(object1, object2, . . .)

which converts object1, object2, and so on to strings and prints them on the same line, separated by spaces. The newline character '\n'  can be used to force a new line. For example,

```python
a=1234.56789
```

```python
b=[2,4,6,8]
```

```python
print(a,b)
```

    (1234.56789, [2, 4, 6, 8])

```python
print 'a=',a,'\nb=',b
```

    a= 1234.56789
    b= [2, 4, 6, 8]

The modulo operator '%' can be used to format a tuple. The form of the conversion statement is

'%format1 %format2 ... ' % tuple

where format1, format2 ... are the format specifications for each object in the tuple. Typically used format specifications are:

wd      Integer
w.df    Floating point notation
w.de    Exponential notation

where 'w' is the width of the field and d is the number of digits after the decimal point. The output is right-justified in the specified field and padded with blank spaces (there are provisions for changing the justification and padding). A few examples:

```python
n=9876
```

```python
print('%7.2f'%a)
```

    1234.57

```python
print('n=%6d'%n)  # Pad with spaces
```

    n=  9876

```python
print('n=%06d'%n) # Pad with zeroes
```

    n=009876

```python
print('%12.4e %6d' %(a,n))
```

      1.2346e+03   9876

# Exercise:
Calculate and print on the screen a temperature conversion table from Fahrenheit to Celsius scales. The table should include temperatures from -300 to 210 degrees Fahrenheit (in steps of 5
degrees) and their Celsius equivalents, presented in 2 columns
with appropriate headings. Each column should be 10 characters wide, and each temperature should have 3 digits to the right of the decimal point. The conversion formula is:

C=(F-32)(5/9)
# Reading from and writing to a file
Up to this point, all data we have used has been either hard-coded into the program or has been obtained from or written to the screen. However, often we need to read from an external file, or write it to such a file. This is done through Python 'file' objects.

A file object is created by opening a file with a given filename and mode. The filename could be given as an absolute path (from the root directory/folder up) or relative one (i.e. from the current working directory). 'Mode' is a string with one of the values given below, which indicates what the file will be used for (reading, writing, or both) and what type of data is being read/written (text, binary).

mode        meaning
r           text, read-only (the default)
w           text, write (an existing file with the same name will be overwritten)
a           text, append to an existing file
r+          text, reading and writing
rb          binary, read-only
wb          binary, write (an existing file with the same name will be overwritten)
ab          binary, append to an existing file
rb+         binary, reading and writing

For example:

```python
f=open('my_file.txt','w')   # open a text file for writing
```
file objects are closed with the 'close' method:

```python
f.close()
```
Note: Python automatically closes any open files when a program terminates.The 'write' method of a file object writes a string to the file (in Python3 it also returns the number of characters being written):

```python
f=open('my_file.txt','w')   # open a text file for writing
```

```python
f.write('Hello')
```

```python
f.close()
```
To read n bytes from a file we use the 'read' method, f.read(n). If n is omitted , the entire file is read in (it is your problem if the file is too large, as stated in the official documentation).

'readline()' reads a single line from a file, up to nd including the newline ('\n') character. Each subsequent call to 'readline' reads another line from the file. Both 'read' and 'readline' return an empty string when they reach the end of the file. To read all of the lines into a list of strings in one go, use 'f.readlines()'.

A better method in practice is to use Numpy to read and write data to files (to be discussed later).
# Functions
We have already encountered many internal Python functions. One can also define new, custom functions which behave the same way as the internal ones. The structure of a Python function is

def  func_name(param1,param2,...):
    statements
    return return_values

(again, note the indentations!), where 'param1, param2, ...' are the parameters. A parameter can be any Python object, including another function. Parameters may be given default values, in which case the parameter in the function call is optional. If the return statement or return values are omitted, the function returns the null object (i.e. nothing).

The following example computes numerically the first two derivatives of arctan(x) using finite differences (we will discuss this method later in the course, for now focus on the Python constructs only):

```python
from math import atan
def finite_diff(f,x,h=0.0001): # h has a default value
   df =(f(x+h) - f(x-h))/(2.0*h)
   ddf =(f(x+h) - 2.0*f(x) + f(x-h))/h**2
   return df,ddf

x = 0.5
df,ddf = finite_diff(atan,x) # Uses default value of h
print 'First derivative =',df
print 'Second derivative =',ddf
```

    First derivative = 0.799999999573
    Second derivative = -0.639999991892

Note that the function atan (arctan) was passed to the function finite_diff as a parameter.

The number of input parameters in a function definition may be left arbitrary. For example, in the function definition

def func(x1,x2,*x3)

'x1' and 'x2' are the usual parameters, also called positional parameters, whereas 'x3' is a tuple of arbitrary length containing the excess parameters (indicated by the star in front of x3). Calling this function with e.g.

func(a,b,c,d,e)

results in the following correspondence between the parameters:

\begin{equation}
a \leftrightarrow x1, b \leftrightarrow x2, (c,d,e) \leftrightarrow x3
\end{equation}

The positional parameters must always be listed before the excess parameters!
If a mutable object, such as a list, is passed to a function where it is then modified, the changes will also appear in the calling program. Here is an example:

```python
def squares(a):
    for i in range(len(a)):
       a[i] = a[i]**2

a = [1, 2, 3, 4]
squares(a)
print(a)  # the list a is now modified
```

    [1, 4, 9, 16]

# * syntax
Sometimes we want to call a function with arguments taken from a list
or other sequence. The * syntax, used in a function call unpacks such a sequence into positional arguments to the function.

Here is an example, using the function 'math.hypot(a,b)'  (this is a function, in the 'math' package, which for given 'a' and 'b' returns sqrt{a^2+b^2}. If the two arguments are in a list or tuple, the following call will fail:

```python
import math

t=[3,4]

math.hypot(t)
```
This is because we tried to call the function with a single (list) argument 't', while it requires two arguments. One can just index the list explicitly:

```python
math.hypot(t[0],t[1])
```

    5.0

which works fine, but a more elegant method is to unpack the object:

```python
math.hypot(*t)
```

    5.0

# Lambda  Statement
If a function has the form of an expression, it can be defined with the
'lambda' statement:

func name = lambda param1, param2,...: expression

This is called an anonymous function (function literal) and useful for
defining small helper function that will be used only once. Here is an example:

```python
c = lambda x,y : x**2 + y**2
print c(3,4)
```

    25

# Exercises:
1) A year is a leap year if it is divisible by 4, unless it is a century that is not divisible by 400. Write a function that takes a year as a parameter and returns 'True' if the year is a leap one and 'False' otherwise.

2) Write a function that takes two parameters - a pay rate and the number of hours worked -- and returns the total pay. Any hours over 40 paid at 1.5 times higher rate.

ExamPrep/01 Intro to Python/__EXAMPLES 01 - Intro to Python.ipynb


# Resolviendo sistemas de ecuaciones lineales

En su forma más general un sistema de ecuaciones lineales se ve como

$$
\begin{align}
A_{11} x_{1} + A_{12} x_{2} + \ldots + A_{1M} x_M &= b_1 \nonumber \\
A_{21} x_{1} + A_{22} x_{2} + \ldots + A_{2M} x_M &= b_2  \nonumber \\
&\vdots  \nonumber \\
A_{N1} x_{1} + A_{N2} x_{2} + \ldots + A_{NM} x_M &= b_N  \nonumber \\
\end{align}
$$

donde

- $A_{ij}$ y $b_i$ son los coeficientes del sistema
- $N$ es la cantidad de ecuaciones del sistema
- $M$ es la cantidad de incógnitas del sistema

El sistema anterior puede escribirse de forma matricial como

$$
A x = b
$$

donde $A \in \mathbb{R}^{N \times M}$ y $b \in \mathbb{R}^N$

Revisemos a continuación como se resuelven problemas de este tipo utilizando `scipy.linalg`

```python
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import scipy.linalg
```

## Solución de un sistema cuadrado

Este es un caso particular donde la matriz $A$ tiene igual número de filas y columnas ($N=M$)

Asumiendo que la inversa de $A$ existe podemos resolver este sistema como

$$
\begin{align}
A x &=  b \nonumber \\
A^{-1} A x &= A^{-1} b \nonumber \\
x &= A^{-1} b \nonumber
\end{align}
$$

donde $A A^{-1} = I$

A continuación veremos como calcular la inversa de una matriz y resolver el sistema de ecuaciones cuadrado

**Ejemplo** Sea el sistema de tres ecuaciones y tres incognitas

$$
\begin{align}
x_1 - 2x_2 + 3x_3 &= 4 \\
2x_1 - 5x2 + 12x_3 &= 15 \\
2x_2 - 10x_3 &= -10
\end{align}
$$

que podemos reescribir como

$$
\begin{pmatrix}
1 & -2 & 3 \\
2 & -5 & 12 \\
0 & 2 & -10
\end{pmatrix} \cdot
\begin{pmatrix}
x_1 \\ x_2 \\ x_3
\end{pmatrix} =
\begin{pmatrix}
4 \\
15 \\
-10
\end{pmatrix}
$$

de donde es directo identificar $A$ y $b$

```python
A = np.array([[1, -2, 3], [2, -5, 12], [0, 2, -10]])
b = np.array([4, 15, -10])
```

Dado que $A$ es una matriz cuadrada podemos intentar invertla la función [`inv`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.inv.html#scipy.linalg.inv)

```python
Ainv = scipy.linalg.inv(A)
Ainv
```

    array([[-13. ,   7. ,   4.5],
           [-10. ,   5. ,   3. ],
           [ -2. ,   1. ,   0.5]])

Con la inversa podemos resolver el problema con

```python
np.dot(Ainv, b)
```

    array([8., 5., 2.])

:::{note}

Si $A$ hubiera sido singular, es decir no invertible, la función `inv` hubiera retornado un `LinAlgError`

:::

Podemos verificar la invertibilidad de la matriz comprobando que su determinante sea distinto de cero

```python
scipy.linalg.det(A)
```

    -1.9999999999999976

Otra forma de verificar si una matriz es invertible es comprobar que todas sus columnas sean linealmente independientes (LI)

Esto es equivalente a que su rango sea igual al número de columnas, lo cual se puede verificar con la función de `NumPy`

```python
np.linalg.matrix_rank(A) == A.shape[1]
```

    True

**Resolviendo sistemas cuadrados eficientemente**

En general si sólo nos interesa $x$, podemos no realizar el cálculo explícito de  $A^{-1}$. Si un sistema de ecuaciones es grande es preferible no calcular la inversa de $A$ debido al alto costo computacional necesario

Podemos encontrar $x$ directamente en un sistema cuadrado usando la función [`solve`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.solve.html#scipy.linalg.solve)

```python
scipy.linalg.solve(A, b)
```

    array([8., 5., 2.])

El resultado es idéntico al anterior

Veamos ahora la diferencia en eficiencia utilizando un sistema más grande

```python
N = 2000
A_big = np.random.rand(N, N) # Matriz cuadrada
b_big = np.random.rand(N, 1) # Vector
%timeit -r5 -n5 np.dot(scipy.linalg.inv(A_big), b_big)
%timeit -r5 -n5 scipy.linalg.solve(A_big, b_big)
```

    297 ms ± 15.7 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)
    193 ms ± 11.6 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)

```python
np.allclose(scipy.linalg.solve(A_big, b_big), np.dot(scipy.linalg.inv(A_big), b_big))
```

    True

Usar `solve` toma un poco más de la mitad del tiempo de utilizar `inv`+`dot`

¿Cómo puede ser posible esto?

La respuesta es que `solve` realiza internamente una factorización del tipo

$$
\begin{align}
A x &= b \nonumber \\
LU x &= b \nonumber \\
L z &= b \nonumber
\end{align}
$$

Donde $L$ es una matriz triangular inferior (lower) y $U$ es una matriz triangular superior (upper)

$$
L = \begin{pmatrix}
l_{11} & 0 & 0 & \ldots & 0 & 0 \\
l_{21} & l_{22} & 0 &\ldots & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
l_{N1} & l_{N2} & l_{N3} & \ldots & l_{N(N-1)} & l_{NN} \\
\end{pmatrix} \quad
U = \begin{pmatrix}
u_{11} & u_{11} & u_{13} & \ldots & u_{1(N-1)} & u_{1N} \\
u_{21} & u_{22} & u_{32} &\ldots & u_{2(N-1)} & 0 \\
\vdots & \vdots & \vdots &\ldots & \ddots & \vdots \\
u_{N1} & 0 & 0 & \ldots & 0 & 0\\
\end{pmatrix}
$$

Luego $z$ se puede obtener recursivamente

$$
z_1 = \frac{b_1}{l_{11}}
$$
$$
z_2 = \frac{b_2 - l_{21} z_1}{l_{22}}
$$
$$
z_i = \frac{b_i - \sum_{j=1}^{i-1} l_{ij} z_j}{l_{ii}}
$$

y $x$ se puede obtener recursivamente de $z$

En caso de necesitar los factores LU podemos realizar la factorización en `scipy` con [`linalg.lu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lu.html#scipy.linalg.lu)

```python
P, L, U = scipy.linalg.lu(A)
L, U
```

    (array([[1.  , 0.  , 0.  ],
            [0.  , 1.  , 0.  ],
            [0.5 , 0.25, 1.  ]]),
     array([[  2. ,  -5. ,  12. ],
            [  0. ,   2. , -10. ],
            [  0. ,   0. ,  -0.5]]))

## Solución de un sistema rectangular

Consideremos que

- Las incógnitas de un sistema representan sus grados de libertad
- Las ecuaciones de un sistema representan sus restricciones

Si tenemos un sistema

- con más ecuaciones que incógnitas ($N>M$): el sistema está sobredeterminado
- con más incógnitas que ecuaciones ($M>N$): el sistema está infradeterminado

:::{warning}

En ambos casos la matriz $A$ ya no es cuadrada, es decir ya no podemos calcular la inversa

:::

Sin embargo podemos utilizar otros métodos, como mostraremos a continuación

**Caso N>M**

Sea el vector de error $e = Ax - b$ de un sistema con más ecuaciones que incognitas.

Podemos encontrar una solución aproximada minimizando la norma euclidiana del error

$$
\begin{align}
\hat x &= \min_x \|e\|_2^2 \nonumber \\
& = \min_x e^T e \nonumber \\
& = \min_x (Ax -b)^T (Ax -b) \nonumber \\
\end{align}
$$

Lo cual se conoce como el **Problema de mínimos cuadrados**

Para continuar tomamos la última expresión y derivamos con respecto a $x$

$$
\begin{align}
\frac{d}{dx}  (A x - b)^T (A x -b) &= 2 A^T (A x -b) \nonumber \\
&= 2A^T A x - 2A^T b = 0 \nonumber \\
\rightarrow \hat x &= (A^T A)^{-1} A^T b \nonumber \\
&= A^{\dagger} b \nonumber \\
\end{align}
$$

donde $A^{\dagger} = (A^T A)^{-1} A^T$ se conoce como la pseudo-inversa de [Moore-Penrose](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse)

**Caso M>N**

La consecuencia de que el sistema sea infradeterminado es que $A^T A$ no es invertible.

Para resolver el problema infradeterminado se debe agregar una restricción adicional. La más típica es que el vector solución tenga norma mínima, por ejemplo

$$
\min_\theta \| x \|_2^2 ~\text{s.a.}~ Ax =b
$$

que se resuelve usando $M$ [multiplicadores de Lagrange](https://es.wikipedia.org/wiki/Multiplicadores_de_Lagrange) $\lambda$ como sigue

$$
\begin{align}
\frac{d}{dx} \| x\|_2^2 + \lambda^T (b - Ax) &= 2x - \lambda^T A  \nonumber \\
&= 2Ax - A A^T \lambda \nonumber \\
&= 2b - A A^T \lambda = 0
\end{align}
$$

De donde obtenemos que

$$
\lambda = 2(AA^T)^{-1}b
$$

y por lo tanto

$$
\hat x = \frac{1}{2} A^T \lambda = A^T (A A^T)^{-1} b,
$$

donde $A^T (A A^T)^{-1}$ se conoce como la pseudo-inversa "por la derecha"

**Resolviendo el sistema rectangular con Python**

Sea un sistema de ecuaciones lineales con $N\neq M$. Podemos usar scipy para

- Calcular la matriz pseudo inversa: [`scipy.linalg.pinv`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.pinv.html)
- Obtener la solución del sistema directamente: [`scipy.linalg.lstsq`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lstsq.html)

Si $N>M$ se utiliza la pseudoinversa por la izquierda y la solución de mínimos cuadrados. En cambio si $M>N$ se utiliza la pseudoinversa para la derecha y la solución de multiplicadores de Lagrange

**Ejemplo:** Sea el siguiente set de datos de un estudio realizado en los años 50

Referencia: [A handbook of small datasets](https://www.routledge.com/A-Handbook-of-Small-Data-Sets/Hand-Daly-McConway-Lunn-Ostrowski/p/book/9780367449667)

```python
df = pd.read_csv('data/helados.csv', index_col=0)
df.head()
```

      cons
      income
      price
      temp

      1
      0.386
      78
      0.270
      41

      2
      0.374
      79
      0.282
      56

      3
      0.393
      81
      0.277
      63

      4
      0.425
      80
      0.280
      68

      5
      0.406
      76
      0.272
      69

donde cada fila corresponde a un día las columnas corresponde a

- consumo de helados promedio ([pintas](https://en.wikipedia.org/wiki/Pint) per capita)
- ingreso familiar promedio (dolares)
- temperatura promedio (grados Fahrenheit)
- precio promedio de los helados (dolares)

Ahora consideremos la siguiente pregunta

> ¿Está el consumo de helados influenciado por la temperatura?

Intentemos responder esta pregunta en base al siguiente modelo

$$
\text{cons} = \theta_0 + \theta_1 \cdot \text{temp}
$$

que corresponde a un sistema de ecuaciones lineales de dos incognitas y

```python
f"{len(df)} ecuaciones"
```

    '30 ecuaciones'

Es decir que es un sistema sobredeterminado

$$
\begin{pmatrix}
cons[0] \\ cons[1] \\ \vdots \\ cons[{29}]
\end{pmatrix} =
\begin{pmatrix}
1&temp[0] \\ 1&temp[1] \\ \vdots & \vdots \\ 1 & temp[29]
\end{pmatrix}
\begin{pmatrix}
\theta_0 \\ \theta_1
\end{pmatrix}
$$

que podemos resolver utilizando `lstsq`

```python
A = np.ones(shape=(len(df), 2))
A[:, 1] = df["temp"].values
b = df["cons"].values

theta, residuals, rank, s = scipy.linalg.lstsq(A, b)
theta
```

    array([0.20686215, 0.00310736])

La tupla retornada contiene

- `theta`: El resultado buscado
- `residuals`: La norma del error al cuadrado
- `rank`: El rango de $A$
- `s`: Los valores singulares de $A$

Analicemos gráficamente la solución obtenida

```python
fig, ax = plt.subplots(tight_layout=True)
ax.scatter(df["temp"], df["cons"], s=5, c='k', label='observaciones')
ax.set_xlabel('Temperatura [F]')
ax.set_ylabel('Consumo promedio')
temp_modelo = np.linspace(df["temp"].min(), df["temp"].max(), num=200)
cons_modelo = temp_modelo*theta[1] + theta[0]
ax.plot(temp_modelo, cons_modelo, label='modelo')
ax.legend();
```

De donde podemos observar que el consumo promedio tiende al alza con la temperatura promedio

:::{note}

Lo que acabamos de resolver es un problema conocido como **regresión lineal**. Más adelante veremos una forma más general de este problema

:::

## Análisis de errores y *condition number*

Incluso aunque una matriz sea matemáticamente invertible (determinante distinto de cero), podríamos no ser capaces de resolver el problema numéricamente

Imaginemos una pequeña variación en $b$ denominada $\delta b$. Esta variación provoca a su vez una pequeña variación en $x$ denominada $\delta x$

Se puede encontrar una cota que compara el error relativo de $b$ y $x$ como

$$
\frac{\| \delta x \|}{\|x\|} \leq \frac{\| A^{-1} \|  \|\delta b\|}{\|x\|}  = \|A^{-1}\| \|A\| \frac{\| \delta b \|}{\|b\|}
$$

donde se usó que $A \delta x = \delta b$ (propiedad de linealidad)

:::{note}

Esto significa que un pequeño error relativo en $b$ puede causar un gran error en $x$

:::

El estimador de $\|A^{-1}\| \|A\|$ es lo que se conoce como *condition number*

Un sistema se dice "bien condicionado" si este valor es cercano a $1$ y "mal condicionado" si es mucho mayor que $1$.

Podemos calcular el *condition number* con la función de NumPy `cond` como se muestra a continuación

```python
np.linalg.cond(x, # Arreglo multidimensional
               p # El orden de la norma: 1, 2, 'fro',...
              )
```

```python

```


# Special Series Solutions

```python
%matplotlib inline
from sympy import *
init_printing()
```

```python
x, t = symbols('x, t')
```

SymPy can compute special series like formal power series and fourier series. This is a new feature released in SymPy 1.0

Let's try computing formal power series of some basic functions.

```python
exp_series = fps(exp(x), x)
exp_series
```

This looks very similar to what ``series`` has to offer, but unlike series a formal power series object returns an infinite expansion.

```python
exp_series.infinite # Infinite representation
```

$$\sum_{k=1}^{\infty} \begin{cases} \frac{x^{k}}{k!} & \text{for}\: \operatorname{Mod}{\left (k,1 \right )} = 0 \\0 & \text{otherwise} \end{cases} + 1$$

We can easily find out any term of the expansion (no need to recompute the expansion).

```python
exp_series.term(51) # equivalent to exp_series[51]
```

```python
exp_series.truncate(10) # return a truncated series expansion
```

# Exercise

Try computing the formal power series of $\log(1 + x)$. Try to look at the infinite representation. What is the 51st term in this case? Compute the expansion about 1.

```python
log_series = fps(log(1 + x), x)
log_series
```

```python
# infinite representation
log_series.infinite
```

```python
# 51st term
log_series.term(51)
```

```python
# expansion about 1
fps(log(1 + x), x, x0=1)
```

# Fourier Series

Fourier series for functions can be computed using ``fourier_series`` function.

A sawtooth wave is defined as:
   1. $$ s(x) = x/\pi \in (-\pi, \pi) $$
   2. $$ s(x + 2k\pi) = s(x) \in (-\infty, \infty) $$

Let's compute the fourier series of the above defined wave.

```python
sawtooth_series = fourier_series(x / pi, (x, -pi, pi))
sawtooth_series
```

```python
plot(sawtooth_series.truncate(50))
```

See https://en.wikipedia.org/wiki/Gibbs_phenomenon for why the fourier series has peculiar behavior near jump discontinuties.

Just like formal power series we can index fourier series as well.

```python
sawtooth_series[51]
```

It is easy to shift and scale the series using ``shift`` and ``scale`` methods.

```python
sawtooth_series.shift(10).truncate(5)
```

```python
sawtooth_series.scale(10).truncate(5)
```

# Exercise

Consider a square wave defined over the range of (0, 1) as:
   1. $$ f(t) = 1 \in (0, 1/2] $$
   2. $$ f(t) = -1 \in (1/2, 1) $$
   3. $$ f(t + 1) = f(t) \in (-\infty, \infty) $$

Try computing the fourier series of the above defined function. Also, plot the computed fourier series.

```python
square_wave = Piecewise((1, t <= Rational(1, 2)), (-1, t > Rational(1, 2)))
```

```python
square_series = fourier_series(square_wave, (t, 0, 1))
square_series
```

```python
plot(square_series.truncate(50))
```

# What next?

Try some basic operations like addition, subtraction, etc on formal power series, fourier series and see what happens.

tutorial_exercises/Advanced - Special Series Solutions.ipynb