```python
%matplotlib inline
```

```python
# Write your imports here
import sympy
import math
import numpy as np
import matplotlib.pyplot as plt
```

### Problem 1. Markdown
Jupyter Notebook is a very light, beautiful and convenient way to organize your research and display your results. Let's play with it for a while.

First, you can double-click each cell and edit its content. If you want to run a cell (that is, execute the code inside it), use Cell > Run Cells in the top menu or press Ctrl + Enter.

Second, each cell has a type. There are two main types: Markdown (which is for any kind of free text, explanations, formulas, results... you get the idea), and code (which is, well... for code :D).

Let me give you a...
#### Quick Introduction to Markdown
##### Text and Paragraphs
There are several things that you can do. As you already saw, you can write paragraph text just by typing it. In order to create a new paragraph, just leave a blank line. See how this works below:
```
This is some text.
This text is on a new line, but it will continue the same paragraph (so you can make your paragraphs more easily readable by just continuing on a new line, or just go on and on like this one line is ever continuing).

This text is displayed in a new paragraph.

And this is yet another paragraph.
```
**Result:**

This is some text.
This text is on a new line, but it will continue the same paragraph (so you can make your paragraphs more easily readable by just continuing on a new line, or just go on and on like this one line is ever continuing).

This text is displayed in a new paragraph.

And this is yet another paragraph.

##### Headings
There are six levels of headings. Level one is the highest (largest and most important), and level 6 is the smallest. You can create headings of several types by prefixing the header line with one to six "#" symbols (this is called a pound sign if you are ancient, or a sharp sign if you're a musician... or a hashtag if you're too young :D). Have a look:
```
# Heading 1
## Heading 2
### Heading 3
#### Heading 4
##### Heading 5
###### Heading 6
```

**Result:**

# Heading 1
## Heading 2
### Heading 3
#### Heading 4
##### Heading 5
###### Heading 6

It is recommended that you have **only one** H1 heading - this should be the header of your notebook (or scientific paper). Below that, you can add your name or just jump to the explanations directly.

##### Emphasis
You can create emphasized (stronger) text by using a **bold** or _italic_ font. You can do this in several ways (using asterisks (\*) or underscores (\_)). In order to "escape" a symbol, prefix it with a backslash (\). You can also strike through your text in order to signify a correction.
```
**bold** __bold__
*italic* _italic_

This is \*\*not \*\* bold.

I ~~didn't make~~ a mistake.
```

**Result:**

**bold** __bold__
*italic* _italic_

This is \*\*not\*\* bold.

I ~~didn't make~~ a mistake.

##### Lists
You can add two types of lists: ordered and unordered. Lists can also be nested inside one another. To do this, press Tab once (it will be converted to 4 spaces).

To create an ordered list, just type the numbers. Don't worry if your numbers are wrong - Jupyter Notebook will create them properly for you. Well, it's better to have them properly numbered anyway...
```
1. This is
2. A list
10. With many
9. Items
    1. Some of which
    2. Can
        3. Be nested
42. You can also
    * Mix
    * list
    * types
```

**Result:**
1. This is
2. A list
10. With many
9. Items
    1. Some of which
    2. Can
        3. Be nested
42. You can also
    * Mix
    * list
    * types

To create an unordered list, type an asterisk, plus or minus at the beginning:
```
* This is
* An
    + Unordered
    - list
```

**Result:**
* This is
* An
    + Unordered
        - list

##### Links
There are many ways to create links but we mostly use one of them: we present links with some explanatory text. See how it works:
```
This is [a link](http://google.com) to Google.
```

**Result:**

This is [a link](http://google.com) to Google.

##### Images
They are very similar to links. Just prefix the image with an exclamation mark. The alt(ernative) text will be displayed if the image is not available. Have a look (hover over the image to see the title text):
```
 Do you know that "taco cat" is a palindrome? Thanks to The Oatmeal :)
```

**Result:**

 Do you know that "taco cat" is a palindrome? Thanks to The Oatmeal :)

If you want to resize images or do some more advanced stuff, just use HTML.

Did I mention these cells support HTML, CSS and JavaScript? Now I did.

##### Tables
These are a pain because they need to be formatted (somewhat) properly. Here's a good [table generator](http://www.tablesgenerator.com/markdown_tables). Just select File > Paste table data... and provide a tab-separated list of values. It will generate a good-looking ASCII-art table for you.
```
| Cell1 | Cell2 | Cell3 |
|-------|-------|-------|
| 1.1   | 1.2   | 1.3   |
| 2.1   | 2.2   | 2.3   |
| 3.1   | 3.2   | 3.3   |
```

**Result:**

| Cell1 | Cell2 | Cell3 |
|-------|-------|-------|
| 1.1   | 1.2   | 1.3   |
| 2.1   | 2.2   | 2.3   |
| 3.1   | 3.2   | 3.3   |

##### Code
Just use triple backtick symbols. If you provide a language, it will be syntax-highlighted. You can also use inline code with single backticks.

```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Result:**
```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Now it's your turn to have some Markdown fun.** In the next cell, try out some of the commands. You can just throw in some things, or do something more structured (like a small notebook).

# This is just a test title
## With a subtitle
### And an even smaller subtitle

For more playing arround, please check out the attached notebook

### Problem 2. Formulas and LaTeX
Writing math formulas has always been hard. But scientists don't like difficulties and prefer standards. So, thanks to Donald Knuth (a very popular computer scientist, who also invented a lot of algorithms), we have a nice typesetting system, called LaTeX (pronounced _lah_-tek). We'll be using it mostly for math formulas, but it has a lot of other things to offer.

There are two main ways to write formulas. You could enclose them in single `$` signs like this: `$ ax + b $`, which will create an **inline formula**: $ ax + b $. You can also enclose them in double `$` signs `$$ ax + b $$` to produce $$ ax + b $$.

Most commands start with a backslash and accept parameters either in square brackets `[]` or in curly braces `{}`. For example, to make a fraction, you typically would write `$$ \frac{a}{b} $$`: $$ \frac{a}{b} $$.

[Here's a resource](http://www.stat.pitt.edu/stoffer/freetex/latex%20basics.pdf) where you can look up the basics of the math syntax. You can also search StackOverflow - there are all sorts of solutions there.

You're on your own now. Research and recreate all formulas shown in the next cell. Try to make your cell look exactly the same as mine. It's an image, so don't try to cheat by copy/pasting :D.

Note that you **do not** need to understand the formulas, what's written there or what it means. We'll have fun with these later in the course.

Write your formulas here.

Equation of a line: $$ y = ax + b $$

Roots of the quadratic equation $ax^{2}+bx+c=0$: $$ x_{1,2}=\frac{-b\pm\sqrt[2]{b^{2}-4ac}}{2a} $$

Taylor series expansion: $$f(x)\mid_{x=a}=f(a)+f'(a)(x-a)+\frac{f''(a)}{2!}(x-a)^{2}+...+\frac{f^{n}(a)}{n!}(x-a)^{n}+...$$

Binomial theorem: $$ (x+y)^{n}=\biggl({n \atop 0}\biggr)x^{n}y^{0}+\biggl({n \atop 1}\biggr)x^{n-1}y^{1}+...+\biggl({n \atop n}\biggr)x^{0}y^{n}=\sum^{n}_{k=0}\biggl({n \atop k}\biggr)x^{n-k}y^{k} $$

An integral (this one is a lot of fun to solve :D): $$ \int_{+\infty}^{-\infty} e^{-x^{2}} \,dx=\sqrt{\pi}$$

A short matrix: $$\begin{pmatrix} 2 & 1 & 3 \\ 2 & 6 & 8 \\ 6 & 8 & 18\end{pmatrix}$$

A long matrix: $$\begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\a_{m1} & a_{m2} & \cdots & a_{mn}\end{pmatrix}$$

### Problem 3. Solving with Python
Let's first do some symbolic computation. We need to import `sympy` first.

**Should your imports be in a single cell at the top or should they appear as they are used?** There's not a single valid best practice. Most people seem to prefer imports at the top of the file though. **Note: If you write new code in a cell, you have to re-execute it!**

Let's use `sympy` to give us a quick symbolic solution to our equation. First import `sympy` (you can use the second cell in this notebook):
```python
import sympy
```

Next, create symbols for all variables and parameters. You may prefer to do this in one pass or separately:
```python
x = sympy.symbols('x')
a, b, c = sympy.symbols('a b c')
```

Now solve:
```python
sympy.solve(a * x**2 + b * x + c)
```

```python
# High-School Maths Exercise
## Getting to Know Jupyter Notebook. Python Libraries and Best Practices. Basic Workflow
```

```python
# Write your code here
x = sympy.symbols('x')
a, b, c = sympy.symbols('a b c')
sympy.solve(a * x**2 + b * x + c)
```

    [{a: (-b*x - c)/x**2}]

```python
# Write your code here
sympy.init_printing()
sympy.solve(a * x**2 + b * x + c, x)
```

Hmmmm... we didn't expect that :(. We got an expression for $a$ because the library tried to solve for the first symbol it saw. This is an equation and we have to solve for $x$. We can provide it as a second parameter:
```python
sympy.solve(a * x**2 + b * x + c, x)
```

Finally, if we start with `sympy.init_printing()`, we'll get a LaTeX-formatted result instead of a typed one. This is very useful because it produces better-looking formulas. **Note:** This means we have to add the line BEFORE we start working with `sympy`.

How about a function that takes $a, b, c$ (assume they are real numbers, you don't need to do additional checks on them) and returns the **real** roots of the quadratic equation?

Remember that in order to calculate the roots, we first need to see whether the expression under the square root sign is non-negative.

If $b^2 - 4ac > 0$, the equation has two real roots: $x_1, x_2$

If $b^2 - 4ac = 0$, the equation has one real root: $x_1 = x_2$

If $b^2 - 4ac < 0$, the equation has zero real roots

Write a function which returns the roots. In the first case, return a list of 2 numbers: `[2, 3]`. In the second case, return a list of only one number: `[2]`. In the third case, return an empty list: `[]`.

```python
def format_decimal(func):
    """
    Decorator to convert the sympy output to Python float format
    """
    def inner(*args, **kwargs):
        raw_res = func(*args, **kwargs)
        if isinstance(raw_res, list) and len(raw_res) > 0:
            retval = []
            for el in raw_res:
                retval.append(float(el))
            return retval
        return raw_res
    return inner

@format_decimal
def solve_quadratic_equation(a, b, c):
    """
    Returns the real solutions of the quadratic equation ax^2 + bx + c = 0
    """
    # Delete the "pass" statement below and write your code
    def sqrt_part():
        return b**2 - 4 * a * c

    if a == 0:
        return sympy.solve(b * x + c, x)
    if sqrt_part() < 0:
        return []
    return sympy.solve(a * x**2 + b * x + c, x)
```

```python
# Testing: Execute this cell. The outputs should match the expected outputs. Feel free to write more tests
print(solve_quadratic_equation(1, -1, -2)) # [-1.0, 2.0]
print(solve_quadratic_equation(1, -8, 16)) # [4.0]
print(solve_quadratic_equation(1, 1, 1)) # []
```

    [-1.0, 2.0]
    [4.0]
    []

**Bonus:** Last time we saw how to solve a linear equation. Remember that linear equations are just like quadratic equations with $a = 0$. In this case, however, division by 0 will throw an error. Extend your function above to support solving linear equations (in the same way we did it last time).

```python
# Bonus: Calling the function with a = 0 for a linear equation
print(solve_quadratic_equation(0, -1, -2)) # [-2.0]
print(solve_quadratic_equation(0, -8, 16)) # [2.0]
print(solve_quadratic_equation(0, 1, 1)) # [-1.0]
```

    [-2.0]
    [2.0]
    [-1.0]

### Problem 4. Equation of a Line
Let's go back to our linear equations and systems. There are many ways to define what "linear" means, but they all boil down to the same thing.

The equation $ax + b = 0$ is called *linear* because the function $f(x) = ax+b$ is a linear function. We know that there are several ways to know what one particular function means. One of them is to just write the expression for it, as we did above. Another way is to **plot** it. This is one of the most exciting parts of maths and science - when we have to fiddle around with beautiful plots (although not so beautiful in this case).

The function produces a straight line and we can see it.

How do we plot functions in general? We know that functions take many (possibly infinitely many) inputs. We can't draw all of them. We could, however, evaluate the function at some points and connect them with tiny straight lines. If the points are too many, we won't notice - the plot will look smooth.

Now, let's take a function, e.g. $y = 2x + 3$ and plot it. For this, we're going to use `numpy` arrays. This is a special type of array which has two characteristics:
* All elements in it must be of the same type
* All operations are **broadcast**: if `x = [1, 2, 3, 10]` and we write `2 * x`, we'll get `[2, 4, 6, 20]`. That is, all operations are performed at all indices. This is very powerful, easy to use and saves us A LOT of looping.

There's one more thing: it's blazingly fast because all computations are done in C, instead of Python.

First let's import `numpy`. Since the name is a bit long, a common convention is to give it an **alias**:
```python
import numpy as np
```

Import that at the top cell and don't forget to re-run it.

Next, let's create a range of values, e.g. $[-3, 5]$. There are two ways to do this. `np.arange(start, stop, step)` will give us evenly spaced numbers with a given step, while `np.linspace(start, stop, num)` will give us `num` samples. You see, one uses a fixed step, the other uses a number of points to return. When plotting functions, we usually use the latter. Let's generate, say, 1000 points (we know a straight line only needs two but we're generalizing the concept of plotting here :)).
```python
x = np.linspace(-3, 5, 1000)
```
Now, let's generate our function variable
```python
y = 2 * x + 3
```

We can print the values if we like but we're more interested in plotting them. To do this, first let's import a plotting library. `matplotlib` is the most commnly used one and we usually give it an alias as well.
```python
import matplotlib.pyplot as plt
```

Now, let's plot the values. To do this, we just call the `plot()` function. Notice that the top-most part of this notebook contains a "magic string": `%matplotlib inline`. This hints Jupyter to display all plots inside the notebook. However, it's a good practice to call `show()` after our plot is ready.
```python
plt.plot(x, y)
plt.show()
```

```python
# Write your code here
x = np.linspace(-3, 5, 1000)
y = 2 * x + 3
plt.plot(x, y)
plt.show()
```

It doesn't look too bad bit we can do much better. See how the axes don't look like they should? Let's move them to zero. This can be done using the "spines" of the plot (i.e. the borders).

All `matplotlib` figures can have many plots (subfigures) inside them. That's why when performing an operation, we have to specify a target figure. There is a default one and we can get it by using `plt.gca()`. We usually call it `ax` for "axis".
Let's save it in a variable (in order to prevent multiple calculations and to make code prettier). Let's now move the bottom and left spines to the origin $(0, 0)$ and hide the top and right one.
```python
ax = plt.gca()
ax.spines["bottom"].set_position("zero")
ax.spines["left"].set_position("zero")
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
```

**Note:** All plot manipulations HAVE TO be done before calling `show()`. It's up to you whether they should be before or after the function you're plotting.

This should look better now. We can, of course, do much better (e.g. remove the double 0 at the origin and replace it with a single one), but this is left as an exercise for the reader :).

```python
# Copy and edit your code here
plt.clf()
ax = plt.gca()
ax.spines["bottom"].set_position('zero')
ax.spines["left"].set_position("zero")
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
xticks = ax.xaxis.get_major_ticks()
yticks = ax.yaxis.get_major_ticks()

plt.plot(x, y)

# Let's try to remove the double 0 at the origin.
xloc, labels = plt.xticks()
yloc, labels = plt.yticks()
x_zero_loc = np.where(xloc==0)[0][0]
y_zero_loc = np.where(yloc==0)[0][0]

xticks[x_zero_loc].set_visible(False)
yticks[y_zero_loc].set_visible(False)

plt.show()
```

### * Problem 5. Linearizing Functions
Why is the line equation so useful? The main reason is because it's so easy to work with. Scientists actually try their best to linearize functions, that is, to make linear functions from non-linear ones. There are several ways of doing this. One of them involves derivatives and we'll talk about it later in the course.

A commonly used method for linearizing functions is through algebraic transformations. Try to linearize
$$ y = ae^{bx} $$

Hint: The inverse operation of $e^{x}$ is $\ln(x)$. Start by taking $\ln$ of both sides and see what you can do. Your goal is to transform the function into another, linear function. You can look up more hints on the Internet :).

Write your result here.

We start by taking the ln of both sides: $ \ln(y) = \ln(a) + bx\ln(e) $

The resulting linear function is: $ \ln(y) = \ln(a) + bx $

Where **ln(y)** is the dependent that we plot on the y-axis, **ln(a)** is the constant and **b** is the slope. This equation is commontly presented as $y = mx + b$

### * Problem 6. Generalizing the Plotting Function
Let's now use the power of Python to generalize the code we created to plot. In Python, you can pass functions as parameters to other functions. We'll utilize this to pass the math function that we're going to plot.

Note: We can also pass *lambda expressions* (anonymous functions) like this:
```python
lambda x: x + 2```
This is a shorter way to write
```python
def some_anonymous_function(x):
    return x + 2
```

We'll also need a range of x values. We may also provide other optional parameters which will help set up our plot. These may include titles, legends, colors, fonts, etc. Let's stick to the basics now.

Write a Python function which takes another function, x range and number of points, and plots the function graph by evaluating it at every point.

**BIG hint:** If you want to use not only `numpy` functions for `f` but any one function, a very useful (and easy) thing to do, is to vectorize the function `f` (e.g. to allow it to be used with `numpy` broadcasting):
```python
f_vectorized = np.vectorize(f)
y = f_vectorized(x)
```

```python
def remove_zero_tick(loc, axticks):
    """
    This function will be used in this book to remove the 0 at origin
    """
    try:
        zero_loc = np.where(loc==0)[0][0]
        axticks[zero_loc].set_visible(False)
    except IndexError:
        ax.spines["bottom"].set_position(('data', 1))

def plot_math_function(f, min_x, max_x, num_points):
    x_array = np.linspace(min_x, max_x, num_points)
    f_vectorized = np.vectorize(f)
    y = f_vectorized(x_array)

    plt.clf()
    plt.cla()
    ax = plt.gca()
    ax.spines["bottom"].set_position('zero')
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    xticks = ax.xaxis.get_major_ticks()
    yticks = ax.yaxis.get_major_ticks()

    plt.plot(x_array, y)

    xloc, labels = plt.xticks()
    yloc, labels = plt.yticks()
    remove_zero_tick(xloc, xticks)
    remove_zero_tick(yloc, yticks)

    plt.show()

```

```python
plot_math_function(lambda x: 2 * x + 3, -3, 5, 1000)
plot_math_function(lambda x: -x + 8, -1, 10, 1000)
plot_math_function(lambda x: x**2 - x - 2, -3, 4, 1000)
plot_math_function(lambda x: np.sin(x), -np.pi, np.pi, 1000)
plot_math_function(lambda x: np.sin(x) / x, -4 * np.pi, 4 * np.pi, 1000)
```

### * Problem 7. Solving Equations Graphically
Now that we have a general plotting function, we can use it for more interesting things. Sometimes we don't need to know what the exact solution is, just to see where it lies. We can do this by plotting the two functions around the "=" sign ans seeing where they intersect. Take, for example, the equation $2x + 3 = 0$. The two functions are $f(x) = 2x + 3$ and $g(x) = 0$. Since they should be equal, the point of their intersection is the solution of the given equation. We don't need to bother marking the point of intersection right now, just showing the functions.

To do this, we'll need to improve our plotting function yet once. This time we'll need to take multiple functions and plot them all on the same graph. Note that we still need to provide the $[x_{min}; x_{max}]$ range and it's going to be the same for all functions.

```python
vectorized_fs = [np.vectorize(f) for f in functions]
ys = [vectorized_f(x) for vectorized_f in vectorized_fs]
```

```python
def plot_math_functions(functions, min_x, max_x, num_points):

    x_array = np.linspace(min_x, max_x, num_points)
    if hasattr(functions, "__iter__"):
        vectorized_fs = [np.vectorize(func) for func in functions]
        ys = [vectorized_f(x_array) for vectorized_f in vectorized_fs]
    else:
        ys = [np.vectorize(functions)(x_array)]

    ax = plt.gca()
    ax.spines["bottom"].set_position('zero')
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    xticks = ax.xaxis.get_major_ticks()
    yticks = ax.yaxis.get_major_ticks()

    for y in ys:
        plt.plot(x_array, y)

    xloc, labels = plt.xticks()
    yloc, labels = plt.yticks()
    remove_zero_tick(xloc, xticks)
    remove_zero_tick(yloc, yticks)

    plt.show()
```

```python
# plot_math_functions(4, -3, 5, 1000)
plot_math_functions([lambda x: 3 * x**2 - 2 * x + 5, lambda x: 3 * x + 7], -2, 3, 1000)
```

This is also a way to plot the solutions of systems of equation, like the one we solved last time. Let's actually try it.

```python
plot_math_functions([lambda x: (-4 * x + 7) / 3, lambda x: (-3 * x + 8) / 5, lambda x: (-x - 1) / -2], -1, 4, 1000)
```

### Problem 8. Trigonometric Functions
We already saw the graph of the function $y = \sin(x)$. But then again, how do we define the trigonometric functions? Let's quickly review that.

The two basic trigonometric functions are defined as the ratio of two sides:
$$ \sin(x) = \frac{\text{opposite}}{\text{hypotenuse}} $$
$$ \cos(x) = \frac{\text{adjacent}}{\text{hypotenuse}} $$

And also:
$$ \tan(x) = \frac{\text{opposite}}{\text{adjacent}} = \frac{\sin(x)}{\cos(x)} $$
$$ \cot(x) = \frac{\text{adjacent}}{\text{opposite}} = \frac{\cos(x)}{\sin(x)} $$

This is fine, but using this, "right-triangle" definition, we're able to calculate the trigonometric functions of angles up to $90^\circ$. But we can do better. Let's now imagine a circle centered at the origin of the coordinate system, with radius $r = 1$. This is called a "unit circle".

We can now see exactly the same picture. The $x$-coordinate of the point in the circle corresponds to $\cos(\alpha)$ and the $y$-coordinate - to $\sin(\alpha)$. What did we get? We're now able to define the trigonometric functions for all degrees up to $360^\circ$. After that, the same values repeat: these functions are **periodic**:
$$ \sin(k.360^\circ + \alpha) = \sin(\alpha), k = 0, 1, 2, \dots $$
$$ \cos(k.360^\circ + \alpha) = \cos(\alpha), k = 0, 1, 2, \dots $$

We can, of course, use this picture to derive other identities, such as:
$$ \sin(90^\circ + \alpha) = \cos(\alpha) $$

A very important property of the sine and cosine is that they accept values in the range $(-\infty; \infty)$ and produce values in the range $[-1; 1]$. The two other functions take values in the range $(-\infty; \infty)$ **except when their denominators are zero** and produce values in the same range.

#### Radians
A degree is a geometric object, $1/360$th of a full circle. This is quite inconvenient when we work with angles. There is another, natural and intrinsic measure of angles. It's called the **radian** and can be written as $\text{rad}$ or without any designation, so $\sin(2)$ means "sine of two radians".

It's defined as *the central angle of an arc with length equal to the circle's radius* and $1\text{rad} \approx 57.296^\circ$.

We know that the circle circumference is $C = 2\pi r$, therefore we can fit exactly $2\pi$ arcs with length $r$ in $C$. The angle corresponding to this is $360^\circ$ or $2\pi\ \text{rad}$. Also, $\pi rad = 180^\circ$.

(Some people prefer using $\tau = 2\pi$ to avoid confusion with always multiplying by 2 or 0.5 but we'll use the standard notation here.)

**NOTE:** All trigonometric functions in `math` and `numpy` accept radians as arguments. In order to convert between radians and degrees, you can use the relations $\text{[deg]} = 180/\pi.\text{[rad]}, \text{[rad]} =  \pi/180.\text{[deg]}$. This can be done using `np.deg2rad()` and `np.rad2deg()` respectively.

#### Inverse trigonometric functions
All trigonometric functions have their inverses. If you plug in, say $\pi/4$ in the $\sin(x)$ function, you get $\sqrt{2}/2$. The inverse functions (also called, arc-functions) take arguments in the interval $[-1; 1]$ and return the angle that they correspond to. Take arcsine for example:
$$ \arcsin(y) = x: sin(y) = x $$
$$ \arcsin\left(\frac{\sqrt{2}}{2}\right) = \frac{\pi}{4} $$

Please note that this is NOT entirely correct. From the relations we found:
$$\sin(x) = sin(2k\pi + x), k = 0, 1, 2, \dots $$

it follows that $\arcsin(x)$ has infinitely many values, separated by $2k\pi$ radians each:
$$ \arcsin\left(\frac{\sqrt{2}}{2}\right) = \frac{\pi}{4} + 2k\pi, k = 0, 1, 2, \dots $$

In most cases, however, we're interested in the first value (when $k = 0$). It's called the **principal value**.

Note 1: There are inverse functions for all four basic trigonometric functions: $\arcsin$, $\arccos$, $\arctan$, $\text{arccot}$. These are sometimes written as $\sin^{-1}(x)$, $\cos^{-1}(x)$, etc. These definitions are completely equivalent.

Just notice the difference between $\sin^{-1}(x) := \arcsin(x)$ and $\sin(x^{-1}) = \sin(1/x)$.

#### Exercise
Use the plotting function you wrote above to plot the inverse trigonometric functions. Use `numpy` (look up how to use inverse trigonometric functions).

```python
# Write your code here
plot_math_functions([lambda x: np.arcsin(x), lambda x: np.arccos(x)], -1, 1, 1000)
plot_math_functions(lambda x: np.arctan(x), -1, 1, 1000)

plot_math_functions(lambda x: np.arctan(1 / x), -1, 1, 1000)
plot_math_functions(lambda x: (3.14 / 2) - np.arctan(x), -1, 1, 1000)
```

### ** Problem 9. Perlin Noise
This algorithm has many applications in computer graphics and can serve to demonstrate several things... and help us learn about math, algorithms and Python :).
#### Noise
Noise is just random values. We can generate noise by just calling a random generator. Note that these are actually called *pseudorandom generators*. We'll talk about this later in this course.
We can generate noise in however many dimensions we want. For example, if we want to generate a single dimension, we just pick N random values and call it a day. If we want to generate a 2D noise space, we can take an approach which is similar to what we already did with `np.meshgrid()`.

$$ \text{noise}(x, y) = N, N \in [n_{min}, n_{max}] $$

This function takes two coordinates and returns a single number N between $n_{min}$ and $n_{max}$. (This is what we call a "scalar field").

Random variables are always connected to **distributions**. We'll talk about these a great deal but now let's just say that these define what our noise will look like. In the most basic case, we can have "uniform noise" - that is, each point in our little noise space $[n_{min}, n_{max}]$ will have an equal chance (probability) of being selected.

#### Perlin noise
There are many more distributions but right now we'll want to have a look at a particular one. **Perlin noise** is a kind of noise which looks smooth. It looks cool, especially if it's colored. The output may be tweaked to look like clouds, fire, etc. 3D Perlin noise is most widely used to generate random terrain.

#### Algorithm
... Now you're on your own :). Research how the algorithm is implemented (note that this will require that you understand some other basic concepts like vectors and gradients).

#### Your task
1. Research about the problem. See what articles, papers, Python notebooks, demos, etc. other people have created
2. Create a new notebook and document your findings. Include any assumptions, models, formulas, etc. that you're using
3. Implement the algorithm. Try not to copy others' work, rather try to do it on your own using the model you've created
4. Test and improve the algorithm
5. (Optional) Create a cool demo :), e.g. using Perlin noise to simulate clouds. You can even do an animation (hint: you'll need gradients not only in space but also in time)
6. Communicate the results (e.g. in the Softuni forum)

Hint: [This](http://flafla2.github.io/2014/08/09/perlinnoise.html) is a very good resource. It can show you both how to organize your notebook (which is important) and how to implement the algorithm.

Math/High School Math/High-School Maths Exercise.ipynb


#  Introduction to Neural Networks and Pytorch

    Notebook version: 0.1 (Nov 14, 2020)

    Authors: Jerónimo Arenas García (jarenas@ing.uc3m.es)

    Changes: v.0.1. (Nov 14, 2020) - First version

    Pending changes: - Use epochs instead of iters in first part of notebook
                     - Add an example with dropout
                     - Add theory about CNNs
                     - Define functions for the training of neural nets and display of the results
                       in order to simplify code cells

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

size=18
params = {'legend.fontsize': 'Large',
          'axes.labelsize': size,
          'axes.titlesize': size,
          'xtick.labelsize': size*0.75,
          'ytick.labelsize': size*0.75}
plt.rcParams.update(params)
```

##  1. Introduction and purpose of this Notebook

###  1.1. About Neural Networks

* Neural Networks (NN) have become the state of the art for many machine learning problems
    * Natural Language Processing
    * Computer Vision
    * Image Recognition

* They are in widespread use for many applications, e.g.,
    * Language Translantion (Google Neural Machine Translation System)
    * Automatic Speech recognition (Hey Siri! DNN overview)
    * Autonomous Navigation (Facebook Robot Autonomous 3D Navigation)
    * Automatic Plate recognition

Feed Forward Neural Networks have been around since 1960 but only recently (last 10-12 years) have they met their expectations, and improve other machine learning algorithms

* Computation resources are now available at large scale
* Cloud Computing (AWS, Azure)
* From MultiLayer Perceptrons to Deep Learning
* Big Data sets
* This has also made possible an intense research effort resulting in
    * Topologies better suited to particular problems (CNNs, RNNs)
    * New training strategies providing better generalization

In parallel, Deep Learning Platforms have emerged that make design, implementation, training, and production of DNNs feasible for everyone

###  1.2. Scope

* To provide just an overview of most important NNs and DNNs concepts
* Connecting with already studied methods as starting point
* Introduction to PyTorch
* Providing links to external sources for further study

###  1.3. Outline

1. Introduction and purpose of this Notebook
2. Introduction to Neural Networks
3. Implementing Deep Networks with PyTorch

###  1.4. Other resources

* We point here to external resources and tutorials that are excellent material for further study of the topic
* Most of them include examples and exercises using numpy and PyTorch
* This notebook uses examples and other material from some of these sources

|Tutorial|Description|
|-----|---------------------|
|  |Very general tutorial including videos and an overview of top deep learning platforms|
|  |Very complete book with a lot of theory and examples for MxNET, PyTorch, and TensorFlow|
|  |Official tutorials from the PyTorch project. Contains a 60 min overview, and a very practical *learning PyTorch with examples* tutorial|
|  |Kaggle tutorials covering an introduction to Neural Networks using Numpy, and a second one offering a PyTorch tutorial|

In addition to this, PyTorch MOOCs can be followed for free in main sites: edX, Coursera, Udacity

##  2. Introduction to Neural Networks

In this section, we will implement neural networks from scratch using Numpy arrays

* No need to learn any new Python libraries
* But we need to deal with complexity of multilayer networks
* Low-level implementation will be useful to grasp the most important concepts concerning DNNs
    * Back-propagation
    * Activation functions
    * Loss functions
    * Optimization methods
    * Generalization
    * Special layers and configurations

###  2.0. Data preparation

We start by loading some data sets that will be used to carry out the exercises

### Sign language digits data set

* Dataset is taken from  Kaggle and used in the above referred tutorial
* 2062 digits in sign language. $64 \times 64$ images
* Problem with 10 classes. One hot encoding for the label matrix
* Input data are images, we create also a flattened version

```python
digitsX = np.load('./data/Sign-language-digits-dataset/X.npy')
digitsY = np.load('./data/Sign-language-digits-dataset/Y.npy')
K = digitsX.shape[0]
img_size = digitsX.shape[1]
digitsX_flatten = digitsX.reshape(K,img_size*img_size)

print('Size of Input Data Matrix:', digitsX.shape)
print('Size of Flattned Input Data Matrix:', digitsX_flatten.shape)
print('Size of label Data Matrix:', digitsY.shape)
selected = [260, 1400]
plt.subplot(1, 2, 1), plt.imshow(digitsX[selected[0]].reshape(img_size, img_size)), plt.axis('off')
plt.subplot(1, 2, 2), plt.imshow(digitsX[selected[1]].reshape(img_size, img_size)), plt.axis('off')
plt.show()
print('Labels corresponding to figures:', digitsY[selected,])
```

### Dogs vs Cats data set

* Dataset is taken from  Kaggle
* 25000 pictures of dogs and cats
* Binary problem
* Input data are images, we create also a flattened version
* Original images are RGB, and arbitrary size
* Preprocessed images are $64 \times 64$ and gray scale

```python
# Preprocessing of original Dogs and Cats Pictures
# Adapted from https://medium.com/@mrgarg.rajat/kaggle-dogs-vs-cats-challenge-complete-step-by-step-guide-part-1-a347194e55b1
# RGB channels are collapsed in GRAYSCALE
# Images are resampled to 64x64

"""
import os, cv2  # cv2 -- OpenCV

train_dir = './data/DogsCats/train/'
rows = 64
cols = 64
train_images = sorted([train_dir+i for i in os.listdir(train_dir)])

def read_image(file_path):
    image = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)
    return cv2.resize(image, (rows, cols),interpolation=cv2.INTER_CUBIC)

def prep_data(images):
    m = len(images)
    X = np.ndarray((m, rows, cols), dtype=np.uint8)
    y = np.zeros((m,))
    print("X.shape is {}".format(X.shape))

    for i,image_file in enumerate(images) :
        image = read_image(image_file)
        X[i,] = np.squeeze(image.reshape((rows, cols)))
        if 'dog' in image_file.split('/')[-1].lower():
            y[i] = 1
        elif 'cat' in image_file.split('/')[-1].lower():
            y[i] = 0

        if i%5000 == 0 :
            print("Proceed {} of {}".format(i, m))

    return X,y

X_train, y_train = prep_data(train_images)
np.save('./data/DogsCats/X.npy', X_train)
np.save('./data/DogsCats/Y.npy', y_train)
"""
```

```python
DogsCatsX = np.load('./data/DogsCats/X.npy')
DogsCatsY = np.load('./data/DogsCats/Y.npy')
K = DogsCatsX.shape[0]
img_size = DogsCatsX.shape[1]
DogsCatsX_flatten = DogsCatsX.reshape(K,img_size*img_size)

print('Size of Input Data Matrix:', DogsCatsX.shape)
print('Size of Flattned Input Data Matrix:', DogsCatsX_flatten.shape)
print('Size of label Data Matrix:', DogsCatsY.shape)
selected = [260, 16000]
plt.subplot(1, 2, 1), plt.imshow(DogsCatsX[selected[0]].reshape(img_size, img_size)), plt.axis('off')
plt.subplot(1, 2, 2), plt.imshow(DogsCatsX[selected[1]].reshape(img_size, img_size)), plt.axis('off')
plt.show()
print('Labels corresponding to figures:', DogsCatsY[selected,])
```

###  2.1. Logistic Regression as a Simple Neural Network

* We can consider logistic regression as an extremely simple (1 layer) neural network

* In this context, $\text{NLL}({\bf w})$ is normally referred to as cross-entropy loss

* We need to find parameters $\bf w$ and $b$ to minimize the loss $\rightarrow$ GD / SGD
* Gradient computation can be simplified using the **chain rule**

\begin{align}
\frac{\partial \text{NLL}}{\partial {\bf w}} & = \frac{\partial \text{NLL}}{\partial {\hat y}} \cdot \frac{\partial \hat y}{\partial o} \cdot \frac{\partial o}{\partial {\bf w}} \\
& = \sum_{k=0}^{K-1} \left[\frac{1 - y_k}{1 - \hat y_k} - \frac{y_k}{\hat y_k}\right]\hat y_k (1-\hat y_k) {\bf x}_k \\
& = \sum_{k=0}^{K-1} \left[(1 - y_k) \hat y_k - y_k (1 - \hat y_k) \right] {\bf x}_k \\
\frac{\partial \text{NLL}}{\partial b} & = \sum_{k=0}^{K-1} \left[(1 - y_k) \hat y_k - y_k (1 - \hat y_k) \right]
\end{align}

* Gradient Descent Optimization

$${\bf w}_{n+1} = {\bf w}_n + \rho_n \sum_{k=0}^{K-1} \left[y_k (1 - \hat y_k) - (1 - y_k) \hat y_k \right] {\bf x}_k = {\bf w}_n + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k){\bf x}_k$$
$$b_{n+1} = b_n + \rho_n \sum_{k=0}^{K-1} \left[y_k (1 - \hat y_k) - (1 - y_k) \hat y_k \right] = b_n + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)$$

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

#dataset = 'DogsCats'
dataset = 'digits'

if dataset=='DogsCats':
    X = DogsCatsX_flatten
    y = DogsCatsY

else:
    #Zero and Ones are one hot encoded in columns 1 and 4
    X0 = digitsX_flatten[np.argmax(digitsY, axis=1)==1,]
    X1 = digitsX_flatten[np.argmax(digitsY, axis=1)==4,]
    X = np.vstack((X0, X1))
    y = np.zeros(X.shape[0])
    y[X0.shape[0]:] = 1

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(X)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, shuffle=True)
```

```python
# Define some useful functions
def logistic(t):
    return 1.0 / (1 + np.exp(-t))

def forward(w,b,x):
    #Calcula la salida de la red
    return logistic(x.dot(w)+b)

def backward(y,y_hat,x):
    #Calcula los gradientes
    #w_grad = x.T.dot((1-y)*y_hat - y*(1-y_hat))/len(y)
    #b_grad = np.sum((1-y)*y_hat - y*(1-y_hat))/len(y)
    w_grad = x.T.dot(y_hat-y)/len(y)
    b_grad = np.sum(y_hat-y)/len(y)
    return w_grad, b_grad

def accuracy(y, y_hat):
    return np.mean(y == (y_hat>=0.5))

def loss(y, y_hat):
    return -np.sum(y*np.log(y_hat)+(1-y)*np.log(1-y_hat))/len(y)
```

```python
#Neural Network Training

epochs = 50
rho = .05 #Use this setting for Sign Digits Dataset

#Parameter initialization
w = .1 * np.random.randn(X.shape[1])
b = .1 * np.random.randn(1)

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in np.arange(epochs):
    y_hat_train = forward(w, b, X_train)
    y_hat_val = forward(w, b, X_val)
    w_grad, b_grad = backward(y_train, y_hat_train, X_train)
    w = w - rho * w_grad
    b = b - rho * b_grad

    loss_train[epoch] = loss(y_train, y_hat_train)
    loss_val[epoch] = loss(y_val, y_hat_val)
    acc_train[epoch] = accuracy(y_train, y_hat_train)
    acc_val[epoch] = accuracy(y_val, y_hat_val)
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Exercise

* Study the behavior of the algorithm changing the number of epochs and the learning rate

* Repeat the analysis for the other dataset, trying to obtain as large an accuracy value as possible

* What do you believe are the reasons for the very different performance for both datasets?

Linear logistic regression allowed us to review a few concepts that are key for Neural Networks:

* Network topology (In this case, a linear network with one layer)
* Activation functions
* Parametric approach ($\bf w$/$b$)
* Parameter initialization
* Obtaining the network prediction using *forward* computation
* Loss function
* Parameter gradient calculus using *backward* computation
* Optimization method for parameters update (here, GD)

###  2.2. (Multiclass) SoftMax Regression

* One hot encoding output, e.g., $[0, 1, 0, 0]$, $[0, 0, 0, 1]$
* Used to encode categorial variables without predefined order
* Similar to logistic regression, network tries to predict class probability
$$\hat y_{k,j} = \hat P(y_k=j|{\bf x}_k)$$
* Network output should satisfy "probability constraints"
$$\hat y_{k,j} \in [0,1]\qquad \text{and} \qquad \sum_j \hat y_{k,j} = 1$$

* Softmax regression network topology:

### Notation

In this section, it is important to pay attention to subindexes:

|Notation/ Variable Name|Definition|
|-----------------------|---------------------------------|
|$y_k \in [0,\dots,M-1]$|The label of pattern $k$|
|${\bf y}_k$|One hot encoding of the label of pattern $k$|
|$y_{k,m}$|$m$-th component of vector ${\bf y}_k$|
|$y_{m}$|$m$-th component of generic vector ${\bf y}$ (i.e., for an undefined pattern)|
|$\hat {\bf y}_k$|Network output for pattern $k$|
|$\hat y_{k,m}$|$m$-th network output for pattern $k$|
|$\hat y_{m}$|$m$-th network output for an undefined pattern)|
|$k$|Index used for pattern enumeration|
|$m$|Index used for network output enumeration|
|$j$|Secondary index for selected network output|

### The softmax function

* It is to multiclass problems as the logistic function for binary classification
* Invented in 1959 by the social scientist R. Duncan Luce
* Transforms a set of $M$ real numbers to satisfy "probability" constraints

$${\bf \hat y} = \text{softmax}({\bf o}) \qquad \text{where} \qquad \hat y_j = \frac{\exp(o_j)}{\sum_m \exp(o_m)} $$

* Continuous and **differentiable** function

$$\frac{\partial \hat y_j}{\partial o_j} = \hat y_j (1 - \hat y_j) \qquad \text{and} \qquad \frac{\partial \hat y_j}{\partial o_m} = - \hat y_j \hat y_m$$

* The classifier is still linear, since

$$\arg\max \hat {\bf y} = \arg\max \hat {\bf o} = \arg\max {\bf W} {\bf x} + {\bf b}$$

### Cross-entropy loss for multiclass problems

* Similarly to logistic regression, minimization of the log-likelihood can be stated to obtain ${\bf W}$ and ${\bf b}$

$$\text{Binary}: \text{NLL}({\bf w}, b) = - \sum_{k=0}^{K-1} \log \hat P(y_k|{\bf x}_k)$$
$$\text{Multiclass}: \text{NLL}({\bf W}, {\bf b}) = - \sum_{k=0}^{K-1} \log \hat P(y_k|{\bf x}_k)$$

* Using one hot encoding for the label vector of each sample, e.g., $y_k = 2 \rightarrow {\bf y}_k = [0, 0, 1, 0]$

$$\text{NLL}({\bf W}, {\bf b}) = - \sum_{k=0}^{K-1} \sum_{m=0}^{M-1} y_{k,m} \log \hat P(m|{\bf x}_k)= - \sum_{k=0}^{K-1} \sum_{m=0}^{M-1} y_{k,m} \log \hat y_{k,m} = \sum_{k=0}^{K-1} l({\bf y}_k, \hat {\bf y}_k)$$

* Note that for each pattern, only one element in the inner sum (the one indexed with $m$) is non-zero

* In the context of Neural Networks, this cost is referred to as the cross-entropy loss

$$l({\bf y}, \hat {\bf y}) = - \sum_{m=0}^{M-1} y_{m} \log \hat y_{m}$$

### Network optimization

* Gradient Descent Optimization

$${\bf W}_{n+1} = {\bf W}_n - \rho_n \sum_{k=0}^{K-1} \frac{\partial l({\bf y}_k,{\hat {\bf y}_k})}{\partial {\bf W}}$$
$${\bf b}_{n+1} = {\bf b}_n - \rho_n \sum_{k=0}^{K-1} \frac{\partial l({\bf y}_k,{\hat {\bf y}_k})}{\partial {\bf b}}$$

* We compute derivatives using the chain rule (we ignore dimension mismatchs, and rearrange at the end)

\begin{align}
\frac{\partial l({\bf y},{\hat {\bf y}})}{\partial {\bf W}} &= \frac{\partial l({\bf y},{\hat {\bf y}})}{\partial \hat {\bf y}} \cdot \frac{\partial \hat {\bf y}}{\partial {\bf o}} \cdot \frac{\partial {\bf o}}{\partial {\bf W}} \\ & = \left[\begin{array}{c} 0 \\ 0 \\ \vdots \\ - 1/\hat y_j \\ \vdots \end{array}\right] \left[ \begin{array}{ccccc} \hat y_1 (1 - \hat y_1) & -\hat y_1 \hat y_2 & \dots & -\hat y_1 \hat y_j & \dots \\ -\hat y_2 \hat y_1 & \hat y_2 (1 - \hat y_2) & \dots & -\hat y_2 \hat y_j & \dots \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ - \hat y_j \hat y_1 & -\hat y_j \hat y_2 & \dots & \hat y_j (1-\hat y_j) & \dots \\ \vdots & \vdots & \ddots & \vdots & \vdots \end{array}\right] {\bf x}^\top \\
& = \left[\begin{array}{c}\hat y_1 \\ \hat y_2 \\ \vdots \\ \hat y_j - 1 \\ \vdots \end{array} \right] {\bf x}^\top \\
& = (\hat {\bf y} - {\bf y}){\bf x}^\top \\
\\
\frac{\partial l({\bf y},{\hat {\bf y}})}{\partial {\bf b}} & = (\hat {\bf y} - {\bf y}){\bf 1}^\top
\end{align}

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

dataset = 'digits'

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(digitsX_flatten)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, digitsY, test_size=0.2, random_state=42, shuffle=True)
```

```python
# Define some useful functions
def softmax(t):
    """Compute softmax values for each sets of scores in t."""
    e_t = np.exp(t)
    return e_t / e_t.sum(axis=1)[:,np.newaxis]

def forward(w,b,x):
    #Calcula la salida de la red
    return softmax(x.dot(w.T)+b.T)

def backward(y,y_hat,x):
    #Calcula los gradientes
    W_grad = (y_hat-y).T.dot(x)/len(y)
    b_grad = ((y_hat-y).sum(axis=0)[:,np.newaxis])/len(y)
    return W_grad, b_grad

def accuracy(y, y_hat):
    return np.mean(np.argmax(y, axis=1) == np.argmax(y_hat, axis=1))

def loss(y, y_hat):
    return -np.sum(y * np.log(y_hat))/len(y)
```

```python
#Neural Network Training

epochs = 300
rho = .1

#Parameter initialization
W = .1 * np.random.randn(y_train.shape[1], X_train.shape[1])
b = .1 * np.random.randn(y_train.shape[1],1)

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in np.arange(epochs):
    y_hat_train = forward(W, b, X_train)
    y_hat_val = forward(W, b, X_val)
    W_grad, b_grad = backward(y_train, y_hat_train, X_train)
    W = W - rho * W_grad
    b = b - rho * b_grad

    loss_train[epoch] = loss(y_train, y_hat_train)
    loss_val[epoch] = loss(y_val, y_hat_val)
    acc_train[epoch] = accuracy(y_train, y_hat_train)
    acc_val[epoch] = accuracy(y_val, y_hat_val)
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Exercise

* Study the behavior of the algorithm changing the number of iterations and the learning rate

* Obtain the confusion matrix, and study which classes are more difficult to classify

* Think about the differences between using this 10-class network, vs training 10 binary classifiers, one for each class

As in linear logistic regression note that we covered the following aspects of neural network design, implementation, and training:

* Network topology (In this case, a linear network with one layer and $M$ ouptuts)
* Activation functions (softmax activation)
* Parameter initialization ($\bf W$/$b$)
* Obtaining the network prediction using *forward* computation
* Loss function
* Parameter gradient calculus using *backward* computation
* Optimization method for parameters update (here, GD)

###  2.3. Multi Layer Networks (Deep Networks)

Previous networks are constrained in the sense that they can only implement linear classifiers. In this section we analyze how we can extend them to implement non-linear classification:
* Fixed non-linear transformations of inputs: ${\bf z} = {\bf{f}}({\bf x})$

* Parametrize the transformation using additional non-linear layers

* When counting layers, we normally ignore the input layer, since there is no computation involved
* Intermediate layers are normally referred to as "hidden" layers
* Non-linear activations result in an overall non-linear classifier
* We can still use Gradient Descent Optimization as long as the network loss derivatives with respect to all parameters exist and are continuous
* This is already deep learning. We can have two layers or more, each with different numbers of neurons. But as long as derivatives with respect to parameters can be calculated, the network can be optimized
* Finding an appropriate number of layers for a particular problem, as well as the number of neurons per layer, requires exploration
* The more data we have for training the network, the more parameters we can afford, making feasible the use of more complex topologies

### Example: 2-layer network for binary classification

* Network topology
    * Hidden layer with $n_h$ neurons
    * Hyperbolic tangent activation function for the hidden layer
    $${\bf h} = \text{tanh}({\bf o}^{(1)})= \text{tanh}\left({\bf W}^{(1)} {\bf x} + {\bf b}^{(1)}\right)$$
    * Output layer is linear with logistic activation (as in logistic regression)
    $$\hat y = \text{logistic}(o) = \text{logistic}\left({{\bf w}^{(2)}}^\top {\bf h} + b^{(2)}\right)$$

* Cross-entropy loss

$$l(y,\hat y) = -\left[ y \log(\hat y) + (1 - y ) \log(1 - \hat y) \right], \qquad \text{with } y\in [0,1]$$

* Update of output layer weights as in logistic regression (use ${\bf h}$ instead of ${\bf x}$)

$${\bf w}_{n+1}^{(2)} = {\bf w}_n^{(2)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k){\bf h}_k$$
$$b_{n+1}^{(2)} = b_n^{(2)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)$$

* For updating the input layer parameters we need to use the chain rule (we ignore dimensions and rearrange at the end)

\begin{align}\frac{\partial l(y, \hat y)}{\partial {\bf W}^{(1)}} & = \frac{\partial l(y, \hat y)}{\partial o} \cdot \frac{\partial o}{\partial {\bf h}} \cdot \frac{\partial {\bf h}}{\partial {\bf o}^{(1)}} \cdot \frac{\partial {\bf o}^{(1)}}{\partial {\bf W}^{(1)}} \\
& = (\hat y - y) [{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h})^2] {\bf x}^{\top}
\end{align}

\begin{align}\frac{\partial l(y, \hat y)}{\partial {\bf b}^{(1)}} & = \frac{\partial l(y, \hat y)}{\partial o} \cdot \frac{\partial o}{\partial {\bf h}} \cdot \frac{\partial {\bf h}}{\partial {\bf o}^{(1)}} \cdot \frac{\partial {\bf o}^{(1)}}{\partial {\bf b}^{(1)}} \\
& = (\hat y - y) [{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h})^2]
\end{align}

* GD update rules become
$${\bf W}_{n+1}^{(1)} = {\bf W}_n^{(1)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)[{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h}_k)^2] {\bf x}_k^{\top}$$
$${\bf b}_{n+1}^{(1)} = {\bf b}_n^{(1)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)[{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h}_k)^2]$$

* The process can be implemented as long as the derivatives of the network overall loss with respect to parameters can be computed

* Forward computation graphs represent how the network output can be computed

* We can then reverse the graph to compute derivatives with respect to parameters

* Deep Learning libraries implement automatic gradient camputation
   * We just define network topology
   * Computation of gradients is carried out automatically

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

#dataset = 'DogsCats'
dataset = 'digits'

if dataset=='DogsCats':
    X = DogsCatsX_flatten
    y = DogsCatsY

else:
    #Zero and Ones are one hot encoded in columns 1 and 4
    X0 = digitsX_flatten[np.argmax(digitsY, axis=1)==1,]
    X1 = digitsX_flatten[np.argmax(digitsY, axis=1)==4,]
    X = np.vstack((X0, X1))
    y = np.zeros(X.shape[0])
    y[X0.shape[0]:] = 1

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(X)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, shuffle=True)
```

```python
# Define some useful functions
def logistic(t):
    return 1.0 / (1 + np.exp(-t))

def forward(W1,b1,w2,b2,x):
    #Calcula la salida de la red
    h = x.dot(W1.T)+b1
    y_hat = logistic(h.dot(w2)+b2)
    #Provide also hidden units value for backward gradient step
    return h, y_hat

def backward(y,y_hat,h,x,w2):
    #Calcula los gradientes
    w2_grad = h.T.dot(y_hat-y)/len(y)
    b2_grad = np.sum(y_hat-y)/len(y)
    W1_grad = ((w2[np.newaxis,]*((1-h)**2)*(y_hat - y)[:,np.newaxis]).T.dot(x))/len(y)
    b1_grad = ((w2[np.newaxis,]*((1-h)**2)*(y_hat - y)[:,np.newaxis]).sum(axis=0))/len(y)
    return w2_grad, b2_grad, W1_grad, b1_grad

def accuracy(y, y_hat):
    return np.mean(y == (y_hat>=0.5))

def loss(y, y_hat):
    return -np.sum(y*np.log(y_hat)+(1-y)*np.log(1-y_hat))/len(y)
```

```python
#Neural Network Training
epochs = 1000
rho = .05

#Parameter initialization
n_h = 5
W1 = .01 * np.random.randn(n_h, X_train.shape[1])
b1 = .01 * np.random.randn(n_h)
w2 = .01 * np.random.randn(n_h)
b2 = .01 * np.random.randn(1)

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in np.arange(epochs):
    h, y_hat_train = forward(W1, b1, w2, b2, X_train)
    dum, y_hat_val = forward(W1, b1, w2, b2, X_val)
    w2_grad, b2_grad, W1_grad, b1_grad = backward(y_train, y_hat_train, h, X_train, w2)
    W1 = W1 - rho/10 * W1_grad
    b1 = b1 - rho/10 * b1_grad
    w2 = w2 - rho * w2_grad
    b2 = b2 - rho * b2_grad

    loss_train[epoch] = loss(y_train, y_hat_train)
    loss_val[epoch] = loss(y_val, y_hat_val)
    acc_train[epoch] = accuracy(y_train, y_hat_train)
    acc_val[epoch] = accuracy(y_val, y_hat_val)

    if not ((epoch+1)%(epochs/5)):
        print('Número de iteraciones:', epoch+1)
```

### Results in Dogs vs Cats dataset ($epochs = 1000$ and $\rho = 0.05$)

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Results in Binary Sign Digits Dataset ($epochs = 10000$ and $\rho = 0.001$)

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Exercises

* Train the network using other settings for:
    * The number of iterations
    * The learning step
    * The number of neurons in the hidden layer

* You may find divergence issues for some settings
    * Related to the use of the hyperbolic tangent function in the hidden layer (numerical issues)
    * This is also why learning step was selected smaller for the hidden layer
    * Optimized libraries rely on certain modifications to obtain more robust implementations

* Try to solve both problems using scikit-learn implementation
    * You can also explore other activation functions
    * You can also explore other solvers to speed up convergence
    * You can also adjust the size of minibatches
    * Take a look at the *early_stopping* parameter

###  2.4. Multi Layer Networks for Regression

* Deep Learning networks can be used to solve regression problems with the following common adjustments

    * Linear activation for the output unit

    * Square loss:
    $$l(y, \hat y) = (y - \hat y)^2, \qquad \text{where} \qquad y, \hat y \in \Re$$

###  2.5. Activation Functions

You can refer to the Dive into Deep Learning book for a more detailed discussion on common actiation functions for the hidden units.

We extract some information about the very important **ReLU** function

> *The most popular choice, due to both simplicity of implementation and its good performance on a variety of predictive tasks, is the rectified linear unit (ReLU). ReLU provides a very simple nonlinear transformation. Given an element $x$, the function is defined as the maximum of that element and 0.*

> *When the input is negative, the derivative of the ReLU function is 0, and when the input is positive, the derivative of the ReLU function is 1. When the input takes value precisely equal to 0, we say that the derivative is 0 when the input is 0.*

> *The reason for using ReLU is that its derivatives are particularly well behaved: either they vanish or they just let the argument through. This makes optimization better behaved and it mitigated the well-documented problem of vanishing gradients that plagued previous versions of neural networks.*

```python
x_array = np.linspace(-6,6,100)
y_array = np.clip(x_array, 0, a_max=None)
plt.plot(x_array, y_array)
plt.title('ReLU activation function')
plt.show()
```

##  3. Implementing Deep Networks with PyTorch

* Pytorch is a Python library that provides different levels of abstraction for implementing deep neural networks

* The main features of PyTorch are:

    * Definition of numpy-like n-dimensional *tensors*. They can be stored in / moved to GPU for parallel execution of operations
    * Automatic calculation of gradients, making *backward gradient calculation* transparent to the user
    * Definition of common loss functions, NN layers of different types, optimization methods, data loaders, etc, simplifying NN implementation and training
    * Provides different levels of abstraction, thus a good balance between flexibility and simplicity

* This notebook provides just a basic review of the main concepts necessary to train NNs with PyTorch taking materials from:
    * Learning PyTorch with Examples, by Justin Johnson
    * What is *torch.nn* really?, by Jeremy Howard
    * Pytorch Tutorial for Deep Learning Lovers, by Kaggle user kanncaa1

###  3.0. Installation and PyTorch introduction

* PyTorch can be installed with or without GPU support
    * If you have an Anaconda installation, you can install from the command line, using the instructions of the project website

* PyTorch is also preinstalled in Google Collab with free GPU access
    * Follow RunTime -> Change runtime type, and select GPU for HW acceleration

* Please, refer to Pytorch getting started tutorial for a quick introduction regarding tensor definition, GPU vs CPU storage of tensors, operations, and bridge to Numpy

###  3.1. Torch tensors (very) general overview

* We can create tensors with different construction methods provided by the library, either to create new tensors from scratch or from a Numpy array

```python
import torch

x = torch.rand((100,200))
digitsX_flatten_tensor = torch.from_numpy(digitsX_flatten)

print(x.type())
print(digitsX_flatten_tensor.size())
```

    torch.FloatTensor
    torch.Size([2062, 4096])

* Tensors can be converted back to numpy arrays

* Note that in this case, a tensor and its corresponding numpy array **will share memory**

* Operations and slicing use a syntax similar to numpy

```python
print('Size of tensor x:', x.size())
print('Tranpose of vector has size', x.t().size()) #Transpose and compute size
print('Extracting upper left matrix of size 3 x 3:', x[:3,:3])
print(x.mm(x.t()).size())  #mm for matrix multiplications
xpx = x.add(x)
xpx2 = torch.add(x,x)
print((xpx!=xpx2).sum())   #Since all are equal, count of different terms is zero
```

    Size of tensor x: torch.Size([100, 200])
    Tranpose of vector has size torch.Size([200, 100])
    Extracting upper left matrix of size 3 x 3: tensor([[0.6252, 0.0318, 0.4039],
            [0.0704, 0.7540, 0.4963],
            [0.1201, 0.1661, 0.0053]])
    torch.Size([100, 100])
    tensor(0)

* Adding underscore performs operations "*in place*", e.g., ```x.add_(y)```

* If a GPU is available, tensors can be moved to and from the GPU device

* Operations on tensors stored in a GPU will be carried out using GPU resources and will typically be highly parallelized

```python
if torch.cuda.is_available():
    device = torch.device('cuda')
    x = x.to(device)
    y = x.add(x)
    y = y.to('cpu')
else:
    print('No GPU card is available')
```

    No GPU card is available

###  3.2. Automatic gradient calculation

* PyTorch tensors have a property ```requires_grad```. When true, PyTorch automatic gradient calculation will be activated for that variable

* In order to compute these derivatives numerically, PyTorch keeps track of all operations carried out on these variables, organizing them in a forward computation graph.

* When executing the ```backward()``` method, derivatives will be calculated

* However, this should only be activated when necessary, to save computation

```python
x.requires_grad = True
y = (3 * torch.log(x)).sum()
y.backward()
print(x.grad[:2,:2])
print(3/x[:2,:2])

x.requires_grad = False
x.grad.zero_()
print('Automatic gradient calculation is deactivated, and gradients set to zero')
```

    tensor([[ 4.7981, 94.4270],
            [42.5862,  3.9788]])
    tensor([[ 4.7981, 94.4270],
            [42.5862,  3.9788]], grad_fn=<MulBackward0>)
    Automatic gradient calculation is deactivated, and gradients set to zero

Exercise

* Initialize a tensor ```x``` with the upper right $5 \times 10$ submatrix of flattened digits
* Compute output vector ```y``` applying a function of your choice to ```x```
* Compute scalar value ```z``` as the sum of all elements in ```y``` squared
* Check that ```x.grad``` calculation is correct using the ```backward``` method
* Try to run your cell multiple times to see if the calculation is still correct. If not, implement the necessary mnodifications so that you can run the cell multiple times, but the gradient does not change from run to run

**Note:** The backward method can only be run on scalar variables

###  3.2. Feed Forward Network using PyTorch

* In this section we will change our code for a neural network to use tensors instead of numpy arrays. We will work with the sign digits datasets.

* We will introduce all concepts using a single layer perceptron (softmax regression), and then implement networks with additional hidden layers

###  3.2.1. Using Automatic differentiation

* We start by loading the data, and converting to tensors.

* As a first step, we refactor our code to use tensor operations

* We do not need to pay too much attention to particular details regarding tensor operations, since these will not be necessary when moving to higher PyTorch abstraction levels

* We do not need to implement gradient calculation. PyTorch will take care of that

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

dataset = 'digits'

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(digitsX_flatten)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, digitsY, test_size=0.2, random_state=42, shuffle=True)

#Convert to Torch tensors
X_train_torch = torch.from_numpy(X_train)
X_val_torch = torch.from_numpy(X_val)
y_train_torch = torch.from_numpy(y_train)
y_val_torch = torch.from_numpy(y_val)
```

```python
# Define some useful functions
def softmax(t):
    """Compute softmax values for each sets of scores in t"""
    return t.exp() / t.exp().sum(-1).unsqueeze(-1)

def model(w,b,x):
    #Calcula la salida de la red
    return softmax(x.mm(w) + b)

def accuracy(y, y_hat):
    return (y.argmax(axis=-1) == y_hat.argmax(axis=-1)).float().mean()

def nll(y, y_hat):
    return -(y * y_hat.log()).mean()
```

* Syntaxis is a bit different because input variables are tensors, not arrays

* This time we did not need to implement the backward function

```python
#Parameter initialization
W = .1 * torch.randn(X_train_torch.size()[1], y_train_torch.size()[1])
W.requires_grad_()
b = torch.zeros(y_train_torch.size()[1], requires_grad=True)

epochs = 500
rho = .5

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)
```

```python
# Network training

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = model(W,b,X_train_torch)
    loss = nll(y_train_torch, pred)

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = model(W, b, X_val_torch)
        loss_val[epoch] = nll(y_val_torch, pred_val).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        W -= rho * W.grad
        b -= rho * b.grad
        #Reset gradients
        W.grad.zero_()
        b.grad.zero_()
```

It is important to deactivate gradient updates after the network has been evaluated on training data, and gradients of the loss function have been computed

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

###  3.2.2. Using torch *nn* module

* PyTorch *nn* module provides many attributes and methods that make the implementation and training of Neural Networks simpler

* ```nn.Module``` and ```nn.Parameter``` allow to implement a more concise training loop

* ```nn.Module``` is a PyTorch class that will be used to encapsulate and design a specific neural network, thus, it is central to the implementation of deep neural nets using PyTorch

* ```nn.Parameter``` allow the definition of trainable network parameters. In this way, we will simplify the implementation of the training loop.

* All parameters defined with ```nn.Parameter``` will have ```requires_grad = True```

```python
from torch import nn

class my_multiclass_net(nn.Module):
    def __init__(self, nin, nout):
        """This method initializes the network parameters
        Parameters nin and nout stand for the number of input parameters (features in X)
        and output parameters (number of classes)"""
        super().__init__()
        self.W = nn.Parameter(.1 * torch.randn(nin, nout))
        self.b = nn.Parameter(torch.zeros(nout))

    def forward(self, x):
        return softmax(x.mm(self.W) + self.b)

    def softmax(t):
        """Compute softmax values for each sets of scores in t"""
        return t.exp() / t.exp().sum(-1).unsqueeze(-1)
```

```python
my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])

epochs = 500
rho = .5

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = nll(y_train_torch, pred)

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = nll(y_val_torch, pred_val).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        for p in my_net.parameters():
            p -= p.grad * rho
        #Reset gradients
        my_net.zero_grad()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

* ```nn.Module``` comes with several kinds of pre-defined layers, thus making it even simpler to implement neural networks

* We can also import the Cross Entropy Loss from ```nn.Module```. When doing so:
    - We do not have to compute the softmax, since the ```nn.CrossEntropyLoss``` already does so
    - ```nn.CrossEntropyLoss``` receives two input arguments, the first is the output of the network, and the second is the true label as a 1-D tensor (i.e., an array of integers, one-hot encoding should not be used)

```python
from torch import nn

class my_multiclass_net(nn.Module):
    def __init__(self, nin, nout):
        """Note that now, we do not even need to initialize network parameters ourselves"""
        super().__init__()
        self.lin = nn.Linear(nin, nout)

    def forward(self, x):
        return self.lin(x)

loss_func = nn.CrossEntropyLoss()
```

```python
my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])

epochs = 500
rho = .1

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = loss_func(pred, y_train_torch.argmax(axis=-1))

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        for p in my_net.parameters():
            p -= p.grad * rho
        #Reset gradients
        my_net.zero_grad()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

Note faster convergence is observed in this case. It is actually due to a more convenient initialization of the hidden layer

###  3.2.3. Network Optimization

* We cover in this subsection two different aspects about network training using PyTorch:

    + Using ```torch.optim``` allows an easier and more interpretable encoding of neural network training, and opens the door to more sophisticated training algorithms

    + Using minibatches can speed up network convergence

* ```torch.optim``` provides two convenient methods for neural network training:
    - ```opt.step()``` updates all network parameters using current gradients
    - ```opt.zero_grad()``` resets all network parameters

```python
from torch import optim

my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])
opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 500

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = loss_func(pred, y_train_torch.argmax(axis=-1))

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

    opt.step()
    opt.zero_grad()
```

Note network optimization is carried out outside ```torch.no_grad()``` but network evaluation (other than forward output calculation for the training patterns) still need to deactivate gradient updates

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

###  Exercise

Implement network training with other optimization methods. You can refer to the official documentation and select a couple of methods. You can also try to implement adaptive learning rates using ```torch.optim.lr_scheduler```

* Each epoch of the previous implementation of network training was actually implementing Gradient Descent

* In SGD only a *minibatch* of training patterns are used at every iteration

* In each epoch we iterate over all training patterns sequentially selecting non-overlapping *minibatches*

* Overall, convergence is usually faster than when using Gradient Descent

* Torch provides methods that simplify the implementation of this strategy

```python
from torch.utils.data import TensorDataset, DataLoader

train_ds = TensorDataset(X_train_torch, y_train_torch)
train_dl = DataLoader(train_ds, batch_size=64)
```

```python
from torch import optim

my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])
opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 200

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

###  3.2.4. Multi Layer networks using ```nn.Sequential```

* PyTorch simplifies considerably the implementation of neural network training, since we do not need to implement derivatives ourselves

* We can also make a simpler implementation of multilayer networks using ```nn.Sequential``` function

* It returns directly a network with the requested topology, including parameters **and forward evaluation method**

```python
my_net = nn.Sequential(
    nn.Linear(X_train_torch.size()[1], 200),
    nn.ReLU(),
    nn.Linear(200,50),
    nn.ReLU(),
    nn.Linear(50,20),
    nn.ReLU(),
    nn.Linear(20,y_train_torch.size()[1])
)

opt = optim.SGD(my_net.parameters(), lr=0.1)
```

```python
epochs = 200

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Número de épocas:', epoch+1)

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

```python
print('Validation accuracy with this net:', acc_val[-1])
```

    Validation accuracy with this net: 0.8692494034767151

###  3.3. Generalization

* For complex network topologies (i.e., many parameters), network training can incur in over-fitting issues

* Some common strategies to avoid this are:

    - Early stopping
    - Dropout regularization

Image Source

* Data augmentation can also be used to avoid overfitting, as well as to achieve improved accuracy by providing the network some a priori expert knowledge
    - E.g., if image rotations and scalings do not affect the correct class, we could enlarge the dataset by creating artificial images with these transformations

###  3.5. Convolutional Networks for Image Processing

* PyTorch implements other layers that are better suited for different applications

* In image processing, we normally recur to Convolutional Neural Networks, since they are able to capture the true spatial information of the image

Image Source

```python
dataset = 'digits'

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(digitsX[:,np.newaxis,:,:], digitsY, test_size=0.2, random_state=42, shuffle=True)

#Convert to Torch tensors
X_train_torch = torch.from_numpy(X_train)
X_val_torch = torch.from_numpy(X_val)
y_train_torch = torch.from_numpy(y_train)
y_val_torch = torch.from_numpy(y_val)

train_ds = TensorDataset(X_train_torch, y_train_torch)
train_dl = DataLoader(train_ds, batch_size=64)
```

```python
class Lambda(nn.Module):
    def __init__(self, func):
        super().__init__()
        self.func = func

    def forward(self, x):
        return self.func(x)

my_net = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AvgPool2d(4),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

opt = optim.SGD(my_net.parameters(), lr=0.1)
```

```python
epochs = 2500

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Número de épocas:', epoch+1)

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()
```

    Número de épocas: 500
    Número de épocas: 1000
    Número de épocas: 1500
    Número de épocas: 2000
    Número de épocas: 2500

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

```python

```


# Heteroskedasticity
## Consequences of Heteroskedasticity for OLS

$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Exp}{\mathrm{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Avar}[2][\,\!]{\mathrm{Avar}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}
\newcommand{\FSD}{\text{FSD}}Review$

>Homoskedasticity assumption $\text{MLR}.5$, that $\Var{u\mid x_1,x_2,\dots, x_k} = \sigma^2$, plays no role in showing whether OLS was unbiased or consistent. Only something like omitting an important variable would have this effect.
>
>Also, $R^2$ and $\bar R^2$ are unaffected by the presence of heteroskedasticity. They are the estimators of population $R^2 = 1 - \sigma_u^2/\sigma_y^2$ where the two variances are *un*conditional while heteroskedasticity is under conditioning on $\mathbf{x}$.

Under $\text{MLR}.1$ to $\text{MLR}.4$, $\text{SSR}/n$ consistently estimates $\sigma_u^2$ and $\text{SST}/n$ consistently estimates $\sigma_y^2$. Therefore, $R^2$ and $\bar R^2$ are both consistant estimators of the population $R^2$, whether or not the homoskedasticity assumption holds.

But to do the inference by $t$ statistic and $F$ statistic, we still need that assumption. Besides, if $\Var{u\mid\mathbf{x}}$ is no longer constant, OLS is no longer BLUE.

## Heteroskedasticity-Robust Inference after OLS Estimation

Consider the simple linear regression, the estimator of slope parameter is

$$\hat\beta_1 = \beta_1 + \ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar x}} u_i}{\d{\sum_{i=1}^{n} \P{x_i - \bar x}^2}}$$

$$\Var{u_i\mid x_i} = \sigma_i^2\;{\Longrightarrow}\;\Var{\hat\beta_1} = \ffrac{\sum \P{x_i - \bar x}^2\sigma_i^2}{\text{SST}_x^2}$$

Now we give a valid estimator for general MLR

$$\widehat{\Var{\hat \beta_j}} = \ffrac{\d{\sum_{i=1}^{n}\hat r_{ij}^2 \hat u_i^2}}{\text{SSR}_j^2}$$

where $\hat r_{ij}$ denotes the $i$th residual from regressing $x_j$ on all other independent variables, and $\text{SSR}_j$ is the sum of squared residuals from this regression. More than that, we have the ***heteroskedasticity-robust standard error*** for $\hat\beta_j$:

$$\sqrt{\widehat{\Var{\hat \beta_j}}}= \ffrac{\d{\sqrt{\sum_{i=1}^{n}\hat r_{ij}^2 \hat u_i^2}}}{\text{SSR}_j^2}$$

Here the $\text{SSR}_j^2$ can be replaced by $\text{SST}_j^2\P{1-R_j^2}$, where $\text{SST}_j^2$ is the total sum of squares of $x_j$, and $R_j^2$ is the usual $R^2$ from regressing $x_j$ on all other explanatory variables.

Then the ***heteroskedasticity-robust $t$ statistic***.

$$t = \ffrac{\text{estimate} - \text{hypothesized value}}{\textbf{heteroskedasticity-robust} \text{ standard error}}$$

Using these formulas, the usual $t$ test is valid asymptotically; but the usual $F$-statistic does not work under heteroskedasticity.

### Computing Heteroskedasticity-Robust LM Tests

Skipped, however. OK, the usual LM statistic:

1. Estimate the restricted model to obtain the residual $\tilde u$
2. regress $\tilde u$ on all of the independent variables
3. $\text{LM} = n \cdot R_{\tilde u}^2$, where $R_{\tilde u}^2$ is the $R^2$ from this regression

Then the HR LM statistic in the general case

1. The same: estimate the restricted model to obtain the residuals $\tilde u$
2. Regress each of the independent variables which are excluded under the null hypothesis, on all of the included variables; say there're $q$ excluded variables (restricted model for them: $x_j = \beta_0 + \beta_1 x_1 + \cdots + \beta_{k-q} x_{k-q} + u$, $j = k-q+1,\dots,k$)
3. Obtain $q$ sets of residuals $\P{\tilde r_1,\tilde r_2,\dots, \tilde r_q}$, then find the element-wise production of $\tilde r_j$ and $\tilde u$
4. Regress $y\equiv 1$ on $\beta = 0, \tilde r_1 \tilde u, \tilde r_2 \tilde u,\dots,\tilde r_q \tilde u$
5. The **Heteroskedasticity-Robust LM Statistic** is now $n-\text{SSR}_1$ (I bet this is a minus sign, it could be some other signs though...), where $\text{SSR}_1$ is just the usual sum of squared residuals from the regression in the final step.

By the way, under $H_0$, $\text{LM}$ is distributed approximately as $\chi_q^2$

## Testing for Heteroskedasticity

Model: $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$; assumption: $\text{MLR}.1$ through $\text{MLR}.4$, so that the OLS estimators are still unbiased and consistent.

To test the heteroskedasticity, we have $H_0: \Var{u\mid x_1,x_2,\dots,x_k} = \sigma^2$. Since $u$ has a zero conditional expectation, this is equivalent to

$$H_0: \Exp\SB{u^2\mid x_1,\dots,x_k} = \Exp\SB{u^2} = \sigma^2$$

So we are actually testing weather $u^2$ is related (in expected value) to one or more of the explanatory variables. Then we assume the linear function

$$u^2 = \delta_0 + \delta_1 x_1 + \delta_2 x_2 + \cdots + \delta_k x_k + v$$

where $v$ is an error term with mean zero given the $x_j$. Then we rewrite the null hypothesis of homoskedasticity as $H_0: \delta_1 = \delta_2 = \cdots = \delta_k = 0$.

Then we can use $F$ statistic or $\text{LM}$ to test this. And to do so, we first need to estimate the left side, the residual and since it's unable to obtain, we will use its estimation $\hat u_i$ so actually the equation to be process is actually

$$\hat u^2 = \delta_0 + \delta_1 x_1 + \delta_2 x_2 + \cdots + \delta_k x_k + \text{error}$$

Then apply the $F$ test or $\text{LM}$ test. And to distinguish two different $R$-square, we denote the $F$ statistic in this regression

$$F = \ffrac{\ffrac{R_{\hat u^2}^2}{k}}{\ffrac{1-R_{\hat u^2}^2}{n-k-1}}$$

And the $\text{LM}$ statistic is $\text{LM} = n\cdot R_{\hat u^2}^2$. Under $H_0$, it's distributed asymptotically as $\chi_k^2$. The $\text{LM}$ version of the test is typically called the ***Breusch-Pagan test for heteroskedasticity (BP test)***.

$Remark$

>Larger $R_{\hat u^2}^2$ could be the evidence against the null hypothesis.

**Steps for BP test**:

1. Estimate the model $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$ by OLS, as usual. And for each observation, find the residual $\hat u^2$
2. Regression the equation $\hat u^2 = \delta_0 + \delta_1 x_1 + \delta_2 x_2 + \cdots + \delta_k x_k + \text{error}$, with the $R$-squared $R_{\hat u^2}^2$.
3.  Form either the $F$ statistic or the $\text{LM}$ statistic and compute the $p$-value. Use $F_{k,n-k-1}$ distribution for $F$ statistic and $\chi_k^2$ distribution for the otherone. If the $p$-value is below the chosen significance level, meaning that it's sufficiently small, we reject the null hypothesis and admit the heteroskedasticity.

$Remark$

> If we suspect that heteroskedasticity depends only upon certain independent variables, we can simple modify the BP test that we regress $\hat u^2$ only on the chosen variables and then carry out the appropriate $F$ or $\text{LM}$ test.

### The White Test for Heteroskedasticity

The ***White test for heteroskedasticity*** is the $\text{LM}$ statistic for testing that all of the $\delta_j$ where $j=1,2,\dots$ (no $0$ here), defined by

$$\begin{align}
\hat u^2 &= \P{y - \hat\beta_0 - \hat\beta_1 x_1 - \cdots - \hat\beta_k x_k}^2 \\
&\equiv \delta_0 + \sum_{i=1}^k \delta_i x_i + \sum_{i=1}^k \delta_{k+i} x_i^2  + \sum_{j<k} \delta_{i\P{j,k}} x_jx_k + \text{error}
\end{align}$$

$F$ statistics can also do, however one drawback is that this method is of too many parameters so that with less $df$. So here's a better one.

$$\hat u^2 = \delta_0 + \delta_1 \hat y + \delta_2 \hat y^2 + \text{error}$$

$H_0: \delta_1 = \delta_2 = 0$; $\text{LM} = n \cdot R_{\hat u^2}^2\sim \chi_2^2$

**Case**:

1. Estimate the model $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$ by OLS, as usual. And for each observation, find the residual $\hat u$ and fitted value $\hat y$; then use the two to compute $\hat u^2$ and $\hat y^2$
2. Regress $\hat u^2 = \delta_0 + \delta_1 \hat y + \delta_2 \hat y^2 + \text{error}$. Keep the $R$-squared from this regression, $R_{\hat u^2}^2$
3. Form either the $F$ or $\text{LM}$ statistic and compute the $p$-value. Use $F_{2,n-3}$ or $\chi_2^2$.

## Weighted Least Squares Estimation

If we have correctly specified the form of the variance, then weighted least squares (WLS) is more efficient than OLS, and WLS leads to new $t$ and $F$ statistics that have $t$ and $F$ distributions.

### The Heteroskedasticity is Known up to a Multiplicative Constant

Model: $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$, and let $\mathbf{x}$ denote all the explanatory variables. We assume that

$$\Var{u\mid\mathbf{x}} = \sigma^2 h\P{\mathbf x}$$

where $h\P{\mathbf x}$ is some function of the explanatory variables that determines the heteroskedasticity. It's *known* and *positive*. Only the population parameter $\sigma^2$ is required to be estimated. Then we take the original equation

$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_k x_{ik} + u_i$$

We'll transform it into an equation that has homoskedastic errors under Gauss-Markov Assumptions.

$$\Exp\SB{u_i\mid\mathbf x_i} = 0\Rightarrow\Exp\SB{\ffrac{u_i}{h_i}\mid\mathbf x_i} = 0$$

$$\Var{u_i\mid \mathbf x_i} = \Exp\SB{u_i^2\mid \mathbf x_i} = \sigma^2 h_i$$

$$\Exp\SB{\P{\ffrac{u_i}{\sqrt{h_i}}}^2\mid\mathbf x_i} = \ffrac{\Exp\SB{u_i^2\mid\mathbf x_i}}{h_i} = \ffrac{\sigma^2 h_i}{h_i} = \sigma^2$$

Then inspired by this, we divide $\sqrt{h_i}$ on both sides of the equation.

$$\ffrac{y_i}{\sqrt{h_i}} = \ffrac{\beta_0}{\sqrt{h_i}} + \beta_1\P{\ffrac{x_{i1}}{\sqrt{h_i}}} + \beta_2\P{\ffrac{x_{i2}}{\sqrt{h_i}}} + \cdots + \beta_k\P{\ffrac{x_{ik}}{\sqrt{h_i}}}+ \ffrac{u_i}{\sqrt{h_i}}$$

and we'll rewrite this as

$$y_i^* = \beta_0x_{i0}^* + \beta_1x_{i1}^* + \cdots + \beta_kx_{ik}^* + u_{i}^*,\bspace x_{i0}^* = \ffrac{1}{\sqrt{h_i}}$$

then we can use the Gauss-Markov assumptions. More than that, $\text{MLR}.6$ can be kept holding true after that transform.

The estimators $\beta_j^*$ here are examples of ***generalized least squares (GLS) estimators***. And WLS is one special case that is used to account for heteroskedasticity in the errors. Mathematically, the WLS estimators are the values of the $b_j$ that make

$$\sum_{i=1}^{n}\ffrac{1}{h_i} \P{y_i - b_0 - b_1 x_{i1} - \cdots - b_kx_{ik}}^2$$

as small as possible.

***
One special case that when the observations are reported as averages of certain level, the variables should be weighted by the size of the unit. A simple example:

$$\overline{\text{contrib}}_i = \beta_0 + \beta_1 \overline{\text{earns}}_i + \beta_2 \overline{\text{age}}_i + \beta_3 \text{mrate}_i + \overline u_i$$

It can be considered an example for a average from a firm $i$'s total $m_i$ workers, where $\overline u_i = m_i^{-1} \sum_{e=1}^{m_i} u_{i,e}$.

If, from the respect of individuals, the data are homoskedastic, $\Var{u_{i,e}} = \sigma^2$, then from the respect of firms, we have $\Var{\bar u_i} = \sigma^2/m_i$.

### The Heteroskedasticity Function Must Be Estimated: Feasible GLS

For the unknown heteroscedasticity function, we assume a general one

$$\Var{u\mid \mathbf x} = \sigma^2 \exp\CB{\delta_0 + \delta_1 x_1 + \cdots + \delta_k x_k} = \sigma^2 h\P{\mathbf x}$$

To estimate $\delta_j$, we write

$$u^2 = \sigma^2 \exp\CB{\delta_0 + \delta_1 x_1 + \cdots + \delta_k x_k}v$$

Further, if we assume that $\Exp\SB{v \mid \mathbf x} = 1$, we have

$$\log\P{u^2} = \alpha_0 + \delta_1 x_1 + \cdots + \delta_k x_k + e$$

where $e$ has a zero mean expectation and is independent from $\mathbf x$. This is the one that satisfies all Gauss-Markov assumptions so that we can obtain $\hat \delta_j$. Denote the fitted value in this regression as $\hat g_i$ then we have the weight:

$$\ffrac{1}{h_i} = \ffrac{1}{\hat h_i} = e^{-\hat g_i}$$

**Step**:

1. Regress $y$ on $x_1,x_2,\dots,x_k$ and obtain the residual $\hat u$
2. Obtain $\log\P{\hat u^2}$
3. Regress $\log\P{u^2} = \alpha_0 + \delta_1 x_1 + \cdots + \delta_k x_k + e$, then let $\hat g_i$ denote the fitted value
4. $\hat h = \exp\CB{\hat g} = \exp\CB{\hat \alpha_0 + \hat\delta_1 x_1 + \cdots + \hat\delta_k x_k}$
5. Use weight $1/\hat h$ and then apply WLS on $y =\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$
***
Feasible GLS is consistent and asymptotically more efficient than OLS.
***

### What if the Assumed Heteroskedasticity Function is Wrong?

- WLS is still consistent under $\text{MLR}.1$ through $\text{MLR}.4$
- robust standard errors should be computed
- WLS is consistent under $\text{MLR}.4$ but not necessarily under $\text{MLR}.4'$

### Prediction and Prediction Intervals with Heteroskedasticity

## The Linear Probability Model Revisited

$$\Var{y\mid \mathbf x} = p\P{\mathbf x} \P{1-p\P{\mathbf x}}\Rightarrow \hat h_i = \hat y_i \P{1-\hat y_i}$$

***