```python
import math
import torch
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

import io
import torchtext
from torchtext.utils import download_from_url, extract_archive
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

import pytorch_lightning as pl
from pytorch_lightning.trainer.trainer import Trainer
from pytorch_lightning import seed_everything
seed_everything(42)

```

    42

# Dataset Preparation

This tutorial uses ``torchtext`` to generate Wikitext-2 dataset. The
vocab object is built based on the train dataset and is used to numericalize
tokens into tensors.

With the alphabet as the sequence (total length of 26)
and a batch size of 4, we would divide the alphabet into 4 sequences of
length 6:

\begin{align}\begin{bmatrix}
  \text{A} & \text{B} & \text{C} & \ldots & \text{X} & \text{Y} & \text{Z}
  \end{bmatrix}
  \Rightarrow
  \begin{bmatrix}
  \begin{bmatrix}\text{A} \\ \text{B} \\ \text{C} \\ \text{D} \\ \text{E} \\ \text{F}\end{bmatrix} &
  \begin{bmatrix}\text{G} \\ \text{H} \\ \text{I} \\ \text{J} \\ \text{K} \\ \text{L}\end{bmatrix} &
  \begin{bmatrix}\text{M} \\ \text{N} \\ \text{O} \\ \text{P} \\ \text{Q} \\ \text{R}\end{bmatrix} &
  \begin{bmatrix}\text{S} \\ \text{T} \\ \text{U} \\ \text{V} \\ \text{W} \\ \text{X}\end{bmatrix}
  \end{bmatrix}\end{align}

These columns are treated as independent by the model, which means that
the dependence of ``G`` and ``F`` can not be learned, but allows more
efficient batch processing.

# Inputa and Target

For the language modeling task, the model needs the following words as Target. For example, with a seq_len value of 2, we’d get the following two Variables.

**Note:** It should be noted that the chunks are along dimension 0, consistent with the S dimension in the Transformer model. The batch dimension N is along dimension 1.

```python
class AlarmDataset(Dataset):
    def __init__(self,data,seq_len,batch_size):
        self.length = len(data)//seq_len # how much data i have
        self.data = data
        self.seq_len = seq_len
        self.batch_size = batch_size

    def __getitem__(self, index: int):
        x = self.data[index*self.seq_len:(index*self.seq_len)+seq_len]
        y = self.data[1+index*self.seq_len:1+(index*self.seq_len)+seq_len]
        return x,y

    def __len__(self) -> int:
        return self.length

class MyDataModule(pl.LightningDataModule):

    def __init__(self, data_path:str, batch_size:int, seq_len:int):
        super().__init__()
        self.batch_size = batch_size

        url = data_path
        test_filepath, valid_filepath, train_filepath = extract_archive(download_from_url(url))
        self.tokenizer = get_tokenizer('basic_english')
        self.vocab = build_vocab_from_iterator(map(self.tokenizer,iter(io.open(train_filepath,encoding="utf8"))))

        train_data = self.data_process(iter(io.open(train_filepath, encoding="utf8")))
        val_data = self.data_process(iter(io.open(valid_filepath, encoding="utf8")))
        test_data = self.data_process(iter(io.open(test_filepath, encoding="utf8")))

        # print(train_data[0:100])
        # print([])

        self.train_dataset = AlarmDataset(train_data, seq_len,self.batch_size)
        self.valid_dataset = AlarmDataset(val_data,seq_len,self.batch_size)
        self.test_dataset = AlarmDataset(test_data, seq_len,self.batch_size)

    def data_process(self, raw_text_iter):
        data = [torch.tensor([self.vocab[token] for token in self.tokenizer(item)],dtype=torch.long) for item in raw_text_iter]
        return torch.cat(tuple(filter(lambda t: t.numel() > 0, data)))

    def setup(self, stage: None):
        return None

    def train_dataloader(self) -> DataLoader:
        return DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=False,num_workers=1,drop_last=True, pin_memory=True)

    def val_dataloader(self) -> DataLoader:
        return DataLoader(self.valid_dataset, batch_size=self.batch_size, shuffle=False,num_workers=1,drop_last=True, pin_memory=True)

    def test_dataloader(self):
        return DataLoader(self.test_dataset, batch_size=self.batch_size, shuffle=False,num_workers=1,drop_last=True, pin_memory=True)

```

# Transformer Model

In this tutorial, we train ``nn.TransformerEncoder`` model on a
language modeling task. The language modeling task is to assign a
probability for the likelihood of a given word (or a sequence of words)
to follow a sequence of words. A sequence of tokens are passed to the embedding
layer first, followed by a positional encoding layer to account for the order
of the word (see the next paragraph for more details). The
``nn.TransformerEncoder`` consists of multiple layers of
`nn.TransformerEncoderLayer <https://pytorch.org/docs/master/nn.html?highlight=transformerencoderlayer#torch.nn.TransformerEncoderLayer>`__. Along with the input sequence, a square
attention mask is required because the self-attention layers in
``nn.TransformerEncoder`` are only allowed to attend the earlier positions in
the sequence. For the language modeling task, any tokens on the future
positions should be masked. To have the actual words, the output
of ``nn.TransformerEncoder`` model is sent to the final Linear
layer, which is followed by a log-Softmax function.

# Positional Encoding

``PositionalEncoding`` module injects some information about the
relative or absolute position of the tokens in the sequence. The
positional encodings have the same dimension as the embeddings so that
the two can be summed. Here, we use ``sine`` and ``cosine`` functions of
different frequencies.

```python
class TransformerModel(pl.LightningModule):

    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5, seq_len=None):
        super(TransformerModel, self).__init__()
        self.model_type = 'Transformer'
        self.ntoken = ntoken
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        encoder_layers = torch.nn.TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        self.transformer_encoder = torch.nn.TransformerEncoder(encoder_layers, nlayers)
        self.encoder = torch.nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.decoder = torch.nn.Linear(ninp, ntoken)
        self.src_mask = self.generate_square_subsequent_mask(seq_len)
        self.seq_len = seq_len
        self.init_weights()

    def generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src, src_mask):
        src_mask = src_mask.to(self.device)
        src = self.encoder(src) * math.sqrt(self.ninp)
        src = self.pos_encoder(src)
        src_mask = src_mask.to(self.device)

        output = self.transformer_encoder(src, src_mask)
        output = self.decoder(output)

        return output

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=0.001, weight_decay=0.0000001)
        return optimizer

    def training_step(self,batch,batch_idx):
        x,y = batch
        x = x.T
        y = y.T.reshape(-1)

        # print("Training Shape: ", x.size(),y.size())

        if x.size(0) != self.seq_len:
           self.src_mask =  self.generate_square_subsequent_mask(x.size(0))

        y_hat = self(x,self.src_mask)

        loss = F.cross_entropy(y_hat.view(-1, self.ntoken),y)
        self.log('train_loss', loss,on_step=True, prog_bar=True, logger=True)
        self.log("train_ppl",math.exp(loss.item()),on_step=True, prog_bar=True, logger=True)
        return loss

    def validation_step(self,batch, batch_idx):
        x,y = batch
        x = x.T
        y = y.T.reshape(-1)

        # print("Validation Shape: ", x.size(),y.size())

        if x.size(0) != self.seq_len:
           self.src_mask =  self.generate_square_subsequent_mask(x.size(0))

        y_hat = self(x,self.src_mask)
        # print("> y-hat",y_hat.size())
        loss = F.cross_entropy(y_hat.view(-1, self.ntoken),y)
        self.log('val_loss', loss, on_step=True, prog_bar=True, logger=True)
        self.log("val_ppl",math.exp(loss.item()),on_step=True, prog_bar=True, logger=True)
        return {'val_loss':loss}

    def test_step(self,batch, batch_idx):
        x,y = batch
        x = x.T
        y = y.T.reshape(-1)

        # print("Validation Shape: ", x.size(),y.size())

        if x.size(0) != self.seq_len:
           self.src_mask =  self.generate_square_subsequent_mask(x.size(0))

        y_hat = self(x,self.src_mask)
        # print("> y-hat",y_hat.size())
        loss = F.cross_entropy(y_hat.view(-1, self.ntoken),y)
        self.log('test_loss', loss, on_step=True, prog_bar=True, logger=True)
        self.log("test_ppl",math.exp(loss.item()),on_step=True, prog_bar=True, logger=True)
        return {'test_loss':loss}

    def training_epoch_end(self, outputs):
        avg_loss = torch.stack([d['loss']  for d in outputs]).mean()
        print(f"> Avg Training loss = {avg_loss}")

    def validation_epoch_end(self, outputs):
        # print(outputs)
        avg_loss = torch.stack([d['val_loss'] for d in outputs]).mean()
        print(f"> Average Valid Loss = {avg_loss}")

    def test_epoch_end(self, outputs):
        avg_loss = torch.stack([d['test_loss'] for d in outputs]).mean()
        print(f"> Average Test Loss = {avg_loss}")

class PositionalEncoding(torch.nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = torch.nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
```

# Hyperparameters

```python
url = 'https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip'

bsize = 20
seq_len = 35
dm = MyDataModule(url,bsize,seq_len)

ntokens = len(dm.vocab.stoi) # the size of vocabulary
emsize = 200 # embedding dimension
nhid = 200 # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2 # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 2 # the number of heads in the multiheadattention models
dropout = 0.2 # the dropout value

model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout,seq_len=seq_len)
trainer = Trainer(gpus=1,max_epochs=1,check_val_every_n_epoch=1,deterministic=True, gradient_clip_val=0.5)
trainer.fit(model,dm) # traning and validation
```

    36718lines [00:01, 29608.89lines/s]
    GPU available: True, used: True
    INFO:lightning:GPU available: True, used: True
    TPU available: None, using: 0 TPU cores
    INFO:lightning:TPU available: None, using: 0 TPU cores
    LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
    INFO:lightning:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

      | Name                | Type               | Params
    -----------------------------------------------------------
    0 | pos_encoder         | PositionalEncoding | 0
    1 | transformer_encoder | TransformerEncoder | 484 K
    2 | encoder             | Embedding          | 5.8 M
    3 | decoder             | Linear             | 5.8 M
    -----------------------------------------------------------
    12.0 M    Trainable params
    0         Non-trainable params
    12.0 M    Total params
    INFO:lightning:
      | Name                | Type               | Params
    -----------------------------------------------------------
    0 | pos_encoder         | PositionalEncoding | 0
    1 | transformer_encoder | TransformerEncoder | 484 K
    2 | encoder             | Embedding          | 5.8 M
    3 | decoder             | Linear             | 5.8 M
    -----------------------------------------------------------
    12.0 M    Trainable params
    0         Non-trainable params
    12.0 M    Total params

    Validation sanity check: |          | 0/? [00:00<?, ?it/s]

    > Average Valid Loss = 10.613945960998535

    Training: |          | 0/? [00:00<?, ?it/s]

    Validating: |          | 0/? [00:00<?, ?it/s]

    > Average Valid Loss = 5.521843433380127
    > Avg Training loss = 6.123049259185791

    1

```python
trainer.test(datamodule=dm) # testing
# %%
```

    Testing: |          | 0/? [00:00<?, ?it/s]

    > Average Test Loss = 5.444727897644043
    --------------------------------------------------------------------------------
    DATALOADER:0 TEST RESULTS
    {'test_loss': tensor(5.7169, device='cuda:0'),
     'test_loss_epoch': tensor(5.4447, device='cuda:0'),
     'test_ppl': 303.96751971676264,
     'test_ppl_epoch': tensor(241.8977),
     'val_loss': tensor(5.3272, device='cuda:0'),
     'val_loss_epoch': tensor(5.5218, device='cuda:0'),
     'val_ppl': 205.85485347893808,
     'val_ppl_epoch': tensor(258.1052)}
    --------------------------------------------------------------------------------

    [{'val_loss_epoch': 5.521843910217285,
      'val_ppl_epoch': 258.105224609375,
      'val_loss': 5.327171325683594,
      'val_ppl': 205.85485347893808,
      'test_loss_epoch': 5.444727897644043,
      'test_ppl_epoch': 241.89773559570312,
      'test_loss': 5.716920852661133,
      'test_ppl': 303.96751971676264}]


Unit 10

Case Study 4: Object Recognition with Deep Learning

IST 718 – Big Data Analytics

Daniel E. Acuna

http://acuna.io

# Object recognition problem
- A bit of neuroscience.
- From logistic regression to neural networks.
- The perceptron.
- Multi-layer perceptron.
- Training neural networks.
- Demo with multi-layer perceptron.

# The object recognition problem
- Detect an object present in an image.

Caltech101 dataset (year 2003)
101 categories of objects, each category has 40 to 800 images

# A high dimensional problem
- Each image is 300 x 200 x 3 pixels (width x height x channels.)
  - 180,000 "raw" features.

- Relationship between raw features and classes is highly non-linear.
  - Spatial features that could appear anywhere on the image.
  - Local features (e.g., shapes) vs global features (e.g., contrast.)

# A high dimensional problem (2)

Credit: https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/info/signals/pixels/

# A problem with logistic regression
- We are trying to learn:

$$p(y \mid X) = \frac{1}{1 + \exp(-(\theta_0 + \sum_{j>0}{x_j\theta_j}))}$$

- So only a linear relationship between pixels $x$ and class $y$.

# Classic data science
- Features fed into the algorithms are meaningful and provided by experts.

- Machine learning algorithms only provide 1 or 2 step transformations.

- Complexity of problems is relatively small.

# A bit of neuroscience
- Deep learning takes loose inspiration from how the brain works.

- At an architectural level, the brain works by combining specialized neurons in hierarchies.

- E.g., the visual system is organized into several areas (around 8.)

# A bit of neuroscience (2)
- The brain has relatively "simple" processing units: neurons.

Credit: http://www.intechopen.com/source/html/39067/media/image1.png

$$p(y \mid X) = \frac{1}{1 + \exp(-(\theta_0 + \sum_{j>0}{x_j\theta_j}))}$$
**A model for the probability of an action potential!**

# Of course reality is a lot more complicated

# Of course reality is a lot more complicated

# The general idea of artificial neural networks
- Simple processing units with linear or non-linear functions $f$.

# The general idea of artificial neural networks (2)
- Multiple units belong to layers and those layers are interconnected to other layers.

# A highly non-linear function
- The output now depends on nested functions:

$$y = \underbrace{f_3\left(\;\underbrace{f_2\left(\;\underbrace{f_1(x)}_{\text{first layer}} + b_0\right)}_{\text{second layer}} + b_1\right) + b_2}_{\text{third layer (nested non-linearities)}}$$

# Advantages and disadvantages of ANN
- Pros:
  - Can fit complex non-linear relationships.
  - Easy to try different types of layers.
  - Inspired by a very effective machine (the brain!)

- Cons:
  - Many parameters.
  - Easy to overfit.
  - Needs lots of training data.
  - Takes a long time to fit.
  - The use of brain research is really minimal.

# How to train a neural network?
- We define a loss function and try to find best value:

$$\arg\min_{\Theta} {l(\;f_\Theta(X),y)}$$

- For example:

$$l(\;f_\Theta(X),y) = \sum(\;f_\Theta(X_i) - y_i)^2$$

where $f_\Theta(X)$ is the function represented by the entire network.

# Optimizing the loss function
- There is no close solution to the loss function:

$$\arg\min_{\Theta} {l(\;f_\Theta(X),y)}$$

- In all but the trivial cases, the loss function is *non-convex* (hard to optimize.)

- A very effective approach to perform stochastic gradient descent.

# Gradient descent
- Gradient descent is a simple approach to iteratively minimize a loss function.

- Intuitively, we want to follow the *negative* of the gradient to find the minimum of a loss function:

$$\Theta_{t+1} = \Theta_{t} - \eta \nabla l(\;f_\Theta(X),y)$$

# Gradient descent: example

# Gradient descent: example (2)

# Gradient descent: example (3)

# Gradient descent: example (4)

# Gradient descent: example (5)

# Gradient descent: problems
- It finds the global optimum when function is convex (e.g., a parabola) and the learning rate is appropriate.
- But:

# Stochastic gradient descent
- If we look at a fraction of the training data, we only observe a (noisy) sample of the loss function.

# Stochastic gradient descent (2)
- If we look at a fraction of the training data, we only observe a (noisy) sample of the loss function.

# Stochastic gradient descent (3)
- If we look at a fraction of the training data, we only observe a (noisy) sample of the loss function.

- Stochastic gradient descent is good for big data (we only need pieces at a time.)

- Stochastic gradient descent can escape local minima.

- As long as we can compute gradients, we can use function we want!

# Worked-out example
- A simple neural network with one neuron and one layer.

- More assumptions sigmoid activation $f$ and $a = y$.

# Worked-out example (2)
- A simple neural network with one neuron and one layer.

- More assumptions sigmoid activation $f$ and $a = y$.

**Similar to Logistic Regression!**

# Worked-out example (3)
- Using quadratic loss function, let’s work out the stochastic gradient descent update rule:

$$\Theta_{t+1} = \Theta_{t} - \eta \nabla l(\;f_\Theta(X),y)$$

$$l(\;f_\Theta(X),y) = (\;f(z) - y)^2 = \left(\sigma\left(\sum{x_j\theta_j} - y\right)\right)^2$$

$$\nabla{l(\;f_\Theta(X),y)} = \left[\frac{dl}{d\theta_0}, \frac{dl}{d\theta_1}, \ldots, \frac{dl}{d\theta_m}\right]$$

# Worked-out example (4)
- Let’s pick one the gradients:

$$\begin{align}
\nabla{l(\;f_\Theta(X),y)} &= \left[\frac{dl}{d\theta_0}, \frac{dl}{d\theta_1}, \ldots, \frac{dl}{d\theta_m}\right] \\
\frac{dl}{d\theta_j} &= \frac{d\left(\sigma\left(\sum{x_j\theta_j} - y\right)\right)^2}{d\theta_j}
\end{align}$$

- We are trying to compute the gradient of a nested function, what to do?

# Worked-out example (5)

<left></left>

- We can apply basic calculus.

- By representing each nested function on its own:

$$\frac{d\left(\sigma\left(\sum{x_j\theta_j} - y\right)\right)^2}{d\theta_j} = \frac{d}{d\theta_j} l(a(z(\theta_0, \cdots, \theta_m, x), y)$$

- Where $a$ is the activation, and $z$ is the summation.

- Now we can simply apply the chain rule!

# Worked-out example (6)
- We can apply basic calculus.
- By representing each nested function on its own:
$$\frac{d}{d\theta_j} l(a(z(\theta_0, \cdots, \theta_m, x), y) = \frac{dl}{da}\frac{da}{dz}\frac{dz}{d\theta_j}$$

$$
\frac{dl}{da} = 2(a - y) \qquad
\frac{da}{dz} = a(1-a) \qquad
\frac{dz}{d\theta_j} = x_j
$$

$$\begin{align}
\Theta_j^{\text{new}} &\leftarrow \Theta_j^{\text{old}} - \eta \;{2(a-y)} \;{a(1-a)} \;{x_j} \\
&\leftarrow \Theta_j^{\text{old}} - \eta \;{\left(\sigma\left(\sum{x_j\theta_j}\right) - y\right)} \;{\sigma\left(\sum{x_j\theta_j}\right)} \;{\left(1 - \sigma\left(\sum{x_j\theta_j}\right)\right)} {x_j}
\end{align}$$

# Backpropagation
- The previous example is the basis of stochastic gradient descent for several layers.
- **Backpropagation** is a general algorithm that exploits the structure of the chain rule when applied to layers.
- To understand backpropagation, we need to understand middle layers:

# Backpropagation (2)
- $\theta_{ij}^{L-2}$: connection between neuron $j$ in layer $L-2$ to neuron $i$ in layer $L-1$.

- Representing all connections from $L-2$ to $L-2$ as one matrix $\theta^{L-2}$, we can predict all activation of layer $L-1$ as:

$$a^{L-1} = f(\theta^{L-2} a^{L-2})$$

# Backpropagation (3)
- Then the gradient of a connection $\theta_{ij}^{L-2}$ with respect to the loss function will be:

$$\frac{dl}{d\theta_{ij}^{L-2}} = \frac{dz_i^{L-1}}{d\theta_{ij}{L-2}}\frac{da_i^{L-1}}{dz_i^{L-1}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}}$$

# Backpropagation (4)
- Then the gradient of a connection $\theta_{jk}^{L-3}$ with respect to the loss function will be:

$$\frac{dl}{d\theta_{jk}^{L-3}} = \frac{dz_j^{L-2}}{d\theta_{jk}^{L-3}}\frac{da_j^{L-2}}{dz_j^{L-2}} \sum{\frac{dz_i^{L-1}}{da_j^{L-2}} \frac{da_i^{L-1}}{dz_i^{L-1}}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}} $$

# Backpropagation (5)
- The same structure gets repeated during the chain rule:

$$\frac{dl}{d\theta_{ij}^{L-2}} = \frac{dz_i^{L-1}}{d\theta_{ij}{L-2}}\frac{da_i^{L-1}}{dz_i^{L-1}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}}$$

$$\frac{dl}{d\theta_{jk}^{L-3}} = \frac{dz_j^{L-2}}{d\theta_{jk}^{L-3}}\frac{da_j^{L-2}}{dz_j^{L-2}} \sum{\frac{dz_i^{L-1}}{da_j^{L-2}} \frac{da_i^{L-1}}{dz_i^{L-1}}} \sum{\frac{dz^L}{da_i^{L-1}} \frac{da^L}{dz^L} \frac{dl}{da^L}}$$

- Therefore, we can *backpropagate* the gradient of the loss function.
- Notice also, that we reuse $a$ and $z$ multiple times, therefore we could precompute them.
- Consequently, we could forward propagate $a$ and $z$ and *backpropagate* the errors.

# Backpropagation algorithm

Forward propagation
1. From layer $1$ thought $L$
  1.1 Compute $a$ and $z$

Backpropagation
1. Define $\delta_i^L = a^L-y$ (gradient of loss function)
2. From layer $L–1$ to $1$
  2.1 Define $\delta_j^{L-1} = a_j^{L-1} (1 - a_j^{L-1}) \sum{\theta_{ij}^{L-1} \delta_i^L}$

The gradients will be computed as follows:
$$\frac{dl}{d\theta_{ij}^k} = a_j^k a_i^{k+1} (1 - a_i^{k+1}) \sum{a_i^{k+1} \delta_i^{k+1}} $$

# Demo

# Problems with classic methods
- The more layers, the more complex representations can be learned.

- However, more layers also implies longer chains.

- This produces numerical errors and small changes in learning.

- Groundbreaking paper in 2006 proposes to train deep architectures in a novel way.

# Deep learning
- Also, new developments in:
  - Training techniques (2006)
  - GPU computing (2008)
  - Big data.
  - New types of architectures (new neurons and layers.)

- This opened the door to train increasingly more complex models.

- Next class we will take a look at these advances

# Other architectures
- Multi-layer perceptron looses spatial location (specially in images.)

- Some features are independent of location.

# Convolutional neural networks
- Create translational-invariant spatial filters (share weights.)

http://cs231n.github.io/convolutional-networks/

# Convolutional neural networks (2)
- It can learn filters similar to what is observed in nature!

# Convolutional neural networks (3)
- Digits classification: classic MNIST problem.

http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html

# Dealing with temporal data
- Sometimes we want to capture temporal-invariance:
  - E.g., in chess, the state of the next move only depends on the current move.
  - Makes learning more efficient.
  - Allows to interpret results.

# Recurrent neural networks

# Recurrent neural networks (2)
- Naïve recurrent neural network training leads to numerical problems:
$$y = h(A(A(A(\ldots A(y_{t-T}))$$

- Why? How would the chain rule apply here?

- Other more advanced neural network allows to learn long-term dependencies easier.

# Recurrent neural networks (3)
- Handwriting generation:
  http://www.cs.toronto.edu/~graves/handwriting.html

- Generating Wikipedia articles:

# More complex temporal problems

Can you think of examples for each of this cases?

# More complex temporal problems (2)

# More complex temporal problems (3)

# More complex temporal problems (4)

# Reinforcement learning
- Given semi-supervised feedback, learn how to perform actions so as to maximize total reward.

- Classical framework to study behavior in humans and animals.

# Reinforcement learning (2)

# Reinforcement learning (3)

http://cs.stanford.edu/people/karpathy/convnetjs/demo/rldemo.html

# Inferring intentions

# WaveNet

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

# WaveNet

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

# WaveNet

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

# Adversarial Neural Networks

# Adversarial Neural Networks (2)

# Adversarial Neural Networks (3)

# Adversarial Neural Networks (4)

# Take home messages
- Neural networks try to mimic how the brain works: hierarchical layers of non-linear functions.

- Neural networks take time to train and overfit badly.

- Neural networks need big data.

- Neural networks are hard or impossible to interpret.

- Backpropagation allows efficient computation of gradients.