# Deep Reinforcement Learning

In this session, we will use Open AI Gym to implement Q-learning, a classic algorthim in RL, and Deep Q-Networks (DQN), its deep learning counterpart. Hopefully this will give us some intuition for what we gain by moving from tabular methods to using neural networks for function approximation.

## Setup

First, let's install the necessary dependencies. This does not need to be run again if you restart the runtime, but it does if you factory reset the runtime.

```
!sudo apt-get install -y xvfb ffmpeg
!apt-get install x11-utils
!pip install 'gym==0.17.1'
!pip install 'pyglet==1.4.0'
!pip install pyvirtualdisplay
!pip install --upgrade tensorflow-probability
!pip install imageio-ffmpeg
```

    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    ffmpeg is already the newest version (7:3.4.6-0ubuntu0.18.04.1).
    The following NEW packages will be installed:
      xvfb
    0 upgraded, 1 newly installed, 0 to remove and 25 not upgraded.
    Need to get 784 kB of archives.
    After this operation, 2,266 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.4 [784 kB]
    Fetched 784 kB in 0s (6,337 kB/s)
    debconf: unable to initialize frontend: Dialog
    debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
    debconf: falling back to frontend: Readline
    debconf: unable to initialize frontend: Readline
    debconf: (This frontend requires a controlling tty.)
    debconf: falling back to frontend: Teletype
    dpkg-preconfigure: unable to re-open stdin:
    Selecting previously unselected package xvfb.
    (Reading database ... 144568 files and directories currently installed.)
    Preparing to unpack .../xvfb_2%3a1.19.6-1ubuntu4.4_amd64.deb ...
    Unpacking xvfb (2:1.19.6-1ubuntu4.4) ...
    Setting up xvfb (2:1.19.6-1ubuntu4.4) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following additional packages will be installed:
      libxxf86dga1
    Suggested packages:
      mesa-utils
    The following NEW packages will be installed:
      libxxf86dga1 x11-utils
    0 upgraded, 2 newly installed, 0 to remove and 25 not upgraded.
    Need to get 209 kB of archives.
    After this operation, 711 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]
    Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]
    Fetched 209 kB in 0s (2,013 kB/s)
    Selecting previously unselected package libxxf86dga1:amd64.
    (Reading database ... 144575 files and directories currently installed.)
    Preparing to unpack .../libxxf86dga1_2%3a1.1.4-1_amd64.deb ...
    Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...
    Selecting previously unselected package x11-utils.
    Preparing to unpack .../x11-utils_7.7+3build1_amd64.deb ...
    Unpacking x11-utils (7.7+3build1) ...
    Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...
    Setting up x11-utils (7.7+3build1) ...
    Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    Processing triggers for libc-bin (2.27-3ubuntu1) ...
    /sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

    Requirement already satisfied: gym==0.17.1 in /usr/local/lib/python3.6/dist-packages (0.17.1)
    Requirement already satisfied: cloudpickle<1.4.0,>=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.3.0)
    Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.5.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.12.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.4.1)
    Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym==0.17.1) (1.18.3)
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym==0.17.1) (0.16.0)
    Collecting pyglet==1.4.0
      Downloading https://files.pythonhosted.org/packages/8a/2e/74069cfb668afcb29f0c7777c863d0b1d831accf61558f46cebf34bcfe07/pyglet-1.4.0-py2.py3-none-any.whl (1.0MB)
         |████████████████████████████████| 1.0MB 6.3MB/s
    Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet==1.4.0) (0.16.0)
    Installing collected packages: pyglet
      Found existing installation: pyglet 1.5.0
        Uninstalling pyglet-1.5.0:
          Successfully uninstalled pyglet-1.5.0
    Successfully installed pyglet-1.4.0
    Collecting pyvirtualdisplay
      Downloading https://files.pythonhosted.org/packages/69/ec/8221a07850d69fa3c57c02e526edd23d18c7c05d58ed103e3b19172757c1/PyVirtualDisplay-0.2.5-py2.py3-none-any.whl
    Collecting EasyProcess
      Downloading https://files.pythonhosted.org/packages/32/8f/88d636f1da22a3c573259e44cfefb46a117d3f9432e2c98b1ab4a21372ad/EasyProcess-0.2.10-py2.py3-none-any.whl
    Installing collected packages: EasyProcess, pyvirtualdisplay
    Successfully installed EasyProcess-0.2.10 pyvirtualdisplay-0.2.5
    Requirement already up-to-date: tensorflow-probability in /usr/local/lib/python3.6/dist-packages (0.10.0rc0)
    Requirement already satisfied, skipping upgrade: gast>=0.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (0.3.3)
    Requirement already satisfied, skipping upgrade: cloudpickle>=1.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (1.3.0)
    Requirement already satisfied, skipping upgrade: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (1.12.0)
    Requirement already satisfied, skipping upgrade: decorator in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (4.4.2)
    Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow-probability) (1.18.3)
    Collecting imageio-ffmpeg
      Downloading https://files.pythonhosted.org/packages/0a/45/2472071095310b3e92010c051cbd2e3c655247ad9090851a86b8bfdcfbc5/imageio_ffmpeg-0.4.1-py3-none-manylinux2010_x86_64.whl (22.2MB)
         |████████████████████████████████| 22.2MB 72.5MB/s
    Installing collected packages: imageio-ffmpeg
    Successfully installed imageio-ffmpeg-0.4.1

Now let's import the packages we will be using.

```
from __future__ import absolute_import, division, print_function

import base64
import imageio
import IPython
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import PIL.Image
import pyvirtualdisplay
import random
import tensorflow as tf
import gym

from gym.spaces import Discrete
from gym.spaces import Box
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
from collections import defaultdict
from collections import deque
from pyvirtualdisplay import Display
```

```
print(tf.version.VERSION)
print(gym.version.VERSION)
```

    2.2.0-rc3
    0.17.1

## Introduction to Open AI Gym

Gym provides you with a set of [environments](https://gym.openai.com/envs/#classic_control). If we think of the classic RL framework schematic, Gym takes care of the environment, and you take care of the agent.

From this diagram, we can expect that we will interact with a Gym environment by giving it an action as input, and receiving a next state and reward as output.

It is then our job to implement some algorithm to learn the policy $\pi(a|s)$ for our agent to use to act within the environment.

### Setting up visualization

First, let's get plotting for Gym working in Colab. This will help give us an intuitive feel for how to work with the Gym environments.

```
def embed_mp4(filename):
    """Embeds an mp4 file in the notebook."""
    video = open(filename,'rb').read()
    b64 = base64.b64encode(video)
    tag = '''
    '''.format(b64.decode())

    return IPython.display.HTML(tag)
```

Before we get to implementing algorithms for our agents to learn a good policy, let's visualize an agent acting according to a random policy. At this point the visualizatation is just to give us an idea of what what an environment looks like, and later on we'll come back to see how we generate this video.

```
def create_random_policy_video(env, filename, num_episodes=5, fps=30):
    """Generates a visualization of an agent acting according to a random
    policy in the given environment."""
    display = Display(visible=0, size=(400, 300))
    display.start()
    filename = filename + ".mp4"
    with imageio.get_writer(filename, fps=fps) as video:
        for _ in range(num_episodes):
            done = False
            observation = env.reset()
            video.append_data(env.render(mode='rgb_array'))
            while not done:
                action = env.action_space.sample()
                observation, reward, done, info = env.step(action)
                video.append_data(env.render(mode='rgb_array'))
    display.stop()
    return embed_mp4(filename)
```

```
env = gym.make("MsPacman-v0")
create_random_policy_video(env, "video", num_episodes=1)
```

    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (210, 160) to (224, 160) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

This is pretty cool! In this example Gym gives us a great visualization of our agent playing PacMan (though not very well).

### Gym basics

#### Environments

As we would expect, an environment is defined by its state and action spaces.

First, of all there are two types of Gym [spaces](https://gym.openai.com/docs/#spaces):
- **`observation_space`**: defines the state space of the environment
- **`action_space`**: defines the action space of the environment

Spaces can be:
- `gym.spaces.Discrete`: fixed range of n values
- `gym.spaces.Box`: n-dimensional box

You can inspect the valid range for a `Box` by calling `Box.low()` and `Box.high()`.

```
env = gym.make("MsPacman-v0")
print(f'Action space: {env.action_space}')
print(f'Observation space: {env.observation_space}')
```

    Action space: Discrete(9)
    Observation space: Box(210, 160, 3)

We can see here that for `MsPacman-v0`, the `action_space` consits 9 possible actions, and the `observation_space` is a 210 x 160 x 3 box (rgb-image). We can also extract these dimensions using `Discrete.n` and `Box.shape`.

```
print(env.action_space.n)
print(env.observation_space.shape)
```

    9
    (210, 160, 3)

If we're curious, we can find the action meanings by calling,

```
print(env.unwrapped.get_action_meanings())
```

    ['NOOP', 'UP', 'RIGHT', 'LEFT', 'DOWN', 'UPRIGHT', 'UPLEFT', 'DOWNRIGHT', 'DOWNLEFT']

I would've guessed that the `action_space` would be just up, down, left, right, but apparently this implementation includes combination actions as well. Theoretically you don't "need" to know these details about the environment you're using because your algorithm should learn a good policy given whatever the available action space is, but I think it's still nice to get a sense of.

#### Key functions for interacting with the environment

We will mainly use three functions for interacting with the environment.

**`observation = env.reset()`**

- This function returns the starting state of an environment. We will call this function any time we want to start a new episode.

**`observation, reward, done, info = env.step(action)`**

- This function is how your agent takes actions in the environment; it defines the transition and reward function. It takes in an `action` as an argument, and returns the next `observation`(next_state), the `reward` (**float**), if the episode is `Done` (**bool**), and `info`, which we won't be using here but can contain helpful information for debugging.

**`action = env.action_space.sample()`**
- This is a helpful function for sampling a random action from the `action_space`. We will be using the $\epsilon$-greedy exploration strategy, so we will use this function when we want to select a random action.

If we look back at the code for `create_random_policy_video()`, we can see how we used these three functions to get the data for the video. Stripping away all code for plotting, the main loop is:

```
num_episodes = 10
env = gym.make("MsPacman-v0")

for _ in range(num_episodes):
    observation = env.reset()
    done = False
    while not done:
        action = env.action_space.sample()
        observation, reward, done, info = env.step(action)
```

In this notebook, we will generally replace the term `observation` with `state` becuase this is the wording we're more familiar with.

## Implementing RL algorithms

Now that we have all of the setup done, let's get to the fun part of actually implementing the algorithms!

### CartPole environment

For our implementations we are going to use the `CartPole-v1` environment. This is a simple environment where both of our algorithms (Q-learning and DQN) will be able to learn a good policy within a reasonably short amount of time.

```
env = gym.make("CartPole-v1")
```

The goal in this environment is to move the cart left or right in order to balance the pole so that it remains upright.

```
create_random_policy_video(env, "video", num_episodes= 10)
```

    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

```
print(f'Action space: {env.action_space}')
print(f'State space: {env.observation_space}')
```

    Action space: Discrete(2)
    State space: Box(4,)

This is a dramatically simpler environment than the MsPacman environment. The `observation_space` is a 4 dimensional array, and there are 2 possible actions. The CartPole [documentation](https://github.com/openai/gym/wiki/CartPole-v0) tells us the meanings of the observation and action spaces.

    Observation:
        Type: Box(4)
        Num	Observation               Min             Max
        0	Cart Position             -4.8            4.8
        1	Cart Velocity             -Inf            Inf
        2	Pole Angle                -24 deg         24 deg
        3	Pole Velocity At Tip      -Inf            Inf

    Actions:
        Type: Discrete(2)
        Num	Action
        0	Push cart to the left
        1	Push cart to the right

An episode terminates when the pole falls more than 12 degrees from vertical, the cart position moves off-screen, or the number of steps within the episode exceeds 500.

### Required functions for learning a policy

```
class Agent:

    def create_model(self):
        """This model will be used by the act() method to select actions,
        and will be updated during training """
        pass

    def act(self, state, test=False):
        """This function implements your policy and choose actions based on the
        current model. If test=True, actions chosen without exploration."""
        return action

    def update_model(self):
        """This function specifies how to update the model based on experience
        in the environment"""
        pass

    def train(self):
        """The main loop for training the model by selecting actions,
        interacting with the environment, and updating the model"""
        pass
```

Once we have a trained model, we can evaluate it's performance using a similar loop to the one above used to visualize the random policy. The only difference is that we replace the `env.action_space.sample()` function call with `agent.act()`.

    agent = Agent()
    agent = Agent.train()

    # run trained agent
    for _ in range(num_episodes):
        state = agent.env.reset()
        done = False
        while not done:
            action = agent.env.act(state, test=True)
            next_state, reward, done, info = env.step(action)

### Evaluation functions

Now we can use this loop to generate a video  visualizing the learned policy.

```
def create_learned_policy_video(agent, filename, num_episodes=5, fps=30):
    """Generates a video of the given agent acting accoring to its learned
    policy for the specified number of episodes."""
    display = Display(visible=0, size=(400, 300))
    display.start()
    filename = filename + ".mp4"
    with imageio.get_writer(filename, fps=fps) as video:
        for _ in range(num_episodes):
            done = False
            state = agent.env.reset()
            video.append_data(agent.env.render(mode='rgb_array'))
            while not done:
                action = agent.act(state, test=True)
                state, reward, done, info = agent.env.step(action)
                video.append_data(agent.env.render(mode='rgb_array'))
    display.stop()
    return embed_mp4(filename)
```

We will also want to evaluate the performance of the learned model. For evaluation trials we will not use $\epsilon$-greedy exploration, but instead always choose the best action according to our learned policy.

```
def evaluate_policy(agent, num_episodes=10):
    """Runs the agent through the specified number of episodes and prints the
    average return. """
    reward_history = []
    for _ in range(num_episodes):
        state = agent.env.reset()
        total_reward = 0
        done = False
        while not done:
            action = agent.act(state, test=True)
            next_state, reward, done, _ = agent.env.step(action)
            total_reward += reward
            state = next_state
        reward_history.append(total_reward)

    print("Exploit reward average: {}".format(np.mean(reward_history).round(2)))
```

## Q-Learning

Tabular Q-learning stores and updates a Q-value estimate for each state-action pair, $Q(s,a)$. Each of these Q-values is stored in a look-up table.

**Discretize environment observations**

Since the cartpole environment has an `observation_space` with continuous values, the number of Q-values we need to store and update would quickly explode, and also not be very useful. To avoid this we are going to use a wrapper for the environment that transforms the `obervation_space` from a continuous-valued `Box` to a discrete-valued `Discrete`. I got this wrapper from Lillian Weng's Q-learning [implementation](https://github.com/lilianweng/deep-reinforcement-learning-gym/blob/master/playground/utils/wrappers.py) (understanding the details of how she implements this isn't important for what we're focusing on).

```
class DiscretizedObservationWrapper(gym.ObservationWrapper):
    """This wrapper converts a Box observation into a single integer.
    """
    def __init__(self, env, n_bins=10, low=None, high=None):
        super().__init__(env)
        assert isinstance(env.observation_space, Box)

        low = self.observation_space.low if low is None else low
        high = self.observation_space.high if high is None else high

        self.n_bins = n_bins
        self.val_bins = [np.linspace(l, h, n_bins + 1) for l, h in
                         zip(low.flatten(), high.flatten())]
        self.observation_space = Discrete(n_bins ** low.flatten().shape[0])

    def _convert_to_one_number(self, digits):
        return sum([d * ((self.n_bins + 1) ** i) for i, d in enumerate(digits)])

    def observation(self, observation):
        digits = [np.digitize([x], bins)[0]
                  for x, bins in zip(observation.flatten(), self.val_bins)]
        return self._convert_to_one_number(digits)

env = gym.make('CartPole-v1')
env_discrete = DiscretizedObservationWrapper(env)

print(env_discrete.action_space)
print(env_discrete.observation_space)
```

    Discrete(2)
    Discrete(10000)

### Algorithm

The next step is to implement the q-learning algorithm. The method names give you the skeleton of the implementation but the content is left for you to fill in. We've inserted detailed comments to guide your implementation. We've also left some code in that is not essential to the algorithm (e.g decaying the epsilon parameter each step, keeping track of reward history).

You need to fill in the content for three methods:

- `create_model()` - filled in already
- `act()`
- `update_model()`
- `train()`

#### create_model()

We've left the code for `create_model()` filled in because it is only creating a dictionary for storing the Q values. We are using `defaultdict (float)` rather than `dict` for a more efficient implementation. This automatically initializes any key entry to 0.0, rather than returning `KeyError`.

```
# define environement
env = gym.make("CartPole-v1")
env_discrete = DiscretizedObservationWrapper(
    env,
    n_bins=8,
    low=np.array([-2.4, -2.0, -0.42, -3.5]),
    high=np.array([2.4, 2.0, 0.42, 3.5])
)

# get example state-action pair
state = env_discrete.reset()
action = env_discrete.action_space.sample()

# define defaultdict and query the state-action pair
example = defaultdict(float)
example[state, action]  # *no KeyError*
```

    0.0

#### act()

For our implementation, we will be using the $\epsilon$-greedy exploration policy.

\begin{equation}
  a =
    \begin{cases}
      \text{random} & \text{with probability $\epsilon$}\\
      \arg\max_a Q(s,a)   & \text{otherwise}\\
    \end{cases}
\end{equation}

#### update_model()

This function should update the Q-value estimate using the Q-learning update rule based on the $(s,a,r,s'\text{done})$.

$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[r_t + \gamma \max_{a'} Q(s',a') - Q(s,a) \right]$$

If the state is terminal (`done=True`) the update will be
$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[r_t  - Q(s,a) \right]$$

#### train()

This function will run the main training loop. Here is the pseudocode for the Q-learning algorithm.

    create model (initialize q_values)
    for n_episodes
        initialize state
        while not done
            select action according to policy
            execute action; observe reward and next_state
            update model

This function will be used to train the agent as follows:

    agent = Agent(env)
    agent = Agent.train()

Remember, these are the environment api calls you will need to use in your implementation.
- `observation = env.reset()`
- `observation, reward, done, info = env.step(action)`
- `action = env.action_space.sample()`

#### Implementation

```
class QLearning:

    def __init__(self, env, gamma=0.9, alpha=0.5, epsilon=0.99,
                 epsilon_decay=0.9999, epsilon_min=0.1):
        self.env = env
        self.gamma = gamma
        self.alpha = alpha
        self.epsilon = epsilon
        self.epsilon_decay = epsilon_decay
        self.epsilon_min = epsilon_min
        self.actions = range(self.env.action_space.n)

    def create_model(self):
        """"For Q-learning the model is simply a dictionary for storing the
        tabular Q values."""
        self.Q = defaultdict(float)

    def act(self, state, test=False):
        """Choose action based on your current model using epsilon-greedy
        exploration"""

        # update epsilon
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        # set epsilon to 0 if testing the learned policy
        epsilon = 0 if test else self.epsilon

        # Select a random action with probability epsilon, otherwise select
        # the action with the highest Q-value. In cases where multiple actions
        # have the same highest Q-value, return a random choice between these
        # actions.

        # --------------------------------------------
        # Your code here
        # --------------------------------------------

        return action

    def update_model(self, state, action, reward, next_state, done):
        """Update Q(s,a) using the Q-learning update rule."""

        # Appy the learning rule to update self.Q[state, action].

        # --------------------------------------------
        # Your code here
        # --------------------------------------------

    def train(self, num_episodes=20):
        """This is the main training loop for interacting with the environment
        and updating your model. We've left in code for storing training history."""
        # keep track of reward history
        self.reward_history = []
        # initialize Q-values
        self.create_model()

        for episode in range(num_episodes):
            total_reward = 0.0

            # Implement the training loop for Q-learning (pseudocode above).

            # --------------------------------------------
            # Your code here
            # --------------------------------------------

            # Save total reward from epsisode and print training progress.
            self.reward_history.append(total_reward)
            if episode % 500 == 0:
                print("episode {}: {} average reward".format(
                    episode,
                    np.mean(self.reward_history[max(0,episode-500):episode+1]).round(2)))

```

### Training an agent

Once you have your algorithm implemented, let's train an agent!

```
env = gym.make('CartPole-v1')
env_discrete = DiscretizedObservationWrapper(
    env,
    n_bins=8,
    low=np.array([-2.4, -2.0, -0.42, -3.5]),
    high=np.array([2.4, 2.0, 0.42, 3.5])
)

seed = 0
env.seed(seed)
env.action_space.seed(seed)
np.random.seed(seed)
random.seed(seed)

qlearning_agent = QLearning(env_discrete)
qlearning_agent.train(num_episodes=5000)
```

    episode 0: 52.0 average reward
    episode 500: 42.61 average reward
    episode 1000: 79.64 average reward
    episode 1500: 94.46 average reward
    episode 2000: 87.87 average reward
    episode 2500: 96.78 average reward
    episode 3000: 101.91 average reward
    episode 3500: 118.41 average reward
    episode 4000: 117.37 average reward
    episode 4500: 113.55 average reward

```
# visualize total reward per episode across training
plt.figure(figsize=(7,4))
plt.plot(qlearning_agent.reward_history, alpha=.3, color='teal', label='raw')
plt.plot(np.convolve(qlearning_agent.reward_history, np.ones((50,))/50, mode='valid'), color='purple', label='smoothed')
plt.xlabel('episode #', fontsize=15)
plt.ylabel('total reward per episode', fontsize=15)
plt.legend()
plt.show()
```

###Evaluating the agent

First, let's see what the average reward is across 100 trials when the agent is exploiting its learned policy (not using $\epsilon$-greedy exploration).

Now, let's visualize the agent acting according to its learned policy.

```
evaluate_policy(qlearning_agent, num_episodes=100)
```

    Exploit reward average: 121.67

```
create_learned_policy_video(qlearning_agent, "video", num_episodes=1)
```

    xdpyinfo was not found, X start can not be checked! Please install xdpyinfo!
    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

Woo, it learned something! This is definitely an improvement from random, but it's certainly not optimal (a reward of 500 is optimal). This agent could get better with more training but it would probably take a long time for it to reach optimal performance.

### What is the model learning?

We can tell that the model has learned something, but it would be nice to get some idea of what it's learned. In order to get a sense of this, we can visualize the learned Q-values across a set of states. For this example, we are going to plot Q-values as a function of pole velocity, while the cart position, cart velocity, and pole angle are all 0 (the pole is in the center, not moving, and upright).

Intuitively, the agent should have learned to push the cart right if the pole velocity is to the right (>0), and to push the cart left if the pole velocity is to the left (<0).

```
n_obs = 40
obs_array = np.zeros((n_obs, env.observation_space.shape[0]))
obs_array[:,0] = 0
obs_array[:,1] = 0
obs_array[:,2] = 0
obs_array[:,3] = np.linspace(-5, 5, n_obs)
# run model
q_values = np.zeros((n_obs, env.action_space.n))
for i, obs in enumerate(obs_array):
    obs_disctete = env_discrete.observation(obs)
    q_values[i] = [qlearning_agent.Q[obs_disctete, action] for action in qlearning_agent.actions]
# visualize results
plt.figure(figsize=(8,5))
plt.plot(obs_array[:,3], q_values[:,0], color='purple', label='push cart left', linewidth=2)
plt.plot(obs_array[:,3], q_values[:,1], color='green', label='push cart right', linewidth=2)
plt.vlines(0, q_values.min(), q_values.max(), linestyle='--', color='dimgray')
plt.xlabel('Pole Velocity', fontsize=15)
plt.ylabel('Q value', fontsize=15)
plt.title('Cart Position=0, Cart Velocity=0, Pole Angle=0', fontsize=15)
plt.legend(fontsize=13)
plt.show()
```

It does what we expect! The Q-values for a=right are larger than the Q-values for a=left when the pole velocity is greater than 0, and vice versa for when the pole velocity is less than 0.

## Deep Q-Networks (DQN)

Now that we've implemented Q-learning, let's move on to implementing DQNs!

### Algorithm

Similar to Q-Learning, the method names for the DQN class give you the skeleton of the implementation, while the content is left for you to fill in. For DQNs you will need to write a few more functions than you needed for Q-Learning.

You need to fill in the content for six functions:

- `create_model` - filled in already
- `act()`
- `remember()`
- `update_model()`
- `update_target()` - fillded in already
- `train()`

#### create_model()

For this implementation, we're going to use a two-layer densely connected network. This network will take in a state as input and output a Q-value estimate for each action within this state. Consequently, the input dim is the same as the `observation_space` shape (4 for the CartPole environment), and the output dim is the same as the `action_space` shape (2 for the CartPole environment).

We've left the code for `create_model()` filled in because it is largely determined by learning TensorFlow syntax, which isn't our focus here. We will use ReLu activation function and the Adam optimizer.

We will use mean squared error loss, as specified by the DQN loss function.

    def create_model(self):
        model = Sequential()
        model.add(Dense(24, input_dim=self.state_shape[0], activation="relu"))
        model.add(Dense(16, activation="relu"))
        model.add(Dense(self.env.action_space.n))
        model.compile(loss="mean_squared_error", optimizer=Adam(lr=self.learning_rate))
        return model

#### act()

We will again use the $\epsilon$-greedy exploration policy.

\begin{equation}
  a =
    \begin{cases}
      \text{random} & \text{with probability $\epsilon$}\\
      \arg\max_a Q(s,a;\theta)   & \text{otherwise}\\
    \end{cases}
\end{equation}

To get the Q-values from your Q-network, you can run

    q_values = self.model.predict(state)

#### remember()

After each step, we need to store the $(s, a,r,s', \text{done})$ experience in the replay memory. In this implementation we will store memories in a `deque` with a specified maximum legnth (memory capacity).

```
replay_memory = deque(maxlen=5)
print(replay_memory)

for i in range(7):
    replay_memory.append(i)
    print(replay_memory)

# in your implementation you will append the experience
# [state, action, reward, next_state, done] instead of i
```

    deque([], maxlen=5)
    deque([0], maxlen=5)
    deque([0, 1], maxlen=5)
    deque([0, 1, 2], maxlen=5)
    deque([0, 1, 2, 3], maxlen=5)
    deque([0, 1, 2, 3, 4], maxlen=5)
    deque([1, 2, 3, 4, 5], maxlen=5)
    deque([2, 3, 4, 5, 6], maxlen=5)

#### update_model()

To update the model, you'll need to:

1. sample a batch of experiences $(s_j, a_j, r_j, s_j', \text{done}_j)$ from memory
2. calculate the target output (`done` indicates if $s$ is terminal)

\begin{equation}
  y_j =
    \begin{cases}
      r_j + \gamma \max_{a'} Q(s_j',a'; \theta^-) & \text{if $s$ is not terminal}\\
      r_j   & \text{if $s$ is terminal}\\
    \end{cases}
\end{equation}

3. Perform gradient descent step according to

$$ L(\theta) = \left\langle \big( y_j - Q(s_j,a_j; \theta) \big)^2\right\rangle_{(s,a,r,s') \sim Uniform(Memory)}$$

For the third step, the TensorFlow code you will need is

    model.fit(batch_states, batch_target, epochs=1, verbose=0)

**NOTE**

The `batch_target` must be the same dimensions as the model output. This means you must have a target for every action for each input state in your batch of experiences $(s_j,a_j,r_j,s_j')$. For each action $a$ that is not in the experience batch, use the current output of the target model as the target value, $Q(s,a;\theta^-)$. Therefore, for each state $s_j$,

\begin{equation}
  \text{target} =
    \begin{cases}
      y_j             & \text{if $a$ is in experience batch $(s_j, a_j, r_j, s_j')$}\\
      Q(s,a;\theta^-)   & \text{if $a$ is NOT in experience batch $(s_j, a_j, r_j, s_j')$}\\
    \end{cases}
\end{equation}

**NOTE 2**

Here is a helpful line of code for reformating samples from memory, each of which will be a list `[state, action, reward, next_state, done]`, to a set of `np.array`s with dim (n_batch x __).

    batch_states, batch_actions, batch_rewards, batch_next_states, batch_done = map(np.asarray, zip(*memory_samples))

#### update_tartget()

This function is used to set the target network weights equal to the model network weights. This is only done periodically thoughout training, which reduces variance in the gradient across steps and stabilizes training.

We've left the code for `update_target()` filled in because, again, it is largely just Tensorflow syntax. You'll have to use this function appropriately within the main training loop.

    def update_target(self):
        weights = self.model.get_weights()
        self.target_model.set_weights(weights)

#### train()

This function will run the main training loop. Here is the pseudocode for the DQN algorithm.

    initialize Q-network (create model)
    initialize target network (create model and set weights equal to Q-network)
    for n_episodes
        initialize state
        while not done
            select action according to policy
            execute action; observe reward and next_state
            add experience (state, action, reward, next_state, done) to memory
            sample batch of (state, action, reward, next_state, done) experiences from memory and update model
            every C steps, update target model

Same as for Q-learning, the Gym api calls you will need are:
- `observation = env.reset()`
- `observation, reward, done, info = env.step(action)`
- `action = env.action_space.sample()`

The Tensorflow api calls you will need are:
- `model_output = model.predict(model_input)`
- `model.fit(model_input, model_target, epochs=1, verbose=0)`

#### Implementation

```
class DQN:
    def __init__(self, env, memory_cap=1000, gamma=0.9, epsilon=0.99,
        epsilon_decay=0.995, epsilon_min=0.01, learning_rate=0.005,
        batch_size=32, C=20):

        self.env = env
        self.memory = deque(maxlen=memory_cap)
        self.state_shape = env.observation_space.shape

        self.gamma = gamma
        self.epsilon = epsilon
        self.epsilon_min = epsilon_min
        self.epsilon_decay = epsilon_decay
        self.learning_rate = learning_rate
        self.batch_size = batch_size
        self.C = C

    def create_model(self):
        """We will use a two-layer perceptron. The input dim must equal the
        state space dim and the output dim must equal the action space dim,
        but you can play around with the size of the hidden layers. For DQNs,
        we need mean squared error loss."""
        model = Sequential()
        model.add(Dense(24, input_dim=self.state_shape[0], activation="relu"))
        model.add(Dense(16, activation="relu"))
        model.add(Dense(self.env.action_space.n))
        model.compile(loss="mean_squared_error", optimizer=Adam(lr=self.learning_rate))
        return model

    def act(self, state, test=False):
        """Choose action based on your current model using epsilon-greedy
        exploration"""

        # update epsilon
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        # set epsilon to 0 if testing the learned policy
        epsilon = 0.01 if test else self.epsilon

        # reshape state to feed into model - tensorflow thing, shape must be
        # (1, input_dim), not (input_dim,)
        state = state.reshape((1, self.state_shape[0]))

        # Select a random action with probability epsilon, otherwise select
        # the action with the highest Q-value estimate.

        # --------------------------------------------
        # Your code here
        # --------------------------------------------

        return action

    def remember(self, state, action, reward, new_state, done):
        """Append experience to memory"""
        # --------------------------------------------
        # Your code here
        # --------------------------------------------

    def update_model(self):
        """This function updates the q-network model. You must 1) sample a
        batch of experiences from the replay memory 2) calculate the target for
        each expereince, 3) update the model by calling model.fit()"""

        # only update model once have sufficient number of experiences in memory
        if len(self.memory) < self.batch_size:
            return

        # 1. Sample a batch of experiences from memory
        # 2. Reformat the samples into a set of np.arrays (convenient code for this provided above)
        # 3. Calculate the target for batch of experiences
        # 4. Get the batch_target for fitting model (using target network predictions
        # for actions not in experience batch)
        # 5. Update the model by running model.fit(batch_states, batch_target, epoch=1, verbose=0)

        # --------------------------------------------
        # Your code here
        # --------------------------------------------

    def update_target(self):
        """"Sets target weights equal to model weights."""
        weights = self.model.get_weights()
        self.target_model.set_weights(weights)

    def train(self, num_episodes=50):
        """This function implements the main training loop."""
        # keep track of total reward per episode
        self.reward_history = []
        # initialize model and target model
        self.model = self.create_model()
        self.target_model = self.create_model()
        self.target_model.set_weights(self.model.get_weights())
        # we need to keep track of steps now so we can update the model every
        # C steps
        step = 0

        for episode in range(num_episodes):
            total_reward = 0

            # Implement the training loop for DQN (pseudocode above).

            # --------------------------------------------
            # Your code here
            # --------------------------------------------

            # Save total reward from epsisode and print training progress.
            self.reward_history.append(total_reward)
            print("episode {}: {} reward".format(episode, total_reward))
```

### Training an agent

```
env = gym.make('CartPole-v1')

seed = 2
env.seed(seed)
env.action_space.seed(seed)
np.random.seed(seed)
random.seed(seed)
tf.random.set_seed(seed)

dqn_agent = DQN(env, batch_size=32)
dqn_agent.train(num_episodes=35)
```

    episode 0: 12.0 reward
    episode 1: 14.0 reward
    episode 2: 19.0 reward
    episode 3: 19.0 reward
    episode 4: 10.0 reward
    episode 5: 10.0 reward
    episode 6: 21.0 reward
    episode 7: 16.0 reward
    episode 8: 30.0 reward
    episode 9: 11.0 reward
    episode 10: 11.0 reward
    episode 11: 50.0 reward
    episode 12: 26.0 reward
    episode 13: 23.0 reward
    episode 14: 40.0 reward
    episode 15: 113.0 reward
    episode 16: 24.0 reward
    episode 17: 43.0 reward
    episode 18: 38.0 reward
    episode 19: 8.0 reward
    episode 20: 42.0 reward
    episode 21: 64.0 reward
    episode 22: 54.0 reward
    episode 23: 70.0 reward
    episode 24: 91.0 reward
    episode 25: 30.0 reward
    episode 26: 121.0 reward
    episode 27: 213.0 reward
    episode 28: 500.0 reward
    episode 29: 500.0 reward
    episode 30: 364.0 reward
    episode 31: 401.0 reward
    episode 32: 342.0 reward
    episode 33: 309.0 reward
    episode 34: 290.0 reward

```
# visualize total reward per episode across training
plt.figure(figsize=(7,4))
plt.plot(dqn_agent.reward_history, alpha=1, color='purple', label='raw')
plt.xlabel('episode #', fontsize=15)
plt.ylabel('total reward per episode', fontsize=15)
plt.title('DQN Agent')
plt.show()
```

### Evaluating the agent

```
evaluate_policy(dqn_agent, num_episodes=1)
```

    Exploit reward average: 227.0

```
create_learned_policy_video(dqn_agent, "video", num_episodes=1)
```

    xdpyinfo was not found, X start can not be checked! Please install xdpyinfo!
    WARNING:root:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (400, 600) to (400, 608) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to None (risking incompatibility). You may also see a FFMPEG warning concerning speedloss due to data not being aligned.

Yay, it learned the task! This is also not quite optimal, but its only been 35 sessions! This is a huge change from our Q-learning agent, which took 5000 episode to reach less than the perfomance of our DQN agents. This tells us how useful it is to keep the continous-valued inputs and use a neural network to appromimate the Q-value function.

### What is the model learning?

```
n_obs = 40
obs_array = np.zeros((n_obs, env.observation_space.shape[0]))
obs_array[:,0] = 0
obs_array[:,1] = 0
obs_array[:,2] = 0
obs_array[:,3] = np.linspace(-5, 5, n_obs)
# run model
q_values = dqn_agent.model.predict(obs_array)
# visualize results
plt.figure(figsize=(8,5))
plt.plot(obs_array[:,3], q_values[:,0], color='purple', label='push cart left', linewidth=2)
plt.plot(obs_array[:,3], q_values[:,1], color='green', label='push cart right', linewidth=2)
plt.vlines(0, q_values.min(), q_values.max(), linestyle='--', color='dimgray')
plt.xlabel('Pole Velocity', fontsize=15)
plt.ylabel('Q value', fontsize=15)
plt.title('Cart Position=0, Cart Velocity=0, Pole Angle=0', fontsize=15)
plt.legend(fontsize=13)
plt.show()
```

We can see here that our agent learned the switch between pushing the car right and left right at 0. It was able to learn this behavior from few examples because even if it didn't encounter every state along this continuum, it can generalize from nearby experiences. Pretty cool!

## Summary

To summarize, in this notebook we implemented the Q-Learning and DQN algorithms and trained an agent on each. Moving from Q-learning to DQNs gave us similar performance (slightly better for the DQN) after 5000 episodes for Q-learning and only 35 episodes for DQNs. This is a pretty amazing improvement, and gives us a hands-on feel for how much you gain by using neural networks for function approximation.

We focused on DQNs here, but Policy Gradient algorithms are a very effective class of algorithms for RL tasks. Our "What is the model learning?" plot may give us a hint for why this is. DQNs need to approximate the Q-value for each action, and must get the highest Q-value right in order for the policy to be correct. If the Q-values are very similar this might by tricky and lead to noise in selecting the best action. Perhaps it is easier to directly learn the action output space, as Policy Gradients do.

There is a lot to learn in DeepRL, and this at least gives us a start!

## Citations

A lot of the code in the Colab notebook was inpired by some really helpful posts from other people:

- Lillian Weng's [deep-reinforcement-learning-gym](https://github.com/lilianweng/deep-reinforcement-learning-gym) repo (adapted her Q-learning implementation)
- Anita Hu's [TF2-RL](https://github.com/anita-hu/TF2-RL) repo (adapted her DQN implementation)
- TF [tutorial](https://www.tensorflow.org/agents/tutorials/1_dqn_tutorial) for training DQN with TF-Agents (adapted TF-Agents visualization functions for Gym)


# GAN training tutorial

This is a tutorial training a [GAN](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf) on MNIST and doing a latent traversal. We will focus on the original GAN, but there are other GANs out there (such as Wasserstein GAN).

Structure:

  * basic GAN training
  * latent traversal
  * try bigger learning rate to see what happens
  * optional: add gradient penalty

 Your tasks:
   * finish up generator definition
   * define the discriminator loss
   * define the generator loss
   * define the discriminator and generator update operations
   * run the latent traversal task
   * change the learning rates of the discriminator / generator to see if you can get mode collapse
   * change the training to do 5 discriminator updates for generator update
   * (optional): implement gradient penalties

```
!pip install -q dm-sonnet
!pip install -q seaborn -U
```

## Imports

We will use Sonnet to define the discriminator and generator networks.

```
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math

import tensorflow as tf
import sonnet as snt

import numpy as np

# Plotting library.
from matplotlib import pyplot as plt
import seaborn as sns
```

```
tf.logging.set_verbosity(tf.logging.ERROR)
```

```
# Don't forget to select GPU runtime environment in Runtime -> Change runtime type
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
```

```
sns.set(rc={"lines.linewidth": 2.8}, font_scale=2)
sns.set_style("whitegrid")
```

## Hyperparameters

```
BATCH_SIZE = 64 # @param
NUM_LATENTS = 20 # @param
TRAINING_STEPS = 5000  # @param
```

## Get the data

We will use the MNIST dataset. Luckly, TensorFlow comes with a simple way to load it.

```
mnist = tf.contrib.learn.datasets.load_dataset("mnist")
```

```
print(mnist.train.images.shape)
print(type(mnist.train.images))
```

```
train_images = mnist.train.images.reshape((-1, 28, 28, 1))
print (train_images.shape)
```

## Reshape data to image structure

### Convert data to TensorFlow datasets, shuffle it and batch it

```
dataset = tf.data.Dataset.from_tensor_slices(train_images)
```

```
batched_dataset = dataset.shuffle(100000).repeat().batch(BATCH_SIZE)
```

### Create iterator that goes through the data

```
iterator = batched_dataset.make_one_shot_iterator()
images = iterator.get_next()
```

```
print(images)
```

    Tensor("IteratorGetNext:0", shape=(?, 28, 28, 1), dtype=float32)

### Scale the data

Scale the data between -1 and 1. This helps training stability and improves GAN convergence.

```
real_data = 2 * images - 1
```

## Define the discriminator and generator networks

### Exercise: Generator network last activation

We use a simple Transpose convolutional network. Because the input to the model is 2 dimensional, we first use a linear layer to transform the input latents to a size suitable for the conv network.

```
class MnistGenerator(snt.AbstractModule):

  def __init__(self, name='MnistGenerator'):
    super(MnistGenerator, self).__init__(name=name)

  def _build(self, inputs):
    """Constructs the generator graph.

    Args:
      inputs: `tf.Tensor` with the input of the generator.

    Returns:
      `tf.Tensor`, the generated samples.
    """
    linear = snt.Linear(7 * 7 * 64)
    inputs = linear(inputs)

    # Reshape the data to have rank 4.
    inputs = snt.BatchReshape((7, 7, 64))(inputs)
    inputs = tf.nn.relu(inputs)

    net = snt.nets.ConvNet2DTranspose(
        output_channels=[32, 1],
        output_shapes=[[14, 14], [28, 28]],
        strides=[2],
        paddings=[snt.SAME],
        kernel_shapes=[[5, 5]],
        use_batch_norm=False)

    # TODO: fill in network output to ensure the data
    # is normalized in [-1, 1].
```

### Discrimiantor network

We use a deep discrimiantor network with leaky relu activation functions. Leaky relus have been observed to stabilize GAN training, see [DCGAN](https://arxiv.org/pdf/1511.06434.pdf).

A leaky relu is a piece wise linear function that has a slope $\alpha$:

$$ lreu(x) = \left\{
    \begin{array}{l}
      - \alpha x,  x < 0\\
      x, x > 0
    \end{array}
  \right. $$

Other tricks you can do: add noise to the network activation.

```
x = np.linspace(-5, 5, 200)
lrelu = np.array([max(0.2 * y, y) for y in x])
plt.plot(x, lrelu)
```

```
class MnistDiscriminator(snt.AbstractModule):

  def __init__(self,
               leaky_relu_coeff=0.2, name='MnistDiscriminator'):
    super(MnistDiscriminator, self).__init__(name=name)
    self._leaky_relu_coeff = leaky_relu_coeff

  def _build(self, input_image):
    """Constructs the discriminator graph.

    Args:
      input_image: `tf.Tensor` input to be used by the discriminator.

    Returns:
      `tf.Tensor` output with the decision of the discriminator.
    """
    leaky_relu_activation = lambda x: tf.maximum(self._leaky_relu_coeff * x, x)

    conv2d = snt.nets.ConvNet2D(
        output_channels=[8, 16, 32, 64, 128],
        kernel_shapes=[[5, 5]],
        strides=[2, 1, 2, 1, 2],
        paddings=[snt.SAME],
        activate_final=True,
        activation=leaky_relu_activation,
        use_batch_norm=False)

    convolved = conv2d(input_image)
    # Flatten the data to 2D for the classification layer
    flat_data = snt.BatchFlatten()(convolved)

    # We have two classes: one for real, and oen for fake data.
    classification_logits = snt.Linear(2)(flat_data)
    return classification_logits

```

## Generate samples

```
latents = tf.random_normal((BATCH_SIZE, NUM_LATENTS))
generator = MnistGenerator()
samples = generator(latents)
```

## Set up the losses

The discriminator is a classifier, so it receives a standard cross entropy loss. The game is given by the value function:
\begin{equation}
\max_{G} \min_{D} \mathbb{E}_{p^*(x)} \log D(x) + \mathbb{E}_{p(z)} \log (1- D(G(z)))
\end{equation}

## Exercise: Discriminator loss

Fill in the discriminator loss. Think about the fact that:
   * you need to do a forward pass through the discriminator with the real data
   * add a classification loss for the real data
   * you need to do a forward pass through the discriminator with the samples
   * add a classification loss for the generator data

```
discriminator = MnistDiscriminator()

real_data_loss = ...

samples_loss = ...

# Reduce loss over batch dimension
discriminator_loss = tf.reduce_mean(real_data_loss + samples_loss)

# Ignore for the first exercise, uncomment if you do the last exercise.
# discriminator_loss += 10 * dragan_penalty_loss(discriminator, real_data)
```

##  Generator loss
The generator is trying to fool the discriminator. While the min-max loss suggests using $\log (1 - D(G(x)))$ , in practice $- \log D(G(x))$ is used, as it provides better gradients for the generator early in training, when it is performing poorly.

## Let's visualize the two losses and their derivatives

### Original min-max loss.

When using $\log (1 - D(G(x)))$ , as the loss of the generator, we see that the generator does not receive a strong gradient (right plot), when it is doing badly and the discriminator can easily distinguish between samples and real data. But this is not what we want, as early on in training we want the generator to quickly move to generate better samples.

```
d_x = np.linspace(0.0, 1.0, 200)
loss = np.log(1 - d_x)
grad = 1 / (1 - d_x)

figsize = (18, 6)
fig, axs = plt.subplots(1, 2, figsize=figsize)

# First plot the loss, and then the derivative.
axs[0].plot(d_x, loss, label='log (1 - D(x)')
axs[0].set_xlabel('D(x)')
axs[0].set_ylabel('loss')

axs[1].plot(d_x, grad, label='1 / (1 - D(x)')
axs[1].set_xlabel('D(x)')
axs[1].set_ylabel('derivative')

plt.suptitle('Loss and derivative for generator loss log (1 - D(x))', fontsize=18)

plt.show()
```

### Alternative loss

When using $-log D(G(x))$ we obtain the desired behaviour: the generator gets strong gradients to move when it is bad at sample generation and the discriminator can easily distinguish between samples and real data.

```
d_g_x = np.linspace(0.0, 1.0, 200)
loss = - np.log(d_g_x)
grad = - 1 / d_g_x

figsize = (18, 6)
fig, axs = plt.subplots(1, 2, figsize=figsize)

# First plot the loss, and then the derivative.
axs[0].plot(d_x, loss, label='- log (D(x)')
axs[0].set_xlabel('D(x)')
axs[0].set_ylabel('loss')

axs[1].plot(d_x, grad, label='1 / (1 - D(x)')
axs[1].set_xlabel('D(x)')
axs[1].set_ylabel('derivative')

plt.suptitle('Loss and derivative for generator loss - log (D(x))', fontsize=18)

plt.show()
```

## Connections with Jensen Shannon divergence

The GAN paper shows that if the discriminator is optimal, the generator is minimizing the Jensen Shannon divergence between the true data distribution and the generator distribution:

 $$ JSD(p, q) = \frac{1}{2} KL (p || \frac{p+q}{2})  + \frac{1}{2}  KL (q || \frac{p+q}{2})   $$

 Note that in practice the assumptions required for the proof do not hold:

 * the discriminator is not optimal
      * we do not train the discriminator to optimality, but we do alternating gradient descent (one discriminator update, one generator update).
      * the discriminator is a neural network, which might not have the capacity to estimate the true probabilities.
      * the discriminator does not have access to the entire data distribution (but only a few samples).

## Exercise: the generator loss

Consider what you have seen above to define the generator loss.

```
# Get the probabilities from the discriminator logits.
discriminator_samples_logits = ...     # Calculated in the 'Discriminator loss' exercise.
discriminator_probabilities = tf.nn.softmax(discriminator_samples_logits)

generator_loss = ...
```

## Exercise: Create optimizers and training ops

**Important**: need to pass the list of variables to the TensorFlow optimizer, otherwise the generator and discriminator
variables will receive both the loss of the discrimiantor and that of the generator.

We want to freeze the discriminator when we update the generator, and vice versa.

```
discrimiantor_optimizer = tf.train.AdamOptimizer(0.0001, beta1=0.5, beta2=0.9)
generator_optimizer = tf.train.AdamOptimizer(0.0001, beta1=0.5, beta2=0.9)

# Optimize the discrimiantor.
# Remember: you need to only update the discriminator variables.
discriminator_update_op = ...

# Optimize the generator..
# Remember: you need to only update the generator variables.
generator_update_op = ...
```

## Create the tensorflow session

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.global_variables_initializer())
```

## Training

We train the discriminator and generator by alternating gradient descent runs. We record the losses to plot them later.

Note: by doing alternating gradient descent we are not solving the original min max problem, which requires training the discrimininator to optimaility in the inner loop.

```
disc_losses = []
gen_losses = []

for i in xrange(TRAINING_STEPS):
  sess.run(discriminator_update_op)
  sess.run(generator_update_op)

  if i % 100 == 0:
    disc_loss = sess.run(discriminator_loss)
    gen_loss = sess.run(generator_loss)

    disc_losses.append(disc_loss)
    gen_losses.append(gen_loss)

    print('At iteration {} out of {}'.format(i, TRAINING_STEPS))
```

## Visualize the behaviour of the two losses during training

Note that unlike losses for classifiers, or for VAEs, the losses are not stable and are going up and down, depending on the training dynamics.

```
figsize = (20, 6)
fig, axs = plt.subplots(1, 2, figsize=figsize)

# First plot the loss, and then the derivative.
axs[0].plot(disc_losses, '-')
axs[0].plot([np.log(2)] * len(disc_losses), 'r--', label='Discriminator is being fooled')
axs[0].legend()
axs[0].set_title('Discriminator loss', fontsize=20, y=-0.2)
axs[1].plot(gen_losses, '-')
axs[1].set_title('Generator loss', fontsize=20, y=-0.2)
```

## Visualize training results

After we have trained the discriminator and the generator, we can visualize the samples obtained from the generator, and compare them with real data samples.

```
def gallery(array, ncols=10, rescale=True):
    if rescale:
      array = (array + 1.) / 2
    nindex, height, width, intensity = array.shape
    nrows = nindex//ncols
    assert nindex == nrows*ncols
    # want result.shape = (height*nrows, width*ncols, intensity)
    result = (array.reshape(nrows, ncols, height, width, intensity)
              .swapaxes(1,2)
              .reshape(height*nrows, width*ncols, intensity))
    return result
```

```
real_data_examples = sess.run(real_data)
final_samples = sess.run(samples)
```

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(real_data_examples, ncols=8).squeeze(axis=2))
```

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(final_samples, ncols=8).squeeze(axis=2))
```

## Interpolate between latent space vectors to see latent space traversal

By doing an interpolation in latent space, we can analyze the behaviour of the model. This allows us to see that the model generalizes in different parts of the space, and test how it "moves" from one digit to the next.

```
a = np.linspace(0.0, 1.0, 100)
a = np.expand_dims(a, axis=1)

first_latents = tf.random_normal((1, NUM_LATENTS))
second_latents = tf.random_normal((1, NUM_LATENTS))

# To ensure that the interpolation is still likely under the Gaussian prior,
# we use Gaussian interpolation - rather than linear interpolation.
interpolations =  np.sqrt(a) * first_latents + np.sqrt(1 - a) * second_latents

samples_from_interpolations = generator(interpolations)
```

```
samples_from_interpolations_np = sess.run(samples_from_interpolations)
```

```
print(samples_from_interpolations_np.shape)
```

    (100, 28, 28, 1)

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(samples_from_interpolations_np).squeeze(axis=2))
plt.show()
```

### Difference with VAE latent traversal

Do you notice a difference between VAE and GAN latent traversal results?

## [Optional exercise] Gradient penalties

Add the DRAGAN gradient penalty to the discriminator loss. For more details, see: https://arxiv.org/pdf/1705.07215.pdf. For more information on gradient penalties, see: https://arxiv.org/abs/1710.08446.

You want to ensure add a penalty that the gradients of the discriminator around the data are 1.

What you have to do:

1) Create a noised version of the data, by adding zero mean noise, with a small std (something like 0.5 * the standard deviation of the input data batch).

2) Take the gradients of the discriminator outputs with respect to these interpolations.

3) Take the norm over the batch over these gradients - this should return a vector of size `batch_size`

4) Return the square loss between the obtained average gradients and 1.

```
def batch_l2_norms(x, eps=1e-5):
  reduction_axis = range(1, x.get_shape().ndims)
  squares = tf.reduce_sum(tf.square(x), axis=reduction_axis)
  squares.get_shape().assert_is_compatible_with([None])
  return tf.sqrt(eps + squares)

def dragan_penalty_loss(discriminator, real_data):
  """The gradient penalty loss on noisy data.

  Proposed by https://arxiv.org/pdf/1705.07215.pdf

  Args:
    discriminator: An instance of `AbstractDiscriminator`.
    real_data: A `tf.Tensor` (joint discriminator `tf.Tensor` sequences are not
      yet supported). The data associated as real by the GAN, usually from a
      datasets. Needs to be a valid input for `discriminator`.
  Returns:
    A tf.Tensor scalar, containing the loss.
  """
  pass

```

Change the discriminator loss from the original cell.

Add to `dragan_penalty_loss(discriminator, real_data)` to `discriminator_loss` with a coefficient of 10.

## Close session

```
# sess.close()
```

## Reflections

* are GANs doing divergence minimization?
* how can you measure the quality of GAN samples?
* how can you check if GANs are overfitting?
* how are GANs different to VAEs?

### GAN samples on other datasets

Note: these are samples with the vanilla algorithm, but they can be improved using better complex distributions.

#### CIFAR-10

#### CelebA


```
pip install strawberryfields
```

    Requirement already satisfied: strawberryfields in /usr/local/lib/python3.6/dist-packages (0.14.0)
    Requirement already satisfied: urllib3>=1.25.3 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (1.25.9)
    Requirement already satisfied: appdirs in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (1.4.4)
    Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (2.4)
    Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (1.4.1)
    Requirement already satisfied: numba in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (0.48.0)
    Requirement already satisfied: toml in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (0.10.1)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (2.8.1)
    Requirement already satisfied: sympy>=1.5 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (1.6.1)
    Requirement already satisfied: quantum-blackbird>=0.2.3 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (0.2.4)
    Requirement already satisfied: requests>=2.22.0 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (2.23.0)
    Requirement already satisfied: numpy>=1.17.4 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (1.18.5)
    Requirement already satisfied: thewalrus>=0.12 in /usr/local/lib/python3.6/dist-packages (from strawberryfields) (0.12.0)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.0->strawberryfields) (4.4.2)
    Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba->strawberryfields) (0.31.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from numba->strawberryfields) (49.1.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.8.0->strawberryfields) (1.12.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.5->strawberryfields) (1.1.0)
    Requirement already satisfied: antlr4-python3-runtime==4.7.2 in /usr/local/lib/python3.6/dist-packages (from quantum-blackbird>=0.2.3->strawberryfields) (4.7.2)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->strawberryfields) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->strawberryfields) (2020.6.20)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->strawberryfields) (3.0.4)
    Requirement already satisfied: dask[delayed] in /usr/local/lib/python3.6/dist-packages (from thewalrus>=0.12->strawberryfields) (2.12.0)
    Requirement already satisfied: toolz>=0.7.3; extra == "delayed" in /usr/local/lib/python3.6/dist-packages (from dask[delayed]->thewalrus>=0.12->strawberryfields) (0.10.0)
    Requirement already satisfied: cloudpickle>=0.2.1; extra == "delayed" in /usr/local/lib/python3.6/dist-packages (from dask[delayed]->thewalrus>=0.12->strawberryfields) (1.3.0)

```
import strawberryfields as sf
from strawberryfields.apps import  data, plot, sample
import numpy as np
```

```
from strawberryfields.apps import data, sample

samples = data.Planted()

A = samples.adj
n_mean = samples.n_mean
more_samples = sample.sample(A, n_mean, n_samples=5)
print(more_samples)
```

    [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0], [1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]

```
import networkx as nx

samples = data.Planted()
g = nx.Graph(samples.adj)
samples = sample.postselect(samples, 16, 30)
samples = sample.to_subgraphs(samples, g)
plot.graph(g, samples[7])

```

<meta charset="utf-8" />

```
from strawberryfields.apps import clique

c = clique.shrink(samples[2],g)
plot.graph(g,c)
```

<meta charset="utf-8" />

```
c = clique.grow(c,g)
plot.graph(g,c)
```

<meta charset="utf-8" />

```
c = clique.swap(c,g)
plot.graph(g, c)
```

<meta charset="utf-8" />

```
shrunk_cliques = [clique.shrink(s, g) for s in samples]
search_cliques = [clique.search(s, g, 10) for s in shrunk_cliques]
cliques_plus_lengths = [(len(s), s) for s in shrunk_cliques]
cliques_plus_lengths = sorted(cliques_plus_lengths, reverse=True)
print(cliques_plus_lengths[0])

```

    (8, [21, 22, 24, 25, 26, 27, 28, 29])

```
samples = data.TaceAs()

g = nx.Graph(samples.adj)
samples = sample.postselect(samples, 8,8)
samples = sample.to_subgraphs(samples, g)

cliques = [clique.shrink(s, g) for s in samples]
cliques = [clique.search(s, g, 10) for s in cliques]

print(max(len(c) for c in cliques))
```

    8

```
from strawberryfields.apps import data, sample, vibronic

f = data.Formic()

params_0 = vibronic.gbs_params(f.w, f.wp, f.Ud, f.delta, T=0)
params_300 = vibronic.gbs_params(f.w, f.wp, f.Ud, f.delta, T=300)

samples_0 = sample.vibronic(*params_0, n_samples=10)
samples_300 = sample.vibronic(*params_300, n_samples=10)

energies_0 = vibronic.energies(samples_0, f.w, f.wp)
energies_300 = vibronic.energies(samples_300, f.w, f.wp)

```

```

```


```python
!pip install qiskit==0.16.1
```

    Requirement already satisfied: qiskit==0.16.1 in /usr/local/lib/python3.7/dist-packages (0.16.1)
    Requirement already satisfied: qiskit-aer==0.4.1 in /usr/local/lib/python3.7/dist-packages (from qiskit==0.16.1) (0.4.1)
    Requirement already satisfied: qiskit-ibmq-provider==0.5.0 in /usr/local/lib/python3.7/dist-packages (from qiskit==0.16.1) (0.5.0)
    Requirement already satisfied: qiskit-aqua==0.6.4 in /usr/local/lib/python3.7/dist-packages (from qiskit==0.16.1) (0.6.4)
    Requirement already satisfied: qiskit-ignis==0.2.0 in /usr/local/lib/python3.7/dist-packages (from qiskit==0.16.1) (0.2.0)
    Requirement already satisfied: qiskit-terra==0.12.0 in /usr/local/lib/python3.7/dist-packages (from qiskit==0.16.1) (0.12.0)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.4.1->qiskit==0.16.1) (1.4.1)
    Requirement already satisfied: pybind11>=2.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.4.1->qiskit==0.16.1) (2.6.2)
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.4.1->qiskit==0.16.1) (1.19.5)
    Requirement already satisfied: cython>=0.27.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.4.1->qiskit==0.16.1) (0.29.22)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (2.23.0)
    Requirement already satisfied: arrow>=0.15.5 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (1.0.3)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (1.1.0)
    Requirement already satisfied: websockets<8,>=7 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (7.0)
    Requirement already satisfied: nest-asyncio!=1.1.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (1.5.1)
    Requirement already satisfied: fastdtw in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (0.3.4)
    Requirement already satisfied: dlx in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (1.0.4)
    Requirement already satisfied: docplex in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (2.20.204)
    Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (0.22.2.post1)
    Requirement already satisfied: pyscf; sys_platform == "linux" or (python_version < "3.8" and sys_platform != "win32") in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (1.7.6.post1)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (5.4.8)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (2.5.1)
    Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (2.6.0)
    Requirement already satisfied: cvxopt; python_version < "3.8" or sys_platform != "win32" in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (1.2.6)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (1.7.1)
    Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (2.10.0)
    Requirement already satisfied: quandl in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (3.6.1)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aqua==0.6.4->qiskit==0.16.1) (56.0.0)
    Requirement already satisfied: marshmallow-polyfield<6,>=5.7 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.12.0->qiskit==0.16.1) (5.10)
    Requirement already satisfied: ply>=3.10 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.12.0->qiskit==0.16.1) (3.11)
    Requirement already satisfied: marshmallow<4,>=3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.12.0->qiskit==0.16.1) (3.11.1)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.12.0->qiskit==0.16.1) (0.3.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (1.24.3)
    Requirement already satisfied: python-dateutil>=2.7.0 in /usr/local/lib/python3.7/dist-packages (from arrow>=0.15.5->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (2.8.1)
    Requirement already satisfied: typing-extensions; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from arrow>=0.15.5->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (3.7.4.3)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (1.5.0)
    Requirement already satisfied: cryptography>=1.3 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (3.4.7)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from docplex->qiskit-aqua==0.6.4->qiskit==0.16.1) (1.15.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.6.4->qiskit==0.16.1) (1.0.1)
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.2->qiskit-aqua==0.6.4->qiskit==0.16.1) (4.4.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-aqua==0.6.4->qiskit==0.16.1) (1.2.1)
    Requirement already satisfied: inflection>=0.3.1 in /usr/local/lib/python3.7/dist-packages (from quandl->qiskit-aqua==0.6.4->qiskit==0.16.1) (0.5.1)
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/dist-packages (from quandl->qiskit-aqua==0.6.4->qiskit==0.16.1) (8.7.0)
    Requirement already satisfied: pandas>=0.14 in /usr/local/lib/python3.7/dist-packages (from quandl->qiskit-aqua==0.6.4->qiskit==0.16.1) (1.1.5)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (1.14.5)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.14->quandl->qiskit-aqua==0.6.4->qiskit==0.16.1) (2018.9)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.5.0->qiskit==0.16.1) (2.20)

# Task One : Fuandamentals of Qunatum Computation

## Basic Arithmatic Operations & Complex Numbers

```python
5+9
```

    14

```python
12*4
```

    48

Complex numbers are always of the form

\begin{align}
\alpha = a + bi
\end{align}

```python
1j * 1j
```

    (-1+0j)

```python
z = 3 + 4j
w = 1 - 2j
```

```python
import numpy as np
```

```python
print('z=', z)
print('Real(z)=', np.real(z))
print('Imag(z)=', np.imag(z))
```

    z= (3+4j)
    Real(z)= 3.0
    Imag(z)= 4.0

```python
z+w
```

    (4+2j)

```python
z-w
```

    (2+6j)

```python
z/w
```

    (-1+2j)

## Complex conjugate

```python
z
```

    (3+4j)

```python
np.conj(z)
```

    (3-4j)

## Norms/Absolute Values

\begin{align} ||z|| &= \sqrt{zz^*} = \sqrt{|z|^2},\\ ||w|| &= \sqrt{ww^*} = \sqrt{|w|^2}, \end{align}

```python
print('||z||=', np.abs(z))
```

    ||z||= 5.0

```python
print('||w||=', np.abs(w))
```

    ||w||= 2.23606797749979

## Row Vectors, Column Vectors, and Bra-Ket Notation

\begin{align} \text{Column Vector:} \ \begin{pmatrix}
a_1 \\ a_2 \\ \vdots \\ a_n
\end{pmatrix}
\quad \quad \text{Row Vector:} \ \begin{pmatrix}
a_1, & a_2, & \cdots, & a_n
\end{pmatrix} \end{align}

```python
# Create a vector as a row
row_vector = np.array([2-1j, 7j, -3])

# Create a vector as a column
column_vector = np.array([[2+1j],
                          [-5],
                          [2j]])
```

```python
row_vector
```

    array([ 2.-1.j,  0.+7.j, -3.+0.j])

Row vectors in quantum mechanics are also called **bra-vectors**, and are denoted as follows:

\begin{align} \langle A| = \begin{pmatrix}
a_1, & a_2, \cdots, & a_n
\end{pmatrix} \end{align}

Column vectors are also called **ket-vectors** in quantum mechanics denoted as follows:

\begin{align} |B\rangle = \begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix} \end{align}

In general, if we have a column vector, i.e. a ket-vector:

\begin{align} |A\rangle = \begin{pmatrix}
a_1 \\ a_2 \\ \vdots \\ a_n
\end{pmatrix} \end{align}

the corresponding bra-vector:

\begin{align} \langle A| = \begin{pmatrix}
a_1^*, & a_2^*, & \cdots, & a_n^*
\end{pmatrix} \end{align}

```python
ket_A = np.array([[2-1j],
                  [5]])
ket_B = np.array([[7],
                  [-3j]])
```

```python
print(ket_A + ket_B)
```

    [[9.-1.j]
     [5.-3.j]]

## Inner Product

\begin{align} \langle A| = \begin{pmatrix}
a_1, & a_2, & \cdots, & a_n
\end{pmatrix}, \quad \quad
|B\rangle = \begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix} \end{align}

Taking the inner product of $\langle A|$ and $|B\rangle$ gives the following:

\begin{align} \langle A| B \rangle &= \begin{pmatrix}
a_1, & a_2, & \cdots, & a_n
\end{pmatrix}
\begin{pmatrix}
b_1 \\ b_2 \\ \vdots \\ b_n
\end{pmatrix}\\
&=
a_1b_1 + a_2b_2 + \cdots + a_nb_n\\
&= \sum_{i=1}^n a_ib_i
\end{align}

```python
# Define the 4x1 matrix version of a column vector (instead of using the np.array() version):
A = np.matrix([[1-1j],
               [3],
               [2j],
               [5+1j]])
A
```

    matrix([[1.-1.j],
            [3.+0.j],
            [0.+2.j],
            [5.+1.j]])

```python
# Define B as a 1x4 matrix
B = np.matrix([[-3j, 2+2j, -6j, -7]])

B
```

    matrix([[-0.-3.j,  2.+2.j, -0.-6.j, -7.+0.j]])

```python
# Compute <B|A>
np.dot(B,A)
```

    matrix([[-20.-4.j]])

## Matrices

\begin{align}
M = \begin{pmatrix}
2-i & -3 \\
-5i & 2
\end{pmatrix}
\end{align}

```python
M = np.array([[2-1j, -3],
              [-5j, 2]])

print(M)
```

    [[ 2.-1.j -3.+0.j]
     [-0.-5.j  2.+0.j]]

```python
M = np.matrix([[2-1j, -3],
               [-5j, 2]])

print(M)
```

    [[ 2.-1.j -3.+0.j]
     [-0.-5.j  2.+0.j]]

Hermitian conjugates are given by taking the conjugate transpose of the matrix

```python
print(M.H)
```

    [[ 2.+1.j -0.+5.j]
     [-3.-0.j  2.-0.j]]

## Tensor Products of Matrices

\begin{align}
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix} \otimes
\begin{pmatrix}
x & y \\
z & w
\end{pmatrix} =
\begin{pmatrix}
a \begin{pmatrix}
x & y \\
z & w
\end{pmatrix} & b \begin{pmatrix}
x & y \\
z & w
\end{pmatrix} \\
c \begin{pmatrix}
x & y \\
z & w
\end{pmatrix} & d \begin{pmatrix}
x & y \\
z & w
\end{pmatrix}
\end{pmatrix} =
\begin{pmatrix}
ax & ay & bx & by \\
az & aw & bz & bw \\
cx & cy & dx & dy \\
cz & cw & dz & dw
\end{pmatrix}
\end{align}

```python
A = np.matrix([[1j+8 , 2j-6], []])

```

    /usr/local/lib/python3.7/dist-packages/numpy/matrixlib/defmatrix.py:145: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      arr = N.array(data, dtype=dtype, copy=copy)

# Task Two: Qubits, Bloch Sphere and Basis States

```python
import qiskit
qiskit.__qiskit_version__
```

    {'qiskit': '0.16.1',
     'qiskit-aer': '0.4.1',
     'qiskit-aqua': '0.6.4',
     'qiskit-ibmq-provider': '0.5.0',
     'qiskit-ignis': '0.2.0',
     'qiskit-terra': '0.12.0'}

\begin{align}
|\psi \rangle = \begin{pmatrix}
\alpha \\ \beta
\end{pmatrix}, \quad \text{where } \sqrt{\langle \psi | \psi \rangle} = 1.
\end{align}

Think of Qubit as an Electron:

\begin{align}
\text{spin-up}: \ |0\rangle &= \begin{pmatrix} 1\\0 \end{pmatrix} \\
\text{spin-down}: \ |1\rangle & = \begin{pmatrix} 0\\1 \end{pmatrix}
\end{align}

```python
import numpy as np
from qiskit import *
from qiskit.visualization import plot_bloch_vector
%matplotlib inline
```

Another representation is via Bloch Sphere:

```python
plot_bloch_vector([0,0,1], title="Spin-Up")
```

## Spin + / - :

\begin{align}
\text{spin +}: \ |+\rangle &= \begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix} = \frac{1}{\sqrt{2}} \left(|0\rangle + |1\rangle\right) \\
\text{spin -}: \ |-\rangle & = \begin{pmatrix} 1/\sqrt{2} \\ -1/\sqrt{2} \end{pmatrix} = \frac{1}{\sqrt{2}} \left(|0\rangle - |1\rangle\right)
\end{align}

```python
plot_bloch_vector([1,0,0], title="spin-plus")
```

## Basis States

\begin{align}
 |0\rangle &= \begin{pmatrix} 1\\0 \end{pmatrix} \\
 |1\rangle & = \begin{pmatrix} 0\\1 \end{pmatrix}
\end{align}
Preapring other states from Basis States:
\begin{align}
|00 \rangle &= |0\rangle \otimes |0\rangle = \begin{pmatrix} 1\\0 \end{pmatrix} \otimes \begin{pmatrix} 1\\0 \end{pmatrix} = \begin{pmatrix} 1\\0\\0\\0 \end{pmatrix} \\
|01 \rangle &= |0\rangle \otimes |1\rangle = \begin{pmatrix} 1\\0 \end{pmatrix} \otimes \begin{pmatrix} 0\\1 \end{pmatrix} = \begin{pmatrix} 0\\1\\0\\0 \end{pmatrix} \\
|10 \rangle &= |1\rangle \otimes |0\rangle = \begin{pmatrix} 0\\1 \end{pmatrix} \otimes \begin{pmatrix} 1\\0 \end{pmatrix} = \begin{pmatrix} 0\\0\\1\\0 \end{pmatrix} \\
|11 \rangle &= |1\rangle \otimes |1\rangle = \begin{pmatrix} 0\\1 \end{pmatrix} \otimes \begin{pmatrix} 0\\1 \end{pmatrix} = \begin{pmatrix} 0\\0\\0\\1 \end{pmatrix}
\end{align}

```python
qubit1_state = np.matrix([[1],[0]])
qubit2_state = np.matrix([[0],[1]])

two_qubit_state =np.kron(qubit1_state, qubit2_state)
```

```python

```

# Task Three: Qunatum Gates and Circuits

```python
from qiskit import *
from qiskit.visualization import plot_bloch_multivector
```

## Pauli Matrices

\begin{align}
I = \begin{pmatrix} 1&0 \\ 0&1 \end{pmatrix}, \quad
X = \begin{pmatrix} 0&1 \\ 1&0 \end{pmatrix}, \quad
Y = \begin{pmatrix} 0&i \\ -i&0 \end{pmatrix}, \quad
Z = \begin{pmatrix} 1&0 \\ 0&-1 \end{pmatrix} \quad
\end{align}

## X-gate

The X-gate is represented by the Pauli-X matrix:

$$ X = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} = |0\rangle\langle1| + |1\rangle\langle0| $$

Effect a gate has on a qubit:

$$ X|0\rangle = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix} = |1\rangle$$

```python
# Let's do an X-gate on a |0> qubit
qc = QuantumCircuit(1)
qc.x(0)
qc.draw('mpl')
```

```python
# Let's see the result
backend = Aer.get_backend('statevector_simulator')
out = execute(qc,backend).result().get_statevector()
plot_bloch_multivector(out)
```

## Z & Y-Gate

$$ Y = \begin{bmatrix} 0 & -i \\ i & 0 \end{bmatrix} \quad\quad\quad\quad Z = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} $$

$$ Y = -i|0\rangle\langle1| + i|1\rangle\langle0| \quad\quad Z = |0\rangle\langle0| - |1\rangle\langle1| $$

```python
qc.y(0) # Do Y-gate on qubit 0
qc.z(0) # Do Z-gate on qubit 0
qc.draw('mpl')
```

## Hadamard Gate

$$ H = \tfrac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} $$

We can see that this performs the transformations below:

$$ H|0\rangle = |+\rangle $$

$$ H|1\rangle = |-\rangle $$

```python
qc = QuantumCircuit(3)
# Apply H-gate to each qubit:
for qubit in range(3):
    qc.h(qubit)
# See the circuit:
qc.draw('mpl')
```

## Identity Gate

$$
I = \begin{bmatrix} 1 & 0 \\ 0 & 1\end{bmatrix}
$$

$$ I = XX $$

```python
qc = QuantumCircuit(1)
#qc.i(0)
qc.draw('mpl')
```

 ** Other Gates: S-gate , T-gate, U-gate

## Multiple Qubits, Entanglement

## Multiple Qubits

The state of two qubits :

$$ |psi\rangle = a_{00}|00\rangle + a_{01}|01\rangle + a_{10}|10\rangle + a_{11}|11\rangle = \begin{bmatrix} a_{00} \\ a_{01} \\ a_{10} \\ a_{11} \end{bmatrix} $$

```python
from qiskit import *
```

```python
qc = QuantumCircuit(3)

for qubit in range(3):
    qc.h(qubit)

qc.draw('mpl')
```

Each qubit is in the state $|+\rangle$, so we should see the vector:

$$
|{+++}\rangle = \frac{1}{\sqrt{8}}\begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \\
                              1 \\ 1 \\ 1 \\ 1 \\
              \end{bmatrix}
$$

```python
backend = Aer.get_backend('statevector_simulator')
final_state= execute(qc, backend).result().get_statevector()
print(final_state)
```

    [0.35355339+0.j 0.35355339+0.j 0.35355339+0.j 0.35355339+0.j
     0.35355339+0.j 0.35355339+0.j 0.35355339+0.j 0.35355339+0.j]

```python
qc = QuantumCircuit(2)

qc.x(0)
qc.h(1)

qc.draw('mpl')
```

$$
X|q_1\rangle \otimes H|q_0\rangle = (X\otimes H)|q_1 q_0\rangle
$$

The operation looks like this:

$$
X\otimes H = \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \otimes \tfrac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} = \frac{1}{\sqrt{2}}
\begin{bmatrix} 0 & 0 & 1 & 1 \\
                0 & 0 & 1 & -1 \\
                1 & 1 & 0 & 0 \\
                1 & -1 & 0 & 0 \\
\end{bmatrix}
$$

Which we can then apply to our 4D statevector $|q_1 q_0\rangle$. You will often see the clearer notation:

$$
X\otimes H =
\begin{bmatrix} 0 & H \\
               H & 0\\
\end{bmatrix}
$$

## C-Not Gate

```python
qc = QuantumCircuit(2)

qc.cx(0, 1)

qc.draw('mpl')

```

Classical truth table of C-Not gate:

| Input (t,c) | Output (t,c) |
|:-----------:|:------------:|
| 00          | 00           |
| 01          | 11           |
| 10          | 10           |
| 11          | 01           |

## Entanglement

```python
qc = QuantumCircuit(2)
# Apply H-gate to the first:
qc.h(0)
qc.draw('mpl')
```

```python
# Let's see the result:
backend = Aer.get_backend('statevector_simulator')
final_state = execute(qc,backend).result().get_statevector()
print(final_state)
```

    [0.70710678+0.j 0.70710678+0.j 0.        +0.j 0.        +0.j]

Quantum System Sate is:

$$
|0{+}\rangle = \tfrac{1}{\sqrt{2}}(|00\rangle + |01\rangle)
$$

```python
qc = QuantumCircuit(2)
# Apply H-gate to the first:
qc.h(0)
# Apply a CNOT:
qc.cx(0,1)
qc.draw('mpl')
```

```python
# Let's see the result:
backend = Aer.get_backend('statevector_simulator')
final_state = execute(qc,backend).result().get_statevector()
print(final_state)
```

    [0.70710678+0.j 0.        +0.j 0.        +0.j 0.70710678+0.j]

We see we have this final state (Bell State):

$$
\text{CNOT}|0{+}\rangle = \tfrac{1}{\sqrt{2}}(|00\rangle + |11\rangle)
$$

Other Bell States:

## Bernstein-Vazirani Algorithm

A black-box function $f$, which takes as input a string of bits ($x$), and returns either $0$ or $1$, that is:

$$f(\{x_0,x_1,x_2,...\}) \rightarrow 0 \textrm{ or } 1 \textrm{ where } x_n \textrm{ is }0 \textrm{ or } 1  $$

The function is guaranteed to return the bitwise product of the input with some string, $s$.

In other words, given an input $x$, $f(x) = s \cdot x \, \text{(mod 2)} =\ x_0 * s_0+x_1*s_1+x_2*s_2+...\ $ mod 2

# Classical-solution:

|Input(x)|
|:-----:|
|100...0|
|010...0|
|001...0|
|000...1|

Where each query reveals a different bit of $s$ (the bit $s_i$)

# The quantum Bernstein-Vazirani Oracle:

1. Initialise the inputs qubits to the $|0\rangle^{\otimes n}$ state, and output qubit to $|{-}\rangle$.
2. Apply Hadamard gates to the input register
3. Query the oracle
4. Apply Hadamard gates to the input register
5. Measure

## Example Two Qubits:

 The register of two qubits is initialized to zero:

$$\lvert \psi_0 \rangle = \lvert 0 0 \rangle$$

 Apply a Hadamard gate to both qubits:

$$\lvert \psi_1 \rangle = \frac{1}{2} \left( \lvert 0 0 \rangle + \lvert 0 1 \rangle + \lvert 1 0 \rangle + \lvert 1 1 \rangle \right) $$

 For the string $s=11$, the quantum oracle performs the operation:
$$
|x \rangle \xrightarrow{f_s} (-1)^{x\cdot 11} |x \rangle.
$$

$$\lvert \psi_2 \rangle = \frac{1}{2} \left( (-1)^{00\cdot 11}|00\rangle + (-1)^{01\cdot 11}|01\rangle + (-1)^{10\cdot 11}|10\rangle + (-1)^{11\cdot 11}|11\rangle \right)$$

$$\lvert \psi_2 \rangle = \frac{1}{2} \left( \lvert 0 0 \rangle - \lvert 0 1 \rangle - \lvert 1 0 \rangle + \lvert 1 1 \rangle \right)$$

 Apply a Hadamard gate to both qubits:

$$\lvert \psi_3 \rangle = \lvert 1 1 \rangle$$

 Measure to find the secret string $s=11$

```python
from qiskit import *
%matplotlib inline
from qiskit.tools.visualization import plot_histogram
```

```python
secret_num = '101001'
```

```python
circuit = QuantumCircuit(6+1,6)
circuit.h([0,1,2,3,4,5])

circuit.x(6)
circuit.h(6)

circuit.barrier()

circuit.cx(5,6)
circuit.cx(3,6)
circuit.cx(0,6)

circuit.barrier()
circuit.h([0,1,2,3,4,5])

circuit.measure([0,1,2,3,4,5],[0,1,2,3,4,5])
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f3bde001ad0>

```python
circuit.draw(output= 'mpl')
```

```python
simulator = Aer.get_backend('qasm_simulator')
result = execute(circuit, backend= simulator, shots = 1).result()
counts= result.get_counts()
print(counts)
```

    {'101001': 1}

```python

```