# Deep Reinforcement Learning for the CartPole Environment

```python
# Install packages
import gym
import copy
import torch
from torch.autograd import Variable
import random
import matplotlib.pyplot as plt
from PIL import Image
from IPython.display import clear_output
import math
import torchvision.transforms as T
import numpy as np

import time
```

## Environment

The CartPole environment consists of a pole which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. The state space is represented by four values: cart position, cart velocity, pole angle, and the velocity of the tip of the pole. The action space consists of two actions: moving left or moving right. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.

Source: [https://gym.openai.com/envs/CartPole-v1/](Open AI Gym).

The cell below plots a bunch of example frames from the environment.

```python
env = gym.envs.make("CartPole-v1")
```

```python
# Demonstration
env = gym.envs.make("CartPole-v1")

def get_screen():
    ''' Extract one step of the simulation.'''
    screen = env.render(mode='rgb_array').transpose((2, 0, 1))
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255.
    return torch.from_numpy(screen)

# Speify the number of simulation steps
num_steps = 2

# Show several steps
for i in range(num_steps):
    clear_output(wait=True)
    env.reset()
    plt.figure()
    plt.imshow(get_screen().cpu().permute(1, 2, 0).numpy(),
               interpolation='none')
    plt.title('CartPole-v0 Environment')
    plt.xticks([])
    plt.yticks([])
    plt.show()
```

## Plotting Function

This function will make it possible to analyze how the agent learns over time. The resulting plot consists of two subplots. The first one plots the total reward the agent accumulates over time, while the other plot shows a histogram of the agent's total rewards for the last 50 episodes.

```python
def plot_res(values, title=''):
    ''' Plot the reward curve and histogram of results over time.'''
    # Update the window after each episode
    clear_output(wait=True)

    # Define the figure
    f, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,5))
    f.suptitle(title)
    ax[0].plot(values, label='score per run')
    ax[0].axhline(195, c='red',ls='--', label='goal')
    ax[0].set_xlabel('Episodes')
    ax[0].set_ylabel('Reward')
    x = range(len(values))
    ax[0].legend()
    # Calculate the trend
    try:
        z = np.polyfit(x, values, 1)
        p = np.poly1d(z)
        ax[0].plot(x,p(x),"--", label='trend')
    except:
        print('')

    # Plot the histogram of results
    ax[1].hist(values[-50:])
    ax[1].axvline(195, c='red', label='goal')
    ax[1].set_xlabel('Scores per Last 50 Episodes')
    ax[1].set_ylabel('Frequency')
    ax[1].legend()
    plt.show()
```

## Random Search

Before implementing any deep learning approaches, I wrote a simple strategy where the action is sampled randomly from the action space. This approach will serve as a baseline for other strategies and will make it easier to understand how to work with the agent using the Open AI Gym environment.

```python
def random_search(env, episodes,
                  title='Random Strategy'):
    """ Random search strategy implementation."""
    final = []
    for episode in range(episodes):
        state = env.reset()
        done = False
        total = 0
        while not done:
            # Sample random actions
            action = env.action_space.sample()
            # Take action and extract results
            next_state, reward, done, _ = env.step(action)
            # Update reward
            total += reward
            if done:
                break
        # Add to the final reward
        final.append(total)
        plot_res(final,title)
    return final
```

```python
# Get random search results
episodes = 30
random_s = random_search(env, episodes)
```

The plot above presents the random strategy. As expected, it's impossible to solve the environment using this approach. The agent is not learning from their experience. Despite being lucky sometimes (getting a reward of almost 75), their average performance is as low as 10 steps.

## Deep Q Learning

The main idea behind Q-learning is that we have a function $Q: State \times Action \rightarrow \mathbb{R}$, which can tell the agent what actions will result in what rewards. If we know the value of Q, it is possible to construct a policy that maximizes rewards:

\begin{align}\pi(s) = \arg\!\max_a \ Q(s, a)\end{align}

However, in the real world, we don't have access to full information, that's why we need to come up with ways of approximating Q. One traditional method is creating a lookup table where the values of Q are updated after each of the agent's actions. However, this approach is slow and does not scale to large action and state spaces. Since neural networks are universal function approximators, I will train a network that can approximate $Q$.

The DQL class implementation consists of a simple neural network implemented in PyTorch that has two main methods--predict and update. The network takes the agent's state as an input and returns the Q values for each of the actions. The maximum Q value is selected by the agent to perform the next action.

```python
class DQN():
    ''' Deep Q Neural Network class. '''
    def __init__(self, state_dim, action_dim, hidden_dim=64, lr=0.05):
            self.criterion = torch.nn.MSELoss()
            self.model = torch.nn.Sequential(
                            torch.nn.Linear(state_dim, hidden_dim),
                            torch.nn.LeakyReLU(),
                            torch.nn.Linear(hidden_dim, hidden_dim*2),
                            torch.nn.LeakyReLU(),
                            torch.nn.Linear(hidden_dim*2, action_dim)
                    )
            self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, state, y):
        """Update the weights of the network given a training sample. """
        y_pred = self.model(torch.Tensor(state))
        loss = self.criterion(y_pred, Variable(torch.Tensor(y)))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    def predict(self, state):
        """ Compute Q values for all actions using the DQL. """
        with torch.no_grad():
            return self.model(torch.Tensor(state))
```

The q_learning function is the main loop for all the algorithms that follow.
It has many parameters, namely:

- Env represents the Open Ai Gym environment that we want to solve (CartPole.)
- Episodes stand for the number of games we want to play (from the beginning until the end.)
- Gamma is a discounting factor that is multiplied by future rewards to dampen these rewards' effect on the agent. It is designed to make future rewards worth less than immediate rewards.
- Epsilon represents the proportion of random actions relative to actions that are informed by existing "knowledge" that the agent accumulates during the episode. Before playing the game, the agent doesn't have any experience, so it is common to set epsilon to higher values and then gradually decrease its value.
- Eps_decay parameter indicates the speed at which the epsilon decreases as the agent learns. 0.99 comes from the original DQN paper.

I will explain other parameters later on when we will get to the corresponding agents.
The most straightforward agent updates its Q-values based on its most recent observation. It doesn't have any memory, but it learns by first exploring the environment and the gradually decreasing its epsilon value to make informed decisions:

```python
def q_learning(env, model, episodes, gamma=0.9,
               epsilon=0.3, eps_decay=0.99,
               replay=False, replay_size=20,
               title = 'DQL', double=False,
               n_update=10, soft=False, verbose=True):
    """Deep Q Learning algorithm using the DQN. """
    final = []
    memory = []
    episode_i=0
    sum_total_replay_time=0

    _max = [0.0, 0.0, 0.0, 0.0]

    for episode in range(episodes):
        episode_i+=1
        if double and not soft:
            # Update target network every n_update steps
            if episode % n_update == 0:
                model.target_update()
        if double and soft:
            model.target_update()

        # Reset state
        state = env.reset()
        done = False
        total = 0

        while not done:
            # Implement greedy search policy to explore the state space
            if random.random() < epsilon:
                action = env.action_space.sample()
            else:
                q_values = model.predict(state)
                action = torch.argmax(q_values).item()

            # Take action and add reward to total
            next_state, reward, done, _ = env.step(action)

            # Update total and memory
            total += reward
            memory.append((state, action, next_state, reward, done))
            q_values = model.predict(state).tolist()

            if done:
                if not replay:
                    q_values[action] = reward
                    # Update network weights
                    model.update(state, q_values)
                break

            if replay:
                t0=time.time()
                # Update network weights using replay memory
                model.replay(memory, replay_size, gamma)
                t1=time.time()
                sum_total_replay_time+=(t1-t0)
            else:
                # Update network weights using the last step only
                q_values_next = model.predict(next_state)
                q_values[action] = reward + gamma * torch.max(q_values_next).item()
                model.update(state, q_values)

            for _i in range(len(_max)):
                if np.abs(state[i]) > _max[i]:
                    _max[i] = np.abs(state[i])

            state = next_state

        # Update epsilon
        epsilon = max(epsilon * eps_decay, 0.01)
        final.append(total)
        plot_res(final, title)

        if verbose:
            print("episode: {}, total reward: {}".format(episode_i, total))
            if replay:
                print("Average replay time:", sum_total_replay_time/episode_i)

    print(_max)
    return final
```

### Parameters

```python
# Number of states
n_state = env.observation_space.shape[0]
# Number of actions
n_action = env.action_space.n
# Number of episodes
episodes = 150
# Number of hidden nodes in the DQN
n_hidden = 50
# Learning rate
lr = 0.001
```

```python
# Get DQN results
simple_dqn = DQN(n_state, n_action, n_hidden, lr)
simple = q_learning(env, simple_dqn, episodes, gamma=.9, epsilon=0.3)
```

The graph above shows that the performance of the agent has significantly improved. It got to 175 steps, which, as we've seen before, is impossible for a random agent. The trend line is also positive, and we can see that the performance increases over time. At the same time, the agent didn't manage to get above the goal line after 150 epochs, and its average performance is still around 15 steps, so there is definitely enough room for improvement.

## Replay

The approximation of Q using one sample at a time is not very effective. The graph above is a nice illustration of that. The network managed to achieve a much better performance compared to a random agent. However, it couldn't get to the threshold line of 195 steps. I implemented experience replay to improve network stability and make sure previous experiences are not discarded but used in training.

Experience replay stores the agent's experiences in memory. Batches of experiences are randomly sampled from memory and are used to train the neural network. Such learning consists of two phases--gaining experience and updating the model. The size of the replay controls the number of experiences that are used for the network update. Memory is an array that stores the agent's state, reward, and action, as well as whether the action finished the game and the next state.

```python
# Expand DQL class with a replay function.
class DQN_replay(DQN):
    #old replay function
    #def replay(self, memory, size, gamma=0.9):
        #""" Add experience replay to the DQN network class. """
        # Make sure the memory is big enough
        #if len(memory) >= size:
            #states = []
            #targets = []
            # Sample a batch of experiences from the agent's memory
            #batch = random.sample(memory, size)

            # Extract information from the data
            #for state, action, next_state, reward, done in batch:
                #states.append(state)
                # Predict q_values
                #q_values = self.predict(state).tolist()
                #if done:
                    #q_values[action] = reward
                #else:
                    #q_values_next = self.predict(next_state)
                    #q_values[action] = reward + gamma * torch.max(q_values_next).item()

                #targets.append(q_values)

            #self.update(states, targets)

    #new replay function
    def replay(self, memory, size, gamma=0.9):
        """New replay function"""
        #Try to improve replay speed
        if len(memory)>=size:
            batch = random.sample(memory,size)
            batch_t = list(map(list, zip(*batch))) #Transpose batch list
            states = batch_t[0]
            actions = batch_t[1]
            next_states = batch_t[2]
            rewards = batch_t[3]
            is_dones = batch_t[4]

            states = torch.Tensor(states)
            actions_tensor = torch.Tensor(actions)
            next_states = torch.Tensor(next_states)
            rewards = torch.Tensor(rewards)
            is_dones_tensor = torch.Tensor(is_dones)

            is_dones_indices = torch.where(is_dones_tensor==True)[0]

            all_q_values = self.model(states) # predicted q_values of all states
            all_q_values_next = self.model(next_states)
            #Update q values
            all_q_values[range(len(all_q_values)),actions]=rewards+gamma*torch.max(all_q_values_next, axis=1).values
            all_q_values[is_dones_indices.tolist(), actions_tensor[is_dones].tolist()]=rewards[is_dones_indices.tolist()]

            self.update(states.tolist(), all_q_values.tolist())
```

### replay using old replay function

```python
# Get replay results
dqn_replay = DQN_replay(n_state, n_action, n_hidden, lr)
replay = q_learning(env, dqn_replay,
                    episodes, gamma=.9,
                    epsilon=0.2, replay=True,
                    title='DQL with Replay')
```

### replay using new replay function

```python
# Get replay results
dqn_replay = DQN_replay(n_state, n_action, n_hidden, lr)
replay = q_learning(env, dqn_replay,
                    episodes, gamma=.9,
                    epsilon=0.2, replay=True,
                    title='DQL with Replay')
```

As expected, the neural network with the replay seems to be much more robust and smart compared to its counterpart that only remembers the last action. After approximately 60 episodes, the agent managed to achieve the winning threshold and remain at this level. I also managed to achieve the highest reward possible--500.

## Double Q Learning

Traditional Deep Q Learning tends to overestimate the reward, which leads to unstable training and lower quality policy. Let's consider the equation for the Q value:

The last part of the equation takes the estimate of the maximum value. This procedure results in systematic overestimation, which introduces a maximization bias. Since Q-learning involves learning estimates from estimates, such overestimation is especially worrying.

To avoid such a situation, I will define a new target network. The Q values will be taken from this new network, which is meant to reflect the state of the main DQN. However, it doesn't have identical weights because it's only updated after a certain number of episodes. This idea has been first introduced in Hasselt et al., 2015.
The addition of the target network might slow down the training since the target network is not continuously updated. However, it should have a more robust performance over time.

n_update parameter specifies the interval, after which the target network should be updated.

```python
class DQN_double(DQN):
    def __init__(self, state_dim, action_dim, hidden_dim, lr):
        super().__init__(state_dim, action_dim, hidden_dim, lr)
        self.target = copy.deepcopy(self.model)

    def target_predict(self, s):
        ''' Use target network to make predicitons.'''
        with torch.no_grad():
            return self.target(torch.Tensor(s))

    def target_update(self):
        ''' Update target network with the model weights.'''
        self.target.load_state_dict(self.model.state_dict())

    def replay(self, memory, size, gamma=1.0):
        ''' Add experience replay to the DQL network class.'''
        if len(memory) >= size:
            # Sample experiences from the agent's memory
            data = random.sample(memory, size)
            states = []
            targets = []
            # Extract datapoints from the data
            for state, action, next_state, reward, done in data:
                states.append(state)
                q_values = self.predict(state).tolist()
                if done:
                    q_values[action] = reward
                else:
                    # The only difference between the simple replay is in this line
                    # It ensures that next q values are predicted with the target network.
                    q_values_next = self.target_predict(next_state)
                    q_values[action] = reward + gamma * torch.max(q_values_next).item()

                targets.append(q_values)

            self.update(states, targets)
```

```python
# Get replay results
dqn_double = DQN_double(n_state, n_action, n_hidden, lr)
double =  q_learning(env, dqn_double, episodes, gamma=.9,
                    epsilon=0.2, replay=True, double=True,
                    title='Double DQL with Replay', n_update=10)

```

Double DQL with replay has outperformed the previous version and has consistently performed above 300 steps. The performance also seems to be a bit more stable, thanks to the separation of action selection and evaluation. Finally, let's explore the last modification to the DQL agent.

## Soft Target Update
The method used to update the target network implemented above was introduced in the original DQN paper. In this section, we will explore another well-established method of updating the target network weights. Instead of updating weights after a certain number of steps, we will incrementally update the target network after every run using the following formula:

target_weights = target_weights * (1-TAU) + model_weights * TAU

where 0 < TAU < 1

This method of updating the target network is known as “soft target network updates” and was introduced in  Lillicrap et al., 2016. Method implementation is shown below:

```python
class DQN_double_soft(DQN_double):
    def target_update(self, TAU=0.1):
        ''' Update the targer gradually. '''
        # Extract parameters
        model_params = self.model.named_parameters()
        target_params = self.target.named_parameters()

        updated_params = dict(target_params)

        for model_name, model_param in model_params:
            if model_name in target_params:
                # Update parameter
                updated_params[model_name].data.copy_((TAU)*model_param.data + (1-TAU)*target_params[model_param].data)

        self.target.load_state_dict(updated_params)
```

```python
dqn_double_soft = DQN_double_soft(n_state, n_action, n_hidden, lr)
double =  q_learning(env, dqn_double_soft, episodes, gamma=.9,
                    epsilon=0.2, replay=True, double=True,
                    title='Double DQL with Replay', n_update=10, soft=True)
```

The network with soft target updates performed quite well. However, it doesn't seem to be better than hard weight updates after a certain number of steps.

## Conclusion

The implementation of the experience replay and the target network have significantly improved the performance of a Deep Q Learning agent in the Open AI CartPole environment. Some other modifications to the agent, such as Dueling Network Architectures (Wang et al., 2015), can be added to this implementation to improve the agent's performance. The algorithm is also generalizable to other environments. Thus, it's possible to test how well it performs on other tasks.

## References:

(1) Reinforcement Q-Learning from Scratch in Python with OpenAI Gym. (2019). Learndatasci.com. Retrieved 9 December 2019, from https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/

(2) Paszke, A., (2019). Reinforcement Learning (DQN) tutorial. Retrieved from: https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html

(3) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., ... & Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.

(4) Van Hasselt, H., Guez, A., & Silver, D. (2016, March). Deep reinforcement learning with double q-learning. In Thirtieth AAAI conference on artificial intelligence.

(5) Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., & De Freitas, N. (2015). Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581.

(6) Double DQN Implementation to Solve OpenAI Gym’s CartPole v-0. (2019). Medium. Retrieved 20 December 2019, from https://medium.com/@leosimmons/double-dqn-implementation-to-solve-openai-gyms-cartpole-v-0-df554cd0614d


```python
%load_ext Cython
```

    /home/rotis/anaconda/envs/calphadpy3/lib/python3.5/site-packages/Cython/Distutils/old_build_ext.py:30: UserWarning: Cython.Distutils.old_build_ext does not properly handle dependencies and is deprecated.
      "Cython.Distutils.old_build_ext does not properly handle dependencies "

```cython
%%cython
from pycalphad.core.rksum import RedlichKisterSum
from pycalphad import Database
cimport numpy as np
import numpy as np
from tinydb import where
from sympy import Symbol

cdef np.ndarray[ndim=1, dtype=np.float64_t] _eval_rk_matrix_gradient(double[:,:] coef_mat, double[:,:] symbol_mat,
                                   double[:] eval_row, double[:] parameters):
    cdef np.ndarray[ndim=1, dtype=np.float64_t] out = np.zeros(eval_row.shape[0]-2)
    cdef double result = 0
    cdef double prod_result
    cdef int row_idx1 = 0
    cdef int row_idx2 = 0
    cdef int col_idx = 0
    cdef int dof_idx
    # eval_row order: P,T,ln(P),ln(T),y...
    # dof order: P,T,y...
    # coef_mat order: low_temp,high_temp,P,T,ln(P),ln(T),y...,constant_term,parameter_value
    for dof_idx in range(eval_row.shape[0]-2):
        if coef_mat.shape[1] > 0:
            for row_idx1 in range(coef_mat.shape[0]):
                if (eval_row[1] >= coef_mat[row_idx1, 0]) and (eval_row[1] < coef_mat[row_idx1, 1]):
                    if dof_idx < 2:
                        # special handling for state variables since they also can have a ln term
                        if coef_mat[row_idx1, 2+dof_idx] != 0:
                            prod_result = coef_mat[row_idx1, coef_mat.shape[1]-2] * coef_mat[row_idx1, coef_mat.shape[1]-1]
                            for col_idx in range(coef_mat.shape[1]-4):
                                if col_idx == dof_idx:
                                    prod_result *= (eval_row[col_idx]**(coef_mat[row_idx1, 2+col_idx]-1) * eval_row[2+col_idx]**(coef_mat[row_idx1, 4+col_idx]-1) * (coef_mat[row_idx1, 4+col_idx]+coef_mat[row_idx1, 2+col_idx]*(eval_row[2+col_idx])))
                                else:
                                    prod_result *= (eval_row[col_idx] ** coef_mat[row_idx1, 2+col_idx])
                            out[dof_idx] += prod_result
                    else:
                        if coef_mat[row_idx1, 4+dof_idx] != 0:
                            prod_result = coef_mat[row_idx1, coef_mat.shape[1]-2] * coef_mat[row_idx1, coef_mat.shape[1]-1]
                            for col_idx in range(coef_mat.shape[1]-4):
                                if col_idx == 2+dof_idx:
                                    prod_result *= (coef_mat[row_idx1, 4+dof_idx] * eval_row[col_idx] ** (coef_mat[row_idx1, 2+col_idx] - 1))
                                else:
                                    prod_result *= (eval_row[col_idx] ** coef_mat[row_idx1, 2+col_idx])
                            out[dof_idx] += prod_result
        if symbol_mat.shape[1] > 0:
            for row_idx2 in range(symbol_mat.shape[0]):
                if (eval_row[1] >= symbol_mat[row_idx2, 0]) and (eval_row[1] < symbol_mat[row_idx2, 1]):
                    if dof_idx < 2:
                        # special handling for state variables since they also can have a ln term
                        if coef_mat[row_idx2, 2+dof_idx] != 0:
                            prod_result = coef_mat[row_idx2, coef_mat.shape[1]-2] * parameters[<int>symbol_mat[row_idx2, symbol_mat.shape[1]-1]]
                            for col_idx in range(coef_mat.shape[1]-4):
                                if col_idx == dof_idx:
                                    prod_result *= (eval_row[col_idx]**(coef_mat[row_idx1, 2+col_idx]-1) * eval_row[2+col_idx]**(coef_mat[row_idx1, 4+col_idx]-1) * (coef_mat[row_idx1, 4+col_idx]+coef_mat[row_idx1, 2+col_idx]*(eval_row[2+col_idx])))
                                else:
                                    prod_result *= (eval_row[col_idx] ** coef_mat[row_idx1, 2+col_idx])
                            out[dof_idx] += prod_result
                    else:
                        if coef_mat[row_idx1, 4+dof_idx] != 0:
                            prod_result = coef_mat[row_idx2, coef_mat.shape[1]-2] * parameters[<int>symbol_mat[row_idx2, symbol_mat.shape[1]-1]]
                            for col_idx in range(coef_mat.shape[1]-4):
                                if col_idx == 2+dof_idx:
                                    prod_result *= (coef_mat[row_idx2, 4+dof_idx] * eval_row[col_idx] ** (coef_mat[row_idx2, 2+col_idx] - 1))
                                else:
                                    prod_result *= (eval_row[col_idx] ** coef_mat[row_idx2, 2+col_idx])
                            out[dof_idx] += prod_result
    return out

dbf = Database('alcocr-sandbox.tdb')
param_query = (
    (where('phase_name') == 'LIQUID') & \
    (where('parameter_type') == "G")
)
parameters = {}
all_symbols = dbf.symbols.copy()
# Convert string symbol names to sympy Symbol objects
# This makes xreplace work with the symbols dict
all_symbols = dict([(Symbol(s), val) for s, val in all_symbols.items()])
for param in parameters.keys():
    all_symbols.pop(param, None)
rks = RedlichKisterSum(['AL','CO','CR','VA'], dbf.phases['LIQUID'], dbf.search, param_query, list(parameters.keys()), all_symbols)
exx = _eval_rk_matrix_gradient(rks.output_matrix[2:4,:], np.array([[]]),
                        np.array([1013235, 300, np.log(101325), np.log(300), 0.3, 0.3, 0.4]), np.array([]))
print(exx)
```

    [  0.00000000e+00  -3.55256010e+00  -1.15287101e+04   0.00000000e+00
       0.00000000e+00]

```python
rks.output_matrix[2:4,:]
```

    array([[  2.98150000e+02,   9.33600000e+02,   0.00000000e+00,
              1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.00000000e+00,  -1.18418670e+01],
           [  2.98150000e+02,   7.00000000e+02,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              1.00000000e+00,  -7.97615000e+03]])

```python
-1.18418670e01 * 300 + -7.97615000e03
```

    -11528.7101

```python

```


# Quantum pipeline using t|ket> backend
This performs a shots-based simulation.

```python
import numpy as np

def read_data(filename):
    labels, sentences = [], []
    with open(filename) as f:
        for line in f:
            labels.append([1, 0] if line[0] == '1' else [0, 1])
            sentences.append(line[1:].strip())
    return np.array(labels), sentences

train_labels, train_data = read_data('datasets/mc_train_data.txt')
dev_labels, dev_data = read_data('datasets/mc_dev_data.txt')
test_labels, test_data = read_data('datasets/mc_test_data.txt')
```

### Create diagrams

```python
from lambeq.ccg2discocat import DepCCGParser

reader = DepCCGParser(possible_root_cats=['S[dcl]'])

raw_train_diagrams = reader.sentences2diagrams(train_data)
raw_dev_diagrams = reader.sentences2diagrams(dev_data)
raw_test_diagrams = reader.sentences2diagrams(test_data)
```

```python
from discopy.rigid import Id

def remove_cups(diagram):
    # Remove cups to reduce post-selection in the circuit, for faster execution

    diags = []
    for box, offset in zip(diagram.boxes, diagram.offsets):
        if not box.dom:  # word box
            diags.insert(offset, box)
        else:  # cup (the only other type of box in these diagrams)
            i = 0
            off = offset
            while off != len(diags[i].cod) - 1:
                assert off > 0
                off -= len(diags[i].cod)
                i += 1
            left, right = diags[i:i+2]

            if len(left.cod) == 1:
                new_diag = right >> (left.r.dagger() @ Id(right.cod[1:]))
            else:
                assert len(right.cod) == 1
                new_diag = left >> (Id(left.cod[:-1]) @ right.l.dagger())

            diags[i:i+2] = [new_diag]

    assert len(diags) == 1
    return diags[0]

train_diagrams = [remove_cups(diagram) for diagram in raw_train_diagrams]
dev_diagrams = [remove_cups(diagram) for diagram in raw_dev_diagrams]
test_diagrams = [remove_cups(diagram) for diagram in raw_test_diagrams]

train_diagrams[0].draw()
```

### Create circuits

```python
from lambeq.circuit import IQPAnsatz
from lambeq.core.types import AtomicType

ansatz = IQPAnsatz({AtomicType.NOUN: 1, AtomicType.SENTENCE: 1},
                   n_layers=1, n_single_qubit_params=3)

train_circuits = [ansatz(diagram) for diagram in train_diagrams]
dev_circuits =  [ansatz(diagram) for diagram in dev_diagrams]
test_circuits = [ansatz(diagram) for diagram in test_diagrams]

train_circuits[0].draw(figsize=(9, 12))
```

### Parameterise

```python
from sympy import default_sort_key

all_circuits = train_circuits + dev_circuits + test_circuits

# sort the symbols since they are returned as a set
parameters = sorted(
    {s for circ in all_circuits for s in circ.free_symbols},
    key=default_sort_key)
```

```python
## CONFIGURE BACKEND
# This uses Qiskit, which requires installing `pytket-qiskit`.
# For more information on how to use pytket with different backends, see:
# https://cqcl.github.io/pytket/build/html/extensions/

# Aer is a shots-based backend that is always available.

from pytket.extensions.qiskit import AerBackend
backend = AerBackend()

## Having an IBMQ account gives access to the IBM emulator for noisy simulations:
#
# from pytket.extensions.qiskit import IBMQEmulatorBackend
# backend = IBMQEmulatorBackend('PLACEHOLDER')
#
## To use a real device:
#
# from pytket.extensions.qiskit import IBMQBackend
# backend = IBMQBackend('PLACEHOLDER')

backend_config = {
    'backend': backend,
    'compilation': backend.default_compilation_pass(2),
    'n_shots': 8192  # maximum recommended shots, reduces sampling error
}
```

```python
from discopy.quantum import Circuit, Id, Measure

def randint(rng, low=-1 << 63, high=1 << 63-1):
    return rng.integers(low, high)

def normalise(predictions):
    # apply smoothing to predictions
    predictions = np.abs(predictions) + 1e-9
    return predictions / predictions.sum()

def make_pred_fn(circuits, rng):
    measured_circuits = [c >> Id().tensor(*[Measure()] * len(c.cod)) for c in circuits]
    circuit_fns = [c.lambdify(*parameters) for c in measured_circuits]

    def predict(params):
        outputs = Circuit.eval(*(c_fn(*params) for c_fn in circuit_fns),
                               **backend_config, seed=randint(rng))
        return np.array([normalise(output.array) for output in outputs])
    return predict

SEED = 0
rng = np.random.default_rng(SEED)

train_pred_fn = make_pred_fn(train_circuits, rng)
dev_pred_fn = make_pred_fn(dev_circuits, rng)
test_pred_fn = make_pred_fn(test_circuits, rng)
```

### Train

```python
from noisyopt import minimizeSPSA

def make_cost_fn(pred_fn, labels):
    def cost_fn(params, **kwargs):
        predictions = pred_fn(params)

        cost = -np.sum(labels * np.log(predictions)) / len(labels)  # binary cross-entropy loss
        costs.append(cost)

        acc = np.sum(np.round(predictions) == labels) / len(labels) / 2  # half due to double-counting
        accuracies.append(acc)

        return cost

    costs, accuracies = [], []
    return cost_fn, costs, accuracies

train_cost_fn, train_costs, train_accs = make_cost_fn(train_pred_fn, train_labels)
dev_cost_fn, dev_costs, dev_accs = make_cost_fn(dev_pred_fn, dev_labels)

x0 = np.array(rng.random(len(parameters)))
np.random.seed(SEED)
result = minimizeSPSA(train_cost_fn, x0=x0, a=0.2, c=0.06, niter=80, callback=dev_cost_fn)
```

### Show results

```python
import matplotlib.pyplot as plt

fig, ((ax_tl, ax_tr), (ax_bl, ax_br)) = plt.subplots(2, 2, sharex=True, sharey='row', figsize=(10, 6))
ax_tl.set_title('Training set')
ax_tr.set_title('Development set')
ax_bl.set_xlabel('Iterations')
ax_br.set_xlabel('Iterations')
ax_bl.set_ylabel('Accuracy')
ax_tl.set_ylabel('Loss')

colours = iter(plt.rcParams['axes.prop_cycle'].by_key()['color'])
ax_tl.plot(train_costs[1::2], color=next(colours))  # training evaluates twice per iteration
ax_bl.plot(train_accs[1::2], color=next(colours))   # so take every other entry
ax_tr.plot(dev_costs, color=next(colours))
ax_br.plot(dev_accs, color=next(colours))

# print test accuracy
test_cost_fn, _, test_accs = make_cost_fn(test_pred_fn, test_labels)
test_cost_fn(result.x)
print('Test accuracy:', test_accs[0])
```


# Including clouds

## Powerlaw cloud models

Let's initialize an atmosphere in the usual way, see ["Getting Started"](getting_started.html):

```python
import numpy as np
from petitRADTRANS import Radtrans

atmosphere = Radtrans(line_species = ['H2O', 'CO_all_iso', 'CH4', 'CO2', 'Na', 'K'], \
      rayleigh_species = ['H2', 'He'], \
      continuum_opacities = ['H2-H2', 'H2-He'], \
      wlen_bords_micron = [0.3, 15])

pressures = np.logspace(-6, 2, 100)
atmosphere.setup_opa_structure(pressures)
```

      Read CIA opacities for H2-H2...
      Read CIA opacities for H2-He...
     Done.

**Units in petitRADTRANS:** remember that all units in *petitRADTRANS* are in cgs, **except for pressure**, which is in bars, **and the mean molecular weight (MMW)**, which is in units of atomic mass units.

Next we set up the atmospheric parameters, with a parameter selection identical to the ["Getting Started"](getting_started.html) case.

```python
from petitRADTRANS import nat_cst as nc
R_pl = 1.838*nc.r_jup_mean
gravity = 1e1**2.45
P0 = 0.01

kappa_IR = 0.01
gamma = 0.4
T_int = 200.
T_equ = 1500.
temperature = nc.guillot_global(pressures, kappa_IR, gamma, gravity, T_int, T_equ)

mass_fractions = {}
mass_fractions['H2'] = 0.74 * np.ones_like(temperature)
mass_fractions['He'] = 0.24 * np.ones_like(temperature)
mass_fractions['H2O'] = 0.001 * np.ones_like(temperature)
mass_fractions['CO_all_iso'] = 0.01 * np.ones_like(temperature)
mass_fractions['CO2'] = 0.00001 * np.ones_like(temperature)
mass_fractions['CH4'] = 0.000001 * np.ones_like(temperature)
mass_fractions['Na'] = 0.00001 * np.ones_like(temperature)
mass_fractions['K'] = 0.000001 * np.ones_like(temperature)

MMW = 2.33 * np.ones_like(temperature)
```

**Abundances in petitRADTRANS:** remember that abundances in pRT are in units of **mass fractions**, not number fractions (aka volume mixing ratio, VMR). You can convert between mass fractions and VMRs by using
\begin{equation}
X_i = \frac{\mu_i}{\mu}n_i,
\end{equation}
where $X_i$ is the mass fraction of species $i$, $\mu_i$ the mass of a single molecule/atom/ion/... of species $i$, $\mu$ is the atmospheric mean molecular weight, and $n_i$ is the VMR of species $i$.

Now, let's calculate cloudy spectra! The first available mode is the so-called power law cloud. In this case the following opacity is *added* to the scattering cross-section:
\begin{equation}
\kappa = \kappa_0\left(\frac{\lambda}{\lambda_0}\right)^\gamma
\end{equation}
Where $\kappa_0$ is the opacity in units of cm$^2$/g, at $\lambda_0=0.35 \ {\rm \mu m}$. The power law index $\gamma$ fixes the wavelength dependence, Rayleigh-like scattering would be obtained for $\gamma=-4$. Hence $\kappa_0$ and $\gamma$ are free parameters.

We now calculate a clear atmosphere, and then four power law cloudy models, with $\kappa_0 = 0.01$ cm$^2$/g, and four different $\gamma$ values.

Clear model:

```python
import pylab as plt
plt.rcParams['figure.figsize'] = (10, 6)

# Clear
atmosphere.calc_transm(temperature, mass_fractions, \
                       gravity, MMW, R_pl=R_pl, P0_bar=P0)
clear = atmosphere.transm_rad/nc.r_jup_mean
```

First, $\gamma = -4$ (Rayleigh-like):

```python
kappa_zero = 0.01
gamma_scat = -4.

atmosphere.calc_transm(temperature, mass_fractions, \
                    gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                    kappa_zero = kappa_zero, gamma_scat = gamma_scat)

m4 = atmosphere.transm_rad/nc.r_jup_mean
```

Second, $\gamma = -2$ (weaker scattering power law):

```python
kappa_zero = 0.01
gamma_scat = -2.

atmosphere.calc_transm(temperature, mass_fractions, \
                    gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                    kappa_zero = kappa_zero, gamma_scat = gamma_scat)

m2 = atmosphere.transm_rad/nc.r_jup_mean
```

Third, $\gamma = 0$ (flat opacity):

```python
kappa_zero = 0.01
gamma_scat = 0.

atmosphere.calc_transm(temperature, mass_fractions, \
                    gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                    kappa_zero = kappa_zero, gamma_scat = gamma_scat)

m0 = atmosphere.transm_rad/nc.r_jup_mean
```

Fourth, the exoctic case of, $\gamma = 1$ (positive opacity slope):

```python
kappa_zero = 0.01
gamma_scat = 1.

atmosphere.calc_transm(temperature, mass_fractions, \
                    gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                    kappa_zero = kappa_zero, gamma_scat = gamma_scat)

p1 = atmosphere.transm_rad/nc.r_jup_mean

# Make plot

plt.plot(nc.c/atmosphere.freq/1e-4, clear, label = 'Clear')
plt.plot(nc.c/atmosphere.freq/1e-4, \
         m4, \
         label = r'Powerlaw cloud, $\gamma = -4$')
plt.plot(nc.c/atmosphere.freq/1e-4, \
         m2, \
         label = r'Powerlaw cloud, $\gamma = -2$')
plt.plot(nc.c/atmosphere.freq/1e-4, \
         m0, \
         label = r'Powerlaw cloud, $\gamma = 0$')
plt.plot(nc.c/atmosphere.freq/1e-4, \
         p1,\
         label = r'Powerlaw cloud, $\gamma = 1$')
plt.xscale('log')
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Transit radius ($\rm R_{Jup}$)')
plt.legend(loc = 'best')
plt.show()
plt.clf()
```

## Gray cloud deck and/or scaled Rayhleigh scattering

In addition to power law clouds, petitRADTRANS can also simply add a gray cloud deck, or scale the Rayleigh scattering opacities of the gas by a factor specified by the user. For these examples we will reuse the Radtrans object from the demonstration of the power law cloud above, there's no need to intialize it again.

Let's calculate three spectra:
-  A clear model
-  A cloudy model with a gray cloud deck at 0.01 bar
-  A haze-like model, mimicked by scaling the Rayleigh scattering of the gas by a factor 10
-  A combination of the two above cloud models: gray cloud deck plus haze

```python
import pylab as plt
plt.rcParams['figure.figsize'] = (10, 6)

# Clear
atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, R_pl=R_pl, P0_bar=P0)
plt.plot(nc.c/atmosphere.freq/1e-4, \
         atmosphere.transm_rad/nc.r_jup_mean, label = 'Clear')

# Gray cloud deck at 0.01 bar
atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                      Pcloud = 0.01)
plt.plot(nc.c/atmosphere.freq/1e-4, \
         atmosphere.transm_rad/nc.r_jup_mean, label = 'Gray cloud deck at 0.01 bar')

# Haze (10 x gas Rayleigh scattering)
atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                      haze_factor = 10)
plt.plot(nc.c/atmosphere.freq/1e-4, \
         atmosphere.transm_rad/nc.r_jup_mean, label = 'Rayleigh haze')

# Haze + cloud deck
atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, R_pl=R_pl, P0_bar=P0, \
                      haze_factor = 10, Pcloud = 0.01)
plt.plot(nc.c/atmosphere.freq/1e-4, \
         atmosphere.transm_rad/nc.r_jup_mean, label = 'Rayleigh haze + cloud deck')

plt.xscale('log')
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Transit radius ($\rm R_{Jup}$)')
plt.legend(loc = 'best')
plt.show()
plt.clf()
```

## Condensate clouds from real optical constants

Let's calculate some spectra using opacities derived from optical constants of various materials.
In this example we will start of with forsterite, that is $\rm Mg_2SiO_4$. For the list of available cloud species see ["Available opacity species"](../available_opacities.html).

**Choose your favorite particle setup:** petitRADTRANS offers multiple opacity versions for every given condensate, assuming either spherical (Mie scattering) or irreagularly shaped (DHS method = distribution of hollow spheres) particles. Moreover, opacities assuming a crystalline and/or amorphous internal structure can be used, where available.

We set up the atmosphere like before, this time loading the opacity of crystalline (``c``), irregularly shaped particles (DHS method, abbreviated ``d``). The mode identifier for the $\rm Mg_2SiO_4$ opacity therefore is ``Mg2SiO4(c)_cd``. ``(c)`` stands for condensed. These are all possible modes, note that for some species only crystalline or amorphous cross-sections are available.

| Opacity code | Description |
|------|------|
|  Species(c)_cm  | crystalline Mie opacity of Species|
|  Species(c)_cd  | crystalline DHS opacity of Species|
|  Species(c)_am  | amorphous Mie opacity of Species|
|  Species(c)_ad  | amorphous DHS opacity of Species|

But now, let's start! We simply give one additional ``cloud_species`` list to the Radtrans object, containing the name of the opacity species we want to use.

```python
import numpy as np
from petitRADTRANS import Radtrans

atmosphere = Radtrans(line_species = ['H2O', 'CO_all_iso', 'CH4', 'CO2', 'Na', 'K'], \
      cloud_species = ['Mg2SiO4(c)_cd'], \
      rayleigh_species = ['H2', 'He'], \
      continuum_opacities = ['H2-H2', 'H2-He'], \
      wlen_bords_micron = [0.3, 15])

pressures = np.logspace(-6, 2, 100)
atmosphere.setup_opa_structure(pressures)
```

      Read CIA opacities for H2-H2...
      Read CIA opacities for H2-He...
     Done.

**Units in petitRADTRANS:** remember that all units in *petitRADTRANS* are in cgs, **except for pressure**, which is in bars, **and the mean molecular weight (MMW)**, which is in units of atomic mass units.

Next we set up the atmospheric parameters, with a parameter selection *almost* identical to the ["Getting Started"](getting_started.html) case. We now also add the mass fraction of the cloud species we are interested in. Note that the mass fractions only contains the name ``Mg2SiO4(c)`` now, the ``_cd`` suffix has to be omitted.

```python
from petitRADTRANS import nat_cst as nc
R_pl = 1.838*nc.r_jup_mean
gravity = 1e1**2.45
P0 = 0.01

kappa_IR = 0.01
gamma = 0.4
T_int = 200.
T_equ = 1500.
temperature = nc.guillot_global(pressures, kappa_IR, gamma, gravity, T_int, T_equ)

mass_fractions = {}
mass_fractions['H2'] = 0.74 * np.ones_like(temperature)
mass_fractions['He'] = 0.24 * np.ones_like(temperature)
mass_fractions['H2O'] = 0.001 * np.ones_like(temperature)
mass_fractions['CO_all_iso'] = 0.01 * np.ones_like(temperature)
mass_fractions['CO2'] = 0.00001 * np.ones_like(temperature)
mass_fractions['CH4'] = 0.000001 * np.ones_like(temperature)
mass_fractions['Na'] = 0.00001 * np.ones_like(temperature)
mass_fractions['K'] = 0.000001 * np.ones_like(temperature)
mass_fractions['Mg2SiO4(c)'] = 0.0000005 * np.ones_like(temperature)

MMW = 2.33 * np.ones_like(temperature)
```

**Abundances in petitRADTRANS:** remember that abundances in petitCODE are in units of **mass fractions**, not number fractions (aka volume mixing ratio, VMR). You can convert between mass fractions and VMRs by using
\begin{equation}
X_i = \frac{\mu_i}{\mu}n_i,
\end{equation}
where $X_i$ is the mass fraction of species $i$, $\mu_i$ the mass of a single molecule/atom/ion/... of species $i$, $\mu$ is the atmospheric mean molecular weight, and $n_i$ is the VMR of species $i$.

### Setting the cloud particle size

We have to define a few additional parameters for the clouds, to start the calculation.
We can give the mean particle size and width of the log-normal particle distribution:

```python
radius = {}
radius['Mg2SiO4(c)'] = 0.00005*np.ones_like(temperature) # I.e. a 0.5-micron particle size (0.00005 cm)

sigma_lnorm = 1.05
```

Now, let's calculate a clear and cloudy spectrum to compare:

```python
atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, \
                       R_pl=R_pl, P0_bar=P0, \
                       radius = radius, sigma_lnorm = sigma_lnorm)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.transm_rad/nc.r_jup_mean, label = 'cloudy', zorder = 2)

mass_fractions['Mg2SiO4(c)'] = np.zeros_like(temperature)

atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, \
                       R_pl=R_pl, P0_bar=P0, \
                       radius = radius, sigma_lnorm = sigma_lnorm)
plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.transm_rad/nc.r_jup_mean, label = 'clear', zorder = 1)

plt.xscale('log')
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Transit radius ($\rm R_{Jup}$)')
plt.legend(loc='best')
plt.show()
plt.clf()
```

Here one sees that there is a lot of additional absorption in the optical and near-IR. Also note the silicate (Si-O stretching mode) becoming visible at 10 micron.

### Calculating the particle size

Alternatively, one can specify an eddy diffusion parameter ($K_{zz}$, with units of cm$^2$/s), which expresses how strongly atmospheric mixing is. The cloud particle size is then controlled by specifying $K_{zz}$, and the unitless settling parameter $f_{\rm sed}$, which expresses the particles' mass averaged settling velocity, when compared to the local atmospheric mixing speed. Also, a width for the log-normal particle size distribution must be specified:

```python
Kzz = np.ones_like(temperature)*1e1**7.5
fsed = 2.
sigma_lnorm = 1.05
```

Again, let's calculate a clear and cloudy spectrum to compare:

```python
mass_fractions['Mg2SiO4(c)'] = 0.0000005 * np.ones_like(temperature)

atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, \
                       R_pl=R_pl, P0_bar=P0, \
                       Kzz = Kzz, fsed=fsed, sigma_lnorm = sigma_lnorm)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.transm_rad/nc.r_jup_mean, label = 'cloudy', zorder = 2)

mass_fractions['Mg2SiO4(c)'] = np.zeros_like(temperature)

atmosphere.calc_transm(temperature, mass_fractions, gravity, MMW, \
                       R_pl=R_pl, P0_bar=P0, \
                       Kzz = Kzz, fsed=fsed, sigma_lnorm = sigma_lnorm)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.transm_rad/nc.r_jup_mean, label = 'clear', zorder = 1)

plt.xscale('log')
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Transit radius ($\rm R_{Jup}$)')
plt.legend(loc='best')
plt.show()
plt.clf()
```

The resulting mean particle sizes can be accessed like this:

```python
plt.yscale('log')
plt.xscale('log')

plt.ylim([1e2,1e-6])

plt.ylabel('P (bar)')
plt.xlabel('Average particle size (microns)')

plt.plot(atmosphere.r_g[:,atmosphere.cloud_species.index('Mg2SiO4(c)')]/1e-4, pressures)
plt.show()
plt.clf()
```

**Minimum particle size:** the opacity of particles smaller than 1 nm is set to zero.

## Emission spectra

We can use the Radtrans object created above to calculate emission spectra as well. Here we will also generate an instance of pRT with scattering turned on, (see ["Scattering for Emission Spectra"](emis_scat.html) for an example on how to do this in detail).

```python
mass_fractions['Mg2SiO4(c)'] = 0.0000005 * np.ones_like(temperature)

atmosphere.calc_flux(temperature, mass_fractions, gravity, MMW, \
                       Kzz = Kzz, fsed=fsed, sigma_lnorm = sigma_lnorm)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.flux/1e-6, \
         color = 'black', label = 'cloudy, no scattering', zorder = 1)

# Load scattering version of pRT
atmosphere = Radtrans(line_species = ['H2O', 'CO_all_iso', 'CH4', 'CO2', 'Na', 'K'], \
      cloud_species = ['Mg2SiO4(c)_cd'], \
      rayleigh_species = ['H2', 'He'], \
      continuum_opacities = ['H2-H2', 'H2-He'], \
      wlen_bords_micron = [0.3, 15], \
      do_scat_emis = True)
pressures = np.logspace(-6, 2, 100)
atmosphere.setup_opa_structure(pressures)

atmosphere.calc_flux(temperature, mass_fractions, gravity, MMW, \
                       Kzz = Kzz, fsed=fsed, sigma_lnorm = sigma_lnorm, \
                       add_cloud_scat_as_abs = True)
plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.flux/1e-6, \
         label = 'cloudy, including scattering', zorder = 2)

mass_fractions['Mg2SiO4(c)'] = np.zeros_like(temperature)

atmosphere.calc_flux(temperature, mass_fractions, gravity, MMW, \
                       Kzz = Kzz, fsed=fsed, sigma_lnorm = sigma_lnorm)

plt.plot(nc.c/atmosphere.freq/1e-4, atmosphere.flux/1e-6, '-', \
         color = 'red', label = 'clear', zorder = 0)

plt.legend(loc='best')
plt.xscale('log')
plt.xlabel('Wavelength (microns)')
plt.ylabel(r'Planet flux $F_\nu$ (10$^{-6}$ erg cm$^{-2}$ s$^{-1}$ Hz$^{-1}$)')
plt.show()
plt.clf()
```

Here we plotted the clear spectrum, neglecting the cloud opacity, the cloudy spectrum only considering the absorption of the cloud particles, and the cloudy spectrum with scattering, using the scattering version of pRT.

**Scattering and petitRADTRANS:** remember that scattering is included for emission spectra in petitRADTRANS only if requested specifically when generating the Radtrans object, as it increases the runtime (see ["Scattering for Emission Spectra"](emis_scat.html) for an example how to do this).