```python
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei')
plt.rcParams['axes.grid'] = False

from IPython.display import SVG

def show_image(filename, figsize=None):
    if figsize:
        plt.figure(figsize=figsize)

    plt.imshow(plt.imread(filename))
```

TreeBoost原理和实现（sklearn）简介
===========================

### 0. 前言

TreeBoost是对GBDT做了更深一步的优化，主要贡献在将整颗树的全局权重值细化到每片叶子的局部权重值。它不像xgboost在树生长时就做出指导，而是在树生长后再修正叶子值。对于工程中的GBDT模块，spark目前是传统的Gradient Boosting，而sklearn采用了TreeBoost，所以本文以sklearn为例进行说明。

本文假设读者已经了解GBDT的基本原理。

sklearn代码版本：

```sh
~/W/s/sklearn ❯❯❯ git log -n 1
commit d161bfaa1a42da75f4940464f7f1c524ef53484f
Author: John B Nelson <jnelso11@gmu.edu>
Date:   Thu May 26 18:36:37 2016 -0400

    Add missing double quote (#6831)
```

GBDT模块位于`sklearn/ensemble/gradient_boosting.py`文件，类的关系比较简单。

```python
SVG("./res/Main.svg")
```

### 1. GBDT优化

传统的Gradient Boost算法主要有四步：

1. 对损失导数求偏导；
2. 训练决策树，拟合偏导；
3. 寻优模型权值；
4. 将训练的模型，加到叠加模型中。

可以用数学公式对应表述为[1]：

+ $F_0(x) = \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L(y_i, \rho)$

+ For $m=1$ to $M$ do:

   1. $\tilde{y} = - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)}, \quad i = 1, 2, \dotsc, N$

   2. $\mathbf{a}_m = \operatorname{arg \, min}_{\mathbf{a}, \beta} \displaystyle \sum_{i=1}^N \left [ \tilde{y}_i - \beta h(x_i; \mathbf{a}) \right ]^2$
   3. $\rho_m = \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right)$
   4. $F_m(x) = F_{m-1}(x) + l_r \rho_m h(x; \mathbf{a}_m)$

其中，$L$是损失函数， $l_r$是学习率，$\rho$是模型的权值，$h(x_i; \mathbf{a})$是决策树模型，$\mathbf{a}$是树的参数，$F$是最终累加模型。

利用一些数学方法和损失函数的特性，可以将第3步的权值寻优进行简化。在sklear中单步训练代码对应如下：

```Python
 748     def _fit_stage(self, i, X, y, y_pred, sample_weight, sample_mask,
 749                    random_state, X_idx_sorted, X_csc=None, X_csr=None):
 750         """Fit another stage of ``n_classes_`` trees to the boosting model. """
 751 #+--  8 lines: assert sample_mask.dtype == np.bool----------
 759
 760             residual = loss.negative_gradient(y, y_pred, k=k,
 761                                               sample_weight=sample_weight)
 762
 763             # induce regression tree on residuals
 764             tree = DecisionTreeRegressor(
 765                 criterion='friedman_mse',
 766                 splitter='best',
 767 #+---  7 lines: max_depth=self.max_depth,------------------
 774                 presort=self.presort)
 775
 776 #+---  8 lines: if self.subsample < 1.0:------------------
 784                 tree.fit(X, residual, sample_weight=sample_weight,
 785                          check_input=False, X_idx_sorted=X_idx_sorted)
 786
 787             # update tree leaves
 788 #+--  5 lines: if X_csr is not None:---------------------
 793                 loss.update_terminal_regions(tree.tree_, X, y, residual, y_pred,
 794                                              sample_weight, sample_mask,
 795                                              self.learning_rate, k=k)
 796
 797             # add tree to ensemble
 798             self.estimators_[i, k] = tree
 799
 800         return y_pred
```

其中，

+ 760L是求解偏导。
+ 784L是生成决策树。注意，对于MSE，$\beta=1$。765L树的评价函数是friedman_mse，它是MSE的变种。我猜测$\beta$是一致的，后续有时间再深究。
+ 793L是TreeBoost做的改进，将第3步寻优全局解$\rho$转化到树内部，后面主要内容就在这。
+ 798L是加回到累加模型。

接下来，我们先就平方差和绝对值两种损失函数，对传统Gradient Boost方法进行第3步的优化，为后续TreeBoost铺路。

[1]: Friedman - Greedy function approximation: A gradient boosting machine

#### 1.0 Least squares regression

这种损失函数定义是 $L(y, F) = \frac{1}{2} (y - F)^2$，则其导数为$\frac{\partial L}{\partial F} = -(y - F)$。将偏导代入到第1步有：

\begin{align}
    \tilde{y} &= - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)} \\
    &= y_i - F_{m-1}(x_i), \quad i = 1, 2, \dotsc, N \\
\end{align}

将$L$和$\tilde{y}$代入到第3步中，整理可得：

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right) \\
    &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N \frac{1}{2} \left ( y_i - F_{m-1}(x_i) - \rho h(x_i; \mathbf{a}_m) \right)^2 \\
    &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N \frac{1}{2} \left ( \tilde{y}_i - \rho h(x_i; \mathbf{a}_m) \right)^2 \\
    &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N \left ( \tilde{y}_i - \rho h(x_i; \mathbf{a}_m) \right)^2 \\
    &= \operatorname{arg \, min}_\beta \displaystyle \sum_{i=1}^N \left ( \tilde{y}_i - \beta h(x_i; \mathbf{a}_m) \right)^2 \quad \text{符号替换}\\
    &= \beta_m
\end{align}

也就是说，对于平方差这种损失函数，它的最优权值就是第2步中指导决策树生成时的$\beta$值。

这个算法称为LS_Boost，具体过程为：

+ $F_0(x) = \bar{y}$

+ For $m=1$ to $M$ do:
  1. $\tilde{y}_i = y_i - F_{m-1}(x_i), \quad i=1, N$
  2. $(\rho_m, \mathbf{a}_m) = \operatorname{arg \, min}_{\mathbf{a}, \rho} \displaystyle \sum_{i=1}^N \left [ \tilde{y}_i - \rho h(x_i; \mathbf{a}) \right ]^2$
  3. $F_m(x) = F_{m-1}(x) + l_r \rho_m h(x; \mathbf{a}_m)$

sklearn中代码如下：

```Python
 274 class LeastSquaresError(RegressionLossFunction):
 275     """Loss function for least squares (LS) estimation.
 276     Terminal regions need not to be updated for least squares. """
 277     def init_estimator(self):
 278         return MeanEstimator()
 279
 280     def __call__(self, y, pred, sample_weight=None):
 281         if sample_weight is None:
 282             return np.mean((y - pred.ravel()) ** 2.0)
 283         else:
 284             return (1.0 / sample_weight.sum() *
 285                     np.sum(sample_weight * ((y - pred.ravel()) ** 2.0)))
 286
 287     def negative_gradient(self, y, pred, **kargs):
 288         return y - pred.ravel()
 289
 290     def update_terminal_regions(self, tree, X, y, residual, y_pred,
 291                                 sample_weight, sample_mask,
 292                                 learning_rate=1.0, k=0):
 293         """Least squares does not need to update terminal regions.
 294
 295         But it has to update the predictions.
 296         """
 297         # update predictions
 298         y_pred[:, k] += learning_rate * tree.predict(X).ravel()
 299
 300     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,
 301                                 residual, pred, sample_weight):
 302         pass
```

前面说过sklearn中$\beta=1$，则$l_r \times \beta = l_r$，所以`update_terminal_regions`这里直接用学习率和树预测值相乘。

#### 1.1 Least-absolute-deviation (LAD) regression

损失函数定义为 $L(y, F) = | y - F |$，同样地，可以利用导数算出残差：

\begin{align}
    \tilde{y} &= - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)} \\
    &= \operatorname{sign}(y_i - F_{m-1}(x_i))\\
\end{align}

$L(y, F)$是分段函数，它的导数在正数区间恒为1，负数区间恒为-1，而在间段点$F=0$处是不可导的，人为规定为0，所以可以用sign函数来描述：

\begin{equation}
    \operatorname{sign}(x) := {
        \begin{cases}
            -1 & {\text{if }} x<0, \\
            0  & {\text{if }} x=0, \\
            1  & {\text{if }} x>0.
        \end{cases}
    }
\end{equation}

同样的，将$L$和$\tilde{y}$代入到第3步中，整理可得：

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right) \\
    &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | y_i - F_{m-1}(x_i) - \rho h(x_i; \mathbf{a}_m) \big | \\
    &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | h(x_i; \mathbf{a}_m) \big | \cdot \left | \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} - \rho \right | \\
    &= \operatorname{median}_W \left \{ \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} \right \}_1^N, \quad w_i = \big | h(x_i; \mathbf{a}_m) \big |
\end{align}

这里$\operatorname{median}_W \{ \cdot \}$是带权$w_i$的[weighted median](https://en.wikipedia.org/wiki/Weighted_median)。

注意，weighted median的概念$\operatorname{median}_W (x)$，这里的加权，应理解为$x_i$出现了$w_i$次，而不是$x_i \cdot w_i$数值。 你可以去维基页查看定义，也可以看下面的推导细节。不太好讲，但其实很简单。

在sklearn中，LAD用的TreeBoost去实现，所以没有代码对应。

#### 推导细节

\begin{align}
\rho_m &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | h(x_i; \mathbf{a}_m) \big | \cdot \left | \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} - \rho \right | \\
    &= \operatorname{median}_W \left \{ \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} \right \}_1^N, \quad w_i = \big | h(x_i; \mathbf{a}_m) \big |
\end{align}

最后一步的过程比较跳，我们在这里详细说下推导。

在这之前，我们简化下符号，以利于理解，

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \big | h(x_i; \mathbf{a}_m) \big | \cdot \left | \frac{y_i - F_{m-1}(x_i)}{h(x_i; \mathbf{a}_m)} - \rho \right | \\
     &= \operatorname{arg \, min}_\rho \sum_{i=1}^N w_i \cdot | t_i - \rho | \\
\end{align}

##### 无加权

首先，我们考虑无加权的情况，即$w_i = 1$。此时有$\rho_m = \operatorname{arg \, min}_\rho \sum_{i=1}^N | t_i - \rho |$。

有两种方法来解释为什么$\rho_m = \operatorname{median}(t_i)$:

第一种是几何方法，$| t_i - \rho |$表示两点距离。我们要找的$\rho_m$，本质上就是离所有点$t_i$总距离最短的点。所以通过作图的方法，直观理解。

```python
show_image("./res/rho_m.png")
```

如上图，绿色直线是中位点到各点距离，黄色直线是另一个点的距离，随着这个点外移，总距离越来越长。总结，对于奇数个点，$\rho_m$是中位点；对于偶数点，$\rho_m$在两个中间点闭区间上。

第二种方法是代数方法。

首先我们直观地了感受下结果，对于$\rho_m = \operatorname{arg \, min}_\rho \sum_{i=1}^N | t_i - \rho | = \operatorname{arg \, min} f(\rho)$，导数是$f'(\rho) = \sum_{i=1}^N \operatorname{sign}(t_i - \rho)$。在极值点$f'(\rho) = 0$，而$\operatorname{sign}(x)$只有$1, -1, 0$三种值，所以要和为0，则必然要求$1$和$-1$的数量相同。而中位点正好满足此条件[2]。

然后，在Mathematics上看到一个较为严密的论证[2]。它先算最左端结果，再向右取区间推进，证明整个过程中距离是先单减再单增的过程，从而证明中位点是最小点，具体如下：

令集合$T$含$N$个元素$t_1 < t_2 < \dots < t_N$。

1. 对于最左端$\rho < t_1$，有$f(\rho) = \sum_{i=1}^N | t_i - \rho | = \sum_{i=1}^N (t_i - \rho)$。很明显，对于每个子项，有$t_i - \rho > 0$，且$\rho \to t_1$时，$(t_i - \rho)$单减。也就是说，随着$\rho$从左向右靠近$t_1$点，所有子项都在减小，则总和$f(\rho)$单减。

2. 选定任一区间，$t_k \leq \rho \leq \rho + d \leq t_{k+1}$，有：

\begin{align}
    f(\rho + d) &= \displaystyle \sum_{i=1}^N | t_i - (\rho + d) | \\
                &= \sum_{i=1}^k (\rho + d - t_i) + \sum_{i=k+1}^N (t_i - (\rho + d)) \\
                &= \sum_{i=1}^k (\rho - t_i) + \sum_{i=1}^k d + \sum_{i=k+1}^N (t_i - \rho) + \sum_{i=k+1}^N -d \\
                &= \sum_{i=1}^N | t_i = \rho | + k d + (N - (k+1) + 1) \times -d \\
                &= f(\rho) + d \times ( k - N + (k + 1 ) -1 ) \\
                &= f(\rho) + d \times (2k - N)
\end{align}

则有：

\begin{equation}
    f(\rho + d) = \begin{cases}
        < f(\rho), \quad \text{when } k < \frac{N}{2} \\
        = f(\rho), \quad \text{when } k = \frac{N}{2} \\
        > f(\rho), \quad \text{when } k > \frac{N}{2} \\
    \end{cases}
\end{equation}

也就是说，在$k < \frac{N}{2}$区间，$f(\rho)$单减；在$k > \frac{N}{2}$区间，$f(\rho)$单增。所以，在中位点$\frac{N}{2}$处，$f(\rho)$是最小值。于是得$\rho_m = \operatorname{median}(t_i)$。

[2]: http://math.stackexchange.com/questions/113270/the-median-minimizes-the-sum-of-absolute-deviations

##### 有加权

\begin{equation}
    \rho_m = \operatorname{arg \, min}_\rho \sum_{i=1}^N w_i \cdot | t_i - \rho |
\end{equation}

对于加权，有点无从下手的感觉。我们可以换个角度，会提供点有意思的想法。

假设$w_i$是正整数，则$w_i \cdot | t_i - \rho | = \sum_{k=1}^{w_i} | t_i - \rho |$，也就是说，相当于将集合$T$中的$t_i$点扩增到$w_i$个。那么，可以展开为：

\begin{align}
    \rho_m &= \operatorname{arg \, min}_\rho \sum_{i=1}^N w_i \cdot | t_i - \rho | \\
           &= \operatorname{arg \, min}_\rho \sum_{i=1}^N \sum_{k=1}^{w_i} | t_i - \rho | \\
           &= \operatorname{arg \, min}_\rho \sum_{t_i \in T'} | t_i - \rho | \quad \text{$T'$ 是$t_i$扩增$w_i$倍的集合}
\end{align}

于是，有加权问题就变更到无加权的问题。这时，我们可以将解法表示如下：

1. 将$t_i$按升序排列。
2. 将$w_i$按对应顺序排，计算累积和$c_k = \sum_{i=1}^k w_i$。
3. 找到对应中位点$c_m = \frac{1}{2} c_N$对应的$t_m$，这个$t_m$就是$\operatorname{median}_W \{t_i\}$。

对应的sklearn代码如下：

```Python
 51 def _weighted_percentile(array, sample_weight, percentile=50):
 52     """Compute the weighted ``percentile`` of ``array`` with ``sample_weight``. """
 53     sorted_idx = np.argsort(array)
 54
 55     # Find index of median prediction for each sample
 56     weight_cdf = sample_weight[sorted_idx].cumsum()
 57     percentile_idx = np.searchsorted(
 58         weight_cdf, (percentile / 100.) * weight_cdf[-1])
 59     return array[sorted_idx[percentile_idx]]
```

当然，如果$w_i$是分数，应该怎么想，我暂时也没有思路。可以参考下维基定义[Weighted median](https://en.wikipedia.org/wiki/Weighted_median)，后续有时间，细究下。

### 2. 从GBDT到TreeBoost

#### 2.0 回归决策树

我们先回顾前面的Gradient Boost方法，其训练公式如下：

+ For $m=1$ to $M$ do:

   1. $\tilde{y} = - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)}, \quad i = 1, 2, \dotsc, N$

   2. $\mathbf{a}_m = \operatorname{arg \, min}_{\mathbf{a}, \beta} \displaystyle \sum_{i=1}^N \left [ \tilde{y}_i - \beta h(x_i; \mathbf{a}) \right ]^2$
   3. $\rho_m = \operatorname{arg \, min}_\rho \displaystyle \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \right)$
   4. $F_m(x) = F_{m-1}(x) + \rho_m h(x; \mathbf{a}_m)$
      注意，为了描述方便，我移除了学习率这个乘数。

TreeBoost将外部的寻优参数内化到决策树内部，从而既让模型更精细，又加快了运行速度。而要让外部参数进入到决策树$h(x_i; \mathbf{a}_m)$，首要的问题是得打开这个函数，即使用解析式来描述。换句话说，我们需要对决策树建立数学模型。

对于$J$个叶子的回归决策树，可以表述为累加式：

\begin{equation}
    h(x; \{b_j, R_j\}_1^J) = \displaystyle \sum_{j=1}^J b_j \, \mathbf{1}(x \in R_j)
\end{equation}

其中，$R_j$是各叶子，对应叶子的值是此区域的样本均值$b_j = \operatorname{ave}_{x_i \in R_j} y_i$。因为各个叶子是没有交集的，所以这个公式的含义等价于：如果$x \in R_j$，那么$h(x) = b_j$。

我们将回归决策树的数学式代入Gradient Boost中第四步，可得到：

\begin{align}
    F_m(x) &= F_{m-1}(x) + \rho_m h(x; \mathbf{a}_m) \\
           &= F_{m-1}(x) + \rho_M \displaystyle \sum_{j=1}^J b_{jm} \, \mathbf{1}(x \in R_{jm}) \\
           &= F_{m-1}(x) + \sum_{j=1}^J \color{red}{\rho_M b_{jm}} \, \mathbf{1}(x \in R_{jm}) \\
           &= F_{m-1}(x) + \sum_{j=1}^J \color{red}{\gamma_{jm}} \, \mathbf{1}(x \in R_{jm})
\end{align}

请注意，最后一步定义$\gamma_{jm} = \rho_M b_{jm}$，只是个简单的代数替换。但它的实质是将权重从对整颗树的全局解移动到了对各个叶子的局部解。也就是说，这个权重值更加细化了，以前是对整颗树寻优，现在是针对各个叶子，各自寻优。此时，将定义代入Gradient Boost第三步，就得到局部权重的最优解：

\begin{equation}
    \{\gamma_{jm}\}_1^J = \displaystyle \operatorname{arg \, min}_{\{\gamma_j\}_1^J} \sum_{i=1}^N L \left ( y_i, F_{m-1}(x_i) + \sum_{j=1}^J \gamma_j \mathbf{1}(x \in R_{jm}) \right )
\end{equation}

又因为各叶子$R_{jm}$是互无交集，相互独立的，上式的最优解就是各叶子各自的最优解汇总，故可简写为：

\begin{equation}
    \gamma_{jm} = \displaystyle \operatorname{arg \, min}_{\gamma} \sum_{x_i \in R_{jm}} L(y_i, F_{m-1}(x_i) + \gamma)
\end{equation}

#### 2.1 LAD_TreeBoost

前面已经讨论过，对于损失函数$L(y,F) = |y - F|$，它的最优解是中位值。

故，对于LAD回归，可得：

\begin{equation}
    \gamma_{jm} = \displaystyle \operatorname{median}_{x_i \in R_{jm}} \{ y_i - F_{m-1}(x_i) \}
\end{equation}

综上，可得到LAD_TreeBoost算法如下：

+ $F_0(x) = \operatorname{median}\{y_i\}_1^N$
+ For $m=1$ to $M$ do:
   1. $\tilde{y}_i = \operatorname{sign}(y_i - F_{m-1}(x_i)), \, i=1, N$
   2. $\{R_{jm}\}_1^J = \text{$J$ terminal node tree} \big (\{\tilde{y}_i, x_i\}_1^N \big )$
   3. $\gamma_{jm} = \displaystyle \operatorname{median}_{x_i \in R_{jm}} \{ y_i - F_{m-1}(x_i)\}, \, j=1, J$
   4. $F_m(x) = F_{m-1}(x) + \displaystyle \sum_{j=1}^J \gamma_{jm} \mathbf{1}(x \in R_{jm})$

对应的sklearn中代码为

```python
 305 class LeastAbsoluteError(RegressionLossFunction):
 306     """Loss function for least absolute deviation (LAD) regression. """
 307     def init_estimator(self):
 308         return QuantileEstimator(alpha=0.5)
 309
 310     def __call__(self, y, pred, sample_weight=None):
 311         if sample_weight is None:
 312             return np.abs(y - pred.ravel()).mean()
 313         else:
 314             return (1.0 / sample_weight.sum() *
 315                     np.sum(sample_weight * np.abs(y - pred.ravel())))
 316
 317     def negative_gradient(self, y, pred, **kargs):
 318         """1.0 if y - pred > 0.0 else -1.0"""
 319         pred = pred.ravel()
 320         return 2.0 * (y - pred > 0.0) - 1.0
 321
 322     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,
 323                                 residual, pred, sample_weight):
 324         """LAD updates terminal regions to median estimates. """
 325         terminal_region = np.where(terminal_regions == leaf)[0]
 326         sample_weight = sample_weight.take(terminal_region, axis=0)
 327         diff = y.take(terminal_region, axis=0) - pred.take(terminal_region, axis=0)
 328         tree.value[leaf, 0, 0] = _weighted_percentile(diff, sample_weight, percentile=50)
```

其中320L就是第一步的求导，328L就是第三步将树的叶子改为中位值（percentile=50）。

因为$\tilde{y}_i$只有两种值$\tilde{y} \in \{-1, 1\}$，又叶子值是取中位数，所以这个损失的鲁棒性非常强。但求解中位数不如平均数有快速方法，所以性能会受影响。

#### 2.2 M-Regression

LS_Boost运行快，LDA_TreeBoost鲁棒性好，能不能将两者结合起来呢？M-Regression的初衷正是来源于此，它用阈值$\delta$，比如说三倍方差，将$|y-F|$误差分隔成两部份：在阈值内的认定是正常误差，用LS_Boost来约束；在阈值外认定是长尾和坏值，用LDA_TreeBoost来抵抗。通过调整$\delta$值来控制平衡，从而达到各取其长的好处。

这里具体对的损失函数叫Huber，定义如下：

\begin{equation}
    L(y, F) = \begin{cases}
        \frac{1}{2} (y - F)^2 \quad & |y - F| \leq \delta \\
        \delta \cdot \left (\big |y - F \big | - \frac{\delta}{2} \right ) \quad & |y - F| > \delta
    \end{cases}
\end{equation}

对应的具体取解推导要再参阅论文，如果后面有时间再来填坑。

在sklearn中对应的是HuberLossFunction类。

#### 2.3 二分类逻辑回归树

这里用的损失函数叫negative binomial log-likelihood[3]，定义为：

\begin{equation}
    L(y, F) = \log (1 + e^{-2 y F}), \quad y \in \{-1, 1\}
\end{equation}

其中$F(x) = \frac{1}{2} \log \left [ \frac{\operatorname{Pr}(y = 1 | x)}{\operatorname{Pr}(y = -1 | x)} \right ]$

则第一步残差展开为：

\begin{align}
    \tilde{y_i} &= - \left [ \frac{\partial L (y_i, F(x_i))}{\partial F(x_i)} \right ]_{F(x) = F_{m-1}(x)} \\
              &= - \frac{1}{1 + e^{-2yF}} \cdot 1 \cdot e^{-2yF} \cdot -2y \\
              &= 2y \frac{e^{-2yF}}{1 + e^{-2yF}} \\
              &= \frac{2y}{1 + e^{2yF}} \\
              &= \frac{2 y_i}{1 + e^{2 y_i F_{m-1}(x_i)}} \\
\end{align}

同样地，第三步寻优代入损失函数，变为：

\begin{equation}
    \rho_m = \displaystyle \operatorname{arg \, min}_\rho \sum_{i=1}^{N} \log \left ( 1 + e^{-2 y_i \big ( F_{m-1}(x_i) + \rho h(x_i; \mathbf{a}_m) \big )} \right )
\end{equation}

运用前面LAD_Boost的技巧，很容易得到：

\begin{equation}
    \gamma_{jm} = \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \log(1 + e^{-2 y_i ( F_{m-1}(x_i) + \gamma ) })
\end{equation}

上式没有封闭形式的解，根据[3]，可用一轮最优化中的牛顿迭代法，得到数值解：

\begin{equation}
    \gamma_{jm} = \displaystyle \sum_{x_i \in R_{jm}} \tilde{y}_i \Big / \sum_{x_i \in R_{jm}} |\tilde{y}_i| (2 - |\tilde{y}_i|)
\end{equation}

这一步推导，我也还没有细看，以后有精力再细究。

这个算法被称为L2_TreeBoost，总结如下：

+ $F_0(x) = \frac{1}{2} \log \frac{1 + \bar{y}}{1 - \bar{y}}$

+ For $m=1$ to $M$ do:
   1. $\tilde{y}_i = \frac{2 y_i}{1 + e^{2 y_i F_{m-1}(x_i)}}, \quad i=1, N$
   2. $\{R_{jm}\}_1^J = \text{$J$ terminal node tree} \big (\{\tilde{y}_i, x_i\}_1^N \big )$
   3. $\gamma_{jm} = \displaystyle \sum_{x_i \in R_{jm}} \tilde{y}_i \Big / \sum_{x_i \in R_{jm}} |\tilde{y}_i| (2 - |\tilde{y}_i|), \quad j=1, J$
   4. $F_m(x) = F_{m-1}(x) + \displaystyle \sum_{j=1}^J \gamma_{jm} \mathbf{1}(x \in R_{jm})$

sklearn中代码如下：

```python
 106 class LogOddsEstimator(BaseEstimator):
 107     """An estimator predicting the log odds ratio."""
 108     scale = 1.0
 109
 110     def fit(self, X, y, sample_weight=None):
 111         # pre-cond: pos, neg are encoded as 1, 0
 112         if sample_weight is None:
 113             pos = np.sum(y)
 114             neg = y.shape[0] - pos
 115         else:
 116             pos = np.sum(sample_weight * y)
 117             neg = np.sum(sample_weight * (1 - y))
 118
 119         if neg == 0 or pos == 0:
 120             raise ValueError('y contains non binary labels.')
 121         self.prior = self.scale * np.log(pos / neg)
 122
 123     def predict(self, X):
 124         check_is_fitted(self, 'prior')
 125
 126         y = np.empty((X.shape[0], 1), dtype=np.float64)
 127         y.fill(self.prior)
 128         return y
```

注意，这个初始化$F_0$对公式做了点变形。

```python
 466 class BinomialDeviance(ClassificationLossFunction):
 467 #    """Binomial deviance loss function for binary classification.
 468 #+-- 10 lines: Binary classification is a special case; here, we only need to-------------------------
 478
 479     def init_estimator(self):
 480         return LogOddsEstimator()
 481
 482 #+--  9 lines: def __call__(self, y, pred, sample_weight=None):---------------------------------------
 491
 492     def negative_gradient(self, y, pred, **kargs):
 493         """Compute the residual (= negative gradient). """
 494         return y - expit(pred.ravel())
 495
 496     def _update_terminal_region(self, tree, terminal_regions, leaf, X, y,
 497                                 residual, pred, sample_weight):
 498         """Make a single Newton-Raphson step.
 499
 500         our node estimate is given by:
 501
 502             sum(w * (y - prob)) / sum(w * prob * (1 - prob))
 503
 504         we take advantage that: y - prob = residual
 505         """
 506         terminal_region = np.where(terminal_regions == leaf)[0]
 507         residual = residual.take(terminal_region, axis=0)
 508         y = y.take(terminal_region, axis=0)
 509         sample_weight = sample_weight.take(terminal_region, axis=0)
 510
 511         numerator = np.sum(sample_weight * residual)
 512         denominator = np.sum(sample_weight * (y - residual) * (1 - y + residual))
 513
 514         if denominator == 0.0:
 515             tree.value[leaf, 0, 0] = 0.0
 516         else:
 517             tree.value[leaf, 0, 0] = numerator / denominator
 518
 519     def _score_to_proba(self, score):
 520         proba = np.ones((score.shape[0], 2), dtype=np.float64)
 521         proba[:, 1] = expit(score.ravel())
 522         proba[:, 0] -= proba[:, 1]
 523         return proba
```

[3]: Jerome Friedman - Additive logistic regression: a statistical view of boosting

模型$F_M(x)$输出的分值需要转换成各类的概率值，519L的`_score_to_proba`处理思路来源如下：

通过联理公式，

\begin{align}
    & F(x) = \frac{1}{2} \log \left [ \frac{\operatorname{Pr}(y = 1 | x)}{\operatorname{Pr}(y = -1 | x)} \right ] \\
    & \operatorname{Pr}(y = 1 | x) + \operatorname{Pr}(y = -1 | x) = 1
\end{align}

可以容易解得：

\begin{align}
    & \operatorname{Pr}(y = 1 | x) = \frac{1}{1 + e^{-2 F(x)}} \\
    & \operatorname{Pr}(y = -1 | x) = \frac{1}{1 + e^{2 F(x)}} \\
\end{align}

因为$F_M(x)$和$F(x)$有关联，我们可以借用上式将分值近似转换成概率值：

\begin{align}
    p_{+}(x) &= \operatorname{\widehat{Pr}}(y = 1 | x) = \frac{1}{1 + e^{-2 F_M(x)}} \\
    p_{-}(x) &= \operatorname{\widehat{Pr}}(y = -1 | x) = \frac{1}{1 + e^{2 F_M(x)}} \\
\end{align}

用于二分类时，应用如下公式：

\begin{equation}
    \hat{y}(x) = 2 \cdot \mathbf{1}[c(-1,1) p_{+}(x) > c(1,-1) p_{-}(x)] - 1
\end{equation}

其中，$c(\hat{y}, y)$是将$y$误认为$\hat{y}$的代价。

回过头来看`score_to_proba`函数，它其实是上式的简化版，并没有$c(\cdot)$，同时虽然$p_{+}(x)$和$p_{-}(x)$并非真实概率，但相加等于1，所以只算$p_{+}(x)$。

##### Influence trimming

在寻优$\gamma_{jm}$时，还有优化的空间：

\begin{align}
    \gamma_{jm} &= \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \log(1 + e^{-2 y_i ( F_{m-1}(x_i) + \gamma ) }) \\
                &= \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \log(1 + e^{-2 \color{blue}{y_i F_{m-1}(x_i)}} \cdot e^{-2 y_i \gamma} ) \\
                &= \displaystyle \operatorname{arg \, min}_\gamma \sum_{x_i \in R_{jm}} \phi(x_i)
\end{align}

注意，上式中，若$y_i F_{m-1}(x_i)$非常大，则对应的$\phi(x_i) \to 0$。也就是说，这些样本对寻优不再有贡献。于是，我们可以定义$w_i = e^{-2 y_i F_{m-1}(x_i)}$作为测量函数，如果它的值小于一定阈值，这个样本就不再参与计算。

另外，对于一个$x_i$，其损失函数$L(y, F(x_i))$如果到达极值点，就相当于是常数，对于整体寻优$\operatorname{arg \, min} L(y, F(x))$不再有贡献。而判断极值点可以用二阶导数来度量，所以二阶导也能作为一种测量函数。具体到现在的逻辑回归树，就可以定义为：

\begin{equation}
    w_i = \frac{\partial^2 L}{\partial F^2} =  |\tilde{y}_i| (2 - |\tilde{y}_i|)
\end{equation}

至于这个移除的阈值，可以用比例来约束。即我们定阈值为$w_{l(\alpha)}$，而$l(\alpha)$满足：

\begin{equation}
    \displaystyle \sum_{i=1}^{l(\alpha)} w_{(i)} = \alpha \sum_{i=1}^{N} w_i
\end{equation}

其中，$w_{(i)}$是按升序排列的权值。典型值$\alpha \in [0.05, 0.2]$，也就是说，可以减少10到20倍的计算量。

#### 2.4 多分类逻辑回归树

多分类是二分类的推广，相对而言公式推导比较复杂，不再详述。它的简化，相当于是对每一个类建一个「是、否」的二分类树，最后对每个类逐一评分，输出分值最高的类。`

### 3. 参数与正则
#### 3.0 Regularization
lack of fit(LOF):
+ regression
  - average absolute error
+ classification
  - minus twice log-likelihood(deviance)
  - misclassification error-rate

两种shrinkage strategy:
1. 简单：最终模型：$F_\nu(x) = \bar{y} + \nu \cdot (F_M(x) - \bar{y})$
2. 复杂：每个迭代模型：$F_m(x) = F_{m-1}(x) + \nu \cdot \rho_m h(x; \mathbf{a}_m, \quad 0 < \nu \leq 1$

#### 3.1 Tree boosting的参数

meta-parameter:

+ $M$: the number of iterations.
+ $\nu$: the learning rate.
+ $J$: the fixed number of terminal nodes.
  The best tree size $J$ is governed by the effective interaction order of the target $F*(x)$, while it is unknown. $\to$ cross-validation.

### 4. 可解释性
理解贡献比较大的变量

#### 4.0 Relative importance of input variables

The relative influences $I_j$, of the individual inputs $x_j$, on the variation of $\hat{F}(x)$ over the joint input variable distributation:

\begin{equation}
    I_j = \left ( E_x \left [ \frac{\partial \hat{F}(x)}{\partial x_j} \right ]^2 \cdot \operatorname{var}_x[x_j] \right )^{1/2}
\end{equation}

据此，Breiman提出了适合于决策树的公式：

\begin{equation}
    \hat{I}_j^2(T) = \displaystyle \sum_{t=1}^{J-1} \hat{i}_t^2 \mathbf{1}(v_t = j)
\end{equation}

其中，$v_t$是以$j$作分割特征的中间节点，$\hat{i}_t^2$是对应节点的Friedman MSE评价$i^2(R_l, R_r) = \frac{w_l w_r}{w_l + w_r}(\bar{y}_l - \bar{y}_r)^2$。

对于多颗树，就加和取平均：

\begin{equation}
    \hat{I}^2_j = \frac{1}{M} \displaystyle \sum_{m=1}^M \hat{I}^2_j (T_m)
\end{equation}

这个思路非常朴素，就是计算各特征对决策树评价函数提升的贡献度。在sklearn中实现如下：

```python
1201     def feature_importances_(self):
1202         """Return the feature importances (the higher, the more important the
1203            feature).
1204
1205         Returns
1206         -------
1207         feature_importances_ : array, shape = [n_features]
1208         """
1209         self._check_initialized()
1210
1211         total_sum = np.zeros((self.n_features, ), dtype=np.float64)
1212         for stage in self.estimators_:
1213             stage_sum = sum(tree.feature_importances_
1214                             for tree in stage) / len(stage)
1215             total_sum += stage_sum
1216
1217         importances = total_sum / len(self.estimators_)
1218         return importances
```

此时，我们也就明白了为什么765L决策树使用的friedman_mse损失函数了。

#### 4.1 Partial dependence plots

Partial dependence主要是用于可视化一维或二维变量对于整体总结果的影响。它的计算思路非常简单，令有特征集合$X_s \cup X_c = X$，则$f(X_s = x_s) = \operatorname{Avg}(f(X_s = x_s, X_{ci}))$。也就是对于特定的$x_s$值，算出它的响应均值。

下图是一个示例[sklearn: Partial Dependence Plots](http://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html)。

对于一维变量，可以看出，房价与Medlin(平均收入）成正比，与AveOccup（每户人均数）成反比，而与HouseAge（房龄）、AveRoom（房间均数）没有明显关系。再看二维变量（HouseAge - AveOccup），对于每户人均数大于2的情况，房龄与房价关系不大；但当人均数小于2时，则出现了相关性。

[ref: Partial Dependency Plots and GBM](http://adventuresindm.blogspot.jp/2013/01/partial-dependency-plots-and-gbm.html)

### 5. 总结

我们先讲了对GBDT框架的优化，再借此拓展到tree boosting方法，然后略微提了正则和调参的参数，最后介绍了两种用于解释特征变量的方法。

本文基本上相当于翻译了论文Friedman - Greedy function approximation: A gradient boosting machine，如果有不明晰的地方，可以直接查看原文。

```python

```


**Table of contents**

* [Parts of Speech](#pos)
* [Hidden Markov Models](#hmm)
* [Maximum likelihood estimation for labelled data](#mle)
* [Implementation](#imp)
* [Evaluation](#eval)
    * [Perplexity](#ppl)
    * [Viterbi](#viterbi)
    * [Accuracy](#acc)

**Table of Exercises**

* Theory (10 points)
    * [Exercise 4-4](#ex4-4)
    * [Exercise 4-5](#ex4-5)
    * [Exercise 4-6](#ex4-6)
    * [Exercise 4-7](#ex4-7)
    * [Exercise 4-8](#ex4-8)
    * [Exercise 4-9](#ex4-9)
* Practice (30 points)
    * [Exercise 4-1](#ex4-1)
    * [Exercise 4-2](#ex4-2)
    * [Exercise 4-3](#ex4-3)
    * [Exercise 4-10](#ex4-10)
    * [Exercise 4-11](#ex4-11)
    * [Exercise 4-12](#ex4-12)

**General notes**

* In this notebook you are expected to use $\LaTeX$.
* Use python3.
* Use NLTK to read annotated data.
* **Document your code**: TAs are more likely to understand the steps if you document them. If you don't, it's also difficult to give you partial points for exercises that are not completely correct.

#  Parts of Speech

**Parts of speech** (also known as PoS, word classes) give us information about a word and its neighbors.
PoS can can be divided into: closed class and open class.

Open class words (or **content words**) are nouns, verbs, adjectives, and adverbs, where they refer to objects, actions, and features in the world. They are called open class, since there is no limit to what these words are
new ones are added all the time (email, website, selfie, etc.).

**Nouns** suchs as proper nouns are names of persons of entitnes: *Regina*, *Colorado*,
and *IBM*. Other type are common nouns that refer to objects that for exmaple, can be counted (one car) or homogeneus groups (snow, sand).

**Verbs** consists pf actions and processes, like *draw*, *provide*, and *go*.

***Adjectives** include terms for properties or qualities for concepts like age (*old*, *young*), value (*good*, *bad*).

Closed class words (or **function words**) are pronouns, determiners, prepositions, and connectives. There is a limited number of these.

**Prepositions** occur before noun phrases and indicate spatial or temporal relations, for example, *by* the house.

The PoS are tags that classifiy words. For example in English uses the 36 tags. And these tags are used to manually annotate a wide variety of corpora, the Brown corpus, the Wall Street Journal corpus.

In this Lab we are going to work with the English [Penn Treebank](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) tag set and the annotated corpus.

Our final goal is the task:
* **Part-of-speech tagging** (tagging for short) is the process of assigning a part-ofspeech tag to each word in an input text.

First, we will download the annotated data from [NLTK](https://www.nltk.org/data.html)

```python
# Read annotated corpora with NLTK
# first download data
import nltk
#nltk.download()
# it will open a GUI and you have to double click in "all" to download
# this will download different types of annotated corpora
```

```python
import sys
sys.version

```

    '3.5.2 (default, Nov 23 2017, 16:37:01) \n[GCC 5.4.0 20160609]'

With NLTK, we load the annotated **Penn Treebank** corpus.
This corpus will be used to train and test our PoS taggers. Let's use the last 100 sentences for test.

```python
# inspect PoS from Treebank
# we use the universal tagset
treebank_sents = nltk.corpus.treebank.tagged_sents(tagset='universal')

# we split our corpus on training, dev and test
treebank_training = list(treebank_sents[:3000])
treebank_dev = list(treebank_sents[3000:3100])
treebank_test = list(treebank_sents[3100:])
print(len(treebank_sents))
print(len(treebank_training))
print(len(treebank_dev)) # we use 100 sentences/instances for validation
print(len(treebank_test)) # we use 814 sentences/instances for test
```

    3914
    3000
    100
    814

```python
# which is the vocabulary?
# we can inspect the vocabulary of the corpus
vocabulary = set([w for (w, t) in nltk.corpus.treebank.tagged_words(tagset='universal')])
print(len(vocabulary)) # number of words in our corpus
# we inspect the universal tagset (this is a general mapping becasue each languge use a different tagset.
tagset = set([t for (w, t) in nltk.corpus.treebank.tagged_words(tagset='universal')])

print(tagset) # tags/labes used to annotate the word class
```

    12408
    {'DET', 'ADP', 'CONJ', 'X', 'ADV', 'NOUN', 'ADJ', 'VERB', 'PRON', 'NUM', 'PRT', '.'}

```python
# The observations are pairs: (sentence, tags)
# so we can use this to get just the sentences (when we need them)
def extract_sentences(treebank_corpus):
    sentences = []
    for observations in treebank_corpus:
        sentences.append([x for x, c in observations])
    return sentences

# The observations are pairs: (sentence, tags)
# so we can use this to get just the tags (when we need them)
def extract_tags(treebank_corpus):
    tags = []
    for observations in treebank_corpus:
        tags.append([c for x, c in observations])
    return tags
```

**Exercise 4-1** **[2 points]**

* **[1 point]** Load the PTB corpus and plot the distribution of POS tags.
* **[1 point]** Do the same for the Brown corpus.

```python
import itertools
import matplotlib.pyplot as plt
from collections import defaultdict
from collections import Counter

def distribution(corpus):
    count = defaultdict(int)
    tag_list = list()
    for line in corpus:
        sets, tags = zip(*line)
        tag_list.extend(tags)
    count = Counter(tag_list)

    plt.bar(range(len(count)), count.values(), align="center")
    plt.xticks(range(len(count)), list(count.keys()))
    plt.title('Tag distribution')
    plt.show()

distribution(treebank_sents)
```

```python
brown_sents = nltk.corpus.brown.tagged_sents(tagset='universal')
distribution(brown_sents)

```

**Exercise 4-2** **[1 point]** Load the PTB corpus, find out for each POS tag what is the most likely tag that follows it.

```python
import operator

def followed_max(corpus):
    follow_count = defaultdict(int)
    for line in corpus:
        sent, tags = zip(*line)
        for i, tag in enumerate(tags):
            # Check if index is in bound else use EoS symbol
            if i < len(tags) - 1:
                next_tag = tags[i + 1]
            else: next_tag = '-EOS-'
            if tag not in follow_count:
                follow_count[tag] = defaultdict(int)
            follow_count[tag][next_tag] += 1

    follow_max = {}
    for tag, counts in follow_count.items():
        follow_max[tag] = max(counts.items(), key=operator.itemgetter(1))[0]

    return follow_max

print("PTB POS followed max by: ", followed_max(treebank_sents))
print("Brown POS followed max by: ", followed_max(brown_sents))
```

    PTB POS followed max by:  {'DET': 'NOUN', 'ADP': 'DET', 'CONJ': 'NOUN', 'X': 'VERB', 'ADV': 'VERB', 'NOUN': 'NOUN', 'VERB': 'X', 'PRON': 'VERB', 'NUM': 'NOUN', 'PRT': 'VERB', 'ADJ': 'NOUN', '.': '-EOS-'}
    Brown POS followed max by:  {'DET': 'NOUN', 'ADP': 'DET', 'CONJ': 'NOUN', 'X': 'X', 'ADV': 'VERB', 'NOUN': '.', 'VERB': 'VERB', 'PRON': 'VERB', 'NUM': 'NOUN', 'PRT': 'VERB', 'ADJ': 'NOUN', '.': '-EOS-'}

**Exercise 4-3** **[1 point]** Find the most frequent verb on the PTB and on the Brown corpus.

```python

def max_verb(corpus):
    count = defaultdict(int)
    for line in corpus:
        for w, t in line:
            if t == "VERB":
                count[w] += 1
    return max(count.items(), key=operator.itemgetter(1))

verb, c1 = max_verb(treebank_sents)
verb2, c2 = max_verb(brown_sents)
print("Most frequent verb in PTB is  :",verb, " \nWith count: ", c1)
print("Most frequent verb in Brown is:",verb2, " \nWith count: ", c2)
```

    Most frequent verb in PTB is  : is
    With count:  671
    Most frequent verb in Brown is: is
    With count:  10010

#  Hidden Markov Models

* How can we learn a PoS tagger given this dataset?

The Hidden Markov Model **HMM** is a sequence model. A sequence model or sequence classifier is a
model whose job is to assign a label or class to each unit in a sequence, thus mapping
a sequence of observations to a sequence of labels.

For example, in PoS the HMM will assing a tag to each word in a sentence:

*Mr. Vinken is chairman of Elsevier N.V. the Dutch publishing group.*

The output of a tagger would look like one of the annotated sentences form the Treeban corpus:

```python
print(treebank_training[1])
```

    [('Mr.', 'NOUN'), ('Vinken', 'NOUN'), ('is', 'VERB'), ('chairman', 'NOUN'), ('of', 'ADP'), ('Elsevier', 'NOUN'), ('N.V.', 'NOUN'), (',', '.'), ('the', 'DET'), ('Dutch', 'NOUN'), ('publishing', 'VERB'), ('group', 'NOUN'), ('.', '.')]

Let's start by defining the **HMM**

We consider two phenomena:
* Transition: we move from one "state" to another "state" where our state is the POS tag
* Emission: with a certain "state" in mind, we generate a certain word

This means that in the sentence above, for example, we generate
1. From the state `BoS` (begin of sequence) we generate the state `NOUN`
2. Then from `NOUN` we generate the word `Mr.`
3. We then "forget" the word we just emitted and use the fact that our current state is `NOUN` to generate the next state, which is again a `NOUN`
4. From where we then generate `Vinken`
5. We proceed like that until we exaust both sequences

Let us give names to things, let's model the current class with a random variable $C$ and let's use the random variable $C_{\text{prev}}$ to model the previous category. For the word we will use the random variable $X$.
Both $C$ and $C_{\text{prev}}$ take on values in the enumeration of a tagset containing $t$ tags, that is, $\{1, \ldots, t\}$. $X$ takes on values in the enumeration of a vocabulary containing $v$ words, that is, $\{1, \ldots, v\}$.

The **transition** distribution captures how our beliefs in a class vary as a function of the previous class. We will use Categorical distributions for that. In fact, for each possible previous class we get a Categorical distribution over the complete set of classes.

\begin{align}
(1) \qquad C \mid C_{\text{prev}}=p \sim \text{Cat}(\lambda_1^{(p)}, \ldots, \lambda_t^{(p)})
\end{align}

**Exercise 4-4** **[1 point]** What is the probability value of  $P_{C|C_{\text{prev}}}(c|p)$?

$\lambda_c^p$

**Exercise 4-5** **[2 points]**

* **[1 point]** How many cpds do we need in order to represent all transition distributions?
* **[1 point]** What's the representation cost of such a set of distributions? Use [big-O notation](https://en.wikipedia.org/wiki/Big_O_notation).

For every tag we have a distribution of length t.
- $t$
- $O(t^2)$

The **emission** distribution captures how our beliefs in a word vary as a function of the word's class. We will again use Categorical distributions for that. In fact, for each possible class, we get a Categorical distribution over the complete vocabulary.

\begin{align}
(2) \qquad X \mid C=c \sim \text{Cat}(\theta_1^{(c)}, \ldots, \theta_v^{(c)})
\end{align}

**Exercise 4-6** **[1 point]** What's the probability value of $P_{X|C}(x|c)$?

$\theta_x^c$

**Exercise 4-7** **[2 points]**

* **[1 point]** How many cpds do we need in order to represent all emission distributions?
* **[1 point]** What's the representation cost of such a set of distributions? Use [big-O notation](https://en.wikipedia.org/wiki/Big_O_notation).

For every tag we have a distribution of length v.
- $t$
- $O(vt)$

Now let's turn to the joint distribution $P_{CX|C_{\text{prev}}}$ over classes and words and let's focus on a single step where we can assume the previous class is already available. Then the model factorises as follows:

\begin{align}
(3) \qquad P_{CX|C_{\text{prev}}}(x, c | c_{\text{prev}}) &= P_{C|C_{\text{prev}}}(c|c_{\text{prev}}) P_{X|C}(x|c)
\end{align}

It will turn out useful to know what is the marginal distribution $P_{X|C_{\text{prev}}}$ over words given the previous class ---  where we have marginalised the current class out.

\begin{align}
(4) \qquad P_{X|C_{\text{prev}}}(x| c_{\text{prev}}) &= \sum_{c=1}^t P_{CX|C_{\text{prev}}}(x, c | c_{\text{prev}}) \\
 &= \sum_{c=1}^t P_{C|C_{\text{prev}}}(c|c_{\text{prev}}) \times P_{X|C}(x|c)
\end{align}

Note that Equation (4) starts by simply summing Equation (3) for all possible values of the current class, that is, for $c$ from 1 to $t$.

Our ultimate goal with the HMM is to assing a probability to a sentence $x_1^n$. For that we need to combine $n$ pairs $(c_i, x_i)$ and marginalise away their history $c_{i-1}$. Doing so yields the **joint probability**:

\begin{align}
(5) \qquad P_{X_1^nC_1^n|N}(x_1^n, c_1^n|n) &= P_{C|C_{\text{prev}}}(c_1|c_0)P_{X|C}(x_1|c_1)P_{C|C_{\text{prev}}}(c_2|c_1)P_{X|C}(x_2|c_1)\cdots P_{C|C_{\text{prev}}}(c_n|c_{n-1})P_{X|C}(x_n|c_n) \\
    &= \prod_{i=1}^n P(c_i|c_{i-1})P(x_i|c_i)
\end{align}

We can get the **marginal probability** of a sentence by summing over all possible values of all possible class variables. This requires summing for each and every $c_i$ from 1 to $t$ as follows:

\begin{align}
(6) \qquad P_{S|N}(x_1^n|n) &= \sum_{c_1=1}^t \cdots \sum_{c_n=1}^t P(c_1|c_0)P(x_1|c_1)P(c_2|c_1)P(x_2|c_1)\cdots P(c_n|c_{n-1})P(x_n|c_n) \\
&= \sum_{c_1=1}^t \cdots \sum_{c_n=1}^t  \prod_{i=1}^n P(c_i|c_{i-1})P(x_i|c_i)
\end{align}

This looks pretty bad! If we have to enumerate all possible tag sequences, there would be just too many of them. That is, in the first sum, $c_1$ takes 1 of $t$ values, then for each of those values $c_2$ will take 1 of $t$ values, and so on. This leads to $t^n$ different tag sequences. An exponential number of them!!! We will never manage to enumerate them, compute their joint probabilities and then sum them up.

Luckily, it turns out that in the HMM only two variables interact at a time, that is, $c_i$ interacts with $c_{i-1}$, then $c_i$ interacts with $x_i$, thus to characterise the distribution over $x_i$ we only need to know $c_{i-1}$ and $c_i$ --- which Equation (4) also shows. We can simplify Equation (6) by rearranging the sums and products as in Equation (7) below.

\begin{align}
(7) \qquad P_{S|N}(x_1^n|n) &= \sum_{c_1=1}^t \cdots \sum_{c_n=1}^t  \prod_{i=1}^n P(c_i|c_{i-1})P(x_i|c_i) \\
&= \prod_{i=1}^n \sum_{c_{i-1}=1}^t \sum_{c_i=1}^t P(c_i|c_{i-1})P(x_i|c_i)
\end{align}

This is great news because it reveals that to marginalise the tag sequences we only need to scan over the sentence $i=1, \ldots, n$ and then scan over the possible combinations of two tags from the tagset.

**Exercise 4-8** **[3 points]** What is the algorithmic complexity of computing the marginal probability of a sentence in the HMM model?

$O(nt^2)$
where $n$ is the number of words in the sentence and $t$ is the number of tags in the corpus

Equation (5) gives us a simple algorithm to assign joint probabilities to sequences of the kind $(c_1^n, x_1^n)$.
Because we chose categorical distributions as in Equations (1) and (2), we can say that

\begin{align}
P_{CX|C_{\text{prev}}}(x, c | p) = \lambda_{c}^{(p)} \times \theta_{x}^{(p)}
\end{align}

and therefore Equation (5) is simply

\begin{align}
(8) \qquad P_{X_1^nC_1^n|N}(x_1^n, c_1^n|n) &= \prod_{i=1}^n P(c_i|c_{i-1})P(x_i|c_i) \\
 &= \prod_{i=1}^n  \lambda_{c_i}^{(c_{i-1})} \times \theta_{x_i}^{(c_i)}
\end{align}

Similarly, Equation (7) is great because it gives us an algorithm to assign probabilities to sentences.
If we now use our parameters to rewrite Equation (7)  we get

\begin{align}
(9) \qquad P_{S|N}(x_1^n|n) &= \prod_{i=1}^n \sum_{c_{i-1}=1}^t \sum_{c_i=1}^t P(c_i|c_{i-1})P(x_i|c_i) \\
&= \prod_{i=1}^n \sum_{c_{i-1}=1}^t \sum_{c_i=1}^t  \lambda_{c_i}^{(c_{i-1})} \times \theta_{x_i}^{(c_i)}
\end{align}

#  Maximum likelihood estimation for labelled data

As we know by now, MLE for Categorical CPDs is just a matter of counting and dividing.

The MLE solution for the transition distribution is therefore

\begin{align}
(10) \qquad \lambda_c^{(p)} = \frac{\text{count(p, c)}}{\text{count}(p)}
\end{align}

**Exercise 4-9** **[1 points]** What is the MLE solution for the emission distribution? Note: use the same style as in Equation (10), that is, state the parameter and the solution.

\begin{align}
(10) \qquad \theta_v^{(c)} = \frac{\text{count(c, v)}}{\text{count}(c)}
\end{align}

Now you will **implement** the HMM, this means you will implement *transition distributions*, *emission distributions*, *Laplace smoothing*, *joint probability*, *marginal probability*, and an algorithm for making *predictions*.

#  Implementation

Here you will implement a language model based on an HMM POS tagger.

**Exercise 4-10** **[17 points]**

You will need to complete the skeleton class below. Read it through before coding. Check the documentation for additional information. Document your steps (this will increase your chance of earning points in case you make mistakes along the way).

* **[8 points]** Implement `estimate_model`: this should take an annotated corpus (such as PTB or Brown) and estimate the CPDs in the model using Laplace smoothing.
* **[2 points]** Implement `transition_parameter`: see method's documentation
* **[2 points]** Implement `emission_parameter`: see method's documentation
* **[1 points]** Implement `joint_parameter`: see method's documentation
* **[2 points]** Implement `log_joint`: see method's documentation
* **[2 points]** Implement `log_marginal`: see method's documentation

Show that you methods work by training a model and testing a few examples. This is an example

```python
treebank_hmm = HMMLM()
treebank_hmm.estimate_model(treebank_training)
print(treebank_hmm.joint_parameter('DET', 'NOUN', 'book'))
sentence = [x for x, _ in treebank_dev[0]]
tag_sequence = [c for _, c in treebank_dev[0]]
print(' '.join(sentence))
print(' '.join(tag_sequence))
print(treebank_hmm.log_joint(sentence, tag_sequence))
```

for which we get

```
9.959527643028553e-05
At Tokyo , the Nikkei index of 225 selected issues , which *T*-1 gained 132 points Tuesday , added 14.99 points to 35564.43 .
ADP NOUN . DET NOUN NOUN ADP NUM VERB NOUN . DET X VERB NUM NOUN NOUN . VERB NUM NOUN PRT NUM .
-193.71018537
```

```python
import numpy as np
from collections import defaultdict

class HMMLM:
    """
    This is our HMM language model class.

    It will be responsible for estimating parameters by MLE
    as well as computing probabilities using the HMM.

    We will use Laplace smoothing by default (because we do not want to assign 0 probabilities).

    GUIDELINES:
        - by convention we will use the string '-UNK-' for an unknown POS tag
            - and '<unk>' for an unknown word
        - don't forget that with Laplace smoothing the unknown symbols have to be in the support of distributions
        - now you will have 2 types of distributions, so you should deal with unknown symbols for both of them
        - we also need padding for sentences and tag sequences, by convention we will use
            - '-BOS-' and '-EOS-' for padding tag sequences
            - '<s>' and '</s>' for padding sentences
        - do recall that '-BOS-' is **not** a valid tag
            in other words we never *generate* '-BOS-' tags, we only pretend they occur at
            the 0th position of the tag sequence in order to provide conditioning context
            for the first actual tag
        - similarly, '<s>' is not a valid word
            in other words, we never *generate* '<s>' as a word
            in fact '<s>' is optional as no emission event is based on it
        - on the other hand, '-EOS-' is a valid tag
            you should model it as the last event of a tag sequence
        - similarly, '</s>' is a valid word
            you should consider it as the last event of a sentence

    You can use whatever data structures you like for cpds
        - we suggest python dict or collections.defaultdict
            but you are free to experiment with list and/or np.array if you like
    """

    def __init__(self, transition_alpha=1.0, emission_alpha=1.0):
        self._vocab = set()
        self._tagset = set()
        self._emission_cpds = dict()
        self._transition_cpds = dict()
        self._transition_alpha = transition_alpha
        self._emission_alpha = emission_alpha

    def tagset(self):
        """
        Return the tagset: a set of all tags seen by the model (including '-UNK-').

        You can modify this if you judge necessary (for example, because you decided  to
            use different datastructures, but do note that we provide you an implementation
            of the Viterbi algorithm that expects this functionality).
        """
        # the -BOS- tag is just something for internal representation
        #  in case you have added it to the tagset, we are removing it here
        #  as keeping it would be bad for algorithms such as Viterbi
        # the -UNK- tag must be in the support (due to Laplace smoothing)
        #  thus in case you forgot it, we are adding it now
        return self._tagset - {'-BOS-'} | {'-UNK-'}

    def addTag(self, tag):
        """
        Adds a tag to the tagset variable
        """
        self._tagset.add(tag)

    def vocab(self):
        """
        Return the vocabulary of words: all words seen by the model (including '<unk>').

        You can modify this if you judge necessary (for example, because you decided  to
            use different datastructures, but do note that we provide you an implementation
            of the Viterbi algorithm that expects this functionality).
        """
        # the <s> token is just something for internal representation
        #  in case you have added it to the vocabulary, we are removing it here
        # the <unk> word must be in the support (due to Laplace smoothing)
        #  thus in case you forgot it, we are adding it now
        return self._vocab - {'<s>'} | {'<unk>'}

    def addWord(self, word):
        """
        Adds a word to the vocab variable
        """

        self._vocab.add(word)

    def preprocess_sentence(self, sentence, bos=True, eos=True):
        """
        Preprocess a sentence by lowercasing its words and possibly padding it.

        :param sentence: a list of tokens (each a string)
        :param bos: if True you will get <s> at the beginning
        :param eos: if True you will get </s> at the end
        :returns: a list of tokens (lowercased strings)
        """
        # lowercase
        sentence = [x.lower() for x in sentence]
        # optional padding
        if bos:
            sentence = ['<s>'] + sentence
        if eos:
            sentence = sentence + ['</s>']
        return sentence

    def preprocess_tag_sequence(self, tag_sequence, bos=True, eos=True):
        """
        Preprocess a tag sequence with optional padding.

        :param tag_sequence: a list of tags (each a string)
        :param bos: if True you will get -BOS- at the beginning
        :param eos: if True you will get -EOS- at the end
        :returns: a list of tokens
        """
        # optional padding
        if bos:
            tag_sequence = ['-BOS-'] + tag_sequence
        if eos:
            tag_sequence = tag_sequence + ['-EOS-']
        return tag_sequence

    def estimate_model(self, treebank):
        """
        :param treebank: a sequence of observations as provided by nltk
            each observation is a list of pairs (x_i, c_i)
            and they have not yet been pre-processed

        Estimate the model parameters.

        This method does not have to return anything, it simply computes the necessary cpds.
        """

        # Create count table for emission and transition(defaultdict(int))
        emis_count_table = {'-UNK-' : {'<unk>': 0}}
        tran_count_table = {'-UNK-' : {'-UNK-': 0}}

        print("Start counting")
        for i, tag_sent in enumerate(treebank):
            # Preprocess the sentence and tag_sequence
            sentence, tags = map(list, zip(*tag_sent))
            sentence = self.preprocess_sentence(sentence)
            tags = self.preprocess_tag_sequence(tags)

            # Fill count tables
            for i, word in enumerate(sentence[1:], 1):
                tag = tags[i]
                tag_prev = tags[i-1]

                # Add word to tag emission_count: P(word|tag)
                if tag not in emis_count_table:
                    emis_count_table[tag] = defaultdict(int)
                emis_count_table[tag][word] += 1
                if '<unk>' not in emis_count_table[tag]:
                    emis_count_table[tag]['<unk>'] = 0

                # Add tag to prevtag in transition_count: P(tag|tag_prev)
                if tag_prev not in tran_count_table:
                    tran_count_table[tag_prev] = defaultdict(int)
                tran_count_table[tag_prev][tag] += 1
                if '-UNK-' not in tran_count_table[tag_prev]:
                    tran_count_table[tag_prev]['-UNK-'] = 0

                # Add tag and word to tagset andd vocab
                self.addTag(tag)
                self.addWord(word)

        print("Start calculating cpd's")
        # Parse count tables and convert them to CPD's
        for i, (tag, word_count) in enumerate(emis_count_table.items()):
            print('.', end='')
            self._emission_cpds[tag] = defaultdict(float)
            total_count = sum(word_count.values())
            for word, count in word_count.items():
                prob = (float(count) + self._emission_alpha) / \
                       (total_count + self._emission_alpha * len(self.vocab()))
                self._emission_cpds[tag][word] = prob

        for tag_prev, tag_count in tran_count_table.items():
            self._transition_cpds[tag_prev] = defaultdict(float)
            total_count = sum(tag_count.values())

            for tag, count in tag_count.items():
                prob = (float(count) + self._transition_alpha) / \
                       (total_count + self._transition_alpha * len(self.tagset()))
                self._transition_cpds[tag_prev][tag] = prob

        print("\nFinished cpd's")

    def transition_parameter(self, previous_tag, current_tag):
        """
        This method returns the transition probability for tag given the previous tag.

        Tips: do not forget that we have a smoothed model, thus
            - if the either tag was never seen, you should pretend it to be '-UNK-'

        :param previous_tag: the previous tag (str)
        :param current_tag: the current tag (str)
        :return: transition parameter
        """
        if previous_tag not in self._transition_cpds:
            previous_tag = '-UNK-'
        if current_tag not in self._transition_cpds[previous_tag]:
            current_tag = '-UNK-'
        return self._transition_cpds[previous_tag][current_tag]

    def emission_parameter(self, tag, word):
        """
        This method returns the emission probability for a word given a tag.
        Tips: do not forget that we have a smoothed model, thus
            - if the tag was never seen, you should pretend it to be '-UNK-'
            - similarly, if the word was never seen, you shoud pretend it to be '<unk>'

        :param tag: the current tag (str)
        :param word: the current word (str)
        :return: the emission probability
        """
        if tag not in self._emission_cpds:
            tag = '-UNK-'
        if word not in self._emission_cpds[tag]:
            word = '<unk>'
        return self._emission_cpds[tag][word]

    def joint_parameter(self, previous_tag, current_tag, word):
        """
        This method returns the joint probability of (current tag, word) given the previous tag
            according to Equation (3)

        :param previous_tag: the previous tag (str)
        :param current_tag: the current tag (str)
        :param word: the current word (str)
        :returns: P(word, current_tag|previous_tag)
        """
        pcp = self.transition_parameter(previous_tag, current_tag)
        pxc = self.emission_parameter(current_tag, word)
        return pcp * pxc

    def marginal_x_given_cprev(self, previous_tag, word):
        """
        Return P(x|prev) as defined in Equation (4) by marginalising current tag.

        :param previous_tag: the previous tag (str)
        :param word: the current word (str)
        """
        return np.sum([self.joint_parameter(previous_tag, c, word) for c in self._tagset])

    def log_joint(self, sentence, tag_sequence):
        """
        Implement the logarithm of the joint probability over a sentence and tag sequence as in Equation (8)

        :param sentence: a sequence of words (each a string) not yet preprocessed
        :param tag_sequence: a sequence of tags (eac a string) not yet preprocessed
        :returns: log P(x_1^n, c_1^n|n) as defined in Equation (8)
        """
        sentence = self.preprocess_sentence(sentence)
        tag_sequence = self.preprocess_tag_sequence(tag_sequence)

        joint_prob = 0
        for i, word in enumerate(sentence[1:], 1):
            tag = tag_sequence[i]
            prev_tag = tag_sequence[i - 1]
            joint_prob += np.log(self.joint_parameter(prev_tag, tag, word))
        return joint_prob

    def log_marginal(self, sentence):
        """
        Implement the logarithm of the marginal probability of a sentence as in Equation (9)
            by marginalisation of all possible tag sequences.

        :param sentence: a sequence of words (each a string) not yet preprocessed
        :returns: log P(x_1^m|n) as defined in Equation (9)
        """
        sentence = self.preprocess_sentence(sentence)

        return sum([np.log(sum([self.marginal_x_given_cprev(p,w) for p in self.tagset()])) for w in sentence])
```

```python
treebank_hmm = HMMLM()
treebank_hmm.estimate_model(treebank_training)

print(treebank_hmm.joint_parameter('DET', 'NOUN', 'book'))
sentence, tag_sequence = map(list, zip(*treebank_dev[0]))
print(' '.join(sentence))
print(' '.join(tag_sequence))
print(treebank_hmm.log_joint(sentence, tag_sequence))
print(treebank_hmm.log_marginal(sentence))

```

    Start counting
    Start calculating cpd's
    ..............
    Finished emission cpd's
    ..............
    Finished transition cpd's
    9.959527643028553e-05
    At Tokyo , the Nikkei index of 225 selected issues , which *T*-1 gained 132 points Tuesday , added 14.99 points to 35564.43 .
    ADP NOUN . DET NOUN NOUN ADP NUM VERB NOUN . DET X VERB NUM NOUN NOUN . VERB NUM NOUN PRT NUM .
    -193.71018536983053
    -123.19211447683843

```python
print(treebank_hmm.joint_parameter('DET', 'VERB', 'cat'))

```

    7.062115274614483e-06

#  Evaluation

We can evaluate our models by the computing the log-perplexity, like with the LM or by comparing the predictions of the trained model with an annotated test set.

##  Perplexity

Perplexity of a model on a test set is the inverse probability of the test set, normalized
by the number of words. Perplexity is a notion of average branching factor, thus a model with low perplexity can be thought of as a *less confused*. That is, each time it introduces a word given some history it picks from a reduced subset of the entire vocabulary (in other words, it is more certain of how to continue).

If a dataset contains $t$ tokens where $t = \sum_{k=1}^m n_k$, then the perplexity of the dataset is

\begin{equation}
(11) \qquad \text{PP}(\mathcal T) = \left( \prod_{k=1}^m P_{S|N}(\langle x_1^{(k)}, \ldots, x_{n_k}^{(k)} \rangle|n_k; \boldsymbol \theta) \right)^{-1/t}
\end{equation}

where we have already discarded the length distribution (since it's held constant across models). And the probability of the sentence requires marginalising tag sequences, as shown in Equation (9).

It's again convenient to use log and define log-perplexity

\begin{equation}
(12) \qquad \log \text{PP}(\mathcal T) = - \frac{1}{t} \sum_{k=1}^m \log P_{S|N}(\langle x_1^{(k)}, \ldots, x_{n_k}^{(k)} \rangle|n_k; \boldsymbol \theta)
\end{equation}

You can compare models in terms of the log-perplexity they assign to the same test data. The lower the perplexity, the better the model is.

**Exercise 4-11** **[4 points]** Implement `log_perplexity` below. Train models of PTB and Brown and test both of them on their respective test sets as well as on each other's test set. Report all results.

To help you have an idea whether you implemented it right, this is an excerpt of what we got with our implementation

```python
dev_sentences = extract_sentences(treebank_dev)
log_perplexity(dev_sentences, treebank_hmm)
```

```
101.80708039918399
```

By the way, this is how you load the Brown corpus. Let's use the last 1000 sentences for test. You can reduce the size of the training set if your computer cannot handle what we suggest below.

```python
# load the Brown corpus wiht the universal tag set
brown_sentences = nltk.corpus.brown.tagged_sents(tagset='universal')

# we split our corpus on training, dev and test
brown_training = list(brown_sentences[:56000])
brown_dev = list(brown_sentences[56000:56340])
brown_test = list(brown_sentences[56340:])
print(len(brown_sentences), len(brown_training), len(brown_dev), len(brown_test))
```

    57340 56000 340 1000

```python
def log_perplexity(sentences, hmm):
    """
    For a dataset of sentences (each sentence is a list of words)
        and an instance of the HMMLM class
        return the log perplexity as defined in Equation (12)
    """

    t = sum([len(s) + 2 for s in sentences])
    return -1.0 / t * sum([hmm.log_marginal(s) for s in sentences])
```

```python
dev_sentences = extract_sentences(treebank_dev)
print(log_perplexity(dev_sentences, treebank_hmm))

brown_hmm = HMMLM()
brown_hmm.estimate_model(brown_training)

brown_dev_sentences = extract_sentences(brown_dev)
print(log_perplexity(brown_dev_sentences, brown_hmm))
```

    4.689496628253122
    Start counting
    Start calculating cpd's
    ..............
    Finished emission cpd's
    ..............
    Finished transition cpd's
    4.668776756438641

##  Viterbi

The Viterbi algorithm is used to search through the space of possible tag sequences and find the one that score highest.

The space of tag sequences is very large, consider that we can select any of $t$ tags for each position, thus by simple counting, we are left with $t^n$ possible sequences. Searching through an exponential space is intractable in general.

Not by chance we designed the HMM with certain conditional independence assumptions. In fact, in our HMM model the probability of each word observation really only depends on two decisions, namely, its tag and its preceding tag. We can use that to derive a *tractable* dynamic program.

[Dynamic programming](https://en.wikipedia.org/wiki/Dynamic_programming) is a kind of *divide and conquer* strategy. We have to identify sub-problems that we can solve efficiently and maintain a memory of partial solutions that we use to build the final one.

For the Viterbi algorithm we need to solve the problem "what is the best probability so far". We will approach this problem with a recursion that computes the probability of the best path of a certain length.
We will use the recursive function $\alpha(i, j)$ which returns the best probability for a path of length $i$ assuming that its last tag is $C_i=j$. For efficiency in our implementation we will map tags to integers.

The Viterbi recursion is pretty simple, if a path has length $0$ we will assume it has probability $1$. If a path has length $i$ (more than $0$), we reckon that its maximum probability is based on the maximum probability assigned to paths of size $i-1$, where we extend those paths with one steps and check which step yields the best probability.

The Viterbi recursion is formalised below:

\begin{align}
(13)\qquad \alpha(i, j) &=
\begin{cases}
1 & \text{if }i = 0 \\
\max_{p \in \{1, \ldots, t\}} \alpha(i-1, p) \times \lambda_{j}^{p} \times \theta_{x_i}^{j} & \text{otherwise}
\end{cases}
\end{align}

Note that in the second line, we look for paths of size $i-1$ (which my have ended in one of $t$ possible previous tags) and we try to extend them with tag $j$. The probability of each hypothetical extension is the probability of the path we mean to extend $\alpha(i-1, p)$ times the joint parameters associated with the extension. That is, continuing from previous class $C_{i-1}=p$ with current class $C_i=j$ (that is a transition parameter), and generating the current word $X_i=x_i$ from the current class $C_i=j$ ( that is an emission parameter).

Note that the recursive function takes 2 inputs, $i$ which ranges from 1 to $n$, and $j$ which ranges from 1 to $t$. Thus we need to make at most $nt$ calls to this function. But also note that each time we call it, we have to solve (in the second line) a maximisation over $t$ possible assignments of the previous tag. Therefore the overall complexity of the algorithm is $O(nt^2)$. That's quite an improvement from $O(t^n)$, isn't it?

Of course, crucial to the success of the algorithm is that we use [memoization](https://en.wikipedia.org/wiki/Memoization), a technique by which we store partial solutions and never recompute things.

We provide you with a **recursive** implementation of the Viterbi algorithm which uses the interface of the HMMLM class that you designed above. You can use this implementation if you want to experiment with PTB and Brown corpus in the exercises below. Read the implementation carefully to learn from it. We have provided extensive documentation.
One important note is that we compute everything in log space, that's to rely on better numerical properties of log probabilities.

The Viterbi recursion gives us a way to compute the *probability* of the best path. To find the actual best path we just need to traverse the table of $\alpha(i, j)$ values looking for the decision (tag) that is best at each point.

```python
def viterbi_recursion(sentence, hmm):
    """
    Computes the best possible tag sequence for a given input
    and also returns it log probability.

    This implementation uses recursion.

    :returns: tag sequence, log probability
    """
    # here we pad the sentence with </s> only
    sentence = hmm.preprocess_sentence(sentence, bos=False, eos=True)
    # this is the length (but recall that padding added 1 token)
    n = len(sentence)
    # this is the complete tagset, which for convenience we will turn into a list
    tagset = list(hmm.tagset())
    t = len(tagset)
    # We need a table to store log alpha(i, j) values
    # - where i is an integer from 0 to n-1 which refers to a position in the list `sentence`
    #   i.e. sentence[i]
    # - and j is an integer from 0 to t-1 that refers to a tag in the list `tagset`
    #   i.e. tagset[j]
    # - together (i, j) means that we are setting `C_i = tagset[j]`
    # - we will be exploring the space of possible tags per position
    #   thus our table has as many as n * t cells
    # - Recall that the value \log \alpha(i, j)
    #   corresponds to the log probability value of the best
    #   path (C_1, ..., C_i) such that C_i = j
    #   in other words the log probability of the best sequence up to the ith token where C_i = j
    # At the beginning path probabilities have not been computed, we use a probability of 0 to indicate that
    #  as we will be computing log probabilities, we use -inf instead
    #  numpy arrays are very handy and we can actually use the quantity -inf
    log_alpha_table = np.full([n, t], -float('inf'))
    # In a best path algorithm we are interested in two things
    #  the best score (or best log probability)
    #  as well as the path that corresponds to the best score
    # We compute the best score by moving i forward from 0 to n-1 computing the maximum value
    #  and we traverse the table backwards following the path that led to the maximum
    #  thus we create a table of "back pointers"
    #  this is an integer for each cell (i, j) that tells us which tag `p` for position `i - 1`
    #   leads to the score stored in `log_alpha_table[i, j]`
    back_pointer_table = np.full([n, t], -1, dtype=int)

    # Here we define the log alpha recursion
    def log_alpha(i, j):
        """
        This function returns
                max_{c_1, ..., c_i=j} log P(c_1, ..., c_i=j)
            where i is a (0-based) position in `sentence`
            and j is a (0-based) position in `tagset`
        """
        if i == 0:  # we do not need to tag the 0th position and it should not affect the probability
            return 0.  # np.log(1)
        # When we implement dynamic programs, we like to re-use computations already made
        # thus first of all we test if we have already computed a value for this cell
        # if so, it will not have a zero probability (-inf in log space)
        if log_alpha_table[i, j] != -float('inf'):
            # then we can simply return it
            return log_alpha_table[i, j]
        # At this point we know we have not yet computed a score for this path
        #  thus we proceed to compute it
        # We will have to figure out the log prob of the best prefix
        #  and which tag best continues from it
        # There are exactly t classes that may tag this position
        #  thus we just go over the tagset trying one at a time
        #  and memorise the score we would have if we would select them
        path_max_log_prob = np.full(t, -float('inf'))
        for p in range(t):
            # this is the essential part of the recursion
            # we ask for the best score associated with the previous position
            #  had it been tagged with p
            #  and we incorporate the probability of C_i = tagset[j] given that C_{i-1} = tagset[p]
            #   as well as the probability of X_i = sentence[i] given that C_i = tagset[j]
            path_max_log_prob[p] = log_alpha(i - 1, p) + np.log(hmm.joint_parameter(tagset[p], tagset[j], sentence[i]))
        # From all possibilities, we are only interested in the best
        log_alpha_table[i, j] = np.max(path_max_log_prob)
        # and we also want to store a pointer to the best
        back_pointer_table[i, j] = np.argmax(path_max_log_prob)
        return log_alpha_table[i, j]

    # Let's get the index associated with -EOS-
    #  which is the tag for the </s> symbol in sentence[-1]
    eos_index = tagset.index('-EOS-')
    # We want the last word in the sentence (</s>) to have the tag -EOS-
    #  thus we ask "what's the probability of the best path that ends in -EOS-?"
    max_log_prob = log_alpha(n - 1, eos_index)

    # Here we retrieve the backpointers for the best analysis
    #  the best analisys has n tags
    bwd_argmax = [None] * n
    #  the last tag is the -EOS- symbol
    bwd_argmax[-1] = eos_index
    # Here we maintain the "current tag" c_i
    c_i = eos_index
    for i in range(n - 1, 0, -1):  # we go backwards from c_{n-1} to c_1
        # and set the value of c_{i-1} for the current c_i
        bwd_argmax[i - 1] = back_pointer_table[i, c_i]
        # we need, of course, to update c_i
        c_i = bwd_argmax[i - 1]

    # Here we translate from ids back to actual tags (strings)
    #  we leave the -EOS- symbol out, since it was just a convenience
    #  and return both the tag sequence and the total log probability
    return [tagset[c] for c in bwd_argmax[:-1]], max_log_prob
```

```python
# let's tag a sentence
viterbi_path, viterbi_log_prob = viterbi_recursion(['i', 'wish', 'i', 'had', 'a', 'book', '.'], treebank_hmm)
print(viterbi_path, viterbi_log_prob)
```

    ['PRON', 'VERB', 'PRON', 'VERB', 'DET', 'NOUN', '.'] -42.817371568242855

##   Accuracy

We can evaluate the performance of our tagger by comparing its Viterbi predictions to human annotation.

For this, we will use the gold standard test data (e.g. treebank_test). Evaluation metrics compute a score for a given model (e.g. our HMM tagger) by comparing the predicted labels that the model generated with the test data againts the gold standard annotation.

The **Accuracy** metric computes the percentage of instances in the test data that our tagger labeled correctly.
If we have a dataset of $m$ labelled sequences
\begin{equation}
\left( \langle x_1^{(k)}, \ldots, x_{n_k}^{(k)}\rangle, \langle \star_1^{(k)}, \ldots, \star_{n_k}^{(k)}\rangle \right)_{k=1}^m
\end{equation}
we can produce all Viterbi predictions
\begin{equation}
\left( \langle x_1^{(k)}, \ldots, x_{n_k}^{(k)}\rangle, \langle c_1^{(k)}, \ldots, c_{n_k}^{(k)}\rangle \right)_{k=1}^m
\end{equation}

and compute

\begin{align}
(14)\qquad \text{accuracy} &= \frac{\sum_{k=1}^m \sum_{i=1}^{n_k} [c_i^{(k)} = \star_i^{(k)}]}{\sum_{k=1}^m n_k}\\
&= \frac{\text{number of correct predictions}}{\text{total tokens}}
\end{align}

**Exercise 4-12** **[5 points]**

* **[1 point]** Implement the accuracy function.
* **[1 point]** Compute accuracy on PTB's test set using a PTB-trained model
* **[1 point]** Compute accuracy on PTB's test set using a Brown-trained model
* **[1 point]** Compute accuracy on Brown's test set using a PTB-trained model
* **[1 point]** Compute accuracy on Brown's test set using a Brown-trained model

For this exercise you can use the Viterbi implementation we provided (or your own -- its up to you). With our own implementation we got $0.8446$ accuracy for PTB-training and PTB-test, and $0.8960$ for Brown-training and Brown-test.

```python
def accuracy(gold_sequences, pred_sequences):
    """
    Return percentage of instances in the test data that our tagger labeled correctly.

    :param gold_sequences: a list of tag sequences that can be assumed to be correct
    :param pred_sequences: a list of tag sequences predicted by Viterbi
    """
    count_correct, count_total = 0, 0
    for i, combined in enumerate(zip(pred_sequences, gold_sequences)):
        for p, g in list(zip(*combined)):
            if p == g:
                count_correct += 1
            count_total += 1
    if count_total:
        return count_correct / count_total
    return None

def predict_corpus(test_set, hmm):
    """
    Returns viterbi predictions for all sentences in a given corpus

    :param test_set: A corpus of tagged sentences
    :param hmm     : A language model
    """
    gold_sequences, pred_sequences = list(), list()
    print('Making predictions', end='')
    for i, sequence in enumerate(test_set):
        if i % round(len(test_set) / 10) == 0:
            print('.', end='')
        sentence , tags = map(list, zip(*sequence))
        viterbi_tags, _ = viterbi_recursion(sentence, hmm)
        gold_sequences.append(tags)
        pred_sequences.append(viterbi_tags)
    return gold_sequences, pred_sequences

print("PTB -> PTB accuracy: ",round(accuracy(*predict_corpus(treebank_test, treebank_hmm)),4))
print("PTB -> Brown accuracy: ",round(accuracy(*predict_corpus(treebank_test, brown_hmm)),4))
print("Brown -> PTB accuracy: ",round(accuracy(*predict_corpus(brown_test, treebank_hmm)),4))
print("Brown -> Brown accuracy: ",round(accuracy(*predict_corpus(brown_test, brown_hmm)),4))

```


```python
%matplotlib inline
```

강화 학습 (DQN) 튜토리얼
=====================================
**Author**: `Adam Paszke <https://github.com/apaszke>`_
  **번역**: `황성수 <https://github.com/adonisues>`_

이 튜토리얼에서는 `OpenAI Gym <https://gym.openai.com/>`__
CartPole-v0 태스크의 DQN (Deep Q Learning) 에이전트를 학습하는데
PyTorch를 사용하는 방법을 보여드립니다.

**태스크**

에이전트는 연결된 막대가 똑바로 서 있도록 카트를 왼쪽이나 오른쪽으로
움직이는 두 가지 동작 중 하나를 선택해야합니다.
다양한 알고리즘과 시각화 기능을 갖춘 공식 순위표를
`Gym website <https://gym.openai.com/envs/CartPole-v0>`__ 에서 찾을 수 있습니다.

.. figure:: /_static/img/cartpole.gif
   :alt: cartpole

   cartpole

에이전트가 현재 환경 상태를 관찰하고 행동을 선택하면
환경이 새로운 상태로 *전환* 되고 작업의 결과를 나타내는 보상도 반환됩니다.
이 태스크에서는 막대가 지나치게 떨어지면 환경이 종료됩니다.

카트폴 태스크는 에이전트에 대한 입력이 환경 상태(위치, 속도 등)를 나타내는
4개의 실제 값이 되도록 설계되었습니다. 그러나 신경망은 순수하게 그 장면을 보고
태스크를 해결할 수 있습니다 따라서 카트 중심의 화면 패치를 입력으로 사용합니다.
이 때문에 우리의 결과는 공식 순위표의 결과와 직접적으로 비교할 수 없습니다.
우리의 태스크는 훨씬 더 어렵습니다.
불행히도 모든 프레임을 렌더링해야되므로 이것은 학습 속도를 늦추게됩니다.

엄밀히 말하면, 현재 스크린 패치와 이전 스크린 패치 사이의 차이로 상태를 표시 할 것입니다.
이렇게하면 에이전트가 막대의 속도를 한 이미지에서 고려할 수 있습니다.

**패키지**

먼저 필요한 패키지를 가져옵니다. 첫째, 환경을 위해
`gym <https://gym.openai.com/docs>`__ 이 필요합니다.
(`pip install gym` 을 사용하여 설치하십시오).
또한 PyTorch에서 다음을 사용합니다:

-  신경망 (``torch.nn``)
-  최적화 (``torch.optim``)
-  자동 미분 (``torch.autograd``)
-  시각 태스크를 위한 유틸리티들 (``torchvision`` - `a separate
   package <https://github.com/pytorch/vision>`__).

```python
import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from collections import namedtuple
from itertools import count
from PIL import Image

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T

env = gym.make('CartPole-v0').unwrapped

# matplotlib 설정
is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython:
    from IPython import display

plt.ion()

# GPU를 사용할 경우
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

재현 메모리(Replay Memory)
-------------------------------

우리는 DQN 학습을 위해 경험 재현 메모리를 사용할 것입니다.
에이전트가 관찰한 전환(transition)을 저장해고 나중에 이 데이터를
재사용 할 수 있습니다. 무작위로 샘플링하면 배치를 구성한는 전환들이
비상관(decorrelated)하게 됩니다. 이것이 DQN 학습 절차를 크게 안정시키고
향상시키는 것으로 나타났습니다.

이를 위해서 두개의 클래스가 필요합니다:

-  ``Transition`` - 우리 환경에서 단일 전환을 나타내도록 명명된 튜플
-  ``ReplayMemory`` - 최근 관찰된 전이를 보관 유지하는 제한된 크기의 순환 버퍼.
   또한 학습을 위한 전환의 무작위 배치를 선택하기위한
   ``.sample ()`` 메소드를 구현합니다.

```python
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """전환 저장"""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)
```

이제 모델을 정의합시다. 그러나 먼저 DQN이 무엇인지 간단히 요약해 보겠습니다.

DQN 알고리즘
-------------

우리의 환경은 결정론적이므로 여기에 제시된 모든 방정식은 단순화를 위해
결정론적으로 공식화됩니다. 강화 학습 자료은 환경에서 확률론적 전환에
대한 기대값(expectation)도 포함 할 것입니다.

우리의 목표는 할인된 누적 보상 (discounted cumulative reward)을
극대화하려는 정책(policy)을 학습하는 것입니다.
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, 여기서
$R_{t_0}$ 는 *반환(return)* 입니다. 할인 상수,
$\gamma$, 는 $0$ 과 $1$ 의 상수이고 합계가
수렴되도록 보장합니다. 에이전트에게 불확실한 먼 미래의 보상이
가까운 미래의 것에 비해 덜 중요하게 만들고, 이것은 상당히 합리적입니다.

Q-learning의 주요 아이디어는 만일 함수 $Q^*: State \times Action \rightarrow \mathbb{R}$ 를
가지고 있다면 반환이 어덯게 될지 알려줄 수 있고,
만약 주어진 상태(state)에서 행동(action)을 한다면, 보상을 최대화하는
정책을 쉽게 구축할 수 있습니다:

\begin{align}\pi^*(s) = \arg\!\max_a \ Q^*(s, a)\end{align}

그러나 세계(world)에 관한 모든 것을 알지 못하기 때문에,
$Q^*$ 에 도달할 수 없습니다. 그러나 신경망은
범용 함수 근사자(universal function approximator)이기 때문에
간단하게 생성하고 $Q^*$ 를 닮도록 학습 할 수 있습니다.

학습 업데이트 규칙으로, 일부 정책을 위한 모든 $Q$ 함수가
Bellman 방정식을 준수한다는 사실을 사용할 것입니다:

\begin{align}Q^{\pi}(s, a) = r + \gamma Q^{\pi}(s', \pi(s'))\end{align}

평등(equality)의 두 측면 사이의 차이는
시간차 오류(temporal difference error), $\delta$ 입니다.:

\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

오류를 최소화하기 위해서 `Huber
loss <https://en.wikipedia.org/wiki/Huber_loss>`__ 를 사용합니다.
Huber loss 는 오류가 작으면 평균 제곱 오차( mean squared error)와 같이
동작하고 오류가 클 때는 평균 절대 오류와 유사합니다.
- 이것은 $Q$ 의 추정이 매우 혼란스러울 때 이상 값에 더 강건하게 합니다.
재현 메모리에서 샘플링한 전환 배치 $B$ 에서 이것을 계산합니다:

\begin{align}\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)\end{align}

\begin{align}\text{where} \quad \mathcal{L}(\delta) = \begin{cases}
     \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
     |\delta| - \frac{1}{2} & \text{otherwise.}
   \end{cases}\end{align}

Q-네트워크
^^^^^^^^^^^

우리 모델은 현재와 이전 스크린 패치의 차이를 취하는
CNN(convolutional neural network) 입니다. 두가지 출력 $Q(s, \mathrm{left})$ 와
$Q(s, \mathrm{right})$ 가 있습니다. (여기서 $s$ 는 네트워크의 입력입니다)
결과적으로 네트워크는 주어진 현재 입력에서 각 행동의 *품질* 을 예측하려고 합니다.

```python
class DQN(nn.Module):

    def __init__(self):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)
        self.head = nn.Linear(448, 2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))
```

입력 추출
^^^^^^^^^^^^^^^^

아래 코드는 환경에서 렌더링 된 이미지를 추출하고 처리하는 유틸리티입니다.
이미지 변환을 쉽게 구성 할 수 있는 ``torchvision`` 패키지를 사용합니다.
셀(cell)을 실행하면 추출한 예제 패치가 표시됩니다.

```python
resize = T.Compose([T.ToPILImage(),
                    T.Resize(40, interpolation=Image.CUBIC),
                    T.ToTensor()])

# 이것은 gym 코드를 기반으로 합니다.
screen_width = 600

def get_cart_location():
    world_width = env.x_threshold * 2
    scale = screen_width / world_width
    return int(env.state[0] * scale + screen_width / 2.0)  # 카트의 중간

def get_screen():
    screen = env.render(mode='rgb_array').transpose(
        (2, 0, 1))  # transpose into torch order (CHW)
    # Strip off the top and bottom of the screen
    screen = screen[:, 160:320]
    view_width = 320
    cart_location = get_cart_location()
    if cart_location < view_width // 2:
        slice_range = slice(view_width)
    elif cart_location > (screen_width - view_width // 2):
        slice_range = slice(-view_width, None)
    else:
        slice_range = slice(cart_location - view_width // 2,
                            cart_location + view_width // 2)
    # Strip off the edges, so that we have a square image centered on a cart
    screen = screen[:, :, slice_range]
    # Convert to float, rescare, convert to torch tensor
    # (this doesn't require a copy)
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return resize(screen).unsqueeze(0).to(device)

env.reset()
plt.figure()
plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),
           interpolation='none')
plt.title('Example extracted screen')
plt.show()
```

학습
--------

하이퍼 파라미터와 유틸리티
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
이 셀은 모델과 최적화기를 인스턴스화하고 일부 유틸리티를 정의합니다:

-  ``select_action`` - Epsilon Greedy 정책에 따라 행동을 선택합니다.
   간단히 말해서, 가끔 모델을 사용하여 행동을 선택하고 때로는 단지 하나를
   균일하게 샘플링 할 것입니다. 임의의 액션을 선택할 확률은
   ``EPS_START`` 에서 시작해서 ``EPS_END`` 를 향해 지수적으로 감소 할 것입니다.
   ``EPS_DECAY``는 감쇠 속도를 제어합니다.
-  ``plot_durations`` - 지난 100개 에피소드의 평균(공식 평가에서 사용 된 수치)에 따른
   에피소드의 지속을 도표로 그리기 위한 헬퍼. 도표는 기본 훈련 루프가
   포함 된 셀 밑에 있으며, 매 에피소드마다 업데이트됩니다.

```python
BATCH_SIZE = 128
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 10

policy_net = DQN().to(device)
target_net = DQN().to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.RMSprop(policy_net.parameters())
memory = ReplayMemory(10000)

steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = EPS_END + (EPS_START - EPS_END) * \
        math.exp(-1. * steps_done / EPS_DECAY)
    steps_done += 1
    if sample > eps_threshold:
        with torch.no_grad():
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(2)]], device=device, dtype=torch.long)

episode_durations = []

def plot_durations():
    plt.figure(2)
    plt.clf()
    durations_t = torch.tensor(episode_durations, dtype=torch.float)
    plt.title('Training...')
    plt.xlabel('Episode')
    plt.ylabel('Duration')
    plt.plot(durations_t.numpy())
    # 100개의 에피소드 평균을 가져 와서 도표 그리기
    if len(durations_t) >= 100:
        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)
        means = torch.cat((torch.zeros(99), means))
        plt.plot(means.numpy())

    plt.pause(0.001)  # 도표가 업데이트되도록 잠시 멈춤
    if is_ipython:
        display.clear_output(wait=True)
        display.display(plt.gcf())
```

학습 루프
^^^^^^^^^^^^^

최종적으로 모델 학습을 위한 코드.

여기서, 최적화의 한 단계를 수행하는 ``optimize_model`` 함수를 찾을 수 있습니다.
먼저 배치 하나를 샘플링하고 모든 Tensor를 하나로 연결하고
$Q(s_t, a_t)$ 와  $V(s_{t+1}) = \max_a Q(s_{t+1}, a)$ 를 계산하고
그것들을 손실로 합칩니다. 우리가 설정한 정의를 따르면 만약 $s$ 가
마지막 상태라면 $V(s) = 0$ 이다.
또한 안정성 추가 위한 $V(s_{t+1})$ 계산을 위해 목표 네트워크를 사용합니다.
목표 네트워크는 대부분의 시간 동결 상태로 유지되지만, 가끔 정책
네트워크의 가중치로 업데이트됩니다.
이것은 대개 설정한 스텝 숫자이지만 단순화를 위해 에피소드를 사용합니다.

```python
def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    # Transpose the batch (자세한 설명을 위해 http://stackoverflow.com/a/19343/3343043 를 보십시오).
    batch = Transition(*zip(*transitions))

    # 최종 상태가 아닌 마스크를 계산하고 배치 요소를 연결합니다.
    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.uint8)
    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)

    # Q(s_t, a) 계산 - 모델이 Q(s_t)를 계산하고, 취한 행동의 칼럼을 선택한다.
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # 모든 다음 상태를 위한 V(s_{t+1}) 계산
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # 기대 Q 값 계산
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Huber 손실 계산
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # 모델 최적화
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
```

아래에서 주요 학습 루프를 찾을 수 있습니다. 처음으로 환경을
재설정하고 ``상태`` Tensor를 초기화합니다. 그런 다음 행동을
샘플링하고, 그것을 실행하고, 다음 화면과 보상(항상 1)을 관찰하고,
모델을 한 번 최적화합니다. 에피소드가 끝나면 (모델이 실패)
루프를 다시 시작합니다.

아래에서 `num_episodes` 는 작게 설정됩니다. 노트북을 다운받고
더많은 에피소드를 실행해 보십시오

```python
num_episodes = 50
for i_episode in range(num_episodes):
    # Initialize the environment and state
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    for t in count():
        # 행동 선택과 수행
        action = select_action(state)
        _, reward, done, _ = env.step(action.item())
        reward = torch.tensor([reward], device=device)

        # 새로운 상태 관찰
        last_screen = current_screen
        current_screen = get_screen()
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None

        # 메모리에 변이 저장
        memory.push(state, action, next_state, reward)

        # 다음 상태로 이동
        state = next_state

        # 최적화 한단계 수행(목표 네트워크에서)
        optimize_model()
        if done:
            episode_durations.append(t + 1)
            plot_durations()
            break
    # 목표 네트워크 업데이트
    if i_episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())

print('Complete')
env.render()
env.close()
plt.ioff()
plt.show()
```