Update README.md
Browse files
README.md
CHANGED
|
@@ -30,7 +30,7 @@ The paper reports experiments at **GPT-2** scale on **LM1B** and **OpenWebText**
|
|
| 30 |
|
| 31 |
## Method Summary
|
| 32 |
|
| 33 |
-
For a clean token
|
| 34 |
|
| 35 |
$$
|
| 36 |
q(z_t \mid x) = \operatorname{Cat}\left(z_t;, \gamma_t\left(\rho_t x + (1-\rho_t)u\right) + (1-\gamma_t)m\right).
|
|
@@ -38,10 +38,10 @@ $$
|
|
| 38 |
|
| 39 |
where:
|
| 40 |
|
| 41 |
-
*
|
| 42 |
-
*
|
| 43 |
-
*
|
| 44 |
-
*
|
| 45 |
|
| 46 |
|
| 47 |
|
|
|
|
| 30 |
|
| 31 |
## Method Summary
|
| 32 |
|
| 33 |
+
For a clean token \\(x\\), SCDD uses the following marginal forward distribution:
|
| 34 |
|
| 35 |
$$
|
| 36 |
q(z_t \mid x) = \operatorname{Cat}\left(z_t;, \gamma_t\left(\rho_t x + (1-\rho_t)u\right) + (1-\gamma_t)m\right).
|
|
|
|
| 38 |
|
| 39 |
where:
|
| 40 |
|
| 41 |
+
* \\(m\\) is the `[MASK]` token.
|
| 42 |
+
* \\(u\\) is the uniform distribution over non-`[MASK]` tokens.
|
| 43 |
+
* \\(\gamma_t\\) is the marginal probability that \\(z_t\\) is not `[MASK]`.
|
| 44 |
+
* \\(\rho_t\\) is the marginal probability that \\(z_t\\) retains the clean token among the non-`[MASK]` mass.
|
| 45 |
|
| 46 |
|
| 47 |
|