# Data for Lecture 6

`phrases.txt` and `variable_text.txt` were generated deterministically with Python
`random.Random(6270)`. Each contains 640 examples. Their vocabulary describes
colored shapes and directions; no external dataset is needed.

| Color | Shape | Direction |
| --- | --- | --- |
| red | circle | left |
| blue | square | right |
| green | triangle | up |
| gold | star | down |

The fixed-length corpus has four tokens per line. The variable-length corpus
uses `COLOR SHAPE`, `COLOR SHAPE moves DIRECTION`, or the latter followed by
`slowly` or `twice`. The color, shape, and direction remain correlated. This
makes invalid independent combinations easy to inspect.

Training uses a seeded shuffle and 80/20 example split. The grammar has a small
support, so examples repeat and both partitions contain the same grammar.
Training-support fraction measures membership in this small support; it is
not a test of open-ended language generalization.

For a custom corpus, supply a UTF-8 text file with one whitespace-tokenized
sequence per line. Fixed-length methods require equal token counts. The
expanding method accepts variable lengths up to 32. Vocabulary and maximum
length are saved in the checkpoint, so sampling does not need the original file.

The continuous dataset is sampled from four equally weighted Gaussians centered
at `(−1.5,−1.5)`, `(−1.5,1.5)`, `(1.5,−1.5)`, and `(1.5,1.5)`, with coordinate
standard deviation 0.22. The latent example embeds these points on
`(x, y, 0.3(x²−y²))` before learning its autoencoder. SSFM uses analytic
Ornstein-Uhlenbeck marginals with initial variance one, drift `−x`, and diffusion
coefficient 0.7.