Update README.md
Browse files
README.md
CHANGED
|
@@ -568,81 +568,6 @@ human-written sentence token-for-token is the wrong question to ask of open-ende
|
|
| 568 |
story continuation, where many different continuations are equally valid. The
|
| 569 |
overlap and diversity columns are what carry the signal.
|
| 570 |
|
| 571 |
-
### Beam search — time complexity and measured cost
|
| 572 |
-
|
| 573 |
-
**Theory.** Let `W` be the beam width, `T` the number of generated tokens, `V` the
|
| 574 |
-
vocabulary size and `C` the cost of one forward pass for one sequence. Each step
|
| 575 |
-
runs the model on all `W` live hypotheses, costing `O(WC)`; scores `W·V` candidate
|
| 576 |
-
extensions, costing `O(WV)`; and selects the best `W` of them, which is `O(WV)`
|
| 577 |
-
with a partial selection and `O(WV log WV)` with a full sort. Over `T` steps the
|
| 578 |
-
total is **`O(TW(C + V))` — linear in the beam width**. Memory is linear in `W` as
|
| 579 |
-
well, and is often the binding constraint in practice, since the KV cache must
|
| 580 |
-
hold `W` copies of the prompt.
|
| 581 |
-
|
| 582 |
-
**Measured.** Relative to width 1: W=1 is 1.00×, W=2 is 1.09×, W=4 is 1.45×
|
| 583 |
-
(31 s, 34 s and 45 s respectively for the full test split). The measured scaling
|
| 584 |
-
is *sub*-linear — quadrupling the beam costs 1.45×, not 4×. The asymptotic
|
| 585 |
-
analysis is not contradicted; the constant factor is hiding it. All `W` hypotheses
|
| 586 |
-
are evaluated in a single batched forward pass of `B·W` rows, and at `B` = 32 with
|
| 587 |
-
a 160M-parameter model the GPU is nowhere near saturated at `W` = 1. Widening the
|
| 588 |
-
beam therefore fills idle parallel capacity instead of adding serial work. The
|
| 589 |
-
linear term reasserts itself once the device is actually busy — at larger batch, a
|
| 590 |
-
larger model, or large enough `W` — and the memory cost is linear in `W` from the
|
| 591 |
-
first token regardless.
|
| 592 |
-
|
| 593 |
-
**Is it worth it?** Going from width 1 to width 4 changes F1 by −0.019 and the
|
| 594 |
-
repetition rate by +0.062, for 1.45× the compute and 4× the cache memory. A wider
|
| 595 |
-
beam is strictly better at what it optimises — it finds sequences of higher
|
| 596 |
-
log-probability, as the score check confirms — but on this task higher
|
| 597 |
-
probability means more of the generic, repetitive text that greedy already
|
| 598 |
-
produces. Beam search earns its cost in tasks with a genuinely peaked target
|
| 599 |
-
distribution, such as translation or speech recognition; open-ended story
|
| 600 |
-
continuation is not one of them.
|
| 601 |
-
|
| 602 |
-
---
|
| 603 |
-
|
| 604 |
-
## Known issues
|
| 605 |
-
|
| 606 |
-
Three bugs were found during the runs and are documented here rather than quietly
|
| 607 |
-
fixed, because each produced plausible-looking numbers rather than an obvious
|
| 608 |
-
failure:
|
| 609 |
-
|
| 610 |
-
1. **Beam search double-normalising its score by length.** Caught by the
|
| 611 |
-
width-1-equals-greedy check.
|
| 612 |
-
2. **Muon's internal AdamW missing the learning-rate schedule.** The fallback path
|
| 613 |
-
for embeddings, the tied head and 1D gains did not receive the warmup/cosine
|
| 614 |
-
schedule.
|
| 615 |
-
3. **Sophia's curvature estimator being inactive**, from two independent causes:
|
| 616 |
-
a missing batch-scale factor, and the sweep harness omitting the periodic
|
| 617 |
-
refresh step. The sampled loss that produces the curvature estimate is a mean
|
| 618 |
-
over tokens, so its gradient carries a `1/B` factor and its square a `1/B²`;
|
| 619 |
-
with the denominator four orders of magnitude too small, the clip saturated on
|
| 620 |
-
99.9% of coordinates, every step became exactly ±lr, and Sophia silently
|
| 621 |
-
degenerated into sign-SGD while still appearing to train normally. Nothing
|
| 622 |
-
errored, the loss went down, and only the suspiciously Lion-like selected
|
| 623 |
-
learning rate hinted that the second-order machinery was not actually running.
|
| 624 |
-
|
| 625 |
-
The general lesson from the third: a hyperparameter search is only valid if its
|
| 626 |
-
inner loop is the same algorithm as the final run, including any step that looks
|
| 627 |
-
like bookkeeping.
|
| 628 |
-
|
| 629 |
-
## Limitations
|
| 630 |
-
|
| 631 |
-
- Every result is a **single seed** at ~27M parameters, on a token budget well
|
| 632 |
-
short of compute-optimal (Part 2 runs at 0.077× Chinchilla), on a GPU shared
|
| 633 |
-
with unrelated jobs. Differences of a few hundredths of perplexity should not
|
| 634 |
-
be read as reliable orderings.
|
| 635 |
-
- Wall-clock figures from the training runs are indicative only; per-step
|
| 636 |
-
optimizer cost is the controlled measurement.
|
| 637 |
-
- The Part 2 sweep covered the peak learning rate alone.
|
| 638 |
-
- Part 1 BLEU is computed over 800 greedy generations (400 per direction), which
|
| 639 |
-
is a small evaluation set by MT standards.
|
| 640 |
-
- Part 3 uses a frozen 160M-parameter model; conclusions about decoding may not
|
| 641 |
-
transfer to larger models with stronger long-range coherence.
|
| 642 |
-
- These are research artefacts for a course assignment, not production models.
|
| 643 |
-
The Part 1 models are small translation models trained on 40M tokens and will
|
| 644 |
-
produce low-quality translations; the Part 2 models are language models trained
|
| 645 |
-
on 41.8M tokens and are not instruction-tuned or safety-aligned.
|
| 646 |
|
| 647 |
## Reproduction
|
| 648 |
|
|
|
|
| 568 |
story continuation, where many different continuations are equally valid. The
|
| 569 |
overlap and diversity columns are what carry the signal.
|
| 570 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 571 |
|
| 572 |
## Reproduction
|
| 573 |
|