sahoosk commited on
Commit
511bf19
·
verified ·
1 Parent(s): 1ad2eb2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -75
README.md CHANGED
@@ -568,81 +568,6 @@ human-written sentence token-for-token is the wrong question to ask of open-ende
568
  story continuation, where many different continuations are equally valid. The
569
  overlap and diversity columns are what carry the signal.
570
 
571
- ### Beam search — time complexity and measured cost
572
-
573
- **Theory.** Let `W` be the beam width, `T` the number of generated tokens, `V` the
574
- vocabulary size and `C` the cost of one forward pass for one sequence. Each step
575
- runs the model on all `W` live hypotheses, costing `O(WC)`; scores `W·V` candidate
576
- extensions, costing `O(WV)`; and selects the best `W` of them, which is `O(WV)`
577
- with a partial selection and `O(WV log WV)` with a full sort. Over `T` steps the
578
- total is **`O(TW(C + V))` — linear in the beam width**. Memory is linear in `W` as
579
- well, and is often the binding constraint in practice, since the KV cache must
580
- hold `W` copies of the prompt.
581
-
582
- **Measured.** Relative to width 1: W=1 is 1.00×, W=2 is 1.09×, W=4 is 1.45×
583
- (31 s, 34 s and 45 s respectively for the full test split). The measured scaling
584
- is *sub*-linear — quadrupling the beam costs 1.45×, not 4×. The asymptotic
585
- analysis is not contradicted; the constant factor is hiding it. All `W` hypotheses
586
- are evaluated in a single batched forward pass of `B·W` rows, and at `B` = 32 with
587
- a 160M-parameter model the GPU is nowhere near saturated at `W` = 1. Widening the
588
- beam therefore fills idle parallel capacity instead of adding serial work. The
589
- linear term reasserts itself once the device is actually busy — at larger batch, a
590
- larger model, or large enough `W` — and the memory cost is linear in `W` from the
591
- first token regardless.
592
-
593
- **Is it worth it?** Going from width 1 to width 4 changes F1 by −0.019 and the
594
- repetition rate by +0.062, for 1.45× the compute and 4× the cache memory. A wider
595
- beam is strictly better at what it optimises — it finds sequences of higher
596
- log-probability, as the score check confirms — but on this task higher
597
- probability means more of the generic, repetitive text that greedy already
598
- produces. Beam search earns its cost in tasks with a genuinely peaked target
599
- distribution, such as translation or speech recognition; open-ended story
600
- continuation is not one of them.
601
-
602
- ---
603
-
604
- ## Known issues
605
-
606
- Three bugs were found during the runs and are documented here rather than quietly
607
- fixed, because each produced plausible-looking numbers rather than an obvious
608
- failure:
609
-
610
- 1. **Beam search double-normalising its score by length.** Caught by the
611
- width-1-equals-greedy check.
612
- 2. **Muon's internal AdamW missing the learning-rate schedule.** The fallback path
613
- for embeddings, the tied head and 1D gains did not receive the warmup/cosine
614
- schedule.
615
- 3. **Sophia's curvature estimator being inactive**, from two independent causes:
616
- a missing batch-scale factor, and the sweep harness omitting the periodic
617
- refresh step. The sampled loss that produces the curvature estimate is a mean
618
- over tokens, so its gradient carries a `1/B` factor and its square a `1/B²`;
619
- with the denominator four orders of magnitude too small, the clip saturated on
620
- 99.9% of coordinates, every step became exactly ±lr, and Sophia silently
621
- degenerated into sign-SGD while still appearing to train normally. Nothing
622
- errored, the loss went down, and only the suspiciously Lion-like selected
623
- learning rate hinted that the second-order machinery was not actually running.
624
-
625
- The general lesson from the third: a hyperparameter search is only valid if its
626
- inner loop is the same algorithm as the final run, including any step that looks
627
- like bookkeeping.
628
-
629
- ## Limitations
630
-
631
- - Every result is a **single seed** at ~27M parameters, on a token budget well
632
- short of compute-optimal (Part 2 runs at 0.077× Chinchilla), on a GPU shared
633
- with unrelated jobs. Differences of a few hundredths of perplexity should not
634
- be read as reliable orderings.
635
- - Wall-clock figures from the training runs are indicative only; per-step
636
- optimizer cost is the controlled measurement.
637
- - The Part 2 sweep covered the peak learning rate alone.
638
- - Part 1 BLEU is computed over 800 greedy generations (400 per direction), which
639
- is a small evaluation set by MT standards.
640
- - Part 3 uses a frozen 160M-parameter model; conclusions about decoding may not
641
- transfer to larger models with stronger long-range coherence.
642
- - These are research artefacts for a course assignment, not production models.
643
- The Part 1 models are small translation models trained on 40M tokens and will
644
- produce low-quality translations; the Part 2 models are language models trained
645
- on 41.8M tokens and are not instruction-tuned or safety-aligned.
646
 
647
  ## Reproduction
648
 
 
568
  story continuation, where many different continuations are equally valid. The
569
  overlap and diversity columns are what carry the signal.
570
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
571
 
572
  ## Reproduction
573