Update model card and export table

README.md

@@ -396,19 +396,57 @@ checkpoints and cannot run lambda.
 
 A vLLM serving path for GPU servers is in preparation.
 
+### The finisher continuation
+
+After lambda stopped, its final checkpoint (`exports/900B-tokens_step73973`, 900.8B tokens) was continued on a
+single 8-GPU server for a further 66.8B tokens, without the fleet. The mid-training mixture replays part of
+lambda's data and adds higher-quality web and reference text, books and other creative writing, scientific
+papers, code, math, instruction-style and question-answering text, synthetic long-document retrieval tasks and
+reasoning traces. The training context was raised in stages from 4,096 to 8,192, 16,384, 32,768 and 65,536
+tokens (23.8B tokens over the first three lengths with dense global attention; then 6B at 16,384 tokens, during
+which, after about 1B tokens, the sparse global-attention selection was switched on; then 10B at 32,768 and 12B
+at 65,536 tokens with it on), followed by a closing 15B-token learning-rate anneal over a mix of 4,096- to
+65,536-token sequences. The longest trained
+context is 65,536 tokens; a planned 131,072-token stage was skipped. The architecture, tokenizer and expert
+format are unchanged; the RoPE base of the global-attention layers was raised from 1e6 to 1.67e7 for the longer
+contexts. The finisher's final export is `exports/finisher-967B-tokens_step31888` (967.6B tokens in total,
+finisher step 31888); like the lambda exports it is a base model with no instruction, chat or safety tuning.
+The *Model* and *Training* sections above describe the original lambda run.
+
+Benchmarks of the finisher's final export, in percent, from the same lm-eval harness as the lambda numbers
+above; the lambda column repeats the final checkpoint's earlier scores on that harness for comparison:
+
+| Benchmark | Finisher final | Lambda final (900.8B) |
+|---|---:|---:|
+| MMLU (continuation, 5-shot) | 34.6 | 35.4 |
+| ARC-Challenge (25-shot) | 45.8 | 47.4 |
+| HellaSwag (0-shot) | 59.0 | 59.8 |
+| GSM8K (5-shot) | 14.7 | 14.3 |
+| HumanEval (pass@1) | 12.2 | 12.2 |
+| MBPP (3-shot) | 16.0 | 15.4 |
+
+Long context, finisher final only: needle-in-a-haystack retrieval 100, 100 and 98 at 16k, 32k and 64k tokens,
+and 63 at 128k (past the trained context), in percent over 100 items each; the RULER seven-task mean was 79.3, 72.4, 60.5,
+49.7, 33.5 and 26.3 at 2k, 4k, 8k, 16k, 32k and 64k. The needle test uses the finisher's own synthetic
+long-context format, so it is in-distribution and measures retrieval the model was trained for; RULER is the
+external benchmark. These long-context numbers were measured with the training code with sparse global attention,
+not with the GGUF runtime below.
+
 ### Ready-made GGUF
 
-`parallax-8b-lambda-878B-tokens.gguf` at the repository root (2,133,006,592 bytes, 2.13 GB, sha256
-`ad38864d341fe0ab70fab02d91c3320e6d36424171e20076ebba022737091a98`, also in the `.sha256` file next to it) was
-converted from `exports/878B-tokens_step72097` (878.3B training tokens, step 72097).
-It keeps the export's bf16 trunk and stores the routed experts' exact ternary codes and per-row scales;
-no weight is requantized. Each GGUF is a snapshot of one export and is not updated as new exports are added.
-To use a later export, convert it yourself (see `tools/parallax/README.md` on the branch).
+`parallax-8b-lambda-finisher-967B-tokens.gguf` at the repository root (2,133,006,592 bytes, 2.13 GB, sha256
+`4a3ed3cf2d4ca959a7a93f0dac9e4cd52b45e39355e4aa63244080b56916cd74`, also in the `.sha256` file next to it) was
+converted from `exports/finisher-967B-tokens_step31888`, the finisher's final export (967.6B training tokens in
+total, finisher step 31888). It keeps the export's bf16 trunk and stores the routed experts' exact ternary codes and
+per-row scales; no weight is requantized. Its global-attention layers run with sparse block selection and the
+finisher's RoPE base, as trained; the lambda GGUFs below run dense global attention. Each GGUF is a snapshot of one
+export. To use another export, convert it yourself (see `tools/parallax/README.md` on the branch).
 
-The earlier snapshots remain at the repository root:
+The earlier snapshots, converted from lambda exports, remain at the repository root:
 
 | File | Export | sha256 |
 |---|---|---|
+| `parallax-8b-lambda-878B-tokens.gguf` | `exports/878B-tokens_step72097` | `ad38864d341fe0ab70fab02d91c3320e6d36424171e20076ebba022737091a98` |
 | `parallax-8b-lambda-860B-tokens.gguf` | `exports/860B-tokens_step70559` | `a65ecadcb38cb8ed34bd73ecc8fedf564393609ac2bb87f1f9f48b6149a70ba7` |
 | `parallax-8b-lambda-790B-tokens.gguf` | `exports/790B-tokens_step64456` | `489cd1d23f6c36ac394dc558fc266b26b14f5fb29b68ef9cbea0bb93b0354ef4` |
 | `parallax-8b-lambda-725B-tokens.gguf` | `exports/725B-tokens_step58814` | `eca13c0bf0bb4a5fbdf33b1616c7f33823ee84eb40fe7edb8852c566c7f6275f` |
@@ -418,7 +456,7 @@ The earlier snapshots remain at the repository root:
 | `parallax-8b-lambda-381B-tokens.gguf` | `exports/381B-tokens_step30942` | `86312194ffc72b7b8d87f37389fe0fdfe7c510e2f6066f0c424cc0eaf592aedf` |
 
 ```bash
-hf download chutesai/parallax-8b-lambda parallax-8b-lambda-878B-tokens.gguf --local-dir .
+hf download chutesai/parallax-8b-lambda parallax-8b-lambda-finisher-967B-tokens.gguf --local-dir .
 hf download NousResearch/Meta-Llama-3-8B tokenizer.json --local-dir .   # Llama 3 tokenizer
 ```
 
@@ -435,12 +473,12 @@ pip install tokenizers
 
 # Apple GPU
 python tools/parallax/run.py --binary build/bin/llama-parallax \
-  --model ../parallax-8b-lambda-878B-tokens.gguf --tokenizer ../tokenizer.json \
+  --model ../parallax-8b-lambda-finisher-967B-tokens.gguf --tokenizer ../tokenizer.json \
   --backend metal --experts lut9 --prompt 'The capital of France is' --predict 64
 
 # CPU (x86-64 or ARM64)
 python tools/parallax/run.py --binary build/bin/llama-parallax \
-  --model ../parallax-8b-lambda-878B-tokens.gguf --tokenizer ../tokenizer.json \
+  --model ../parallax-8b-lambda-finisher-967B-tokens.gguf --tokenizer ../tokenizer.json \
   --backend cpu --experts lut8 --threads 8 --prompt 'The capital of France is' --predict 64
 ```
 
@@ -451,7 +489,7 @@ prompt scoring and benchmark options.
 
 Measured on an Apple M5 Max on AC power (batch 1, 128 generated tokens) with the bf16-trunk GGUF of the earlier
 `270B-tokens_step23035` export, which has the same architecture, using the same runtime code; speed does not
-depend on the weight values. Prompt times used `--prefill-batch 128`:
+depend on the weight values. The finisher GGUF additionally runs the sparse-attention indexer and block selection in its global-attention layers; it was not re-timed. Prompt times used `--prefill-batch 128`:
 
 | Backend | Experts | Tokens/s, short context | Tokens/s after 4,096 tokens | 2,048-token prompt |
 |---|---|---:|---:|---:|
@@ -466,40 +504,34 @@ same results but is not tuned for speed: on a shared, loaded 64-core AMD EPYC 95
 
 ### Fidelity
 
-The 878B GGUF was checked on CPU with `lut` against the training code's own model loaded from the same
+The finisher GGUF was checked on CPU with `lut` against the training code's own model loaded from the same
 export, with FP32 activations in both, on the same 2,555 held-out positions from public text and code
-passages as before. Overall mean KL was 4.1e-7 nats/token; the top-1 next token agreed at 2,554 of 2,555 positions,
-and three 48-token greedy continuations matched exactly.
-
-The larger differences are in two passages (segments 3 and 4, maximum logit error 0.347 and 1.085;
-0.00067, 0.00075 and 0.00080 in segments 0, 1 and 2). In each affected passage they begin at a token where
-the router's 12th- and 13th-ranked expert-selection scores were near-tied, so the two implementations
-picked different experts. In segment 3 the first flip occurs in layer 7 at position 465; in segment 4 it
-occurs in layer 17 at position 187. The differences carry forward to later positions in both passages.
-Top-1 predictions agree throughout segment 3; the single top-1 disagreement is at position 319 in segment 4,
-after its first flip. In segment 3, the score margins were 1.4e-6 in the training code and 9.5e-7 in the
-runtime near -4.70 (FP32 spacing 4.8e-7, respectively 3 and 2 units of that spacing); in segment 4, they
-were 1.9e-6 in both near -3.92 (FP32 spacing 2.4e-7, 8 units of that spacing in both). Forcing the runtime's
-selection at just the first flip token/layer in the training code reduced mean KL to 1.9e-9 and 2.2e-9
-in those passages, with top-1 agreement at every position.
-Conservatively excluding every position from those tokens onward in those passages (2,185 positions
-remain), mean KL was 2.1e-9 nats/token with top-1 agreement at every position.
+passages as the earlier releases, with sparse global attention in both. Overall mean KL was 1.7e-9 nats/token
+(per passage 1.5e-9 to 1.8e-9); the top-1 next token agreed at all 2,555 positions; the largest logit
+difference at any position was 0.0010; and three 48-token greedy continuations matched exactly. No passage
+showed the divergence seen with the lambda GGUFs, where a router near-tie made the two implementations pick
+different experts from one token onward.
 
 The training code itself runs in bf16; against that bf16 forward the top-1 agreement was 94.9% (mean
 KL 0.0027, measured on the earlier `270B-tokens_step23035` export), which reflects the bf16 rounding
 of the reference, not an error of the runtime.
 
-On the 878B GGUF, the largest per-layer relative RMS difference between Metal and CPU was about 2.2e-5
-at the final position of a 512-token text, and the two gave identical 48-token greedy continuations on three prompts. `lut8` (CPU)
-quantizes expert lookup tables to 8 bits and is slightly approximate; `lut` and `lut9` are exact.
+On the finisher GGUF, the largest per-layer relative RMS difference between Metal and CPU was about 2.7e-5
+(logits 6.8e-6) at the final position of a 512-token text, and the two gave identical 48-token greedy continuations
+on three prompts. `lut8` (CPU) quantizes expert lookup tables to 8 bits and is slightly approximate; `lut` and
+`lut9` are exact.
 
 ### Limitations of the runtime
 
 - One sequence per process, greedy or simple sampled generation. No chat template, batching or server.
 - Not integrated into `llama-cli`, `llama-server` or the standard llama.cpp model loader.
-- Context is limited by `-c`. Inference uses a 2,048-token window for all sliding-window layers and dense
-  global attention, the settings the run is evaluated with. Lambda was trained on 4,096-token sequences;
-  longer inputs run but were not qualified.
+- Context is limited by `-c`. Inference uses a 2,048-token window for all sliding-window layers. With the
+  finisher GGUF the global-attention layers attend to up to 16 selected blocks of 128 tokens per query, always
+  including the current block (so inputs up to 2,048 tokens are attended in full); with the lambda GGUFs they are
+  dense. The finisher was trained on sequences
+  up to 65,536 tokens and lambda on 4,096, but the runtime was checked only on 512-token passages and 48-token
+  continuations; longer inputs run (raise `-c`) but were not qualified in the runtime, and the long-context
+  numbers above come from the training code.
 
 ## Tech report
 
@@ -515,4 +547,4 @@ alignment or safety tuning, and it is small and far from converged.
 
 MIT.
 
-<sub>Table updated 2026-10-10 11:19 UTC.</sub>
+<sub>Table updated 2026-10-21 23:47 UTC.</sub>