Upload folder using huggingface_hub

390M_1M.py

@@ -0,0 +1,1927 @@
+
+#!/usr/bin/env python3
+# =============================================================================
+# train_muon.py — the MUON arm of train.py. Same model, same data order, same
+# schedule shape, same seed: the ONLY thing that changes is how a gradient
+# becomes a weight update.
+#
+# Fork discipline — everything outside section 5 is byte-identical to train.py,
+# so `diff train.py train_muon.py` is exactly the experiment being run. If you
+# edit a kernel or the model in one file, port it to the other or the comparison
+# stops meaning anything.
+#
+# The optimizer itself is torch.optim.Muon — stock, not reimplemented. This file
+# supplies only the two things it does not do for you: the ROUTING (which
+# parameters it may have) and the fp32 MASTER WEIGHTS (it updates the master,
+# not the bf16 param). Requires a torch new enough to ship it; main() checks and
+# says so.
+#
+# What Muon does: AdamW rescales each weight ENTRY by its own gradient history.
+# Muon rescales each weight MATRIX by its singular values — it takes the
+# momentum buffer M and replaces it with the nearest semi-orthogonal matrix
+# (all singular values driven to ~1) via a 5-step Newton-Schulz iteration, then
+# steps along that. Directions the gradient has barely explored get the same
+# step size as the dominant ones, which is where the reported speedup comes
+# from. Only 2D hidden weights can be orthogonalized, so this is a HYBRID:
+#
+#     Muon   : every 2D Linear weight inside the blocks   (75.5M params, 66%)
+#     AdamW  : the token embedding / lm_head              (38.6M params)
+#              + RMSNorm weights + QK-norm gammas (1D)    (22.3k params)
+#
+# The embedding is deliberately NOT given to Muon: its rows are per-token, not a
+# linear map between two feature spaces, and orthogonalizing across the vocab
+# axis mixes unrelated tokens. That split is the standard Muon recipe, and it
+# keeps the AdamW arm's exact update on 34% of the parameters — so a difference
+# in the loss curve is attributable to the hidden matrices alone.
+#
+# Everything train_ddp.py / train_zero1.py import from blu_common / blu_data /
+# blu_kernels / blu_model is inlined here, in that order, so this file runs on
+# its own:
+#     section 1  fused QK-norm + RoPE (Triton)  <- blu_kernels.py
+#     section 2  config + WSD schedule          <- blu_common.py
+#     section 3  data loader                    <- blu_data.py
+#     section 4  model                          <- blu_model.py
+#     section 5  optimizer / clipping / checkpointing
+#     section 6  training
+#
+# Architecture:
+#   token embedding
+#   -> N x { RMSNorm -> [fused QK-norm + RoPE] -> causal GQA (flash) -> Wo -> +x
+#            RMSNorm -> SwiGLU FFN (gate_up -> silu(gate)*up -> down)  -> +x }
+#   -> final RMSNorm
+#   -> lm_head (weight-tied with the token embedding)
+#   -> cross-entropy
+#   -> backward -> AdamW (decoupled weight decay) on a WSD (warmup / stable /
+#      cosine-decay) schedule
+#
+# Kept from the AdamW arm, unchanged, because they are confounds if they move:
+#   * fp32 master weights for every non-fp32 param — Muon updates the MASTER,
+#     reading the same fp32 landing buffer AdamW reads, so both optimizers see
+#     bit-identical gradients;
+#   * decoupled weight decay 0.1 on dim>=2 params, applied to the master;
+#   * ONE global-norm gradient clip over every parameter, BEFORE the optimizer.
+#     (Muon's orthogonalization makes the update scale-invariant, so clipping is
+#     nearly a no-op for its group — but it still gates the AdamW group, and
+#     removing it would change two things at once.)
+#
+# Memory, at the 114M config this repo actually runs (printed at startup):
+#
+#     parameters (bf16 Linear + fp32 embed/norms)  0.28 GiB
+#     gradients (same dtypes as the params)        0.28 GiB
+#     fp32 master weights (the bf16 Linears)       0.28 GiB
+#     fp32 gradient landing buffers                0.28 GiB
+#     Muon momentum   (75.5M params, fp32 x1)      0.28 GiB
+#     AdamW m + v     (38.7M params, fp32 x2)      0.29 GiB
+#     -------------------------------------------------------
+#     static total                                 1.70 GiB   (AdamW arm: 1.98)
+#
+# Muon carries ONE state tensor per param where Adam carries two, so the saving
+# scales with how much of the model is hidden matrices: 0.28 GiB here (14% of
+# static), because this config spends a third of its parameters on a 50304-row
+# embedding that stays on AdamW. At the 1.73B config in the original header the
+# same split saves ~5.5 GiB.
+#
+# Activations dominate either way (63.96 GiB measured at B=16 x T=4096), so the
+# OOM escape hatches are unchanged: --B 8, --loss-chunks 4, --activation-checkpoint.
+#
+# Fidelity notes (why a run here is statistically but not bit-wise identical):
+#   * init draws come from torch's Philox, not the C++ single advancing
+#     std::mt19937, so the two runs start from different samples of the same
+#     distributions;
+#   * QK-norm reads the projection output in bf16 here (autocast) where the C++
+#     casts it up to fp32 first; the reduction and the rotation are fp32 in both.
+#
+# Determinism: two runs of the same launch line on the same GPU produce
+# bit-identical losses. Seeding is only half of that — the default CUDA kernels
+# for the embedding backward and the flash/mem-efficient attention backward
+# accumulate partial sums with atomicAdd, so the order they land in (and with it
+# the fp32 rounding) follows the scheduler, not the program. --determinism
+# strict (the default) forces the ordered variants of those kernels; it costs a
+# few percent of step time, which --determinism off buys back.
+#
+# Env overrides (the bluscript_single.cpp names, so an existing launch line keeps
+# working; the equivalent CLI flag wins if both are given):
+#   BLU_DEVICE=<idx>          CUDA device to train on            (default 0)
+#   BLU_MAX_STEPS=<n>         short benchmark run (rescales WSD) (default off)
+#   BLU_B=<n>                 micro-batch size                   (default 4)
+#   BLU_GLOBAL_BATCH=<tokens> tokens per optimizer step          (default 524288)
+#   BLU_DIAG=1                per-micro-batch loss + depth probes
+#
+# Multi-GPU: DDP, one process per GPU, via torchrun. RANK / LOCAL_RANK /
+# WORLD_SIZE (set by torchrun, not this file) switch the bootstrap in main();
+# their absence is the single-process path above, unchanged. --device /
+# BLU_DEVICE are ignored under torchrun — LOCAL_RANK picks each rank's GPU.
+# --global-batch is still the TOTAL tokens per optimizer step across every
+# rank (grad_accum = global_batch / (B*T*world_size)), so the same
+# --global-batch reproduces the same schedule at any world size. Data is
+# rank-sharded (DataLoaderLite reads a disjoint, rank-offset slice of the
+# shards); gradients and the reported train/val loss are averaged across
+# ranks; only rank 0 writes the CSV and checkpoints. Same launch line as
+# single-GPU otherwise:
+#   torchrun --standalone --nproc_per_node=8 train_muon.py
+#
+# Learning rate — the one knob that decides whether the comparison is fair.
+#
+# Muon's update is spectrally normalized, so its natural scale has nothing to do
+# with AdamW's. torch.optim.Muon ships both published conventions behind its
+# `adjust_lr_fn`; --muon-adjust-lr passes the string straight through:
+#
+#   match_rms_adamw      Moonshot's: gamma <- 0.2 * gamma * sqrt(max(A,B)). The
+#     (DEFAULT HERE)     Newton-Schulz output has singular values ~1, so its
+#                        entries have RMS 1/sqrt(max(A,B)); this lifts them to
+#                        RMS 0.2, which is what a converged AdamW update looks
+#                        like. The consequence: --muon-lr is on the SAME SCALE
+#                        as --max-lr, and the default (2e-4, equal to the AdamW
+#                        arm's max_lr) gives both arms the same step size on the
+#                        first run — only the update DIRECTION differs. That is
+#                        the controlled experiment; run it before tuning.
+#   original             Keller's modded-nanogpt scaling, sqrt(max(1, A/B)).
+#                        Entries stay at RMS 1/sqrt(max(A,B)), so an equivalent
+#                        LR is ~5.5x larger at d_model=768 (0.2*sqrt(768)).
+#                        Use it to reproduce published Muon numbers.
+#
+# NOTE: torch's own default for adjust_lr_fn is `original`. This file defaults
+# to `match_rms_adamw` instead, because an LR-matched first run is the point of
+# having two arms. If you cite a number from here, cite the flag with it.
+#
+# Muon usually tolerates 2-4x more LR than AdamW once it is working, so the
+# honest protocol is: baseline run, matched run, then a short --max-steps 200
+# sweep over --muon-lr before claiming a win.
+#
+# Launch (the two arms; separate CSV + checkpoint dirs so they never collide):
+#   python3 train.py                                  # AdamW baseline
+#   python3 train_muon.py                             # Muon, LR-matched
+#   python3 train_muon.py --muon-lr 6e-4              # after a sweep
+#   python3 train_muon.py --max-steps 200 --muon-lr 4e-4   # short sweep point
+# =============================================================================
+
+import argparse
+import glob
+import math
+import os
+import random
+import re
+import signal
+import time
+from contextlib import nullcontext
+from dataclasses import dataclass
+
+# cuBLAS reads CUBLAS_WORKSPACE_CONFIG once, when it creates its handle on the
+# first GEMM, and torch refuses to run fp32 cuBLAS GEMMs under
+# use_deterministic_algorithms() unless it is one of the two allowed values. It
+# has to be in the environment before torch touches CUDA, hence: before the
+# import, not inside main(). An explicit setting from the launch line wins.
+os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
+
+import numpy as np
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.attention import SDPBackend, sdpa_kernel
+from torch.nn.parallel import DistributedDataParallel as DDP
+from torch.utils.checkpoint import checkpoint
+
+try:
+    import triton
+    import triton.language as tl
+    _HAS_TRITON = True
+except ImportError:                                   # eager fallback below
+    _HAS_TRITON = False
+
+
+# =============================================================================
+# 1. Fused QK-norm (RMSNorm over head_dim) + RoPE, in Triton.
+#
+# PyTorch counterpart of the C++ `autograd::fused_rms_rope`. One kernel does,
+# per (batch, head, position) row:
+#
+#     s = rsqrt(mean(x^2) + eps)          # RMS over head_dim
+#     y = x * s * gamma                   # learnable per-head_dim scale
+#     o = rope_neox(y, cos[t], sin[t])    # NeoX / Llama half-split rotation
+#
+# Attention is deliberately NOT fused in here — it stays a separate op
+# (F.scaled_dot_product_attention on the flash backend), mirroring the C++ split
+# between `fused_rms_rope` and `gqa_fused_flash_attn_forward/backward`.
+#
+# Why fuse: the eager formulation (`rms_rope_eager`) materialises an fp32 copy of
+# x, the squared tensor, the normalised tensor, the gamma-scaled tensor and a
+# `cat` of the two rotated halves — roughly 6 extra [B,H,T,D] buffers held for
+# backward. The fused version saves only x (which the projection's backward
+# already holds) plus one fp32 scalar per row.
+#
+# Layout notes
+#   - x is [B,H,T,D] and may be a non-contiguous transpose of a [B,T,H,D] view;
+#     arbitrary (b,h,t) strides are honoured, D must be stride-1.
+#   - D must be a power of two (the C++ kernel requires 64 or 128).
+#   - cos/sin are [T_ctx, D//2] fp32, as produced by `build_rope_cache`.
+#   - the row index is r = (b*H + h)*T + t, so the output is contiguous
+#     [B,H,T,D] in that same order.
+# =============================================================================
+if _HAS_TRITON:
+
+    @triton.jit
+    def _fused_rms_rope_fwd(
+        X, GAMMA, COS, SIN, OUT, RSTD,
+        sx_b, sx_h, sx_t,                   # input strides, in elements
+        T, H, N,                            # seq len, n_heads, total rows
+        eps,
+        D: tl.constexpr, HALF: tl.constexpr, BLOCK_R: tl.constexpr,
+    ):
+        pid = tl.program_id(0)
+        rows = pid * BLOCK_R + tl.arange(0, BLOCK_R)          # [R]
+        mask_r = rows < N
+
+        # r = (b*H + h)*T + t  ->  recover t (for RoPE) and the (b, h) base.
+        t = rows % T
+        bh = rows // T
+        b = bh // H
+        h = bh % H
+        base = b * sx_b + h * sx_h + t * sx_t                  # [R]
+
+        dh = tl.arange(0, HALF)                               # [HALF]
+        p1 = X + base[:, None] + dh[None, :]
+        x1 = tl.load(p1, mask=mask_r[:, None], other=0.0).to(tl.float32)
+        x2 = tl.load(p1 + HALF, mask=mask_r[:, None], other=0.0).to(tl.float32)
+
+        # RMS over the FULL head_dim (both halves), fp32 accumulation.
+        ss = tl.sum(x1 * x1, 1) + tl.sum(x2 * x2, 1)          # [R]
+        rstd = 1.0 / tl.sqrt(ss / D + eps)                    # [R]
+
+        g1 = tl.load(GAMMA + dh).to(tl.float32)
+        g2 = tl.load(GAMMA + HALF + dh).to(tl.float32)
+        y1 = x1 * rstd[:, None] * g1[None, :]
+        y2 = x2 * rstd[:, None] * g2[None, :]
+
+        cp = t[:, None] * HALF + dh[None, :]
+        c = tl.load(COS + cp, mask=mask_r[:, None], other=0.0)
+        s = tl.load(SIN + cp, mask=mask_r[:, None], other=0.0)
+
+        # NeoX half-split rotation.
+        o1 = y1 * c - y2 * s
+        o2 = y2 * c + y1 * s
+
+        op = OUT + rows[:, None] * D + dh[None, :]
+        tl.store(op, o1.to(OUT.dtype.element_ty), mask=mask_r[:, None])
+        tl.store(op + HALF, o2.to(OUT.dtype.element_ty), mask=mask_r[:, None])
+        tl.store(RSTD + rows, rstd, mask=mask_r)
+
+    # -------------------------------------------------------------------------
+    # backward
+    #
+    #   RoPE is an orthogonal rotation, so its vjp is the inverse rotation:
+    #       dy1 = do1*cos + do2*sin ;  dy2 = do2*cos - do1*sin
+    #   then, with n = x * rstd (the normalised row),
+    #       dgamma += dy * n
+    #       dn      = dy * gamma
+    #       dx      = rstd * (dn - n * mean(dn * n))
+    # -------------------------------------------------------------------------
+    @triton.jit
+    def _fused_rms_rope_bwd(
+        X, GAMMA, COS, SIN, DOUT, RSTD, DX, DGP,
+        sx_b, sx_h, sx_t,                   # input strides
+        sd_b, sd_h, sd_t,                   # dx strides
+        T, H, N,
+        D: tl.constexpr, HALF: tl.constexpr, BLOCK_R: tl.constexpr,
+    ):
+        pid = tl.program_id(0)
+        rows = pid * BLOCK_R + tl.arange(0, BLOCK_R)
+        mask_r = rows < N
+
+        t = rows % T
+        bh = rows // T
+        b = bh // H
+        h = bh % H
+        base = b * sx_b + h * sx_h + t * sx_t
+
+        dh = tl.arange(0, HALF)
+        p1 = X + base[:, None] + dh[None, :]
+        x1 = tl.load(p1, mask=mask_r[:, None], other=0.0).to(tl.float32)
+        x2 = tl.load(p1 + HALF, mask=mask_r[:, None], other=0.0).to(tl.float32)
+
+        rstd = tl.load(RSTD + rows, mask=mask_r, other=1.0)
+        n1 = x1 * rstd[:, None]
+        n2 = x2 * rstd[:, None]
+
+        dp = DOUT + rows[:, None] * D + dh[None, :]
+        do1 = tl.load(dp, mask=mask_r[:, None], other=0.0).to(tl.float32)
+        do2 = tl.load(dp + HALF, mask=mask_r[:, None], other=0.0).to(tl.float32)
+
+        cp = t[:, None] * HALF + dh[None, :]
+        c = tl.load(COS + cp, mask=mask_r[:, None], other=0.0)
+        s = tl.load(SIN + cp, mask=mask_r[:, None], other=0.0)
+
+        # inverse rotation
+        dy1 = do1 * c + do2 * s
+        dy2 = do2 * c - do1 * s
+
+        # per-program partial dgamma; summed by the caller (deterministic, no atomics)
+        tl.store(DGP + pid * D + dh, tl.sum(dy1 * n1, 0))
+        tl.store(DGP + pid * D + HALF + dh, tl.sum(dy2 * n2, 0))
+
+        g1 = tl.load(GAMMA + dh).to(tl.float32)
+        g2 = tl.load(GAMMA + HALF + dh).to(tl.float32)
+        dn1 = dy1 * g1[None, :]
+        dn2 = dy2 * g2[None, :]
+
+        dot = (tl.sum(dn1 * n1, 1) + tl.sum(dn2 * n2, 1)) / D      # [R]
+        dx1 = rstd[:, None] * (dn1 - n1 * dot[:, None])
+        dx2 = rstd[:, None] * (dn2 - n2 * dot[:, None])
+
+        dbase = b * sd_b + h * sd_h + t * sd_t
+        q1 = DX + dbase[:, None] + dh[None, :]
+        tl.store(q1, dx1.to(DX.dtype.element_ty), mask=mask_r[:, None])
+        tl.store(q1 + HALF, dx2.to(DX.dtype.element_ty), mask=mask_r[:, None])
+
+
+_BLOCK_R = 16
+_NUM_WARPS = 4
+
+
+def _launch_fwd(x, gamma, cos, sin, eps):
+    B, H, T, D = x.shape
+    N = B * H * T
+    out = torch.empty((B, H, T, D), dtype=x.dtype, device=x.device)
+    rstd = torch.empty(N, dtype=torch.float32, device=x.device)
+    grid = (triton.cdiv(N, _BLOCK_R),)
+    _fused_rms_rope_fwd[grid](
+        x, gamma, cos, sin, out, rstd,
+        x.stride(0), x.stride(1), x.stride(2),
+        T, H, N, eps,
+        D=D, HALF=D // 2, BLOCK_R=_BLOCK_R, num_warps=_NUM_WARPS,
+    )
+    return out, rstd
+
+
+def _launch_bwd(x, gamma, cos, sin, dout, rstd):
+    B, H, T, D = x.shape
+    N = B * H * T
+    dout = dout.contiguous()
+    dx = torch.empty_like(x)
+    nprog = triton.cdiv(N, _BLOCK_R)
+    dgp = torch.empty((nprog, D), dtype=torch.float32, device=x.device)
+    _fused_rms_rope_bwd[(nprog,)](
+        x, gamma, cos, sin, dout, rstd, dx, dgp,
+        x.stride(0), x.stride(1), x.stride(2),
+        dx.stride(0), dx.stride(1), dx.stride(2),
+        T, H, N,
+        D=D, HALF=D // 2, BLOCK_R=_BLOCK_R, num_warps=_NUM_WARPS,
+    )
+    return dx, dgp.sum(0).to(gamma.dtype)
+
+
+class _FusedRMSRoPE(torch.autograd.Function):
+    """QK-norm + RoPE for q and k in one autograd node (two kernel launches).
+    Mirrors the C++ `fused_rms_rope(q, k, q_gamma, k_gamma, cache)`: q/k keep
+    their incoming dtype (bf16 under autocast), the reduction and the rotation
+    run in fp32 internally, gamma and the cache stay fp32.
+    """
+
+    @staticmethod
+    def forward(ctx, q, k, q_gamma, k_gamma, cos, sin, eps):
+        qo, q_rstd = _launch_fwd(q, q_gamma, cos, sin, eps)
+        ko, k_rstd = _launch_fwd(k, k_gamma, cos, sin, eps)
+        ctx.save_for_backward(q, k, q_gamma, k_gamma, cos, sin, q_rstd, k_rstd)
+        return qo, ko
+
+    @staticmethod
+    def backward(ctx, dqo, dko):
+        q, k, q_gamma, k_gamma, cos, sin, q_rstd, k_rstd = ctx.saved_tensors
+        dq, dqg = _launch_bwd(q, q_gamma, cos, sin, dqo, q_rstd)
+        dk, dkg = _launch_bwd(k, k_gamma, cos, sin, dko, k_rstd)
+        return dq, dk, dqg, dkg, None, None, None
+
+
+def fused_rms_rope(q, k, q_gamma, k_gamma, cos, sin, eps):
+    """q: [B,Hq,T,D], k: [B,Hkv,T,D] -> same shapes, contiguous.
+    `cos`/`sin` are [>=T, D//2] fp32; only the first T rows are read.
+    """
+    assert q.shape[-1] == k.shape[-1], "q and k must share head_dim"
+    D = q.shape[-1]
+    assert D % 2 == 0 and (D & (D - 1)) == 0, f"head_dim must be a power of two, got {D}"
+    assert q.stride(-1) == 1 and k.stride(-1) == 1, "head_dim must be stride-1"
+    T = q.shape[2]
+    return _FusedRMSRoPE.apply(q, k, q_gamma, k_gamma,
+                               cos[:T].contiguous(), sin[:T].contiguous(), eps)
+
+
+def rms_rope_eager(x, gamma, cos, sin, eps):
+    """Unfused QK-norm then NeoX RoPE, fp32 throughout — the reference the fused
+    kernel is measured against (`--rms-rope eager`), and the fallback when
+    Triton is unavailable."""
+    T, D = x.shape[2], x.shape[-1]
+    xf = x.float()
+    xf = xf * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + eps) * gamma
+    x1, x2 = xf[..., : D // 2], xf[..., D // 2:]
+    c = cos[:T][None, None, :, :]
+    s = sin[:T][None, None, :, :]
+    return torch.cat([x1 * c - x2 * s, x2 * c + x1 * s], dim=-1).to(x.dtype)
+
+
+def build_rope_cache(seq_len, head_dim, theta, device):
+    """NeoX / Llama RoPE cache: cos, sin of shape [seq_len, head_dim//2], fp32."""
+    inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, dtype=torch.float32,
+                                             device=device) / head_dim))
+    pos = torch.arange(seq_len, dtype=torch.float32, device=device)
+    freqs = torch.outer(pos, inv_freq)                       # [T, D/2]
+    return torch.cos(freqs).contiguous(), torch.sin(freqs).contiguous()
+
+
+# =============================================================================
+# 2. Configuration — 1:1 with ModelConfig in bluscript_single.cpp — and the WSD
+#    learning-rate schedule.
+# =============================================================================
+@dataclass
+class ModelConfig:
+    # ---- model ----
+    d_model: int = 960
+    n_layers: int = 30
+    q_heads: int = 15
+    kv_heads: int = 5                     # grouped-query: G = q/kv = 4
+    head_dim: int = 64                   # the fused kernels need 64 or 128
+    ffn_hidden: int = 2560                # ~8/3 * d_model, rounded to /256
+    context_length: int = 4096            # RoPE cache length; must be >= T
+    rope_theta: float = 500000.0
+    vocab_size: int = 50304               # 50257 (tiktoken) padded for the GPU
+    weight_tying: bool = False
+
+    rms_eps: float = 1e-5                 # block RMSNorm epsilon
+    qk_eps: float = 1e-6                  # QK-norm (fused) epsilon
+
+    # ---- optimization / data ----
+    B: int = 8                           # micro-batch
+    T: int = 4096                         # sequence length (<= context_length)
+    global_batch: int = 1048576          # tokens per optimizer step
+    grad_clip: float = 1.0
+    weight_decay: float = 0.1
+
+    # WSD: linear warmup -> constant at max_lr -> cosine decay to min_lr.
+    # Requires warmup_steps + stable_steps < max_steps - 1 (the remainder is
+    # the decay tail).
+    max_lr: float = 0.001
+    min_lr: float = 0.0001                # 10% of max_lr
+    
+    muon_lr: float = 0.001
+    muon_min_lr: float = 0.0001
+    
+    max_steps: int = 13351
+
+    warmup_steps: int = 500
+    stable_steps: int = int(max_steps * 0.99) - warmup_steps           # steps held at max_lr after warmup
+     
+
+    # ---- Muon (hidden 2D weights); the AdamW fields above still drive the
+    #      embedding / lm_head / 1D group, unchanged from the AdamW arm ----
+    # Default == max_lr: under adjust_lr_fn=match_rms_adamw the two are the same unit,
+    # so out of the box both arms take the same size step and differ only in
+    # direction. Same 10% floor, same WSD shape, so the schedules stay aligned.
+   
+    # The rest are torch.optim.Muon's own arguments, passed through unchanged.
+    # momentum/nesterov/ns_steps are torch's defaults; adjust_lr_fn is not (see
+    # the header — torch defaults to "original").
+    muon_momentum: float = 0.95           # beta on the momentum buffer
+    muon_ns_steps: int = 5                # Newton-Schulz iterations
+    muon_adjust_lr: str = "match_rms_adamw"   # "match_rms_adamw" | "original"
+
+    val_freq: int = 250
+    val_loss_steps: int = 20
+
+    # ---- checkpointing ----
+    checkpointing: bool = True
+    ckpt_freq: int = 1000                 # unused by the save trigger (milestone-based
+                                           # now, see target_losses below) — kept in case
+                                           # interval saving is wanted again later
+    ckpt_keep: int = 40                    # retain the most-recent N checkpoints
+    # Separate from the AdamW arm's dir: the two checkpoint formats differ (Muon
+    # momentum vs Adam m/v) and auto-resume picks the newest file it finds.
+    ckpt_dir: str = "ckpt_blu_muon"
+    ckpt_prefix: str = "blumodel_muon"
+
+    # ---- runtime knobs (not in the C++ struct) ----
+    fused_rms_rope: bool = True
+    activation_checkpoint: bool = False
+    loss_chunks: int = 1
+    # Milestones to record (step + elapsed hours) AND checkpoint on, as the
+    # EMA-smoothed train loss passes each one — the same shared target-loss
+    # set used across this project's batch-size ablation arms. Raw per-step
+    # loss is noisy, so the comparison runs against an EMA, not loss_val
+    # directly — same convention as batchsize_scaling.py's EMA_BETA=0.99.
+    # A checkpoint is saved the step each one is first crossed (replacing
+    # interval-based saving): this ablation's whole point is comparing
+    # batch sizes at matched loss levels, so that's the checkpoint that's
+    # actually useful, not whatever happens to land on a round step number.
+    target_losses: tuple = (11.035316, 2.690595, 2.554042, 2.505404,
+                            2.480678, 2.450056, 2.437589)
+    target_loss_ema_beta: float = 0.99
+    # Depth probe: which layers --diag reports gradient / weight statistics for.
+    probe_layers: tuple = (0, 8, 17, 26, 35)
+
+
+def get_lr(step, cfg, max_lr=None, min_lr=None):
+    """WSD: linear Warmup -> Stable (constant max_lr) -> cosine Decay to min_lr.
+    Byte-for-byte the C++ get_lr().
+    `max_lr`/`min_lr` override the config pair so the Muon group can ride the
+    SAME schedule shape — same warmup length, same stable window, same cosine
+    tail — at its own amplitude. Called with no override it is identical to the
+    AdamW arm's get_lr(), which is what the embedding/norm group uses.
+    """
+    max_lr = cfg.max_lr if max_lr is None else max_lr
+    min_lr = cfg.min_lr if min_lr is None else min_lr
+    if step < cfg.warmup_steps:
+        return max_lr * (step + 1) / cfg.warmup_steps
+    if step < cfg.stable_steps + cfg.warmup_steps:
+        return max_lr
+    tail = cfg.max_steps - (cfg.stable_steps + cfg.warmup_steps) - 1
+    decay_ratio = (step - (cfg.stable_steps + cfg.warmup_steps)) / max(tail, 1)
+    decay_ratio = min(max(decay_ratio, 0.0), 1.0)
+    coeff = 0.5 * (1.0 + math.cos(math.pi * decay_ratio))       # 1 -> 0
+    return min_lr + coeff * (max_lr - min_lr)
+
+
+# =============================================================================
+# 3. Data — port of the C++ DataLoaderLite (Tensor-Implementations/include/
+#    DataLoader.h), including its pinned-host / side-stream prefetch ring.
+#
+# Shards are HEADERLESS flat uint16 token files. A shard is "val" if "val"
+# appears anywhere in its basename, "train" otherwise — this covers both an
+# explicit flux_train_*/flux_val_* naming and an unmarked-train naming like
+# flux_20b_shard_*.bin alongside a separately named *val* shard. For a batch
+# at position p:
+#     x = tokens[p : p+B*T]      y = tokens[p+1 : p+B*T+1]
+# p strides by B*T (world_size == 1 here). A shard tail that cannot fit a full
+# stride is dropped and the loader wraps to the next shard.
+#
+# The prefetch mirrors the C++ ring: NBUF pinned staging buffers and NBUF device
+# tensors, H2D issued on a side stream, CUDA events ordering producer against
+# consumer. Each slot also carries the (shard, token offset) it was filled from,
+# which is what --diag prints to name the bytes behind a loss spike.
+# =============================================================================
+class DataLoaderLite:
+    NBUF = 4
+
+    def __init__(self, B, T, split, data_root, device, master=True,
+                 max_tokens_per_shard=100_000_000, rank=0, world_size=1):
+        assert split in ("train", "val")
+        self.B, self.T = B, T
+        self.split = split
+        self.max_tokens = max_tokens_per_shard
+        self.device = device
+        # DDP: every rank reads the SAME shard list but a DISJOINT, interleaved
+        # slice of it (offset by rank, strided by world_size), so a training
+        # step's world_size*grad_accum micro-batches never repeat a window.
+        self.rank = rank
+        self.world_size = world_size
+
+        # Shard naming isn't consistent across datasets: some mark the split
+        # explicitly in both names (flux_train_*.bin / flux_val_*.bin), others
+        # only mark val and leave train shards unmarked (flux_20b_shard_*.bin
+        # + a separate *val* file). Treat "contains val" as the val split and
+        # everything else as train so both conventions resolve correctly.
+        self.shards = sorted(
+            p for p in glob.glob(os.path.join(data_root, "*.bin"))
+            if ("val" in os.path.basename(p).lower()) == (split == "val")
+        )
+        if not self.shards:
+            raise RuntimeError(f"no .bin shards found for split {split} in {data_root}")
+        if master:
+            print(f"found {len(self.shards)} shards for split {split}")
+
+        BT = B * T
+        self.pin_x = [torch.empty(BT, dtype=torch.int64).pin_memory() for _ in range(self.NBUF)]
+        self.pin_y = [torch.empty(BT, dtype=torch.int64).pin_memory() for _ in range(self.NBUF)]
+        self.dev_x = [torch.empty(B, T, dtype=torch.int64, device=device) for _ in range(self.NBUF)]
+        self.dev_y = [torch.empty(B, T, dtype=torch.int64, device=device) for _ in range(self.NBUF)]
+        self.copy_stream = torch.cuda.Stream(device=device)
+        self.h2d_done = [torch.cuda.Event() for _ in range(self.NBUF)]
+        self.h2d_recorded = [False] * self.NBUF
+        self.where = [(0, 0)] * self.NBUF        # (shard_index, token_offset)
+        self.reset()
+
+    # -- shard plumbing ------------------------------------------------------
+    def _load_shard(self, i):
+        self.cur = i
+        arr = np.memmap(self.shards[i], dtype=np.uint16, mode="r")
+        self.tok = arr
+        self.ntok = min(len(arr), self.max_tokens)
+
+    def _advance_shard(self):
+        self._load_shard((self.cur + 1) % len(self.shards))
+        self.pos = self.B * self.T * self.rank
+
+    def reset(self):
+        self.copy_stream.synchronize()
+        self._load_shard(0)
+        self.pos = self.B * self.T * self.rank
+        self.primed = False
+        self.buf = 0
+
+    # -- prefetch ------------------------------------------------------------
+    def _stage(self, idx):
+        BT = self.B * self.T
+        stride = BT * self.world_size
+        if self.ntok < BT + 1:
+            raise RuntimeError(f"shard too small for one batch: {self.shards[self.cur]}")
+        if self.pos + BT + 1 > self.ntok:
+            self._advance_shard()
+
+        # Host-side WAR guard: the pinned buffer may still be the source of the
+        # H2D issued the last time this slot was filled (that copy is gated on
+        # the device, not the host, so a host running ahead of the GPU would
+        # otherwise overwrite bytes still in flight and hand the model a batch
+        # spliced from two different windows).
+        if self.h2d_recorded[idx]:
+            self.h2d_done[idx].synchronize()
+
+        self.where[idx] = (self.cur, self.pos)
+        buf = np.asarray(self.tok[self.pos: self.pos + BT + 1]).astype(np.int64)
+        self.pin_x[idx].copy_(torch.from_numpy(buf[:-1]))
+        self.pin_y[idx].copy_(torch.from_numpy(buf[1:]))
+
+        with torch.cuda.stream(self.copy_stream):
+            self.dev_x[idx].view(-1).copy_(self.pin_x[idx], non_blocking=True)
+            self.dev_y[idx].view(-1).copy_(self.pin_y[idx], non_blocking=True)
+            self.h2d_done[idx].record(self.copy_stream)
+        self.h2d_recorded[idx] = True
+
+        self.pos += stride
+        if self.pos + BT + 1 > self.ntok:
+            self._advance_shard()
+
+    def next_batch(self):
+        """-> (x, y, shard_index, token_offset), x/y int64 [B,T] on device."""
+        if not self.primed:
+            for i in range(self.NBUF):
+                self._stage(i)
+            self.primed = True
+            self.buf = 0
+        else:
+            # Refill the slot returned LAST call. Its consumer's kernels are
+            # already queued on the compute stream, so the producer must wait
+            # for them before overwriting the device tensor.
+            self.copy_stream.wait_stream(torch.cuda.current_stream())
+            self._stage((self.buf - 1) % self.NBUF)
+
+        torch.cuda.current_stream().wait_event(self.h2d_done[self.buf])
+        i = self.buf
+        self.buf = (self.buf + 1) % self.NBUF
+        return self.dev_x[i], self.dev_y[i], self.where[i][0], self.where[i][1]
+
+    def skip_batches(self, n):
+        """Fast-forward `n` batches without touching the GPU (resume path)."""
+        self.copy_stream.synchronize()
+        self.primed = False
+        BT = self.B * self.T
+        stride = BT * self.world_size
+        for _ in range(n):
+            if self.pos + BT + 1 > self.ntok:
+                self._advance_shard()
+            self.pos += stride
+            if self.pos + BT + 1 > self.ntok:
+                self._advance_shard()
+
+
+# =============================================================================
+# 4. Model.
+#
+# Precision (bf16 mixed precision; both --precision modes run under autocast):
+#   bf16-params : Linear weights stored bf16, embedding / RMSNorm / QK-norm
+#                 scales stay fp32, fp32 master weights live in the optimizer
+#                 — the exact scheme bluscript_single.cpp uses (default).
+#   bf16        : all params fp32, autocast casts per-op (standard PyTorch AMP).
+# In both cases: matmuls + attention bf16, RMSNorm / QK-norm reductions, RoPE,
+# softmax and the loss in fp32. No GradScaler (bf16 has fp32's exponent range).
+# =============================================================================
+_FLASH = [SDPBackend.FLASH_ATTENTION, SDPBackend.EFFICIENT_ATTENTION]
+
+
+class RMSNorm(nn.Module):
+    """RMSNorm over d_model, fp32 weight. F.rms_norm is a fused ATen op and is
+    not autocast-eligible, so it runs in the input dtype — fp32 on the residual
+    stream, matching the C++ nn::RMSNorm."""
+
+    def __init__(self, dim, eps):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(dim))
+        self.eps = eps
+        self.dim = (dim,)
+
+    def forward(self, x):
+        return F.rms_norm(x, self.dim, self.weight, self.eps)
+
+
+class CausalGQA(nn.Module):
+    """RMSNorm -> fused(QK-norm + RoPE) -> causal grouped-query attention -> Wo -> +x"""
+
+    def __init__(self, cfg, resid_scale):
+        super().__init__()
+        self.cfg = cfg
+        self.norm = RMSNorm(cfg.d_model, cfg.rms_eps)
+        self.wQ = nn.Linear(cfg.d_model, cfg.q_heads * cfg.head_dim, bias=False)
+        self.wK = nn.Linear(cfg.d_model, cfg.kv_heads * cfg.head_dim, bias=False)
+        self.wV = nn.Linear(cfg.d_model, cfg.kv_heads * cfg.head_dim, bias=False)
+        self.wO = nn.Linear(cfg.q_heads * cfg.head_dim, cfg.d_model, bias=False)
+        # QK-norm scales, initialised to 1.0 (identity RMS scale), fp32.
+        self.q_gamma = nn.Parameter(torch.ones(cfg.head_dim))
+        self.k_gamma = nn.Parameter(torch.ones(cfg.head_dim))
+        # GPT-2/Llama init: projections std=0.02, residual proj scaled by
+        # (2*n_layers)^-0.5 to keep residual-stream variance ~constant.
+        for lin in (self.wQ, self.wK, self.wV):
+            nn.init.normal_(lin.weight, mean=0.0, std=0.02)
+        nn.init.normal_(self.wO.weight, mean=0.0, std=0.02 * resid_scale)
+
+    def forward(self, x, cos, sin):
+        cfg = self.cfg
+        B, T, _ = x.shape
+        h = self.norm(x)
+
+        # [B,T,H*D] -> [B,H,T,D]. The transpose stays a view; the fused kernel
+        # honours arbitrary (b,h,t) strides, so no .contiguous() is paid here.
+        q = self.wQ(h).view(B, T, cfg.q_heads, cfg.head_dim).transpose(1, 2)
+        k = self.wK(h).view(B, T, cfg.kv_heads, cfg.head_dim).transpose(1, 2)
+        v = self.wV(h).view(B, T, cfg.kv_heads, cfg.head_dim).transpose(1, 2)
+
+        if cfg.fused_rms_rope:
+            q, k = fused_rms_rope(q, k, self.q_gamma, self.k_gamma, cos, sin, cfg.qk_eps)
+        else:
+            q = rms_rope_eager(q, self.q_gamma, cos, sin, cfg.qk_eps)
+            k = rms_rope_eager(k, self.k_gamma, cos, sin, cfg.qk_eps)
+
+        # Separate fused flash kernel. enable_gqa maps query head i to kv head
+        # i // (q_heads/kv_heads), the same grouping as the C++ kernel.
+        with sdpa_kernel(_FLASH):
+            o = F.scaled_dot_product_attention(
+                q.to(v.dtype), k.to(v.dtype), v, is_causal=True, enable_gqa=True)
+
+        o = o.transpose(1, 2).reshape(B, T, cfg.q_heads * cfg.head_dim)
+        return x + self.wO(o)
+
+
+class SwiGLUMLP(nn.Module):
+    """RMSNorm -> gate_up -> silu(gate)*up -> down -> +x   ([gate | up] layout)"""
+
+    def __init__(self, cfg, resid_scale):
+        super().__init__()
+        self.norm = RMSNorm(cfg.d_model, cfg.rms_eps)
+        self.gate_up = nn.Linear(cfg.d_model, 2 * cfg.ffn_hidden, bias=False)
+        self.down = nn.Linear(cfg.ffn_hidden, cfg.d_model, bias=False)
+        nn.init.normal_(self.gate_up.weight, mean=0.0, std=0.02)
+        nn.init.normal_(self.down.weight, mean=0.0, std=0.02 * resid_scale)
+
+    def forward(self, x):
+        gate, up = self.gate_up(self.norm(x)).chunk(2, dim=-1)
+        return x + self.down(F.silu(gate) * up)
+
+
+class Block(nn.Module):
+    def __init__(self, cfg, resid_scale):
+        super().__init__()
+        self.attn = CausalGQA(cfg, resid_scale)
+        self.mlp = SwiGLUMLP(cfg, resid_scale)
+
+    def forward(self, x, cos, sin):
+        return self.mlp(self.attn(x, cos, sin))
+
+
+class BLU(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.cfg = cfg
+        resid_scale = 1.0 / math.sqrt(2.0 * cfg.n_layers)
+
+        self.wte = nn.Embedding(cfg.vocab_size, cfg.d_model)
+        nn.init.normal_(self.wte.weight, mean=0.0, std=0.02)
+
+        self.blocks = nn.ModuleList([Block(cfg, resid_scale) for _ in range(cfg.n_layers)])
+        self.norm_f = RMSNorm(cfg.d_model, cfg.rms_eps)
+        if not cfg.weight_tying:
+            self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
+            nn.init.normal_(self.lm_head.weight, mean=0.0, std=0.02)
+
+        cos, sin = build_rope_cache(cfg.context_length, cfg.head_dim, cfg.rope_theta, "cpu")
+        self.register_buffer("rope_cos", cos, persistent=False)
+        self.register_buffer("rope_sin", sin, persistent=False)
+
+    def apply_param_dtype(self, mode):
+        """`bf16-params`: store Linear weights in bf16 (embedding, RMSNorm and
+        QK-norm scales stay fp32) — the two-dtype-group layout of the C++ model.
+        Under autocast this is transparent: the cast to bf16 becomes a no-op."""
+        if mode != "bf16-params":
+            return
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                m.weight.data = m.weight.data.to(torch.bfloat16)
+
+    def forward(self, idx, targets=None):
+        cfg = self.cfg
+        T = idx.shape[1]
+        cos, sin = self.rope_cos[:T], self.rope_sin[:T]
+
+        x = self.wte(idx)                                    # fp32 [B,T,d_model]
+        for blk in self.blocks:
+            if cfg.activation_checkpoint and self.training:
+                x = checkpoint(blk, x, cos, sin, use_reentrant=False)
+            else:
+                x = blk(x, cos, sin)
+        x = self.norm_f(x)
+
+        head_w = self.wte.weight if cfg.weight_tying else self.lm_head.weight
+        if targets is None:
+            return F.linear(x, head_w), None
+        if cfg.loss_chunks > 1:
+            return None, _chunked_ce(x, head_w, targets, cfg.loss_chunks)
+        # Matches the C++: bf16 matmul, logits promoted to fp32 for the loss
+        # (cross_entropy is an autocast-fp32 op, so the promotion is implicit).
+        logits = F.linear(x, head_w)
+        loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.reshape(-1))
+        return logits, loss
+
+
+def _chunked_ce(x, weight, targets, chunks):
+    """lm_head + cross-entropy in row chunks, logits recomputed in backward.
+    Peak logit memory drops from [B*T, V] to [B*T/chunks, V] at the cost of one
+    extra lm_head matmul. Off by default (`--loss-chunks 1`) so the memory
+    profile matches the C++ script, which materialises the full fp32 logits.
+    """
+    xf = x.reshape(-1, x.shape[-1])
+    tf = targets.reshape(-1)
+    n = xf.shape[0]
+    size = (n + chunks - 1) // chunks
+    total = xf.new_zeros((), dtype=torch.float32)
+    for lo in range(0, n, size):
+        hi = min(lo + size, n)
+        part = checkpoint(_ce_chunk, xf[lo:hi], weight, tf[lo:hi], use_reentrant=False)
+        total = total + part * (hi - lo)
+    return total / n
+
+
+def _ce_chunk(x, weight, targets):
+    return F.cross_entropy(F.linear(x, weight).float(), targets)
+
+
+# =============================================================================
+# 5. Optimizer (Muon on the hidden matrices, AdamW on the rest), gradient
+#    clipping, checkpointing.
+# =============================================================================
+def _is_muon_param(name, p):
+    """Muon orthogonalizes a matrix, so it needs the parameter to BE a linear
+    map between two feature spaces:
+      * dim >= 2 — RMSNorm weights and QK-norm gammas are 1D and have no
+        singular values to equalise;
+      * not the embedding / lm_head — its rows index tokens, not features, so
+        the left singular vectors mix unrelated vocabulary entries. Weight
+        tying makes this one tensor serving both ends; either way it is AdamW's.
+    Everything else — wQ/wK/wV/wO, gate_up, down — is Muon's: 75.5M of the
+    114.2M parameters at the default config.
+    """
+    return p.dim() >= 2 and not name.startswith(("wte.", "lm_head."))
+
+
+def _state_bytes(opt):
+    """Bytes held in a torch optimizer's per-parameter state — Adam's m/v, or
+    Muon's momentum buffer. Reads 0 before the first step(): both allocate
+    lazily."""
+    return sum(v.numel() * v.element_size()
+               for st in opt.state.values()
+               for v in st.values() if torch.is_tensor(v))
+
+
+class MasterMuon:
+    """torch.optim.Muon on the hidden matrices + torch.optim.AdamW on the rest,
+    both driving fp32 master weights.
+    Neither optimizer is reimplemented here. What this class owns is the part
+    torch cannot know about:
+      * ROUTING — torch.optim.Muon accepts whatever you hand it and the docs are
+        explicit that only 2D hidden weights belong ("bias, and embedding,
+        should be optimized by a standard method such as AdamW"). The split is
+        `_is_muon_param`.
+      * MASTER WEIGHTS — the parameters are bf16 under --precision bf16-params,
+        and a bf16 tensor cannot absorb an lr*update of 1e-4 relative size. So
+        both optimizers are constructed over the fp32 MASTERS, fed from the fp32
+        landing buffers, and the masters are cast back over the params after the
+        step. Unchanged from the AdamW arm — that machinery is what makes the
+        two arms comparable, since both optimizers then read a bit-identical
+        fp32 gradient and write a bit-identical bf16 parameter.
+    Weight decay is decoupled in both (torch.optim.Muon applies it before the
+    update, as AdamW does) and is applied to the master for the same reason: a
+    decay written to the bf16 param is overwritten by the master write-back
+    microseconds later and silently no-ops.
+    State cost: Muon holds one momentum buffer per param, against Adam's two.
+    """
+
+    def __init__(self, named_params, cfg):
+        self.cfg = cfg
+        self.triples = []                 # (param, master, landing_grad), ALL params
+        self.muon_masters = []            # masters routed to Muon
+        decay, nodecay = [], []
+        self.n_muon = self.n_decay = self.n_nodecay = 0
+        for name, p in named_params:
+            if p.dtype == torch.float32:
+                m, g = p, None
+            else:
+                m = p.detach().float().clone()
+                g = torch.zeros_like(m)
+                m.grad = g
+            self.triples.append((p, m, g))
+            if _is_muon_param(name, p):
+                self.muon_masters.append(m)
+                self.n_muon += m.numel()
+            elif p.dim() >= 2:
+                decay.append(m)
+                self.n_decay += m.numel()
+            else:
+                nodecay.append(m)
+                self.n_nodecay += m.numel()
+        self.opt = torch.optim.AdamW(
+            [{"params": decay, "weight_decay": cfg.weight_decay},
+             {"params": nodecay, "weight_decay": 0.0}],
+            lr=cfg.max_lr, betas=(0.9, 0.95), eps=1e-8, fused=True)
+        # Everything below is a torch.optim.Muon default except adjust_lr_fn
+        # (torch: "original") and weight_decay, which is pinned to the AdamW
+        # arm's 0.1 so the decay is identical across both groups and both arms.
+        self.muon = torch.optim.Muon(
+            self.muon_masters,
+            lr=cfg.muon_lr,
+            weight_decay=cfg.weight_decay,
+            momentum=cfg.muon_momentum,
+            nesterov=True,
+            ns_steps=cfg.muon_ns_steps,
+            adjust_lr_fn=cfg.muon_adjust_lr)
+
+    @torch.no_grad()
+    def step(self, lr, muon_lr):
+        for p, m, g in self.triples:
+            if g is not None:
+                g.copy_(p.grad)               # bf16 grad -> fp32 landing buffer
+        # AdamW group (embedding / lm_head / norms / gammas) — identical to the
+        # baseline's update, on identical gradients.
+        for grp in self.opt.param_groups:
+            grp["lr"] = lr
+        self.opt.step()
+        # Muon group. Its nesterov path may consume the gradient in place; that
+        # is harmless here because grad_norm and the --diag probes are read
+        # before step() and the landing buffers are overwritten wholesale on the
+        # next one.
+        for grp in self.muon.param_groups:
+            grp["lr"] = muon_lr
+        self.muon.step()
+        for p, m, g in self.triples:
+            if g is not None:
+                p.data.copy_(m)               # fp32 master -> bf16 param
+
+    def state_breakdown(self):
+        """-> (master_bytes, landing_grad_bytes, adam_mv_bytes, muon_mom_bytes).
+        The first two are zero under --precision bf16, where the master IS the
+        parameter and the gradient is already fp32. Both optimizers allocate
+        their state lazily on the first step(), so the last two read 0 until
+        then — the startup report uses the expected_* predictors instead.
+        """
+        master = landing = 0
+        for p, m, g in self.triples:
+            if m is not p:
+                master += m.numel() * m.element_size()
+                landing += g.numel() * g.element_size()
+        mv = _state_bytes(self.opt)
+        mom = _state_bytes(self.muon)
+        return master, landing, mv, mom
+
+    def state_bytes(self):
+        return sum(self.state_breakdown())
+
+    def expected_mv_bytes(self):
+        """Adam m/v are lazily allocated, so the startup report has to predict
+        them rather than measure them. Muon's params are excluded — they carry a
+        momentum buffer instead, reported separately."""
+        muon_ids = {id(m) for m in self.muon_masters}
+        return sum(m.numel() * m.element_size() * 2
+                   for _, m, _ in self.triples if id(m) not in muon_ids)
+
+    def expected_muon_mom_bytes(self):
+        """One fp32 momentum buffer per Muon parameter — the prediction the
+        startup report prints before step 0 has allocated it."""
+        return sum(m.numel() * m.element_size() for m in self.muon_masters)
+
+    # -- checkpointing -------------------------------------------------------
+    def state_dict(self):
+        # `optimizer` tags the arm: resuming a Muon run from an AdamW checkpoint
+        # would restore the weights and then silently drop the momentum, so
+        # CheckpointManager refuses on a mismatch rather than producing a run
+        # that looks fine and is not comparable to either arm.
+        return {"optimizer": "muon",
+                "adam": self.opt.state_dict(),
+                "muon": self.muon.state_dict(),
+                "masters": [None if m is p else m for p, m, g in self.triples]}
+
+    @torch.no_grad()
+    def load_state_dict(self, blob):
+        """-> True if the fp32 masters were restored from the checkpoint."""
+        if blob.get("optimizer") != "muon":
+            raise SystemExit(
+                "ERROR: this checkpoint was written by the AdamW arm (train.py). "
+                "Point --ckpt-dir somewhere else, or resume it with train.py.")
+        self.opt.load_state_dict(blob["adam"])
+        if blob.get("muon") is not None:
+            self.muon.load_state_dict(blob["muon"])
+        masters = blob.get("masters")
+        if masters is None or len(masters) != len(self.triples):
+            return False
+        for (p, m, g), saved in zip(self.triples, masters):
+            if saved is not None and m is not p:
+                m.copy_(saved.to(m.device))
+        return True
+
+    @torch.no_grad()
+    def resync_masters(self):
+        """Re-seed every master from its parameter.
+        Needed after loading a checkpoint that carries no masters: the masters
+        were built in __init__ from the RANDOM INIT, so without this the first
+        step() would write them straight back over the restored weights and
+        silently throw the run away.
+        """
+        for p, m, g in self.triples:
+            if m is not p:
+                m.copy_(p.detach().float())
+
+
+def clip_grads_(params, max_norm):
+    """ONE global L2 norm over every parameter, then one rescale — the textbook
+    definition. (It differs from the ZeRO script, which clips each dtype group
+    against its own norm and can therefore let the true global norm reach
+    sqrt(n_groups) * grad_clip.) The per-tensor norms are promoted to fp32
+    before they are combined, or a reduction over ~250 bf16 norms would lose
+    ~3 digits."""
+    grads = [p.grad for p in params if p.grad is not None]
+    norms = torch._foreach_norm(grads)
+    total = torch.linalg.vector_norm(torch.stack([n.float() for n in norms]))
+    if max_norm and max_norm > 0:
+        clip = (max_norm / (total + 1e-6)).clamp(max=1.0)
+        torch._foreach_mul_(grads, clip)
+    return total
+
+
+class CheckpointManager:
+    """Synchronous save / keep-N / auto-resume, mirroring the C++
+    CheckpointManager. A checkpoint holds the model parameters, Adam's m/v, the
+    fp32 masters, and the step it should resume at."""
+
+    def __init__(self, ckpt_dir, prefix, keep):
+        self.dir, self.prefix, self.keep = ckpt_dir, prefix, keep
+        os.makedirs(ckpt_dir, exist_ok=True)
+
+    def _path(self, step):
+        return os.path.join(self.dir, f"{self.prefix}_step{step:08d}.pt")
+
+    def _existing(self):
+        out = []
+        for p in glob.glob(os.path.join(self.dir, f"{self.prefix}_step*.pt")):
+            m = re.search(r"_step(\d+)\.pt$", os.path.basename(p))
+            if m:
+                out.append((int(m.group(1)), p))
+        return sorted(out)
+
+    def save(self, step, model, opt, loss, last_val, cfg):
+        tmp = self._path(step) + ".tmp"
+        torch.save({"step": step, "loss": loss, "last_val": last_val,
+                    # B/T/global_batch pin the data stride this checkpoint's
+                    # step count assumes. world_size is deliberately NOT
+                    # recorded: grad_accum = global_batch // (B*T*world_size)
+                    # already makes tokens-per-step (and skip_batches'
+                    # target offset) independent of how many ranks are
+                    # running, so a GPU-count change alone is not a mismatch.
+                    "run_cfg": {"B": cfg.B, "T": cfg.T,
+                                "global_batch": cfg.global_batch},
+                    "model": model.state_dict(),
+                    "opt": opt.state_dict()}, tmp)
+        # Rename last: a crash mid-write leaves a .tmp, never a truncated
+        # checkpoint that the next load_latest would pick as the newest.
+        os.replace(tmp, self._path(step))
+        if self.keep > 0:
+            for _, stale in self._existing()[:-self.keep]:
+                os.remove(stale)
+
+    def load_latest(self, model, opt, cfg, master=True):
+        """-> (start_step, loss, last_val). (0, 0.0, nan) when there is
+        nothing to resume."""
+        existing = self._existing()
+        if not existing:
+            return 0, 0.0, float("nan")
+        _, path = existing[-1]
+        return self.load(path, model, opt, cfg, master)
+
+    def load(self, path, model, opt, cfg, master=True):
+        """Load a specific checkpoint file (e.g. from --resume-from), rather
+        than auto-discovering the latest one in self.dir.
+        -> (start_step, loss, last_val)."""
+        blob = torch.load(path, map_location="cpu", weights_only=True)
+        model.load_state_dict(blob["model"])
+        if not opt.load_state_dict(blob["opt"]):
+            opt.resync_masters()
+        run_cfg = blob.get("run_cfg")
+        if master and run_cfg is not None:
+            mismatches = [f"{k} checkpoint={v!r} current={getattr(cfg, k)!r}"
+                         for k, v in run_cfg.items() if getattr(cfg, k) != v]
+            if mismatches:
+                print("[WARN] resume config differs from the checkpoint's — "
+                      "the data offset this step count assumes will be wrong:")
+                for m in mismatches:
+                    print(f"         {m}")
+        print(f"[resume] loaded {path}")
+        return (int(blob["step"]), float(blob["loss"]),
+                float(blob.get("last_val", float("nan"))))
+
+
+# =============================================================================
+# 6. Training.
+# =============================================================================
+def _env_int(name, default=0):
+    v = os.environ.get(name)
+    if v in (None, ""):
+        return default
+    try:
+        return int(v)
+    except ValueError:
+        print(f"[WARN] ignoring non-integer {name}={v!r}")
+        return default
+
+
+def seed_everything(seed):
+    """Every RNG the run can draw from, off one seed.
+    torch's CPU generator is the one that matters here (the model is built on
+    CPU and moved), but `random` / numpy are seeded too so that anything added
+    later — a shuffled shard order, a sampled probe — is reproducible by
+    default rather than by accident.
+    """
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+
+
+def apply_determinism(mode):
+    """Pin the run-to-run drift that seeding does not cover.
+    Three sources, all on the backward pass:
+      * nn.Embedding's dense backward scatters one atomicAdd per token into
+        [vocab, d_model]; with weight tying that is also the lm_head gradient,
+        so it is the largest reduction in the step;
+      * the flash / mem-efficient attention backward splits dq over the key
+        blocks and atomicAdds them together;
+      * cuDNN picks kernels by wall-clock timing when `benchmark` is on, so the
+        chosen algorithm can differ between runs of the same shape.
+    Each is a float reduction in an order the scheduler chooses, so two runs
+    diverge in the last mantissa bits of the first step and, through the
+    optimizer, visibly by a few hundred steps.
+    `strict` raises if some op has no deterministic kernel — better than
+    reporting a reproducible run that is not one. `warn` downgrades that to a
+    warning and keeps training; `off` restores the default fast kernels.
+    """
+    if mode == "off":
+        return
+    torch.backends.cudnn.benchmark = False
+    torch.backends.cudnn.deterministic = True
+    # Routes embedding-backward and SDPA-backward to their ordered variants, and
+    # enforces the cuBLAS workspace setting made at the top of this file.
+    torch.use_deterministic_algorithms(True, warn_only=(mode == "warn"))
+
+
+def build_argparser():
+    ap = argparse.ArgumentParser(
+        description="BLU single-GPU training — PyTorch port of bluscript_single.cpp")
+    ap.add_argument("--data-root",
+                    default=("/root/.cache/huggingface/hub/datasets--jayasriii--flux_20b/"
+                             "/snapshots/ddc6cc7e43365a6b9096c30053df0fb001c94f4c"))
+    ap.add_argument("--device", type=int, default=None,
+                    help="CUDA device index (env: BLU_DEVICE)")
+    ap.add_argument("--precision", choices=["bf16-params", "bf16"], default="bf16-params",
+                    help="bf16-params (default): Linear weights stored bf16 with "
+                         "fp32 master weights in the optimizer — the C++ scheme. "
+                         "bf16: fp32 params + autocast (standard PyTorch AMP).")
+    ap.add_argument("--rms-rope", choices=["fused", "eager"], default="fused",
+                    help="fused = Triton QK-norm+RoPE kernel (default), "
+                         "eager = unfused reference, for A/B on the kernel")
+    ap.add_argument("--compile", action="store_true", help="torch.compile each block")
+    ap.add_argument("--activation-checkpoint", action="store_true",
+                    help="recompute each block in backward (trades ~30%% time for memory)")
+    ap.add_argument("--loss-chunks", type=int, default=1,
+                    help="chunk lm_head+CE into N pieces with recompute; 1 (default) "
+                         "materialises the full logits, as the C++ does")
+    # bluscript_single.cpp defaults unless overridden.
+    ap.add_argument("--B", type=int, default=None, help="micro-batch (env: BLU_B)")
+    ap.add_argument("--T", type=int, default=None)
+    ap.add_argument("--d-model", type=int, default=None, help="smoke-test override")
+    ap.add_argument("--n-layers", type=int, default=None, help="smoke-test override")
+    ap.add_argument("--q-heads", type=int, default=None, help="smoke-test override")
+    ap.add_argument("--kv-heads", type=int, default=None, help="smoke-test override")
+    ap.add_argument("--ffn-hidden", type=int, default=None, help="smoke-test override")
+    ap.add_argument("--vocab-size", type=int, default=None, help="smoke-test override")
+    ap.add_argument("--global-batch", type=int, default=None,
+                    help="tokens per optimizer step (env: BLU_GLOBAL_BATCH)")
+    ap.add_argument("--max-steps", type=int, default=None,
+                    help="caps the run for benchmarking; warmup/stable shrink "
+                         "proportionally so the WSD decay tail stays non-empty "
+                         "(env: BLU_MAX_STEPS)")
+    ap.add_argument("--target-losses", type=float, nargs="+", default=None,
+                    help="loss milestones to record (step + elapsed hours) and "
+                         "checkpoint on, as the EMA-smoothed train loss passes "
+                         "each one; default is the shared target-loss set used "
+                         "across this project's batch-size ablation arms.")
+    ap.add_argument("--target-loss-ema-beta", type=float, default=None,
+                    help="EMA smoothing beta applied before comparing against "
+                         "--target-losses (default 0.99)")
+    # ---- Muon ----
+    ap.add_argument("--muon-lr", type=float, default=None,
+                    help="peak LR for the hidden 2D weights. Under the default "
+                         "--muon-adjust-lr match_rms_adamw this is the same unit as "
+                         "--max-lr, and defaults to the same value (2e-4) so "
+                         "both arms take a matched first run; sweep it upward "
+                         "from there (Muon usually takes 2-4x AdamW's LR).")
+    ap.add_argument("--muon-min-lr", type=float, default=None,
+                    help="floor of the Muon cosine tail (default 10%% of --muon-lr)")
+    ap.add_argument("--muon-momentum", type=float, default=None,
+                    help="torch.optim.Muon momentum (default 0.95, torch's own)")
+    ap.add_argument("--muon-ns-steps", type=int, default=None,
+                    help="torch.optim.Muon ns_steps, Newton-Schulz iterations "
+                         "(default 5, torch's own); fewer = looser "
+                         "orthogonalization, cheaper")
+    ap.add_argument("--muon-adjust-lr", choices=["match_rms_adamw", "original"],
+                    default=None,
+                    help="torch.optim.Muon adjust_lr_fn. match_rms_adamw "
+                         "(default HERE): 0.2*sqrt(max(A,B)), putting --muon-lr "
+                         "on the same scale as --max-lr. original (default in "
+                         "torch): sqrt(max(1,A/B)), needs a ~5.5x larger "
+                         "--muon-lr at d_model=768.")
+    ap.add_argument("--val-freq", type=int, default=None)
+    ap.add_argument("--val-loss-steps", type=int, default=None)
+    ap.add_argument("--diag", action="store_true",
+                    help="per-micro-batch loss + depth probes (env: BLU_DIAG=1)")
+    ap.add_argument("--warmup-measure", type=int, default=3,
+                    help="steps excluded from the steady-state throughput summary")
+    ap.add_argument("--csv", default="blu_muon_log.csv",
+                    help="CSV log path; appended (not truncated) on resume. The "
+                         "first 8 columns match the AdamW arm's CSV exactly, so "
+                         "the two files plot against each other directly; "
+                         "muon_lr is appended as a 9th.")
+    ap.add_argument("--seed", type=int, default=1234)
+    ap.add_argument("--determinism", choices=["strict", "warn", "off"], default="strict",
+                    help="strict (default): deterministic CUDA kernels, so a rerun "
+                         "of the same launch line reproduces the loss curve bit for "
+                         "bit; raises if an op has no deterministic implementation. "
+                         "warn: same, but non-deterministic ops warn instead of "
+                         "raising. off: fastest kernels, results drift between runs.")
+    # checkpointing
+    ap.add_argument("--no-checkpointing", action="store_true")
+    ap.add_argument("--ckpt-dir", default=None)
+    ap.add_argument("--ckpt-prefix", default=None)
+    ap.add_argument("--ckpt-freq", type=int, default=None)
+    ap.add_argument("--ckpt-keep", type=int, default=None)
+    ap.add_argument("--resume-from", default=None,
+                    help="path to a specific *_step*.pt checkpoint to resume "
+                         "from, overriding auto-discovery of the latest "
+                         "checkpoint in --ckpt-dir")
+    return ap
+
+
+def config_from_args(args):
+    cfg = ModelConfig()
+    for f in ("B", "T", "d_model", "n_layers", "q_heads", "kv_heads", "ffn_hidden",
+              "vocab_size", "global_batch", "val_freq", "val_loss_steps",
+              "ckpt_dir", "ckpt_prefix", "ckpt_freq", "ckpt_keep",
+              "muon_lr", "muon_min_lr", "muon_momentum", "muon_ns_steps",
+              "muon_adjust_lr", "target_losses", "target_loss_ema_beta"):
+        v = getattr(args, f, None)
+        if v is not None:
+            setattr(cfg, f, v)
+    # A --muon-lr given without a floor keeps the 10% ratio the WSD tail assumes,
+    # instead of silently decaying to the default 2e-5 from wherever it starts.
+    if args.muon_lr is not None and args.muon_min_lr is None:
+        cfg.muon_min_lr = 0.1 * cfg.muon_lr
+    cfg.fused_rms_rope = (args.rms_rope == "fused")
+    cfg.activation_checkpoint = args.activation_checkpoint
+    cfg.loss_chunks = args.loss_chunks
+    cfg.checkpointing = not args.no_checkpointing
+    cfg.context_length = max(cfg.context_length, cfg.T)
+
+    # Mirrors the BLU_MAX_STEPS branch in bluscript_single.cpp: shrink warmup and
+    # stable with the run, or a short benchmark leaves no decay tail (the C++
+    # asserts on it; get_lr here would clamp, which is worse — it would silently
+    # report a schedule the config does not describe).
+    if args.max_steps is not None:
+        ms = args.max_steps
+        cfg.max_steps = ms
+        cfg.warmup_steps = max(1, ms // 10)
+        cfg.stable_steps = max(1, ms // 2)
+        if cfg.warmup_steps + cfg.stable_steps >= cfg.max_steps - 1:
+            cfg.stable_steps = max(1, cfg.max_steps - cfg.warmup_steps - 2)
+        print(f"[bench] max_steps={ms} (warmup={cfg.warmup_steps} "
+              f"stable={cfg.stable_steps})")
+    return cfg
+
+
+def validate_config(cfg, world_size=1, master=True):
+    """Cheap checks; each one is a silent-corruption or out-of-bounds failure
+    mode if it does not hold."""
+    bad = []
+    if cfg.T > cfg.context_length:
+        bad.append(f"T ({cfg.T}) > context_length ({cfg.context_length}): "
+                   "the RoPE cache is too short")
+    if cfg.q_heads % cfg.kv_heads != 0:
+        bad.append(f"q_heads ({cfg.q_heads}) not divisible by kv_heads ({cfg.kv_heads})")
+    if cfg.head_dim not in (64, 128):
+        bad.append(f"the fused kernels support head_dim 64 or 128, got {cfg.head_dim}")
+    # Under DDP each rank runs B*T tokens per micro-step, so the world's tokens
+    # per micro-step are B*T*world_size; global_batch must land on a whole
+    # number of those or grad_accum below truncates and quietly under-shoots
+    # the configured optimizer-step token count.
+    if cfg.global_batch % (cfg.B * cfg.T * world_size) != 0:
+        bad.append(f"global_batch ({cfg.global_batch}) not divisible by "
+                   f"B*T*world_size ({cfg.B * cfg.T * world_size})")
+    if cfg.warmup_steps < 1:
+        bad.append("warmup_steps must be >= 1 (get_lr divides by it)")
+    if cfg.warmup_steps + cfg.stable_steps >= cfg.max_steps - 1:
+        bad.append(f"warmup_steps + stable_steps "
+                   f"({cfg.warmup_steps + cfg.stable_steps}) leaves no decay tail "
+                   f"before max_steps ({cfg.max_steps})")
+    if not hasattr(torch.optim, "Muon"):
+        bad.append(f"torch {torch.__version__} has no torch.optim.Muon — it first "
+                   "appears in the 2.9 docs. Upgrade torch, or run the AdamW "
+                   "arm (train.py).")
+    if cfg.muon_ns_steps < 1:
+        bad.append("muon_ns_steps must be >= 1 (0 leaves the update unorthogonalized, "
+                   "which is not Muon)")
+    if cfg.muon_momentum <= 0.0 or cfg.muon_momentum >= 1.0:
+        bad.append(f"muon_momentum must be in (0,1), got {cfg.muon_momentum}")
+    if cfg.q_heads * cfg.head_dim != cfg.d_model and master:
+        print(f"WARNING: q_heads*head_dim ({cfg.q_heads * cfg.head_dim}) "
+              f"!= d_model ({cfg.d_model})")
+    if bad:
+        raise SystemExit("ERROR: " + "\nERROR: ".join(bad))
+
+
+def main():
+    args = build_argparser().parse_args()
+
+    # Env overrides, for parity with the C++ launch line. An explicit CLI flag
+    # wins over the environment.
+    if args.B is None:
+        args.B = _env_int("BLU_B", 0) or None
+    if args.global_batch is None:
+        args.global_batch = _env_int("BLU_GLOBAL_BATCH", 0) or None
+    if args.max_steps is None:
+        args.max_steps = _env_int("BLU_MAX_STEPS", 0) or None
+    diag = args.diag or _env_int("BLU_DIAG", 0) != 0
+
+    if not torch.cuda.is_available():
+        raise SystemExit("ERROR: no CUDA device available")
+
+    # ---- DDP bootstrap ----
+    # torchrun sets RANK/LOCAL_RANK/WORLD_SIZE; their absence means a plain
+    # single-process launch, where --device / BLU_DEVICE pick the GPU instead.
+    # torch.cuda.set_device MUST happen before init_process_group: NCCL creates
+    # its context on whatever device is current at that call, so doing it after
+    # would leave every rank on this node contending for device 0.
+    ddp = int(os.environ.get("RANK", -1)) != -1
+    if ddp:
+        ddp_rank = int(os.environ["RANK"])
+        ddp_local_rank = int(os.environ["LOCAL_RANK"])
+        ddp_world_size = int(os.environ["WORLD_SIZE"])
+        master_process = ddp_rank == 0
+        if ddp_local_rank >= torch.cuda.device_count():
+            raise SystemExit(f"ERROR: LOCAL_RANK={ddp_local_rank} but only "
+                             f"{torch.cuda.device_count()} CUDA device(s) visible")
+        torch.cuda.set_device(ddp_local_rank)
+        device = torch.device("cuda", ddp_local_rank)
+        dist.init_process_group(backend="nccl")
+        if master_process and (args.device is not None
+                               or os.environ.get("BLU_DEVICE") not in (None, "")):
+            print("[WARN] --device / BLU_DEVICE is ignored under torchrun — "
+                  "LOCAL_RANK selects each rank's GPU.")
+    else:
+        ddp_rank, ddp_local_rank, ddp_world_size = 0, 0, 1
+        master_process = True
+        if args.device is None:
+            args.device = _env_int("BLU_DEVICE", 0)
+        if args.device >= torch.cuda.device_count():
+            raise SystemExit(f"ERROR: BLU_DEVICE={args.device} but only "
+                             f"{torch.cuda.device_count()} CUDA device(s) visible")
+        torch.cuda.set_device(args.device)
+        device = torch.device("cuda", args.device)
+
+    seed_everything(args.seed)
+    apply_determinism(args.determinism)
+    # TF32 loses mantissa bits but is deterministic, so it is orthogonal to the
+    # setting above and stays on.
+    torch.set_float32_matmul_precision("high")
+
+    cfg = config_from_args(args)
+    if cfg.fused_rms_rope and not _HAS_TRITON:
+        if master_process:
+            print("[WARN] Triton not available — falling back to the eager QK-norm+RoPE")
+        cfg.fused_rms_rope = False
+    validate_config(cfg, ddp_world_size, master_process)
+
+    # Tokens consumed per optimizer step = B * T * grad_accum * world_size.
+    grad_accum = cfg.global_batch // (cfg.B * cfg.T * ddp_world_size)
+    tokens_per_step = cfg.B * cfg.T * grad_accum * ddp_world_size
+
+    if master_process:
+        print("================ Configuration ================")
+        mode_str = (f"DDP x{ddp_world_size} (torchrun, NCCL)" if ddp
+                   else "SINGLE GPU (no torchrun / DDP / ZeRO)")
+        print(f"  mode            : {mode_str}")
+        print(f"  cuda device     : {ddp_local_rank if ddp else args.device} "
+              f"({torch.cuda.get_device_name(device)})")
+        print(f"  d_model         : {cfg.d_model}")
+        print(f"  n_layers        : {cfg.n_layers}")
+        print(f"  q_heads/kv_heads: {cfg.q_heads} / {cfg.kv_heads} "
+              f"(G={cfg.q_heads // cfg.kv_heads})")
+        print(f"  head_dim        : {cfg.head_dim}")
+        print(f"  ffn_hidden      : {cfg.ffn_hidden}  (SwiGLU)")
+        print(f"  context_length  : {cfg.context_length}")
+        print(f"  rope_theta      : {cfg.rope_theta:g}  (NeoX half-split)")
+        print(f"  vocab_size      : {cfg.vocab_size}")
+        print(f"  weight_tying    : {'ON' if cfg.weight_tying else 'OFF'}")
+        print(f"  B x T           : {cfg.B} x {cfg.T}")
+        print(f"  global_batch    : {cfg.global_batch}  (grad_accum={grad_accum}, "
+              f"world_size={ddp_world_size})")
+        print("  optimizer       : MUON (hidden 2D) + AdamW (embed/head/1D)")
+        print(f"  max_lr / min_lr : {cfg.max_lr:g} / {cfg.min_lr:g}   (AdamW group)")
+        print(f"  muon lr / min   : {cfg.muon_lr:g} / {cfg.muon_min_lr:g}   "
+              f"(adjust_lr_fn={cfg.muon_adjust_lr})")
+        print(f"  muon momentum   : {cfg.muon_momentum:g}  | NS steps {cfg.muon_ns_steps}")
+        print(f"  warmup / steps  : {cfg.warmup_steps} / {cfg.max_steps}")
+        print(f"  WSD stable_steps: {cfg.stable_steps}  "
+              f"(decay tail={cfg.max_steps - cfg.warmup_steps - cfg.stable_steps})")
+        tl_str = ", ".join(f"{t:g}" for t in cfg.target_losses)
+        print(f"  target losses   : [{tl_str}]  (ema beta {cfg.target_loss_ema_beta:g}, "
+              f"checkpointed on crossing — ckpt_freq is unused)")
+        print(f"  precision       : {args.precision}")
+        print(f"  seed / determ.  : {args.seed} / {args.determinism}")
+        print(f"  rms+rope        : {'fused' if cfg.fused_rms_rope else 'eager'}"
+              f" | compile {args.compile} | act-ckpt {cfg.activation_checkpoint}"
+              f" | loss_chunks {cfg.loss_chunks}")
+        print("===============================================", flush=True)
+
+    # ---- model ----
+    # Built and seeded identically on every rank (same seed, deterministic CPU
+    # init); DDP's constructor broadcasts rank 0's parameters/buffers anyway,
+    # so this is belt-and-suspenders against any future source of divergence.
+    if master_process:
+        print(f"Initializing model on CUDA device {device.index}...", flush=True)
+    raw_model = BLU(cfg).to(device)
+    raw_model.apply_param_dtype(args.precision)
+    params = list(raw_model.parameters())
+    n_params = sum(p.numel() for p in params)
+    if master_process:
+        print(f"Number of parameters: {n_params}")
+    if args.compile:
+        for blk in raw_model.blocks:
+            blk.compile()
+
+    # DDP wraps the (already dtype-cast, already compiled) raw module. Buffer
+    # broadcasting is off: rope_cos/rope_sin are deterministic functions of
+    # cfg, identical on every rank already, so syncing them every forward
+    # would just be wasted communication.
+    # if ddp:
+    #     model = DDP(raw_model, device_ids=[ddp_local_rank], broadcast_buffers=False)
+    # else:
+    #     model = raw_model
+
+    if ddp:
+        model = DDP(raw_model, device_ids=[ddp_local_rank], broadcast_buffers=False)
+        torch.cuda.synchronize()   # let DDP's constructor-time NCCL broadcast fully land
+                                # before DataLoaderLite's independent copy_stream starts
+                                # issuing its own async H2D work — see [crash investigation]
+    else:
+        model = raw_model
+
+
+    # ---- optimizer ----
+    # named_parameters(), not parameters(): the routing is by name (the
+    # embedding and lm_head stay on AdamW). Duplicates are dropped by default,
+    # so a weight-tied head is visited once, as `wte.weight`. Built from
+    # raw_model so names never carry DDP's "module." prefix.
+    opt = MasterMuon(raw_model.named_parameters(), cfg)
+    if master_process:
+        print(f"muon params: {opt.n_muon} | adamw decayed: {opt.n_decay}, "
+              f"non-decayed: {opt.n_nodecay}")
+    # Static footprint. Nothing here is sharded — plain DDP replicates the
+    # full optimizer state on every rank (the whole cost of not running
+    # ZeRO-1), so this is the per-GPU charge before a single activation is
+    # allocated.
+    gb = 2 ** 30
+    param_bytes = sum(p.numel() * p.element_size() for p in params)
+    grad_bytes = param_bytes                       # grads share the param dtype
+    master_bytes, landing_bytes, _, _ = opt.state_breakdown()
+    # Both optimizers allocate their state on the first step(), so at this point
+    # the measured values are 0 and the report has to predict them.
+    mv_bytes = opt.expected_mv_bytes()
+    mom_bytes = opt.expected_muon_mom_bytes()
+    static_bytes = (param_bytes + grad_bytes + master_bytes + landing_bytes
+                    + mv_bytes + mom_bytes)
+    if master_process:
+        print(f"[{'ddp' if ddp else 'single'}] static footprint  (per GPU: "
+              f"params + grads + optimizer state)")
+        print(f"    parameters              : {param_bytes / gb:7.2f} GiB")
+        print(f"    gradients               : {grad_bytes / gb:7.2f} GiB")
+        print(f"    fp32 master weights     : {master_bytes / gb:7.2f} GiB")
+        print(f"    fp32 grad landing bufs  : {landing_bytes / gb:7.2f} GiB")
+        print(f"    Muon momentum           : {mom_bytes / gb:7.2f} GiB")
+        print(f"    Adam exp_avg + exp_avg_sq: {mv_bytes / gb:6.2f} GiB")
+        print(f"    static total            : {static_bytes / gb:7.2f} GiB", flush=True)
+
+    # ---- data ----
+    # Sharded by rank: each rank reads a disjoint, interleaved slice of the
+    # same shard list (see DataLoaderLite), so DDP's cross-rank gradient
+    # average is over world_size DIFFERENT windows, not world_size copies of
+    # the same one.
+    train_loader = DataLoaderLite(cfg.B, cfg.T, "train", args.data_root, device,
+                                  master=master_process, rank=ddp_rank,
+                                  world_size=ddp_world_size)
+    # rank=0, world_size=1 UNCONDITIONALLY, on every rank: val_loss_steps
+    # windows at fixed token offsets 0, BT, 2BT, ... Sharding this by
+    # ddp_rank/ddp_world_size like the train loader would make the reported
+    # val_loss a function of how many GPUs happen to be running (world_size=2
+    # averages 2x as many, differently-offset windows as world_size=1) —
+    # the two numbers would not be comparable across a GPU-count change, which
+    # is what produced the apparent val_loss jump on the 2-GPU -> 1-GPU
+    # resume. Every rank now redundantly evaluates the identical fixed
+    # windows, so the metric means the same thing regardless of world_size.
+    #
+    # Batch size is hardcoded to 4 here (NOT cfg.B) for the same reason: the
+    # batch-size ablation varies cfg.B/global_batch across runs, and if val
+    # windows scaled with it (B*T tokens per val_loss_steps iteration) the
+    # val_loss numbers would stop being comparable across ablation arms. A
+    # fixed val batch size means every run evaluates the same fixed set of
+    # tokens (4*T*val_loss_steps), regardless of what B that run trains with.
+    val_loader = DataLoaderLite(4, cfg.T, "val", args.data_root, device,
+                                master=master_process, rank=0, world_size=1)
+
+    # ---- checkpointing: manager + auto-resume ----
+    # Every rank independently loads the same checkpoint file from the same
+    # (assumed shared) filesystem — a read, so no race — which keeps model,
+    # optimizer state, and start_step identical across ranks without an
+    # explicit broadcast. Only rank 0 ever WRITES one (below).
+    start_step, ckpt = 0, None
+    last_val = float("nan")
+    if cfg.checkpointing or args.resume_from:
+        ckpt = CheckpointManager(cfg.ckpt_dir, cfg.ckpt_prefix, cfg.ckpt_keep)
+        if args.resume_from:
+            if master_process:
+                print(f"[resume] loading explicit checkpoint: {args.resume_from}")
+            start_step, resume_loss, last_val = ckpt.load(
+                args.resume_from, raw_model, opt, cfg, master_process)
+        else:
+            start_step, resume_loss, last_val = ckpt.load_latest(
+                raw_model, opt, cfg, master_process)
+        if start_step > 0:
+            if master_process:
+                print(f"[resume] at step {start_step} (loss {resume_loss:.6f}, "
+                      f"last val {last_val:.6f})")
+            # Skip the batches already consumed by steps [0, start_step).
+            # global_batch // (B*T*world_size) makes tokens-per-step (and so
+            # this target offset) independent of world_size, so a GPU-count
+            # change alone lands on the correct byte offset as long as B, T,
+            # and global_batch are unchanged (checked above).
+            train_loader.skip_batches(start_step * grad_accum)
+
+    # ---- CSV log ----
+    # Appended (not truncated) when resuming so a restart does not erase the
+    # history of the run it is continuing. Rank 0 only — every rank would
+    # otherwise open and write the same path.
+    log_file = None
+    if args.csv and master_process:
+        resuming = start_step > 0 and os.path.exists(args.csv)
+        log_file = open(args.csv, "a" if resuming else "w")
+        if not resuming:
+            log_file.write("step,train_loss,val_loss,tok_per_sec,grad_norm,dt_ms,lr,"
+                           "elapsed_min,muon_lr\n")
+        log_file.flush()
+
+    amp = torch.autocast(device_type="cuda", dtype=torch.bfloat16)
+    probes = [L for L in cfg.probe_layers if L < cfg.n_layers]
+    steady_dts = []
+    t_train_start = time.perf_counter()
+    ema_loss = None   # EMA-smoothed train loss, checked against target_losses
+    # Milestones not yet reached, highest loss first (the order the EMA passes
+    # them in as it decreases). Each entry becomes (step, elapsed_hours) once crossed.
+    pending_targets = sorted(cfg.target_losses, reverse=True)
+    milestones = {}   # target loss -> (step, elapsed_hours)
+
+    # ---- Ctrl-C: finish the in-flight step, save a checkpoint, exit clean ----
+    # Registered here (not earlier) so a Ctrl-C during setup/data-loading still
+    # gets Python's default immediate-abort behaviour — only the training loop
+    # below gets the graceful path. A second Ctrl-C re-arms the default handler
+    # and re-raises, for when the checkpoint save itself needs interrupting.
+    interrupted = False
+
+    def _handle_sigint(signum, frame):
+        nonlocal interrupted
+        if interrupted:
+            signal.signal(signal.SIGINT, signal.SIG_DFL)
+            raise KeyboardInterrupt
+        interrupted = True
+        if master_process:
+            print("\n[ctrl-c] finishing this step, then checkpointing and exiting "
+                  "(press again to abort immediately)...", flush=True)
+
+    # A terminal Ctrl-C is delivered to every process in the foreground
+    # process group, so under torchrun every rank sets `interrupted` at
+    # roughly the same time and the ranks stay in lockstep for the collective
+    # ops (DDP's grad all-reduce, the checkpoint barrier below) still ahead.
+    signal.signal(signal.SIGINT, _handle_sigint)
+
+    if master_process:
+        print("\nStarting training" +
+              (f" (resumed at step {start_step})" if start_step else "") + "...\n", flush=True)
+
+    for step in range(start_step, cfg.max_steps):
+        # ---------- periodic validation ----------
+        if cfg.val_loss_steps > 0 and (step % cfg.val_freq == 0 or step == cfg.max_steps - 1):
+            model.eval()
+            val_loader.reset()
+            acc = 0.0
+            with torch.no_grad(), amp:
+                for _ in range(cfg.val_loss_steps):
+                    x, y, _, _ = val_loader.next_batch()
+                    acc += model(x, y)[1].item() / cfg.val_loss_steps
+            model.train()
+            # val_loader is rank=0/world_size=1 on every rank (see its
+            # construction above), so every rank just computed the identical
+            # fixed windows redundantly — no cross-rank reduction needed, and
+            # val_loss no longer depends on how many ranks are running.
+            last_val = acc
+            if master_process:
+                print(f"  [val] step {step} | val_loss {last_val:.6f}", flush=True)
+
+        # ---------- training step ----------
+        t0 = time.perf_counter()
+        model.zero_grad(set_to_none=True)
+        # Fresh zeros rather than an in-place scale of last step's accumulator:
+        # multiplying by zero does NOT clear a NaN, so a single non-finite
+        # micro-batch would poison the accumulator for the rest of the run.
+        loss_accum = torch.zeros((), device=device)
+        micro_loss, micro_where = [], []
+
+        for micro in range(grad_accum):
+            x, y, shard, offset = train_loader.next_batch()
+            # DDP allreduces gradients on every .backward(); for grad_accum > 1
+            # only the LAST micro-batch of the step should pay for that — the
+            # others accumulate purely locally under no_sync().
+            sync_ctx = (nullcontext() if (not ddp or micro == grad_accum - 1)
+                       else model.no_sync())
+            with sync_ctx:
+                with amp:
+                    _, loss = model(x, y)
+                loss_accum += loss.detach()
+                # Per-micro-batch loss (--diag). The reported train loss is the mean
+                # over grad_accum micro-steps, so a single pathological micro-batch
+                # is diluted while its gradient dominates the accumulated norm
+                # almost undiluted — which is how a modest loss bump can sit next to
+                # a large norm spike. `micro_where` names the shard and token offset
+                # so the responsible bytes can be decoded offline.
+                #
+                # Costs a device sync per micro-step, so it is off by default.
+                if diag:
+                    micro_loss.append(loss.item())
+                    micro_where.append((shard, offset))
+                (loss / grad_accum).backward()
+
+        if ddp:
+            # Average each rank's accumulated (summed) loss BEFORE dividing by
+            # grad_accum, so loss_val is the mean over world_size*grad_accum
+            # micro-batches — the true loss behind this optimizer step, not
+            # just rank 0's slice. Also doubles as a sync point: a NaN on any
+            # rank poisons the average for every rank, so the check below
+            # raises everywhere instead of only where it originated (which
+            # would otherwise hang the survivors on the next collective).
+            dist.all_reduce(loss_accum, op=dist.ReduceOp.AVG)
+        loss_val = loss_accum.item() / grad_accum
+        if math.isnan(loss_val) or math.isinf(loss_val):
+            if log_file:
+                log_file.close()
+            # Every rank reaches this identically (the all_reduce above is a
+            # sync point and NaN propagates through AVG), so every rank exits
+            # here together rather than the survivors hanging on the next
+            # collective.
+            if ddp:
+                dist.destroy_process_group()
+            raise SystemExit(f"ERROR: NaN/Inf loss at step {step}")
+
+        # EMA-smoothed train loss vs. the pending milestones. loss_val is
+        # already all-reduced across ranks above, so ema_loss (and the
+        # milestones it crosses) comes out identical on every rank without an
+        # extra collective.
+        beta = cfg.target_loss_ema_beta
+        ema_loss = loss_val if ema_loss is None else beta * ema_loss + (1 - beta) * loss_val
+        elapsed_hr = (time.perf_counter() - t_train_start) / 3600.0
+        target_reached = False
+        while pending_targets and ema_loss <= pending_targets[0]:
+            t = pending_targets.pop(0)
+            milestones[t] = (step, elapsed_hr)
+            target_reached = True
+            if master_process:
+                print(f"\n[target] loss <= {t:.6f} reached at step {step} "
+                      f"({elapsed_hr:.3f} hrs)\n", flush=True)
+
+        # Depth probe (--diag), read BEFORE the update, so these are the
+        # pre-update gradients. Compare the profile on a spiking step against a
+        # quiet one: a blow-up that begins mid-stack and grows toward the loss
+        # points at the forward, while a flat multiplicative lift across all
+        # depths points at the optimizer/update.
+        depth = []
+        if diag:
+            for L in probes:
+                a = raw_model.blocks[L].attn
+                gn = a.wQ.weight.grad
+                depth.append((L,
+                              gn.float().norm().item() if gn is not None else -1.0,
+                              a.q_gamma.abs().max().item(),
+                              a.wQ.weight.abs().max().float().item()))
+
+        grad_norm = clip_grads_(params, cfg.grad_clip).item()
+        lr = get_lr(step, cfg)                                   # AdamW group
+        muon_lr = get_lr(step, cfg, cfg.muon_lr, cfg.muon_min_lr)
+        opt.step(lr, muon_lr)
+
+        torch.cuda.synchronize()   # make the per-step timing reflect completed GPU work
+        dt = time.perf_counter() - t0
+        tok_per_sec = tokens_per_step / dt
+        elapsed_min = (time.perf_counter() - t_train_start) / 60.0
+        if step - start_step >= args.warmup_measure:
+            steady_dts.append(dt)
+
+        # ---------- per-step report (rank 0 only — every rank's numbers agree
+        # on loss/grad_norm post-allreduce, so printing from every rank would
+        # only duplicate lines, not add information) ----------
+        if master_process:
+            vs = "     -  " if last_val != last_val else f"{last_val:.6f}"
+            print(f"step {step:6d} | train {loss_val:.6f} | val {vs} "
+                  f"| tok/s {tok_per_sec:10.2f} | norm {grad_norm:.4f} "
+                  f"| dt {dt * 1e3:8.2f} ms | lr {lr:.4e} | mlr {muon_lr:.4e} "
+                  f"| mem {torch.cuda.max_memory_allocated() / gb:5.2f}/"
+                  f"{torch.cuda.max_memory_reserved() / gb:5.2f} GiB", flush=True)
+
+            if diag and micro_loss:
+                # Flag micro-batches that stand out against this step's own median.
+                med = sorted(micro_loss)[len(micro_loss) // 2]
+                marks = " ".join(("*" if m > med + 0.5 else "") + f"{m:.3f}"
+                                 for m in micro_loss)
+                print(f"  [micro] med {med:.4f} | {marks}")
+                for m, (ml, (shard, offset)) in enumerate(zip(micro_loss, micro_where)):
+                    if ml > med + 0.5:
+                        print(f"  [micro] OUTLIER m={m} loss {ml:.4f} (median {med:.4f}) "
+                              f"from shard {shard} token offset {offset}")
+            if diag and depth:
+                print("  [depth] ||g(wQ)|| by layer:"
+                      + "".join(f" L{L} {g:.3e}" for L, g, _, _ in depth)
+                      + " | max|q_gamma|:"
+                      + "".join(f" L{L} {gm:.4f}" for L, _, gm, _ in depth)
+                      + " | max|wQ|:"
+                      + "".join(f" L{L} {w:.4f}" for L, _, _, w in depth), flush=True)
+
+            if log_file:
+                log_file.write(f"{step},{loss_val:.6f},{last_val:.6f},{tok_per_sec:.6f},"
+                               f"{grad_norm:.6f},{dt * 1e3:.6f},{lr:.6f},{elapsed_min:.6f},"
+                               f"{muon_lr:.6f}\n")
+                log_file.flush()
+
+        if ddp:
+            # `interrupted` is set by this process's own SIGINT handler, so in
+            # principle one rank could see it before another (a signal isn't
+            # guaranteed simultaneous delivery, even though in practice a
+            # terminal Ctrl-C hits every rank in the same process group at
+            # once). Left asymmetric, it would desync the checkpoint barrier
+            # and the `break` below against ranks still issuing DDP's regular
+            # per-step collectives — a hang, not just a wrong answer. This
+            # runs unconditionally every step (never gated on a per-rank
+            # value), so it can never itself mismatch across ranks.
+            flag = torch.tensor(1.0 if interrupted else 0.0, device=device)
+            dist.all_reduce(flag, op=dist.ReduceOp.MAX)
+            interrupted = flag.item() > 0
+
+        # ---------- checkpoint on target-loss crossing (also end-of-run / Ctrl-C) ----------
+        # Labeled step+1 (the next iteration to run) so load_latest resumes
+        # exactly where this save left off — no duplicated or skipped iteration.
+        # Captures weights + Adam m/v AFTER this step's update. Rank 0 only —
+        # every rank's raw_model/opt state is identical post-allreduce, so a
+        # second writer would only race the first on the same path. The
+        # barrier holds every rank at this point until the save (or its
+        # failure) is done, so a slow write can't let other ranks race ahead
+        # into the next step's collectives before the checkpoint is durable.
+        #
+        # target_reached replaces interval-based saving: this ablation's whole
+        # point is comparing batch sizes at matched loss levels, so the
+        # checkpoint that's actually useful is the one at each milestone, not
+        # whatever happens to land on a round step number. target_reached is
+        # derived from loss_val, which is already all-reduced above, so it's
+        # identical on every rank without its own collective.
+        if cfg.checkpointing and (target_reached
+                                  or step == cfg.max_steps - 1
+                                  or interrupted):
+            if master_process:
+                try:
+                    ckpt.save(step + 1, raw_model, opt, loss_val, last_val, cfg)
+                except Exception as e:                   # noqa: BLE001 — advisory
+                    print(f"[WARN] checkpoint save at step {step + 1} failed: {e}")
+            if ddp:
+                dist.barrier()
+
+        if interrupted:
+            if not cfg.checkpointing and master_process:
+                print(f"[ctrl-c] checkpointing is disabled (--no-checkpointing); "
+                      f"stopping at step {step} without saving.", flush=True)
+            break
+
+        # Reclaim the extra allocator high-water mark from the first couple of
+        # steps' one-time warmup allocations (autograd graph caches, etc.).
+        if step in (start_step, start_step + 1):
+            torch.cuda.synchronize()
+            torch.cuda.empty_cache()
+            torch.cuda.reset_peak_memory_stats()
+
+    torch.cuda.synchronize()
+    if log_file:
+        log_file.close()
+
+    # ---- target-loss milestone summary ----
+    if master_process:
+        print("\n" + "=" * 52)
+        print("  TARGET LOSS MILESTONES")
+        print("=" * 52)
+        print(f"  {'target loss':>12}   {'step':>10}   {'hours':>10}")
+        for t in sorted(cfg.target_losses, reverse=True):
+            if t in milestones:
+                step_hit, hrs = milestones[t]
+                print(f"  {t:>12.6f}   {step_hit:>10}   {hrs:>10.3f}")
+            else:
+                print(f"  {t:>12.6f}   {'not reached':>10}   {'-':>10}")
+        print("=" * 52)
+
+    # ---- steady-state summary ----
+    # dt/throughput are this rank's own timing; under DDP all ranks run in
+    # lockstep (grad sync + the checkpoint barrier), so rank 0's numbers are
+    # representative and `tokens_per_step` already counts every rank's tokens.
+    if steady_dts and master_process:
+        xs = sorted(steady_dts)
+        med = xs[len(xs) // 2]
+        mean = sum(xs) / len(xs)
+        alloc = torch.cuda.max_memory_allocated() / gb
+        reserved = torch.cuda.max_memory_reserved() / gb
+        # Measured now that Adam's m/v exist, rather than the startup estimate.
+        static = (param_bytes + grad_bytes + opt.state_bytes()) / gb
+        mode_str = f"DDP x{ddp_world_size}" if ddp else "single GPU"
+        print("\n" + "=" * 78)
+        print(f"  STEADY-STATE SUMMARY — {mode_str}  optimizer=muon+adamw  "
+              f"precision={args.precision}  "
+              f"rms+rope={'fused' if cfg.fused_rms_rope else 'eager'}")
+        print(f"  torch.optim.Muon  lr={cfg.muon_lr:g} "
+              f"adjust_lr_fn={cfg.muon_adjust_lr} "
+              f"momentum={cfg.muon_momentum:g} ns_steps={cfg.muon_ns_steps}")
+        print(f"  ({len(xs)} steps, first {args.warmup_measure} excluded)")
+        print("=" * 78)
+        print(f"  step time          : {med * 1e3:10.2f} ms  (mean {mean * 1e3:.2f})")
+        print(f"  throughput         : {tokens_per_step / med:10.1f} tok/s")
+        print(f"  tokens / step      : {tokens_per_step:10d}")
+        print(f"  static (p+g+opt)   : {static:10.2f} GiB  (per GPU)")
+        print(f"  activations (peak) : {max(alloc - static, 0.0):10.2f} GiB")
+        print(f"  peak mem allocated : {alloc:10.2f} GiB")
+        print(f"  peak mem reserved  : {reserved:10.2f} GiB")
+        print("=" * 78)
+
+    if ddp:
+        dist.destroy_process_group()
+    if master_process:
+        print("\n=== Training complete ===")
+
+
+if __name__ == "__main__":
+    main()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

390M_1M/blumodel_muon_step00002933.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b70bdcfe0366e8c4cb26c96a8bc26b3856bb6d856244f170fef135274824943b
+size 4205800344

390M_1M/blumodel_muon_step00003460.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cad905610df859006341a8df8e7c626ef7d4727882eed19ba3aa619896a00b46
+size 4205800344

blu_muon_log.csv

@@ -0,0 +1,529 @@
+step,train_loss,val_loss,tok_per_sec,grad_norm,dt_ms,lr,elapsed_min,muon_lr
+2932,2.518530,3.004766,79131.586627,0.182534,13251.042279,0.001000,0.220851,0.001000
+2933,2.498816,3.004766,103169.320475,0.171943,10163.641625,0.001000,0.425363,0.001000
+2934,2.486779,3.004766,103165.559478,0.151263,10164.012150,0.001000,0.597048,0.001000
+2935,2.515379,3.004766,103143.387685,0.164831,10166.197015,0.001000,0.766492,0.001000
+2936,2.597795,3.004766,103131.441530,0.167779,10167.374609,0.001000,0.935955,0.001000
+2937,2.539912,3.004766,103115.323141,0.159947,10168.963914,0.001000,1.105445,0.001000
+2938,2.749660,3.004766,103110.638247,0.164717,10169.425947,0.001000,1.274942,0.001000
+2939,2.554215,3.004766,103107.466657,0.165757,10169.738759,0.001000,1.444445,0.001000
+2940,2.646553,3.004766,103099.801876,0.175390,10170.494811,0.001000,1.613960,0.001000
+2941,2.654152,3.004766,103092.712322,0.193434,10171.194223,0.001000,1.783487,0.001000
+2942,2.545764,3.004766,103092.886981,0.174831,10171.176991,0.001000,1.953013,0.001000
+2943,2.681435,3.004766,103096.048157,0.167737,10170.865118,0.001000,2.122535,0.001000
+2944,2.612506,3.004766,103094.902917,0.155954,10170.978102,0.001000,2.292058,0.001000
+2945,2.676021,3.004766,103071.255720,0.157018,10173.311586,0.001000,2.461621,0.001000
+2946,2.623533,3.004766,103086.831569,0.179788,10171.774455,0.001000,2.631158,0.001000
+2947,2.526068,3.004766,103091.404279,0.174980,10171.323277,0.001000,2.800688,0.001000
+2948,2.634506,3.004766,103088.093950,0.167745,10171.649895,0.001000,2.970223,0.001000
+2949,2.649794,3.004766,103087.608777,0.166923,10171.697767,0.001000,3.139760,0.001000
+2950,2.591897,3.004766,103092.608157,0.161171,10171.204500,0.001000,3.309287,0.001000
+2951,2.581865,3.004766,103098.137475,0.162181,10170.659002,0.001000,3.478805,0.001000
+2952,2.667576,3.004766,103099.416180,0.176816,10170.532859,0.001000,3.648324,0.001000
+2953,2.607063,3.004766,103099.938637,0.181681,10170.481320,0.001000,3.817839,0.001000
+2954,2.627990,3.004766,103098.202290,0.175428,10170.652608,0.001000,3.987362,0.001000
+2955,2.598402,3.004766,103093.746746,0.172794,10171.092167,0.001000,4.156887,0.001000
+2956,2.569453,3.004766,103095.689897,0.177931,10170.900462,0.001000,4.326411,0.001000
+2957,2.670726,3.004766,103093.034457,0.188311,10171.162441,0.001000,4.495937,0.001000
+2958,2.686060,3.004766,103094.096458,0.175831,10171.057665,0.001000,4.665462,0.001000
+2959,2.553031,3.004766,103099.224975,0.160648,10170.551721,0.001000,4.834978,0.001000
+2960,2.584077,3.004766,103093.851218,0.163573,10171.081860,0.001000,5.004503,0.001000
+2961,2.622176,3.004766,103094.087184,0.159366,10171.058580,0.001000,5.174027,0.001000
+2962,2.604548,3.004766,103095.027866,0.159252,10170.965775,0.001000,5.343550,0.001000
+2963,2.651807,3.004766,103090.524880,0.168856,10171.410042,0.001000,5.513080,0.001000
+2964,2.559568,3.004766,103090.025535,0.180596,10171.459310,0.001000,5.682610,0.001000
+2965,2.644744,3.004766,103084.981977,0.182255,10171.956961,0.001000,5.852150,0.001000
+2966,2.576633,3.004766,103091.584630,0.168877,10171.305483,0.001000,6.021679,0.001000
+2967,2.488044,3.004766,103089.545878,0.188059,10171.506636,0.001000,6.191210,0.001000
+2968,2.527740,3.004766,103090.223061,0.181073,10171.439821,0.001000,6.360741,0.001000
+2969,2.630423,3.004766,103090.975245,0.176638,10171.365607,0.001000,6.530270,0.001000
+2970,2.633029,3.004766,103093.926174,0.165660,10171.074465,0.001000,6.699794,0.001000
+2971,2.632665,3.004766,103094.046326,0.171361,10171.062611,0.001000,6.869318,0.001000
+2972,2.618899,3.004766,103089.034148,0.158224,10171.557127,0.001000,7.038851,0.001000
+2973,2.524681,3.004766,103090.882638,0.152433,10171.374744,0.001000,7.208380,0.001000
+2974,2.626079,3.004766,103092.437056,0.161751,10171.221381,0.001000,7.377907,0.001000
+2975,2.574749,3.004766,103097.234830,0.176625,10170.748049,0.001000,7.547427,0.001000
+2976,2.610802,3.004766,103096.760072,0.172951,10170.794885,0.001000,7.716947,0.001000
+2977,2.529051,3.004766,103096.721391,0.181333,10170.798701,0.001000,7.886467,0.001000
+2978,2.641349,3.004766,103095.843726,0.187481,10170.885286,0.001000,8.055987,0.001000
+2979,2.596373,3.004766,103097.055310,0.191994,10170.765759,0.001000,8.225508,0.001000
+2980,2.619727,3.004766,103093.815620,0.198549,10171.085372,0.001000,8.395032,0.001000
+2981,2.594059,3.004766,103095.052416,0.193291,10170.963353,0.001000,8.564558,0.001000
+2982,2.622165,3.004766,103091.713960,0.169352,10171.292723,0.001000,8.734086,0.001000
+2983,2.576459,3.004766,103093.648387,0.159810,10171.101871,0.001000,8.903614,0.001000
+2984,2.686820,3.004766,103093.420481,0.173824,10171.124356,0.001000,9.073143,0.001000
+2985,2.612260,3.004766,103096.788089,0.175690,10170.792121,0.001000,9.242664,0.001000
+2986,2.614008,3.004766,103091.112428,0.183698,10171.352072,0.001000,9.412194,0.001000
+2987,2.567060,3.004766,103095.838790,0.169787,10170.885773,0.001000,9.581724,0.001000
+2988,2.581716,3.004766,103099.029351,0.162090,10170.571019,0.001000,9.751240,0.001000
+2989,2.499635,3.004766,103097.412373,0.158330,10170.730534,0.001000,9.920767,0.001000
+2990,2.577906,3.004766,103087.844238,0.160210,10171.674534,0.001000,10.090302,0.001000
+2991,2.625936,3.004766,103092.751243,0.159002,10171.190383,0.001000,10.259838,0.001000
+2992,2.637316,3.004766,103094.546165,0.166021,10171.013298,0.001000,10.429361,0.001000
+2993,2.485345,3.004766,103093.136941,0.172688,10171.152330,0.001000,10.598897,0.001000
+2994,2.624996,3.004766,103092.920956,0.178217,10171.173639,0.001000,10.768423,0.001000
+2995,2.577661,3.004766,103098.058266,0.173527,10170.666816,0.001000,10.937941,0.001000
+2996,2.500997,3.004766,103092.694554,0.168445,10171.195976,0.001000,11.107468,0.001000
+2997,2.688860,3.004766,103092.540298,0.169264,10171.211195,0.001000,11.276994,0.001000
+2998,2.579581,3.004766,103103.096362,0.173561,10170.169830,0.001000,11.446504,0.001000
+2999,2.643594,3.004766,103096.698888,0.161783,10170.800921,0.001000,11.616024,0.001000
+3000,2.698153,2.984432,103100.392673,0.176746,10170.436531,0.001000,11.812062,0.001000
+3001,2.603330,2.984432,103098.827675,0.176735,10170.590914,0.001000,11.981581,0.001000
+3002,2.681886,2.984432,103093.456717,0.183900,10171.120781,0.001000,12.151106,0.001000
+3003,2.702585,2.984432,103084.968874,0.176800,10171.958254,0.001000,12.320645,0.001000
+3004,2.607144,2.984432,103099.876040,0.185552,10170.487495,0.001000,12.490162,0.001000
+3005,2.603316,2.984432,103095.588148,0.184665,10170.910500,0.001000,12.659686,0.001000
+3006,2.634832,2.984432,103089.134059,0.182142,10171.547269,0.001000,12.829219,0.001000
+3007,2.569043,2.984432,103096.602246,0.197791,10170.810455,0.001000,12.998740,0.001000
+3008,2.624683,2.984432,103089.684354,0.162698,10171.492973,0.001000,13.168274,0.001000
+3009,2.590243,2.984432,103093.258986,0.152414,10171.140289,0.001000,13.337803,0.001000
+3010,2.652208,2.984432,103091.176109,0.157451,10171.345789,0.001000,13.507334,0.001000
+3011,2.646666,2.984432,103084.937579,0.174894,10171.961342,0.001000,13.676874,0.001000
+3012,2.603009,2.984432,103088.663460,0.163219,10171.593702,0.001000,13.846409,0.001000
+3013,2.556043,2.984432,103093.351152,0.167298,10171.131196,0.001000,14.015934,0.001000
+3014,2.553932,2.984432,103096.706440,0.176029,10170.800176,0.001000,14.185455,0.001000
+3015,2.616219,2.984432,103094.832927,0.174999,10170.985007,0.001000,14.354979,0.001000
+3016,2.623441,2.984432,103092.369511,0.170596,10171.228045,0.001000,14.524507,0.001000
+3017,2.523988,2.984432,103088.549118,0.170719,10171.604984,0.001000,14.694041,0.001000
+3018,2.636304,2.984432,103074.255856,0.165403,10173.015476,0.001000,14.863599,0.001000
+3019,2.742237,2.984432,103089.793338,0.171052,10171.482220,0.001000,15.033130,0.001000
+3020,2.601624,2.984432,103087.242366,0.152314,10171.733921,0.001000,15.202665,0.001000
+3021,2.644132,2.984432,103087.080941,0.162079,10171.749849,0.001000,15.372201,0.001000
+3022,2.666094,2.984432,103087.301370,0.160901,10171.728099,0.001000,15.541736,0.001000
+3023,2.554512,2.984432,103088.328876,0.171974,10171.626715,0.001000,15.711270,0.001000
+3024,2.636309,2.984432,103088.219450,0.179528,10171.637512,0.001000,15.880804,0.001000
+3025,2.729039,2.984432,103081.419188,0.172464,10172.308533,0.001000,16.050349,0.001000
+3026,2.605507,2.984432,103081.621606,0.151041,10172.288558,0.001000,16.219894,0.001000
+3027,2.764655,2.984432,103085.060862,0.180815,10171.949177,0.001000,16.389433,0.001000
+3028,2.607679,2.984432,103077.167586,0.172908,10172.728108,0.001000,16.558986,0.001000
+3029,2.610501,2.984432,103083.975921,0.171522,10172.056235,0.001000,16.728527,0.001000
+3030,2.568144,2.984432,103088.283370,0.169987,10171.631205,0.001000,16.898062,0.001000
+3031,2.622425,2.984432,103081.150326,0.164871,10172.335065,0.001000,17.067607,0.001000
+3032,2.566345,2.984432,103084.579853,0.160595,10171.996641,0.001000,17.237148,0.001000
+3033,2.632028,2.984432,103088.667950,0.162795,10171.593259,0.001000,17.406681,0.001000
+3034,2.548529,2.984432,103077.774133,0.166652,10172.668248,0.001000,17.576232,0.001000
+3035,2.665282,2.984432,103087.816975,0.162836,10171.677224,0.001000,17.745767,0.001000
+3036,2.607318,2.984432,103088.647660,0.168045,10171.595261,0.001000,17.915300,0.001000
+3037,2.627769,2.984432,103092.384036,0.178026,10171.226612,0.001000,18.084827,0.001000
+3038,2.615799,2.984432,103084.801386,0.189367,10171.974781,0.001000,18.254367,0.001000
+3039,2.586826,2.984432,103095.111743,0.178166,10170.957500,0.001000,18.423889,0.001000
+3040,2.646325,2.984432,103095.755813,0.174693,10170.893959,0.001000,18.593411,0.001000
+3041,2.607988,2.984432,103076.211940,0.161738,10172.822422,0.001000,18.762964,0.001000
+3042,2.614072,2.984432,103069.221166,0.163707,10173.512404,0.001000,18.932532,0.001000
+3043,2.649881,2.984432,103085.757323,0.164804,10171.880454,0.001000,19.102072,0.001000
+3044,2.712023,2.984432,103089.579344,0.158996,10171.503334,0.001000,19.271603,0.001000
+3045,2.560740,2.984432,103085.664614,0.156987,10171.889602,0.001000,19.441142,0.001000
+3046,2.748270,2.984432,103085.991215,0.152658,10171.857375,0.001000,19.610679,0.001000
+3047,2.642848,2.984432,103081.837137,0.165336,10172.267289,0.001000,19.780224,0.001000
+3048,2.535511,2.984432,103087.018238,0.185537,10171.756036,0.001000,19.949760,0.001000
+3049,2.602525,2.984432,103080.049142,0.184409,10172.443734,0.001000,20.119307,0.001000
+3050,2.606381,2.984432,103085.420731,0.179806,10171.913667,0.001000,20.288846,0.001000
+3051,2.592681,2.984432,103086.773944,0.179042,10171.780141,0.001000,20.458382,0.001000
+3052,2.647141,2.984432,103088.944037,0.169544,10171.566018,0.001000,20.627915,0.001000
+3053,2.669973,2.984432,103087.575859,0.171833,10171.701015,0.001000,20.797450,0.001000
+3054,2.607740,2.984432,103089.206251,0.175586,10171.540146,0.001000,20.966983,0.001000
+3055,2.734015,2.984432,103088.066971,0.184090,10171.652557,0.001000,21.136517,0.001000
+3056,2.626555,2.984432,103083.009589,0.165505,10172.151591,0.001000,21.306060,0.001000
+3057,2.625455,2.984432,103086.951451,0.168108,10171.762626,0.001000,21.475596,0.001000
+3058,2.663499,2.984432,103084.451625,0.181594,10172.009294,0.001000,21.645137,0.001000
+3059,2.551778,2.984432,103088.193089,0.174325,10171.640113,0.001000,21.814671,0.001000
+3060,2.603032,2.984432,103090.823741,0.168156,10171.380555,0.001000,21.984201,0.001000
+3061,2.654394,2.984432,103091.554680,0.183867,10171.308438,0.001000,22.153729,0.001000
+3062,2.603266,2.984432,103093.514248,0.190890,10171.115105,0.001000,22.323254,0.001000
+3063,2.584371,2.984432,103095.308184,0.194130,10170.938120,0.001000,22.492776,0.001000
+3064,2.600125,2.984432,103090.398645,0.187447,10171.422497,0.001000,22.662307,0.001000
+3065,2.598893,2.984432,103089.574033,0.191131,10171.503858,0.001000,22.831839,0.001000
+3066,2.579171,2.984432,103094.816375,0.193260,10170.986640,0.001000,23.001363,0.001000
+3067,2.698627,2.984432,103085.900056,0.177171,10171.866370,0.001000,23.170901,0.001000
+3068,2.558210,2.984432,103085.509852,0.170588,10171.904873,0.001000,23.340439,0.001000
+3069,2.574222,2.984432,103084.902738,0.166820,10171.964780,0.001000,23.509979,0.001000
+3070,2.670976,2.984432,103088.143803,0.169623,10171.644976,0.001000,23.679513,0.001000
+3071,2.568007,2.984432,103086.318457,0.168190,10171.825085,0.001000,23.849050,0.001000
+3072,2.530254,2.984432,103089.302129,0.172831,10171.530686,0.001000,24.018581,0.001000
+3073,2.611373,2.984432,103086.649643,0.184703,10171.792406,0.001000,24.188118,0.001000
+3074,2.640197,2.984432,103086.355194,0.183050,10171.821460,0.001000,24.357655,0.001000
+3075,2.691546,2.984432,103086.956640,0.198880,10171.762114,0.001000,24.527193,0.001000
+3076,2.527573,2.984432,103088.359433,0.199787,10171.623700,0.001000,24.696726,0.001000
+3077,2.578455,2.984432,103081.231718,0.185396,10172.327033,0.001000,24.866271,0.001000
+3078,2.541167,2.984432,103091.068410,0.172467,10171.356415,0.001000,25.035801,0.001000
+3079,2.585632,2.984432,103087.547411,0.172038,10171.703822,0.001000,25.205336,0.001000
+3080,2.620516,2.984432,103082.901553,0.177857,10172.162252,0.001000,25.374879,0.001000
+3081,2.612582,2.984432,103087.577673,0.159248,10171.700836,0.001000,25.544415,0.001000
+3082,2.559495,2.984432,103085.949248,0.160227,10171.861516,0.001000,25.713954,0.001000
+3083,2.587861,2.984432,103085.832247,0.174669,10171.873061,0.001000,25.883493,0.001000
+3084,2.633636,2.984432,103086.484967,0.172384,10171.808655,0.001000,26.053030,0.001000
+3085,2.622861,2.984432,103091.100144,0.152158,10171.353284,0.001000,26.222559,0.001000
+3086,2.623696,2.984432,103077.753077,0.157878,10172.670326,0.001000,26.392110,0.001000
+3087,2.561499,2.984432,103082.789746,0.170322,10172.173285,0.001000,26.561654,0.001000
+3088,2.583263,2.984432,103088.171258,0.176649,10171.642267,0.001000,26.731187,0.001000
+3089,2.656955,2.984432,103098.285807,0.172580,10170.644369,0.001000,26.900705,0.001000
+3090,2.537261,2.984432,103094.473784,0.166796,10171.020439,0.001000,27.070229,0.001000
+3091,2.704237,2.984432,103090.879526,0.173015,10171.375051,0.001000,27.239758,0.001000
+3092,2.595150,2.984432,103093.307486,0.167397,10171.135504,0.001000,27.409283,0.001000
+3093,2.587607,2.984432,103092.991816,0.159635,10171.166648,0.001000,27.578809,0.001000
+3094,2.603078,2.984432,103088.895268,0.163308,10171.570830,0.001000,27.748341,0.001000
+3095,2.568929,2.984432,103088.279296,0.160694,10171.631607,0.001000,27.917874,0.001000
+3096,2.601391,2.984432,103085.595801,0.156325,10171.896392,0.001000,28.087412,0.001000
+3097,2.637080,2.984432,103094.332851,0.167607,10171.034343,0.001000,28.256936,0.001000
+3098,2.561841,2.984432,103102.872175,0.160904,10170.191944,0.001000,28.426445,0.001000
+3099,2.753889,2.984432,103095.755692,0.169177,10170.893971,0.001000,28.595966,0.001000
+3100,2.597961,2.984432,103090.206571,0.154905,10171.441448,0.001000,28.765496,0.001000
+3101,2.641227,2.984432,103098.344976,0.158605,10170.638532,0.001000,28.935013,0.001000
+3102,2.566831,2.984432,103098.042676,0.184382,10170.668354,0.001000,29.104531,0.001000
+3103,2.548577,2.984432,103097.051194,0.188476,10170.766165,0.001000,29.274050,0.001000
+3104,2.639575,2.984432,103095.706155,0.163960,10170.898858,0.001000,29.443571,0.001000
+3105,2.627871,2.984432,103096.217597,0.184160,10170.848402,0.001000,29.613091,0.001000
+3106,2.617517,2.984432,103099.317972,0.159821,10170.542547,0.001000,29.782606,0.001000
+3107,2.606517,2.984432,103099.471386,0.165939,10170.527413,0.001000,29.952122,0.001000
+3108,2.577323,2.984432,103089.148400,0.157723,10171.545854,0.001000,30.121653,0.001000
+3109,2.601144,2.984432,103087.276013,0.157020,10171.730601,0.001000,30.291188,0.001000
+3110,2.631731,2.984432,103096.270530,0.167376,10170.843180,0.001000,30.460709,0.001000
+3111,2.524446,2.984432,103095.802248,0.171719,10170.889378,0.001000,30.630231,0.001000
+3112,2.612978,2.984432,103087.852437,0.167311,10171.673725,0.001000,30.799765,0.001000
+3113,2.573527,2.984432,103090.654896,0.174831,10171.397214,0.001000,30.969297,0.001000
+3114,2.571442,2.984432,103092.337493,0.172743,10171.231204,0.001000,31.138824,0.001000
+3115,2.593837,2.984432,103093.177747,0.171706,10171.148304,0.001000,31.308352,0.001000
+3116,2.664948,2.984432,103093.214835,0.183435,10171.144645,0.001000,31.477879,0.001000
+3117,2.625830,2.984432,103086.278416,0.183334,10171.829036,0.001000,31.647418,0.001000
+3118,2.563191,2.984432,103095.905122,0.167796,10170.879229,0.001000,31.816941,0.001000
+3119,2.618722,2.984432,103098.471880,0.171723,10170.626013,0.001000,31.986458,0.001000
+3120,2.566702,2.984432,103101.290477,0.160701,10170.347967,0.001000,32.155972,0.001000
+3121,2.647903,2.984432,103094.688173,0.175365,10170.999288,0.001000,32.325496,0.001000
+3122,2.607458,2.984432,103096.606311,0.179975,10170.810054,0.001000,32.495018,0.001000
+3123,2.644614,2.984432,103091.433682,0.162044,10171.320376,0.001000,32.664546,0.001000
+3124,2.541438,2.984432,103095.332085,0.163028,10170.935762,0.001000,32.834076,0.001000
+3125,2.605136,2.984432,103088.930071,0.176615,10171.567396,0.001000,33.003609,0.001000
+3126,2.585379,2.984432,103096.289738,0.166346,10170.841285,0.001000,33.173129,0.001000
+3127,2.603942,2.984432,103094.248144,0.156537,10171.042700,0.001000,33.342652,0.001000
+3128,2.519915,2.984432,103095.180386,0.159535,10170.950728,0.001000,33.512175,0.001000
+3129,2.654501,2.984432,103089.008435,0.162632,10171.559664,0.001000,33.681708,0.001000
+3130,2.586948,2.984432,103089.547348,0.156246,10171.506491,0.001000,33.851240,0.001000
+3131,2.518124,2.984432,103094.973759,0.162953,10170.971113,0.001000,34.020763,0.001000
+3132,2.571071,2.984432,103090.030106,0.165506,10171.458859,0.001000,34.190294,0.001000
+3133,2.586661,2.984432,103086.232496,0.161674,10171.833567,0.001000,34.359833,0.001000
+3134,2.607214,2.984432,103087.205648,0.156817,10171.737544,0.001000,34.529369,0.001000
+3135,2.530256,2.984432,103093.286455,0.155453,10171.137579,0.001000,34.698896,0.001000
+3136,2.550986,2.984432,103096.515123,0.160149,10170.819050,0.001000,34.868416,0.001000
+3137,2.603904,2.984432,103097.208251,0.175141,10170.750671,0.001000,35.037935,0.001000
+3138,2.556391,2.984432,103098.334130,0.171023,10170.639602,0.001000,35.207452,0.001000
+3139,2.589101,2.984432,103090.445825,0.164176,10171.417842,0.001000,35.376983,0.001000
+3140,2.601631,2.984432,103092.689212,0.178431,10171.196503,0.001000,35.546516,0.001000
+3141,2.586335,2.984432,103094.387647,0.185074,10171.028937,0.001000,35.716039,0.001000
+3142,2.606042,2.984432,103091.088954,0.178916,10171.354388,0.001000,35.885571,0.001000
+3143,2.598014,2.984432,103099.697829,0.180011,10170.505075,0.001000,36.055087,0.001000
+3144,2.610785,2.984432,103091.366088,0.192013,10171.327045,0.001000,36.224622,0.001000
+3145,2.519475,2.984432,103095.827781,0.168475,10170.886859,0.001000,36.394143,0.001000
+3146,2.533395,2.984432,103089.584635,0.162975,10171.502812,0.001000,36.563683,0.001000
+3147,2.566407,2.984432,103095.976695,0.171495,10170.872168,0.001000,36.733204,0.001000
+3148,2.610042,2.984432,103095.839499,0.156708,10170.885703,0.001000,36.902731,0.001000
+3149,2.644021,2.984432,103091.636484,0.158000,10171.300367,0.001000,37.072259,0.001000
+3150,2.533216,2.984432,103094.079258,0.163857,10171.059362,0.001000,37.241791,0.001000
+3151,2.639207,2.984432,103097.514298,0.174491,10170.720479,0.001000,37.411311,0.001000
+3152,2.525908,2.984432,103092.053512,0.172016,10171.259222,0.001000,37.580845,0.001000
+3153,2.536929,2.984432,103092.916710,0.174304,10171.174058,0.001000,37.750371,0.001000
+3154,2.550884,2.984432,103097.626430,0.187723,10170.709417,0.001000,37.919896,0.001000
+3155,2.696508,2.984432,103094.319482,0.181622,10171.035662,0.001000,38.089422,0.001000
+3156,2.591186,2.984432,103083.906696,0.175910,10172.063066,0.001000,38.258969,0.001000
+3157,2.546389,2.984432,103088.787289,0.164579,10171.581484,0.001000,38.428503,0.001000
+3158,2.664961,2.984432,103091.214887,0.162974,10171.341963,0.001000,38.598033,0.001000
+3159,2.573251,2.984432,103090.027471,0.168941,10171.459119,0.001000,38.767564,0.001000
+3160,2.575205,2.984432,103088.095440,0.176494,10171.649748,0.001000,38.937099,0.001000
+3161,2.654025,2.984432,103097.968829,0.173073,10170.675639,0.001000,39.106617,0.001000
+3162,2.620358,2.984432,103099.822597,0.164692,10170.492767,0.001000,39.276132,0.001000
+3163,2.512127,2.984432,103091.155402,0.165089,10171.347832,0.001000,39.445661,0.001000
+3164,2.609811,2.984432,103100.597304,0.154963,10170.416345,0.001000,39.615175,0.001000
+3165,2.626517,2.984432,103093.187508,0.157933,10171.147341,0.001000,39.784701,0.001000
+3166,2.558947,2.984432,103090.035772,0.165843,10171.458300,0.001000,39.954232,0.001000
+3167,2.671727,2.984432,103088.043336,0.177124,10171.654889,0.001000,40.123767,0.001000
+3168,2.543355,2.984432,103089.668290,0.162969,10171.494558,0.001000,40.293299,0.001000
+3169,2.580249,2.984432,103086.985483,0.146277,10171.759268,0.001000,40.462835,0.001000
+3170,2.566869,2.984432,103085.869288,0.161386,10171.869406,0.001000,40.632373,0.001000
+3171,2.685902,2.984432,103090.297180,0.196569,10171.432508,0.001000,40.801904,0.001000
+3172,2.592744,2.984432,103088.947078,0.183943,10171.565718,0.001000,40.971437,0.001000
+3173,2.737288,2.984432,103092.205861,0.181967,10171.244191,0.001000,41.140965,0.001000
+3174,2.608429,2.984432,103091.485332,0.165078,10171.315280,0.001000,41.310493,0.001000
+3175,2.598910,2.984432,103086.518968,0.149408,10171.805300,0.001000,41.480031,0.001000
+3176,2.633080,2.984432,103090.613361,0.156918,10171.401312,0.001000,41.649561,0.001000
+3177,2.594174,2.984432,103087.412001,0.161623,10171.717183,0.001000,41.819097,0.001000
+3178,2.521140,2.984432,103081.994786,0.169700,10172.251732,0.001000,41.988641,0.001000
+3179,2.597688,2.984432,103089.238582,0.183148,10171.536956,0.001000,42.158174,0.001000
+3180,2.679385,2.984432,103095.036634,0.166889,10170.964910,0.001000,42.327697,0.001000
+3181,2.654499,2.984432,103086.509746,0.155840,10171.806210,0.001000,42.497233,0.001000
+3182,2.592400,2.984432,103087.883480,0.158456,10171.670662,0.001000,42.666768,0.001000
+3183,2.595298,2.984432,103089.766024,0.170117,10171.484915,0.001000,42.836299,0.001000
+3184,2.526134,2.984432,103086.070051,0.163458,10171.849596,0.001000,43.005837,0.001000
+3185,2.551760,2.984432,103082.456011,0.174805,10172.206218,0.001000,43.175381,0.001000
+3186,2.644013,2.984432,103082.871972,0.187951,10172.165171,0.001000,43.344924,0.001000
+3187,2.659145,2.984432,103097.234637,0.184003,10170.748068,0.001000,43.514444,0.001000
+3188,2.611820,2.984432,103090.675218,0.186403,10171.395209,0.001000,43.683974,0.001000
+3189,2.595366,2.984432,103095.286665,0.191293,10170.940243,0.001000,43.853496,0.001000
+3190,2.520628,2.984432,103090.848573,0.180429,10171.378105,0.001000,44.023026,0.001000
+3191,2.503470,2.984432,103084.852401,0.166106,10171.969747,0.001000,44.192566,0.001000
+3192,2.521365,2.984432,103088.735327,0.168118,10171.586611,0.001000,44.362099,0.001000
+3193,2.628481,2.984432,103080.670224,0.190791,10172.382443,0.001000,44.531645,0.001000
+3194,2.548547,2.984432,103087.848636,0.179966,10171.674100,0.001000,44.701180,0.001000
+3195,2.664605,2.984432,103092.988806,0.181798,10171.166945,0.001000,44.870706,0.001000
+3196,2.604728,2.984432,103090.225017,0.164223,10171.439628,0.001000,45.040236,0.001000
+3197,2.606874,2.984432,103091.720740,0.170106,10171.292054,0.001000,45.209765,0.001000
+3198,2.628014,2.984432,103085.610365,0.173624,10171.894955,0.001000,45.379303,0.001000
+3199,2.570484,2.984432,103092.256934,0.171421,10171.239152,0.001000,45.548831,0.001000
+3200,2.590347,2.984432,103085.870433,0.161671,10171.869293,0.001000,45.718369,0.001000
+3201,2.567670,2.984432,103089.420973,0.159363,10171.518960,0.001000,45.887901,0.001000
+3202,2.559825,2.984432,103097.300941,0.174437,10170.741527,0.001000,46.057420,0.001000
+3203,2.551521,2.984432,103089.116718,0.165715,10171.548980,0.001000,46.226953,0.001000
+3204,2.495384,2.984432,103084.186943,0.162745,10172.035412,0.001000,46.396494,0.001000
+3205,2.603484,2.984432,103086.819772,0.162552,10171.775619,0.001000,46.566030,0.001000
+3206,2.552297,2.984432,103088.631211,0.169195,10171.596884,0.001000,46.735563,0.001000
+3207,2.604045,2.984432,103092.594149,0.179763,10171.205882,0.001000,46.905090,0.001000
+3208,2.628468,2.984432,103092.628874,0.181469,10171.202456,0.001000,47.074617,0.001000
+3209,2.580190,2.984432,103096.128589,0.165718,10170.857183,0.001000,47.244138,0.001000
+3210,2.655056,2.984432,103090.998759,0.157271,10171.363287,0.001000,47.413667,0.001000
+3211,2.591047,2.984432,103093.460964,0.151146,10171.120362,0.001000,47.583192,0.001000
+3212,2.756294,2.984432,103091.479687,0.176161,10171.315837,0.001000,47.752721,0.001000
+3213,2.602210,2.984432,103092.874383,0.160915,10171.178234,0.001000,47.922247,0.001000
+3214,2.786813,2.984432,103089.840983,0.168389,10171.477519,0.001000,48.091779,0.001000
+3215,2.534050,2.984432,103084.765683,0.162248,10171.978304,0.001000,48.261318,0.001000
+3216,2.600623,2.984432,103093.781361,0.167744,10171.088752,0.001000,48.430843,0.001000
+3217,2.595063,2.984432,103094.369595,0.171927,10171.030718,0.001000,48.600367,0.001000
+3218,2.532391,2.984432,103092.575378,0.168399,10171.207734,0.001000,48.769893,0.001000
+3219,2.705562,2.984432,103090.777483,0.174912,10171.385119,0.001000,48.939423,0.001000
+3220,2.564148,2.984432,103096.313822,0.177727,10170.838909,0.001000,49.108944,0.001000
+3221,2.638940,2.984432,103097.217547,0.163667,10170.749754,0.001000,49.278463,0.001000
+3222,2.727661,2.984432,103095.423636,0.160952,10170.926730,0.001000,49.447985,0.001000
+3223,2.573673,2.984432,103090.675866,0.169289,10171.395145,0.001000,49.617514,0.001000
+3224,2.607946,2.984432,103095.427964,0.184303,10170.926303,0.001000,49.787036,0.001000
+3225,2.561997,2.984432,103092.471031,0.187815,10171.218029,0.001000,49.956564,0.001000
+3226,2.580966,2.984432,103090.487542,0.183206,10171.413726,0.001000,50.126094,0.001000
+3227,2.541920,2.984432,103101.757521,0.228216,10170.301896,0.001000,50.295605,0.001000
+3228,2.610976,2.984432,103092.889637,0.167759,10171.176729,0.001000,50.465132,0.001000
+3229,2.695706,2.984432,103093.020014,0.172140,10171.163866,0.001000,50.634659,0.001000
+3230,2.651542,2.984432,103096.430949,0.161633,10170.827354,0.001000,50.804179,0.001000
+3231,2.673234,2.984432,103098.127744,0.161898,10170.659962,0.001000,50.973696,0.001000
+3232,2.598337,2.984432,103097.015990,0.153534,10170.769638,0.001000,51.143216,0.001000
+3233,2.592498,2.984432,103099.470332,0.144578,10170.527517,0.001000,51.312731,0.001000
+3234,2.595832,2.984432,103092.847553,0.160791,10171.180881,0.001000,51.482261,0.001000
+3235,2.551354,2.984432,103091.157065,0.166091,10171.347668,0.001000,51.651791,0.001000
+3236,2.611280,2.984432,103100.150514,0.171584,10170.460419,0.001000,51.821307,0.001000
+3237,2.563223,2.984432,103098.919496,0.184419,10170.581856,0.001000,51.990823,0.001000
+3238,2.635842,2.984432,103101.133378,0.192776,10170.363464,0.001000,52.160336,0.001000
+3239,2.573549,2.984432,103091.684607,0.182937,10171.295619,0.001000,52.329864,0.001000
+3240,2.585678,2.984432,103096.315059,0.180083,10170.838787,0.001000,52.499386,0.001000
+3241,2.579909,2.984432,103093.482939,0.169588,10171.118194,0.001000,52.668911,0.001000
+3242,2.553871,2.984432,103100.253255,0.171468,10170.450284,0.001000,52.838427,0.001000
+3243,2.602914,2.984432,103098.302523,0.173439,10170.642720,0.001000,53.007954,0.001000
+3244,2.513550,2.984432,103099.176641,0.163924,10170.556489,0.001000,53.177470,0.001000
+3245,2.582921,2.984432,103101.141041,0.166617,10170.362708,0.001000,53.346983,0.001000
+3246,2.665513,2.984432,103099.252000,0.171187,10170.549055,0.001000,53.516499,0.001000
+3247,2.565963,2.984432,103096.673435,0.175181,10170.803432,0.001000,53.686019,0.001000
+3248,2.601732,2.984432,103090.232426,0.172596,10171.438897,0.001000,53.855550,0.001000
+3249,2.583050,2.984432,103095.784225,0.169520,10170.891156,0.001000,54.025071,0.001000
+3250,2.570794,2.969588,103096.586940,0.171784,10170.811965,0.001000,54.221099,0.001000
+3251,2.747301,2.969588,103086.321963,0.178640,10171.824739,0.001000,54.390637,0.001000
+3252,2.485192,2.969588,103090.121121,0.173197,10171.449879,0.001000,54.560168,0.001000
+3253,2.622178,2.969588,103090.442328,0.173137,10171.418187,0.001000,54.729700,0.001000
+3254,2.590855,2.969588,103093.894539,0.171568,10171.077586,0.001000,54.899224,0.001000
+3255,2.672008,2.969588,103090.406074,0.168380,10171.421764,0.001000,55.068755,0.001000
+3256,2.586234,2.969588,103099.766052,0.170648,10170.498345,0.001000,55.238270,0.001000
+3257,2.662860,2.969588,103092.415568,0.161537,10171.223501,0.001000,55.407797,0.001000
+3258,2.563779,2.969588,103095.086393,0.170526,10170.960001,0.001000,55.577320,0.001000
+3259,2.570365,2.969588,103092.833505,0.172421,10171.182267,0.001000,55.746846,0.001000
+3260,2.625733,2.969588,103086.835015,0.175147,10171.774115,0.001000,55.916397,0.001000
+3261,2.538575,2.969588,103092.205526,0.160732,10171.244224,0.001000,56.085924,0.001000
+3262,2.631830,2.969588,103092.132661,0.149515,10171.251413,0.001000,56.255451,0.001000
+3263,2.564277,2.969588,103095.926581,0.156544,10170.877112,0.001000,56.424972,0.001000
+3264,2.526287,2.969588,103093.905962,0.172236,10171.076459,0.001000,56.594497,0.001000
+3265,2.556432,2.969588,103100.694774,0.180205,10170.406730,0.001000,56.764010,0.001000
+3266,2.529977,2.969588,103095.594757,0.169233,10170.909848,0.001000,56.933531,0.001000
+3267,2.590171,2.969588,103093.477871,0.164432,10171.118694,0.001000,57.103056,0.001000
+3268,2.533563,2.969588,103094.866610,0.164482,10170.981684,0.001000,57.272579,0.001000
+3269,2.641843,2.969588,103096.887205,0.156791,10170.782343,0.001000,57.442099,0.001000
+3270,2.558879,2.969588,103089.365037,0.171622,10171.524479,0.001000,57.611631,0.001000
+3271,2.624928,2.969588,103095.031465,0.188049,10170.965420,0.001000,57.781153,0.001000
+3272,2.576524,2.969588,103092.013112,0.174689,10171.263208,0.001000,57.950681,0.001000
+3273,2.502206,2.969588,103083.859198,0.159856,10172.067753,0.001000,58.120221,0.001000
+3274,2.615892,2.969588,103094.794916,0.177806,10170.988757,0.001000,58.289745,0.001000
+3275,2.537559,2.969588,103098.202209,0.174900,10170.652616,0.001000,58.459262,0.001000
+3276,2.608619,2.969588,103054.855639,0.174769,10174.930560,0.001000,58.628850,0.001000
+3277,2.561466,2.969588,103088.479521,0.169460,10171.611851,0.001000,58.798384,0.001000
+3278,2.572806,2.969588,103089.586794,0.169421,10171.502599,0.001000,58.967915,0.001000
+3279,2.611094,2.969588,103092.178302,0.172682,10171.246910,0.001000,59.137442,0.001000
+3280,2.662110,2.969588,103090.587466,0.173734,10171.403867,0.001000,59.306974,0.001000
+3281,2.553421,2.969588,103091.398106,0.163034,10171.323886,0.001000,59.476502,0.001000
+3282,2.636236,2.969588,103088.509176,0.168987,10171.608925,0.001000,59.646038,0.001000
+3283,2.615084,2.969588,103054.894603,0.167801,10174.926713,0.001000,59.815626,0.001000
+3284,2.587587,2.969588,103088.883491,0.164784,10171.571992,0.001000,59.985160,0.001000
+3285,2.527129,2.969588,103090.836441,0.178241,10171.379302,0.001000,60.154689,0.001000
+3286,2.544123,2.969588,103091.198387,0.184112,10171.343591,0.001000,60.324218,0.001000
+3287,2.524717,2.969588,103089.321142,0.177995,10171.528810,0.001000,60.493750,0.001000
+3288,2.568714,2.969588,103090.931116,0.186290,10171.369961,0.001000,60.663280,0.001000
+3289,2.566547,2.969588,103086.078250,0.183865,10171.848787,0.001000,60.832817,0.001000
+3290,2.558284,2.969588,103084.195526,0.182632,10172.034565,0.001000,61.002359,0.001000
+3291,2.586743,2.969588,103087.658447,0.177487,10171.692866,0.001000,61.171894,0.001000
+3292,2.644320,2.969588,103089.643175,0.170283,10171.497036,0.001000,61.341426,0.001000
+3293,2.550558,2.969588,103091.176960,0.170479,10171.345705,0.001000,61.510956,0.001000
+3294,2.622975,2.969588,103093.250715,0.178989,10171.141105,0.001000,61.680481,0.001000
+3295,2.624788,2.969588,103097.090322,0.169964,10170.762305,0.001000,61.850001,0.001000
+3296,2.561498,2.969588,103091.997259,0.154094,10171.264772,0.001000,62.019528,0.001000
+3297,2.634943,2.969588,103100.487010,0.152361,10170.427225,0.001000,62.189042,0.001000
+3298,2.608403,2.969588,103088.828072,0.170924,10171.577460,0.001000,62.358575,0.001000
+3299,2.642709,2.969588,103090.625939,0.183776,10171.400071,0.001000,62.528105,0.001000
+3300,2.623695,2.969588,103090.046019,0.168069,10171.457289,0.001000,62.697636,0.001000
+3301,2.499890,2.969588,103092.905246,0.157218,10171.175189,0.001000,62.867163,0.001000
+3302,2.586325,2.969588,103089.925714,0.174570,10171.469159,0.001000,63.036694,0.001000
+3303,2.654237,2.969588,103086.389895,0.161266,10171.818036,0.001000,63.206231,0.001000
+3304,2.648149,2.969588,103087.859835,0.155522,10171.672995,0.001000,63.375767,0.001000
+3305,2.661752,2.969588,103094.344771,0.150082,10171.033167,0.001000,63.545291,0.001000
+3306,2.634814,2.969588,103092.169423,0.155289,10171.247786,0.001000,63.714820,0.001000
+3307,2.653605,2.969588,103086.602852,0.170200,10171.797023,0.001000,63.884356,0.001000
+3308,2.573256,2.969588,103091.202704,0.161669,10171.343165,0.001000,64.053886,0.001000
+3309,2.570794,2.969588,103092.005652,0.150231,10171.263944,0.001000,64.223414,0.001000
+3310,2.535677,2.969588,103088.418185,0.153815,10171.617903,0.001000,64.392949,0.001000
+3311,2.588498,2.969588,103101.383651,0.165910,10170.338776,0.001000,64.562461,0.001000
+3312,2.569311,2.969588,103093.644931,0.163084,10171.102212,0.001000,64.731986,0.001000
+3313,2.531060,2.969588,103095.056471,0.158968,10170.962953,0.001000,64.901509,0.001000
+3314,2.582192,2.969588,103088.345548,0.178172,10171.625070,0.001000,65.071043,0.001000
+3315,2.563098,2.969588,103089.480375,0.179293,10171.513099,0.001000,65.240575,0.001000
+3316,2.682483,2.969588,103094.198366,0.184959,10171.047611,0.001000,65.410099,0.001000
+3317,2.579535,2.969588,103093.719856,0.191275,10171.094820,0.001000,65.579624,0.001000
+3318,2.581599,2.969588,103091.850932,0.183853,10171.279209,0.001000,65.749152,0.001000
+3319,2.609619,2.969588,103086.990703,0.169010,10171.758753,0.001000,65.918688,0.001000
+3320,2.617500,2.969588,103089.060661,0.168682,10171.554511,0.001000,66.088220,0.001000
+3321,2.569565,2.969588,103088.830657,0.170909,10171.577205,0.001000,66.257754,0.001000
+3322,2.554571,2.969588,103089.224676,0.175051,10171.538328,0.001000,66.427286,0.001000
+3323,2.554905,2.969588,103086.226628,0.195399,10171.834146,0.001000,66.596823,0.001000
+3324,2.664313,2.969588,103080.709876,0.197254,10172.378530,0.001000,66.766370,0.001000
+3325,2.558815,2.969588,103089.284828,0.180805,10171.532393,0.001000,66.935902,0.001000
+3326,2.595249,2.969588,103086.828437,0.168312,10171.774764,0.001000,67.105439,0.001000
+3327,2.521863,2.969588,103086.825326,0.173621,10171.775071,0.001000,67.274975,0.001000
+3328,2.581258,2.969588,103092.569276,0.167356,10171.208336,0.001000,67.444502,0.001000
+3329,2.539475,2.969588,103089.913420,0.160485,10171.470372,0.001000,67.614033,0.001000
+3330,2.607611,2.969588,103085.645703,0.172698,10171.891468,0.001000,67.783572,0.001000
+3331,2.533270,2.969588,103090.938575,0.154147,10171.369225,0.001000,67.953102,0.001000
+3332,2.643965,2.969588,103089.487611,0.153099,10171.512385,0.001000,68.122634,0.001000
+3333,2.606556,2.969588,103092.028862,0.170864,10171.261654,0.001000,68.292162,0.001000
+3334,2.513801,2.969588,103084.967648,0.176943,10171.958375,0.001000,68.461701,0.001000
+3335,2.559145,2.969588,103091.561805,0.175172,10171.307735,0.001000,68.631230,0.001000
+3336,2.563275,2.969588,103089.946288,0.175566,10171.467129,0.001000,68.800761,0.001000
+3337,2.547048,2.969588,103088.657369,0.164936,10171.594303,0.001000,68.970295,0.001000
+3338,2.563286,2.969588,103092.803006,0.171397,10171.185276,0.001000,69.139822,0.001000
+3339,2.582546,2.969588,103087.153627,0.167515,10171.742677,0.001000,69.309358,0.001000
+3340,2.573735,2.969588,103091.432192,0.178089,10171.320523,0.001000,69.478887,0.001000
+3341,2.525950,2.969588,103097.618199,0.179708,10170.710229,0.001000,69.648406,0.001000
+3342,2.564679,2.969588,103093.614037,0.176151,10171.105260,0.001000,69.817932,0.001000
+3343,2.585093,2.969588,103088.911179,0.174970,10171.569260,0.001000,69.987464,0.001000
+3344,2.579835,2.969588,103089.234244,0.175421,10171.537384,0.001000,70.156998,0.001000
+3345,2.593774,2.969588,103095.541126,0.179729,10170.915139,0.001000,70.326519,0.001000
+3346,2.585589,2.969588,103098.332994,0.176751,10170.639714,0.001000,70.496039,0.001000
+3347,2.579988,2.969588,103097.536913,0.176583,10170.718248,0.001000,70.665557,0.001000
+3348,2.549825,2.969588,103093.409088,0.169459,10171.125480,0.001000,70.835084,0.001000
+3349,2.619757,2.969588,103094.543692,0.162360,10171.013542,0.001000,71.004608,0.001000
+3350,2.546021,2.969588,103086.090563,0.156246,10171.847572,0.001000,71.174147,0.001000
+3351,2.601354,2.969588,103086.565405,0.161483,10171.800718,0.001000,71.343684,0.001000
+3352,2.607131,2.969588,103088.678389,0.160175,10171.592229,0.001000,71.513218,0.001000
+3353,2.494443,2.969588,103084.016478,0.157094,10172.052233,0.001000,71.682759,0.001000
+3354,2.566845,2.969588,103083.607338,0.166779,10172.092606,0.001000,71.852302,0.001000
+3355,2.651251,2.969588,103079.400547,0.164441,10172.507741,0.001000,72.021852,0.001000
+3356,2.541584,2.969588,103085.659486,0.155977,10171.890108,0.001000,72.191392,0.001000
+3357,2.679792,2.969588,103086.779133,0.177694,10171.779629,0.001000,72.360928,0.001000
+3358,2.599742,2.969588,103090.089316,0.185092,10171.453017,0.001000,72.530461,0.001000
+3359,2.629931,2.969588,103088.496781,0.182437,10171.610148,0.001000,72.699994,0.001000
+3360,2.550574,2.969588,103090.607189,0.178941,10171.401921,0.001000,72.869525,0.001000
+3361,2.548457,2.969588,103091.894373,0.185549,10171.274923,0.001000,73.039053,0.001000
+3362,2.547307,2.969588,103086.574384,0.187774,10171.799832,0.001000,73.208592,0.001000
+3363,2.493783,2.969588,103088.471545,0.173066,10171.612638,0.001000,73.378125,0.001000
+3364,2.612309,2.969588,103086.771957,0.162683,10171.780337,0.001000,73.547662,0.001000
+3365,2.521334,2.969588,103086.470089,0.168883,10171.810123,0.001000,73.717199,0.001000
+3366,2.586009,2.969588,103089.610297,0.160480,10171.500280,0.001000,73.886731,0.001000
+3367,2.475593,2.969588,103093.890951,0.162679,10171.077940,0.001000,74.056256,0.001000
+3368,2.607940,2.969588,103092.663032,0.167234,10171.199086,0.001000,74.225785,0.001000
+3369,2.555333,2.969588,103088.535811,0.179411,10171.606297,0.001000,74.395319,0.001000
+3370,2.636269,2.969588,103085.042986,0.178148,10171.950941,0.001000,74.564858,0.001000
+3371,2.497338,2.969588,103087.918435,0.171019,10171.667213,0.001000,74.734394,0.001000
+3372,2.485677,2.969588,103088.479653,0.164674,10171.611838,0.001000,74.903929,0.001000
+3373,2.549701,2.969588,103092.247002,0.164587,10171.240132,0.001000,75.073458,0.001000
+3374,2.604171,2.969588,103093.158135,0.169097,10171.150239,0.001000,75.242987,0.001000
+3375,2.664688,2.969588,103087.738188,0.173842,10171.684998,0.001000,75.412524,0.001000
+3376,2.595036,2.969588,103093.082258,0.169515,10171.157725,0.001000,75.582056,0.001000
+3377,2.591540,2.969588,103090.962829,0.155678,10171.366832,0.001000,75.751587,0.001000
+3378,2.637111,2.969588,103087.138385,0.160262,10171.744181,0.001000,75.921124,0.001000
+3379,2.590006,2.969588,103086.057808,0.166681,10171.850804,0.001000,76.090663,0.001000
+3380,2.641059,2.969588,103086.169956,0.169148,10171.839738,0.001000,76.260201,0.001000
+3381,2.642158,2.969588,103088.429282,0.163423,10171.616808,0.001000,76.429737,0.001000
+3382,2.552609,2.969588,103092.321762,0.168376,10171.232756,0.001000,76.599264,0.001000
+3383,2.635085,2.969588,103084.983366,0.173904,10171.956824,0.001000,76.768805,0.001000
+3384,2.552792,2.969588,103092.639314,0.157524,10171.201426,0.001000,76.938335,0.001000
+3385,2.544689,2.969588,103094.352302,0.148725,10171.032424,0.001000,77.107859,0.001000
+3386,2.573870,2.969588,103089.349612,0.157312,10171.526001,0.001000,77.277392,0.001000
+3387,2.526491,2.969588,103090.352458,0.152180,10171.427054,0.001000,77.446926,0.001000
+3388,2.537632,2.969588,103092.161659,0.160490,10171.248552,0.001000,77.616455,0.001000
+3389,2.574361,2.969588,103093.895766,0.174615,10171.077465,0.001000,77.785979,0.001000
+3390,2.620977,2.969588,103092.892728,0.176791,10171.176424,0.001000,77.955506,0.001000
+3391,2.613632,2.969588,103090.713549,0.168801,10171.391427,0.001000,78.125037,0.001000
+3392,2.587037,2.969588,103091.039423,0.156722,10171.359275,0.001000,78.294566,0.001000
+3393,2.619840,2.969588,103101.244575,0.170522,10170.352495,0.001000,78.464078,0.001000
+3394,2.529140,2.969588,103101.692783,0.180393,10170.308282,0.001000,78.633590,0.001000
+3395,2.538913,2.969588,103099.673125,0.168844,10170.507512,0.001000,78.803115,0.001000
+3396,2.604615,2.969588,103091.035561,0.175669,10171.359656,0.001000,78.972645,0.001000
+3397,2.601986,2.969588,103097.688122,0.168209,10170.703331,0.001000,79.142163,0.001000
+3398,2.589046,2.969588,103094.049873,0.163592,10171.062261,0.001000,79.311687,0.001000
+3399,2.595712,2.969588,103098.174272,0.180690,10170.655372,0.001000,79.481204,0.001000
+3400,2.617138,2.969588,103102.198911,0.213787,10170.258356,0.001000,79.650714,0.001000
+3401,2.613343,2.969588,103090.953373,0.163428,10171.367765,0.001000,79.820243,0.001000
+3402,2.594885,2.969588,103096.643644,0.153874,10170.806371,0.001000,79.989763,0.001000
+3403,2.644967,2.969588,103100.498313,0.152267,10170.426110,0.001000,80.159276,0.001000
+3404,2.503607,2.969588,103095.231523,0.154717,10170.945683,0.001000,80.328798,0.001000
+3405,2.516342,2.969588,103097.926711,0.160856,10170.679794,0.001000,80.498315,0.001000
+3406,2.548811,2.969588,103092.516236,0.181122,10171.213569,0.001000,80.667842,0.001000
+3407,2.585251,2.969588,103093.795541,0.183136,10171.087353,0.001000,80.837367,0.001000
+3408,2.564416,2.969588,103093.166963,0.166278,10171.149368,0.001000,81.006892,0.001000
+3409,2.550410,2.969588,103092.603879,0.173442,10171.204922,0.001000,81.176418,0.001000
+3410,2.618539,2.969588,103091.754360,0.182610,10171.288737,0.001000,81.345945,0.001000
+3411,2.610064,2.969588,103085.105291,0.176324,10171.944793,0.001000,81.515484,0.001000
+3412,2.613327,2.969588,103089.925420,0.182923,10171.469188,0.001000,81.685015,0.001000
+3413,2.627610,2.969588,103092.879673,0.182910,10171.177712,0.001000,81.854541,0.001000
+3414,2.587284,2.969588,103095.752509,0.159679,10170.894285,0.001000,82.024062,0.001000
+3415,2.643609,2.969588,103106.265970,0.164270,10169.857187,0.001000,82.193574,0.001000
+3416,2.606405,2.969588,103099.861696,0.160696,10170.488910,0.001000,82.363090,0.001000
+3417,2.629639,2.969588,103102.772450,0.151496,10170.201781,0.001000,82.532603,0.001000
+3418,2.602618,2.969588,103102.225543,0.154639,10170.255729,0.001000,82.702115,0.001000
+3419,2.566788,2.969588,103108.258096,0.166768,10169.660698,0.001000,82.871623,0.001000
+3420,2.468071,2.969588,103099.059001,0.161984,10170.568094,0.001000,83.041142,0.001000
+3421,2.596793,2.969588,103104.860592,0.169539,10169.995808,0.001000,83.210653,0.001000
+3422,2.549049,2.969588,103098.162118,0.179597,10170.656571,0.001000,83.380172,0.001000
+3423,2.605649,2.969588,103102.450882,0.199491,10170.233501,0.001000,83.549684,0.001000
+3424,2.648352,2.969588,103102.529368,0.197889,10170.225759,0.001000,83.719196,0.001000
+3425,2.590079,2.969588,103102.598153,0.168969,10170.218974,0.001000,83.888710,0.001000
+3426,2.576338,2.969588,103098.924048,0.163892,10170.581407,0.001000,84.058227,0.001000
+3427,2.632526,2.969588,103104.071151,0.160281,10170.073677,0.001000,84.227744,0.001000
+3428,2.570278,2.969588,103096.404088,0.156802,10170.830004,0.001000,84.397266,0.001000
+3429,2.582601,2.969588,103094.643969,0.156494,10171.003649,0.001000,84.566790,0.001000
+3430,2.592120,2.969588,103088.474667,0.163640,10171.612330,0.001000,84.736323,0.001000
+3431,2.585729,2.969588,103092.605957,0.158074,10171.204717,0.001000,84.905851,0.001000
+3432,2.564941,2.969588,103096.834424,0.152876,10170.787550,0.001000,85.075371,0.001000
+3433,2.657804,2.969588,103097.021423,0.155401,10170.769102,0.001000,85.244891,0.001000
+3434,2.510366,2.969588,103092.474375,0.153440,10171.217699,0.001000,85.414417,0.001000
+3435,2.554708,2.969588,103095.586516,0.162303,10170.910661,0.001000,85.583945,0.001000
+3436,2.555516,2.969588,103097.393651,0.170502,10170.732381,0.001000,85.753465,0.001000
+3437,2.561697,2.969588,103100.055468,0.164062,10170.469795,0.001000,85.922981,0.001000
+3438,2.547385,2.969588,103104.922059,0.170837,10169.989745,0.001000,86.092488,0.001000
+3439,2.559653,2.969588,103097.053252,0.169536,10170.765962,0.001000,86.262011,0.001000
+3440,2.529861,2.969588,103100.284681,0.161489,10170.447184,0.001000,86.431527,0.001000
+3441,2.540334,2.969588,103103.831854,0.155053,10170.097281,0.001000,86.601042,0.001000
+3442,2.608873,2.969588,103103.970704,0.163941,10170.083585,0.001000,86.770551,0.001000
+3443,2.530012,2.969588,103098.046548,0.157614,10170.667972,0.001000,86.940071,0.001000
+3444,2.552046,2.969588,103093.559546,0.161414,10171.110636,0.001000,87.109596,0.001000
+3445,2.621527,2.969588,103098.561378,0.173669,10170.617184,0.001000,87.279115,0.001000
+3446,2.572051,2.969588,103101.408274,0.176576,10170.336347,0.001000,87.448627,0.001000
+3447,2.645121,2.969588,103102.825521,0.171836,10170.196546,0.001000,87.618136,0.001000
+3448,2.562349,2.969588,103097.550496,0.174137,10170.716908,0.001000,87.787656,0.001000
+3449,2.632760,2.969588,103097.892327,0.176089,10170.683186,0.001000,87.957177,0.001000
+3450,2.525856,2.969588,103100.935759,0.164371,10170.382958,0.001000,88.126690,0.001000
+3451,2.597590,2.969588,103101.097380,0.162322,10170.367015,0.001000,88.296204,0.001000
+3452,2.635059,2.969588,103102.146905,0.183219,10170.263486,0.001000,88.465715,0.001000
+3453,2.517982,2.969588,103094.903232,0.178856,10170.978071,0.001000,88.635237,0.001000
+3454,2.577394,2.969588,103100.764914,0.182817,10170.399811,0.001000,88.804750,0.001000
+3455,2.528866,2.969588,103098.143030,0.167373,10170.658454,0.001000,88.974267,0.001000
+3456,2.536610,2.969588,103101.890871,0.157916,10170.288742,0.001000,89.143778,0.001000
+3457,2.636271,2.969588,103096.322195,0.172564,10170.838083,0.001000,89.313298,0.001000
+3458,2.526578,2.969588,103095.694184,0.174519,10170.900039,0.001000,89.482819,0.001000
+3459,2.717874,2.969588,103099.311667,0.188175,10170.543169,0.001000,89.652334,0.001000