Clean model history

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,82 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+tags: [bet, byte-level, recurrent, looped]
+---
+# Cortex — SparkBET-9M
+
+Target repository: **appvoid/cortex**. The current training model has **9,353,876 parameters**: width 324, FFN 864, one prelude block, six physical recurrent body blocks, one coda block, 6 query heads / 2 KV heads, head dimension 54, rank-16 phase LoRA, two loop-level Hyper-Connection lanes, Deep-Delta body residuals, QK normalization, and continuous phase/stride conditioning. The byte vocabulary has 259 IDs (0–255 plus PAD/BOS/EOS), context is 1,024 IDs, and the maximum recurrent budget is L8.
+
+## Run
+
+Import the notebook into Kaggle (two T4 GPUs) or a Modal notebook (A10/A10G or RTX PRO 6000), enable network access, provide `HF_TOKEN`, then **Run All**. Kaggle Secrets and environment variables are both supported. The same FP16 autocast + GradScaler precision policy is used on every supported CUDA family; larger-memory GPUs spend memory on larger physical microbatches before activation checkpointing is enabled.
+
+The trainer runs continuously until interrupted. A fresh run warms up once and then keeps a nonzero constant learning rate. Interrupting the training cell requests a clean stop and waits for a completed optimizer update, local full-state save, Hugging Face full-state upload, metrics upload, and inference export. A hard kill can recover from the most recent durable checkpoint.
+
+All project source remains embedded in readable notebook cells and is materialized by Run All. This includes data streams, the Cortex curriculum, model, trainer, full-state recovery, Hugging Face export, tokenizer/model wrappers, provenance files, and tests. Existing codec utility modules remain packaged for project continuity, but the current training registry contains **no image or audio datasets**.
+
+## Active training data
+
+The active external language sources are:
+
+- `appvoid/rewrite6`
+- Ultra-FineWeb-L3 English multi-style
+- Ultra-FineWeb-L3 English QA
+- DCLM baseline 1.0
+- FineWeb-Edu score 2
+- FineMath 4+
+- `appvoid/no-prompt-oasst`
+- `appvoid/no-prompt-openhermes`
+- SmolLM-Corpus Cosmopedia v2
+- SmolLM-Corpus FineWeb-Edu-dedup
+
+The procedural Cortex curriculum remains active as its own weighted source. Dataset revisions and shard lists are pinned in `dataset_manifest.json`; each source keeps deterministic cursor/epoch/rejection state in full checkpoints.
+
+## Objective and recurrent compute
+
+Every successful optimizer update trains the exact L8 trajectory plus one exact auxiliary trajectory, keeping the same objective:
+
+`CE(L8) + 0.20 * CE(Lr), r in {1,...,7}`.
+
+The auxiliary depth follows a deterministic, repeating **sustained progressive-data curriculum**, not a rapid 28-update depth rotation. With `aux_stage_base_updates=128` the L8+L1 stage receives 128 committed optimizer updates and batch draws, L8+L2 receives 256, then L3=384, L4=512, L5=640, L6=768, and L7=896; one curriculum spans 3,584 updates before repeating. Every update draws another global mixer batch (which can contain repeated finite-dataset samples after epochs, not guaranteed unique records). Hence L7 receives 7× the optimizer updates and example presentations allocated to L1, and **L8 remains supervised in every update**. Tune only `aux_stage_base_updates` to scale the entire schedule; the checkpoint persists this value and the curriculum origin. Depth is independent of microbatch count, DDP partition and overflow retries, and is reconstructed from the committed checkpoint step on resume. Each exact-budget trajectory receives its own continuous phase/stride coordinates; auxiliary losses are not snapshots from L8. Full BPTT is retained; activation checkpointing remains a GPU-memory fallback. More exposure encourages but does not guarantee monotonic validation performance.
+
+The exact previous SparkBET notebook fingerprint and exact original pre-schedule fingerprint are explicitly approved for *verified auxiliary-schedule migration*. Migrated checkpoints begin their new data stages at L1 using their committed update as `aux_curriculum_origin_step`, while preserving model, optimizer, scaler, and data cursors. A subsequent resume restores the saved origin and stage budget instead of restarting L1. Other fingerprints are rejected rather than silently restarting. Set `allow_verified_schedule_migration=False` to forbid even these approved migrations. The new objective fingerprint is stored on the next checkpoint. BET2/LeWorldModel experimental code is included, but `BET2_ABLATION_ENABLED=False` by default; standard training does not create or train the sidecar.
+
+## Hardware policy
+
+Run All identifies T4, A10/A10G, and RTX PRO 6000 explicitly. It first probes the largest divisible physical microbatch without activation checkpointing. If none fits with safety headroom, it retries with checkpointing. This keeps model/context/precision fixed while using larger-memory GPUs for larger physical batches. TF32 remains disabled so the numerical precision policy does not silently change when a run moves between supported GPU families.
+
+## Checkpoints, resume, and Hugging Face
+
+Complete local checkpoint directories contain `training.pt`, `metadata.json`, and `COMPLETE`. They preserve model, AdamW, GradScaler, per-rank RNG, exact mixer/data cursors, update counters, and training contract. Local retention keeps three complete checkpoints.
+
+Hugging Face full-state checkpoints are uploaded to a SparkBET-specific namespace inside **appvoid/cortex** so they cannot be confused with earlier architectures. Hub head retention keeps two complete SparkBET checkpoints; repository history is not rewritten. At startup the trainer compares verified local and Hub candidates newest-first and can recover across sessions. Upload failures preserve local state and are reported rather than deleting a completed checkpoint.
+
+TensorBoard event files are uploaded independently to `runs/`; the model page exposes them under the repository TensorBoard view. A standalone Transformers export is uploaded periodically and on a clean stop. The export contains SafeTensors weights, exact configuration, byte tokenizer, model card, dataset manifest, and custom model code for `trust_remote_code=True`.
+
+## Hugging Face authentication
+
+`HF_TOKEN` is read from Kaggle Secrets first on Kaggle and otherwise from the environment. It is never printed. The token needs read access to any private training source you use and write access to `appvoid/cortex`.
+
+## Inference
+
+The Hub export supports:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+repo = "appvoid/cortex"
+tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).cuda().eval()
+ids = torch.tensor([[257] + tok.encode("The next step is", add_special_tokens=False)], device="cuda")
+with torch.inference_mode(), torch.autocast("cuda", dtype=torch.float16):
+    result = model.generate(ids, max_new_tokens=200, do_sample=False, use_cache=False)
+print(tok.decode(result[0], skip_special_tokens=True))
+```
+
+The current model does not implement a recurrent KV cache, so generation recomputes the retained context for each new byte.
+
+## Validation
+
+The notebook runs source-integrity checks, architecture fingerprint/count checks, semantic/curriculum tests, data-stream tests, checkpoint recovery tests, Hugging Face export structure tests, and DDP equivalence checks before training. GPU capacity is validated with a real full-context L8+L7 backward/Adam preflight before the first committed update.

bet_model.py

@@ -0,0 +1,488 @@
+"""SparkBET-9M: min-spark-style looped core with BET phase conditioning."""
+import math, hashlib
+from dataclasses import dataclass, asdict
+import torch
+from torch import nn
+import torch.nn.functional as F
+from torch.utils.checkpoint import checkpoint
+
+SEQ_LEN = 1024
+MAX_LOOPS = 8
+PAD_ID, BOS_ID, EOS_ID = 256, 257, 258
+VOCAB_SIZE = 259
+EXPECTED_PARAM_COUNT = 9_353_876
+EXPECTED_ARCH_SHAPE_SHA256 = "c85415fc50d03a23f89ccd870ebf53a3a00c5a61017c13a28f6444fdd71b72b9"
+_USE_GRAD_CHECKPOINTING = False
+_STATE_NOISE_SIGMA = 0.0
+
+
+def set_gradient_checkpointing(enabled):
+    global _USE_GRAD_CHECKPOINTING
+    _USE_GRAD_CHECKPOINTING = bool(enabled)
+
+
+def set_state_noise_sigma(value):
+    global _STATE_NOISE_SIGMA
+    value = float(value)
+    if value < 0:
+        raise ValueError("state noise sigma must be >= 0")
+    _STATE_NOISE_SIGMA = value
+
+
+@dataclass(frozen=True)
+class BETConfig:
+    vocab_size: int = VOCAB_SIZE
+    hidden_size: int = 324
+    intermediate_size: int = 864
+    prelude_layers: int = 1
+    body_blocks: int = 6
+    coda_layers: int = 1
+    num_heads: int = 6
+    num_kv_heads: int = 2
+    head_dim: int = 54
+    lora_rank: int = 16
+    hyper_lanes: int = 2
+    max_seq_len: int = SEQ_LEN
+    max_loops: int = MAX_LOOPS
+    rope_theta: float = 10_000.0
+    rms_eps: float = 1e-6
+    ddl_beta_init: float = 1.0
+    ddl_k_eps: float = 1e-2
+    ddl_v_sigmoid_scale: float = 4.0
+
+    @property
+    def q_dim(self):
+        return self.num_heads * self.head_dim
+
+    @property
+    def kv_dim(self):
+        return self.num_kv_heads * self.head_dim
+
+    @property
+    def qkv_dim(self):
+        return self.q_dim + 2 * self.kv_dim
+
+
+CFG = BETConfig()
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(dim))
+        self.eps = eps
+
+    def forward(self, x):
+        dtype = x.dtype
+        y = x.float()
+        y = y * torch.rsqrt(y.pow(2).mean(-1, keepdim=True) + self.eps)
+        return (y * self.weight.float()).to(dtype)
+
+
+def rope_cos_sin(position_ids, dim, theta, dtype):
+    inv = 1.0 / (theta ** (torch.arange(0, dim, 2, device=position_ids.device, dtype=torch.float32) / dim))
+    f = position_ids.float().unsqueeze(-1) * inv
+    return f.cos().unsqueeze(1).to(dtype), f.sin().unsqueeze(1).to(dtype)
+
+
+def apply_rope(x, cos, sin):
+    # RoPE tables may be prepared before the first autocast linear, when the
+    # embedding stream is FP32. Cast them to the projected Q/K dtype here so
+    # attention remains FP16 on every CUDA profile instead of being promoted.
+    cos, sin = cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+    xe, xo = x[..., 0::2], x[..., 1::2]
+    return torch.stack((xe * cos - xo * sin, xe * sin + xo * cos), dim=-1).flatten(-2)
+
+
+def attention_mask_and_positions(input_ids, attention_mask=None):
+    b, t = input_ids.shape
+    if attention_mask is None:
+        pos = torch.arange(t, device=input_ids.device).view(1, t).expand(b, t)
+        return None, pos
+    mask = attention_mask.to(device=input_ids.device, dtype=torch.bool)
+    if mask.shape != input_ids.shape:
+        raise ValueError(f"attention_mask shape {tuple(mask.shape)} != input_ids {tuple(input_ids.shape)}")
+    pos = (mask.long().cumsum(-1) - 1).clamp_min(0)
+    if bool(mask.all()):
+        return None, pos
+    causal = torch.ones((t, t), device=input_ids.device, dtype=torch.bool).tril()[None, None]
+    allowed = causal & mask[:, None, None, :]
+    return allowed, pos
+
+
+class Attention(nn.Module):
+    """6Q/2KV GQA with per-head QK RMSNorm before RoPE."""
+    def __init__(self, c):
+        super().__init__()
+        if c.hidden_size != c.q_dim:
+            raise ValueError("hidden_size must equal num_heads * head_dim")
+        if c.num_heads % c.num_kv_heads:
+            raise ValueError("num_heads must be divisible by num_kv_heads")
+        self.qkv = nn.Linear(c.hidden_size, c.qkv_dim, bias=False)
+        self.out = nn.Linear(c.q_dim, c.hidden_size, bias=False)
+        self.qn = RMSNorm(c.head_dim, c.rms_eps)
+        self.kn = RMSNorm(c.head_dim, c.rms_eps)
+        self.nh, self.nkv, self.hd = c.num_heads, c.num_kv_heads, c.head_dim
+        self.q_dim, self.kv_dim = c.q_dim, c.kv_dim
+
+    def forward(self, x, cos, sin, qkv_delta=None, attn_mask=None):
+        b, t, _ = x.shape
+        qkv = self.qkv(x)
+        if qkv_delta is not None:
+            if qkv_delta.shape != qkv.shape:
+                raise RuntimeError("phase LoRA QKV delta shape mismatch")
+            qkv = qkv + qkv_delta
+        q, k, v = qkv.split([self.q_dim, self.kv_dim, self.kv_dim], dim=-1)
+        q = q.view(b, t, self.nh, self.hd).transpose(1, 2)
+        k = k.view(b, t, self.nkv, self.hd).transpose(1, 2)
+        v = v.view(b, t, self.nkv, self.hd).transpose(1, 2)
+        q = apply_rope(self.qn(q), cos, sin)
+        k = apply_rope(self.kn(k), cos, sin)
+        if self.nkv != self.nh:
+            repeat = self.nh // self.nkv
+            k = k.repeat_interleave(repeat, dim=1)
+            v = v.repeat_interleave(repeat, dim=1)
+        if attn_mask is None:
+            z = F.scaled_dot_product_attention(q, k, v, is_causal=True, dropout_p=0.0)
+        else:
+            z = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, dropout_p=0.0)
+        return self.out(z.transpose(1, 2).contiguous().view(b, t, self.q_dim))
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, c):
+        super().__init__()
+        self.gate_up = nn.Linear(c.hidden_size, 2 * c.intermediate_size, bias=False)
+        self.down = nn.Linear(c.intermediate_size, c.hidden_size, bias=False)
+
+    def forward(self, x):
+        gate, up = self.gate_up(x).chunk(2, dim=-1)
+        return self.down(F.silu(gate) * up)
+
+
+class DeepDeltaResidual(nn.Module):
+    """Scalar Deep-Delta residual update used only in the shared recurrent body."""
+    def __init__(self, c):
+        super().__init__()
+        self.k_eps = c.ddl_k_eps
+        self.v_sigmoid_scale = c.ddl_v_sigmoid_scale
+        self.beta_init = c.ddl_beta_init
+        self.beta = nn.Linear(c.hidden_size, 1, bias=True)
+        self.v_proj = nn.Linear(c.hidden_size, 1, bias=True)
+
+    @torch.no_grad()
+    def reset_beta_bias(self):
+        p = min(max(self.beta_init, 0.0), 2.0) / 2.0
+        p = min(max(p, 1e-6), 1.0 - 1e-6)
+        self.beta.bias.fill_(math.log(p) - math.log(1.0 - p))
+
+    def forward(self, x, *, k_in, context):
+        d = k_in.size(-1)
+        eps_rms = (self.k_eps * self.k_eps) / d
+        k_rms = F.rms_norm(k_in, [d], eps=eps_rms)
+        scale = 1.0 / math.sqrt(d)
+        beta = 2.0 * torch.sigmoid(self.beta(context).float())
+        proj = torch.sum(k_rms * x, dim=-1, keepdim=True, dtype=torch.float32) * scale
+        v = torch.sigmoid(self.v_proj(x).float()) * self.v_sigmoid_scale
+        delta = ((beta * (v - proj)) * scale).to(dtype=x.dtype)
+        return x + delta * k_rms
+
+
+class PlainBlock(nn.Module):
+    def __init__(self, c):
+        super().__init__()
+        self.attn_norm = RMSNorm(c.hidden_size, c.rms_eps)
+        self.attn = Attention(c)
+        self.ffn_norm = RMSNorm(c.hidden_size, c.rms_eps)
+        self.ffn = SwiGLU(c)
+
+    def forward(self, x, cos, sin, attn_mask):
+        x = x + self.attn(self.attn_norm(x), cos, sin, attn_mask=attn_mask)
+        return x + self.ffn(self.ffn_norm(x))
+
+
+class ContinuousLoopConditioner(nn.Module):
+    def __init__(self, d):
+        super().__init__()
+        self.net = nn.Sequential(nn.Linear(8, d), nn.SiLU(), nn.Linear(d, 2 * d))
+
+    @staticmethod
+    def features(t, dt, device, dtype):
+        return torch.tensor([
+            t, dt,
+            math.sin(math.pi * t), math.cos(math.pi * t),
+            math.sin(2 * math.pi * t), math.cos(2 * math.pi * t),
+            math.log(max(dt, 1e-6)), math.log(max(1.0 - t + dt, 1e-6)),
+        ], device=device, dtype=dtype)
+
+    def parameters_for(self, t, dt, device, dtype):
+        f = self.features(t, dt, device, dtype)
+        scale, shift = self.net(f).chunk(2, dim=-1)
+        return f, scale, shift
+
+    @staticmethod
+    def modulate(x, scale, shift):
+        return x * (1.0 + 0.1 * scale) + 0.1 * shift
+
+
+class PhaseLoRA(nn.Module):
+    def __init__(self, c):
+        super().__init__()
+        self.down = nn.Linear(c.hidden_size, c.lora_rank, bias=False)
+        self.up = nn.Linear(c.lora_rank, c.qkv_dim, bias=False)
+        self.gate = nn.Linear(8, c.lora_rank, bias=True)
+
+    def forward(self, x, phase_features):
+        gate = (2.0 * torch.sigmoid(self.gate(phase_features).float())).to(dtype=x.dtype)
+        return self.up(self.down(x) * gate)
+
+
+class LoopedBlock(nn.Module):
+    def __init__(self, c):
+        super().__init__()
+        self.attn_norm = RMSNorm(c.hidden_size, c.rms_eps)
+        self.attn = Attention(c)
+        self.phase_lora = PhaseLoRA(c)
+        self.ddl_attn = DeepDeltaResidual(c)
+        self.ffn_norm = RMSNorm(c.hidden_size, c.rms_eps)
+        self.ffn = SwiGLU(c)
+        self.ddl_ffn = DeepDeltaResidual(c)
+
+    def forward(self, x, cos, sin, phase_scale, phase_shift, phase_features, attn_mask):
+        attn_context = ContinuousLoopConditioner.modulate(x, phase_scale, phase_shift)
+        qkv_delta = self.phase_lora(attn_context, phase_features)
+        x_norm = self.attn_norm(attn_context)
+        x = self.ddl_attn(
+            x,
+            k_in=self.attn(x_norm, cos, sin, qkv_delta=qkv_delta, attn_mask=attn_mask),
+            context=x_norm,
+        )
+        ffn_context = ContinuousLoopConditioner.modulate(x, phase_scale, phase_shift)
+        x_norm = self.ffn_norm(ffn_context)
+        return self.ddl_ffn(x, k_in=self.ffn(x_norm), context=x_norm)
+
+
+class LoopHyperConnection(nn.Module):
+    """Two persistent loop lanes with per-budget read/mix/write scalars."""
+    def __init__(self, c):
+        super().__init__()
+        self.k = int(c.hyper_lanes)
+        self.max_loops = int(c.max_loops)
+        if self.k != 2:
+            raise ValueError("SparkBET is defined for two Hyper-Connection lanes")
+        shape = (self.max_loops, self.max_loops)
+        self.alpha = nn.Parameter(torch.zeros(*shape, self.k))
+        self.mix = nn.Parameter(torch.zeros(*shape, self.k, self.k))
+        self.beta = nn.Parameter(torch.zeros(*shape, self.k))
+        self.reset_parameters()
+
+    @torch.no_grad()
+    def reset_parameters(self):
+        self.alpha.zero_(); self.mix.zero_(); self.beta.zero_()
+        eye = torch.eye(self.k, device=self.mix.device, dtype=self.mix.dtype)
+        for budget in range(1, self.max_loops + 1):
+            b = budget - 1
+            for i in range(budget):
+                self.alpha[b, i, i % self.k] = 1.0
+                self.mix[b, i].copy_(eye)
+                self.beta[b, i].fill_(1.0)
+
+    def init_lanes(self, prelude_state):
+        return prelude_state.unsqueeze(0).expand(self.k, *prelude_state.shape)
+
+    def read(self, lanes, loops, iteration):
+        a = self.alpha[loops - 1, iteration].to(dtype=lanes.dtype)
+        return torch.einsum("k,kbtd->btd", a, lanes)
+
+    def write(self, lanes, branch_delta, loops, iteration):
+        m = self.mix[loops - 1, iteration].to(dtype=lanes.dtype)
+        b = self.beta[loops - 1, iteration].to(dtype=lanes.dtype)
+        mixed = torch.einsum("kj,jbtd->kbtd", m, lanes)
+        return mixed + b[:, None, None, None] * branch_delta.unsqueeze(0)
+
+    @staticmethod
+    def pool(lanes):
+        return lanes.mean(dim=0)
+
+
+class SparkBET(nn.Module):
+    """Prelude -> six shared recurrent blocks -> coda, with exact loop budgets 1..8."""
+    def __init__(self, c=CFG):
+        super().__init__()
+        self.c = c
+        self.embed = nn.Embedding(c.vocab_size, c.hidden_size)
+        self.prelude = nn.ModuleList([PlainBlock(c) for _ in range(c.prelude_layers)])
+        self.body = nn.ModuleList([LoopedBlock(c) for _ in range(c.body_blocks)])
+        self.time_cond = ContinuousLoopConditioner(c.hidden_size)
+        self.loop_hyper = LoopHyperConnection(c)
+        self.coda = nn.ModuleList([PlainBlock(c) for _ in range(c.coda_layers)])
+        self.final_norm = RMSNorm(c.hidden_size, c.rms_eps)
+        self.apply(self._generic_init)
+        self._mechanism_init()
+
+    @staticmethod
+    def _generic_init(m):
+        if isinstance(m, nn.Linear):
+            nn.init.normal_(m.weight, 0.0, 0.02)
+            if m.bias is not None: nn.init.zeros_(m.bias)
+        elif isinstance(m, nn.Embedding):
+            nn.init.normal_(m.weight, 0.0, 0.02)
+
+    @torch.no_grad()
+    def _mechanism_init(self):
+        nn.init.zeros_(self.time_cond.net[-1].weight)
+        nn.init.zeros_(self.time_cond.net[-1].bias)
+        self.loop_hyper.reset_parameters()
+        for block in self.body:
+            nn.init.zeros_(block.phase_lora.up.weight)
+            nn.init.zeros_(block.phase_lora.gate.weight)
+            nn.init.zeros_(block.phase_lora.gate.bias)
+        for block in [*self.prelude, *self.body, *self.coda]:
+            nn.init.zeros_(block.attn.out.weight)
+            nn.init.zeros_(block.ffn.down.weight)
+        for block in self.body:
+            block.ddl_attn.reset_beta_bias(); block.ddl_ffn.reset_beta_bias()
+
+    def _run_plain(self, block, x, cos, sin, attn_mask):
+        if _USE_GRAD_CHECKPOINTING and self.training:
+            return checkpoint(block, x, cos, sin, attn_mask, use_reentrant=False)
+        return block(x, cos, sin, attn_mask)
+
+    def _run_looped(self, block, x, cos, sin, scale, shift, features, attn_mask):
+        if _USE_GRAD_CHECKPOINTING and self.training:
+            return checkpoint(block, x, cos, sin, scale, shift, features, attn_mask, use_reentrant=False)
+        return block(x, cos, sin, scale, shift, features, attn_mask)
+
+    def _readout(self, x, cos, sin, attn_mask):
+        h = x
+        for block in self.coda:
+            h = self._run_plain(block, h, cos, sin, attn_mask)
+        h = self.final_norm(h)
+        return F.linear(h, self.embed.weight)
+
+    @staticmethod
+    def _schedule(step_sizes):
+        if isinstance(step_sizes, int):
+            n = int(step_sizes)
+            step_sizes = uniform_steps(n)
+        if not step_sizes:
+            raise ValueError("empty refinement schedule")
+        values = [float(v) for v in step_sizes]
+        if any(v <= 0 for v in values):
+            raise ValueError("refinement strides must be positive")
+        if abs(sum(values) - 1.0) > 1e-5:
+            raise ValueError("refinement strides must sum to 1")
+        return values
+
+    def _prepare(self, input_ids, step_sizes, attention_mask=None):
+        steps = self._schedule(step_sizes)
+        loops = len(steps)
+        if loops > self.c.max_loops:
+            raise ValueError(f"loops {loops} > configured max_loops {self.c.max_loops}")
+        if input_ids.shape[1] > self.c.max_seq_len:
+            raise ValueError("context exceeds max_seq_len")
+        x = self.embed(input_ids)
+        attn_mask, pos = attention_mask_and_positions(input_ids, attention_mask)
+        cos, sin = rope_cos_sin(pos, self.c.head_dim, self.c.rope_theta, x.dtype)
+        for block in self.prelude:
+            x = self._run_plain(block, x, cos, sin, attn_mask)
+        lanes = self.loop_hyper.init_lanes(x)
+        shared_noise = torch.randn_like(x) if self.training and _STATE_NOISE_SIGMA > 0 else None
+        return steps, lanes, shared_noise, cos, sin, attn_mask
+
+    def _advance(self, lanes, steps, iteration, elapsed, shared_noise, cos, sin, attn_mask):
+        dt = steps[iteration]
+        t_mid = elapsed + 0.5 * dt
+        features, scale, shift = self.time_cond.parameters_for(t_mid, dt, lanes.device, lanes.dtype)
+        branch_input = self.loop_hyper.read(lanes, len(steps), iteration)
+        if shared_noise is not None:
+            t_end = elapsed + dt
+            sigma = _STATE_NOISE_SIGMA * max(0.0, 1.0 - t_end)
+            if sigma:
+                branch_input = branch_input + sigma * shared_noise
+        h = branch_input
+        for block in self.body:
+            h = self._run_looped(block, h, cos, sin, scale, shift, features, attn_mask)
+        return self.loop_hyper.write(lanes, h - branch_input, len(steps), iteration)
+
+    def _run_recurrence(self, input_ids, step_sizes, attention_mask=None, collect_states=False):
+        steps, lanes, noise, cos, sin, attn_mask = self._prepare(input_ids, step_sizes, attention_mask)
+        states = [] if collect_states else None
+        elapsed = 0.0
+        for i, dt in enumerate(steps):
+            lanes = self._advance(lanes, steps, i, elapsed, noise, cos, sin, attn_mask)
+            elapsed += dt
+            if collect_states: states.append(self.loop_hyper.pool(lanes))
+        return self.loop_hyper.pool(lanes), states, cos, sin, attn_mask
+
+    def forward(self, input_ids, step_sizes=None, attention_mask=None):
+        if step_sizes is None: step_sizes = uniform_steps(self.c.max_loops)
+        x, _, cos, sin, attn_mask = self._run_recurrence(input_ids, step_sizes, attention_mask, False)
+        return self._readout(x, cos, sin, attn_mask)
+
+    def forward_loop_exits(self, input_ids, step_sizes=None, attention_mask=None):
+        if step_sizes is None: step_sizes = uniform_steps(self.c.max_loops)
+        _, states, cos, sin, attn_mask = self._run_recurrence(input_ids, step_sizes, attention_mask, True)
+        return [self._readout(h, cos, sin, attn_mask) for h in states]
+
+    def count_params(self):
+        return sum(p.numel() for p in self.parameters())
+
+
+# Preserve the old trainer/import name while changing the implementation.
+BETFog = SparkBET
+
+
+def uniform_steps(n):
+    n = int(n)
+    if not 1 <= n <= MAX_LOOPS:
+        raise ValueError(f"loop budget must be in [1,{MAX_LOOPS}]")
+    return [1.0 / n] * n
+
+
+def architecture_shape_signature(model):
+    lines = [f"{k}:{tuple(v.shape)}:{v.dtype}" for k, v in model.state_dict().items()]
+    return hashlib.sha256("\n".join(lines).encode()).hexdigest()
+
+
+@torch.no_grad()
+def verify_architecture(model, rank=0):
+    expected = dict(
+        vocab_size=259, hidden_size=324, intermediate_size=864,
+        prelude_layers=1, body_blocks=6, coda_layers=1,
+        num_heads=6, num_kv_heads=2, head_dim=54,
+        lora_rank=16, hyper_lanes=2, max_seq_len=1024, max_loops=8,
+        rope_theta=10_000.0, rms_eps=1e-6,
+        ddl_beta_init=1.0, ddl_k_eps=1e-2, ddl_v_sigmoid_scale=4.0,
+    )
+    actual = asdict(model.c)
+    for k, v in expected.items():
+        if actual[k] != v:
+            raise AssertionError(f"Architecture drift: {k}={actual[k]} expected {v}")
+    if model.count_params() != EXPECTED_PARAM_COUNT:
+        raise AssertionError(f"Parameter drift: {model.count_params():,} != {EXPECTED_PARAM_COUNT:,}")
+    sig = architecture_shape_signature(model)
+    if sig != EXPECTED_ARCH_SHAPE_SHA256:
+        raise AssertionError(f"Architecture SHA drift: {sig} != {EXPECTED_ARCH_SHAPE_SHA256}")
+    if not torch.allclose(model.time_cond.net[-1].weight, torch.zeros_like(model.time_cond.net[-1].weight)):
+        raise AssertionError("time conditioner must start as identity")
+    for i, block in enumerate(model.body):
+        if not torch.allclose(block.phase_lora.up.weight, torch.zeros_like(block.phase_lora.up.weight)):
+            raise AssertionError(f"phase LoRA {i} up projection must start zero")
+        for name, ddl in (("attn", block.ddl_attn), ("ffn", block.ddl_ffn)):
+            beta = (2.0 * torch.sigmoid(ddl.beta.bias.float())).item()
+            if abs(beta - 1.0) > 1e-6:
+                raise AssertionError(f"body {i} {name} DDL beta init={beta}")
+    hc = model.loop_hyper
+    eye = torch.eye(hc.k, device=hc.mix.device, dtype=hc.mix.dtype)
+    for budget in range(1, model.c.max_loops + 1):
+        for i in range(budget):
+            expected_alpha = torch.zeros_like(hc.alpha[budget - 1, i]); expected_alpha[i % hc.k] = 1
+            if not torch.allclose(hc.alpha[budget - 1, i], expected_alpha): raise AssertionError("Hyper alpha init drift")
+            if not torch.allclose(hc.mix[budget - 1, i], eye): raise AssertionError("Hyper mix init drift")
+            if not torch.allclose(hc.beta[budget - 1, i], torch.ones_like(hc.beta[budget - 1, i])): raise AssertionError("Hyper beta init drift")
+    if rank == 0:
+        print("[verify] SparkBET architecture PASSED")
+        print(f"[verify] params: {model.count_params():,}")
+        print(f"[verify] physical blocks: 1 prelude + 6 shared body + 1 coda; L8 applications=50")
+        print(f"[verify] architecture SHA256: {sig}")

checkpoints/sparkbet9m/checkpoint-000000221000/COMPLETE

@@ -0,0 +1 @@
+complete

checkpoints/sparkbet9m/checkpoint-000000221000/metadata.json

@@ -0,0 +1,10 @@
+{
+  "format": "cortex-sparkbet9m-full-v1",
+  "step": 221000,
+  "lineage": "cc0c9b5484e046e3b892214e660d2b7d",
+  "architecture": "c85415fc50d03a23f89ccd870ebf53a3a00c5a61017c13a28f6444fdd71b72b9",
+  "pipeline": "0a5a3d695613ed57222d0a20ae6c0c70f27ed7c33f86c9e7d0ba4651bc87f062",
+  "sha256": "076b418994565028f27ed43bb6597c1e2b0f56ce1baa06827eb71921579e3fbb",
+  "bytes": 115622259,
+  "saved_at": 1789957868.5571094
+}

checkpoints/sparkbet9m/checkpoint-000000221000/training.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:076b418994565028f27ed43bb6597c1e2b0f56ce1baa06827eb71921579e3fbb
+size 115622259

checkpoints/sparkbet9m/checkpoint-000000222000/COMPLETE

@@ -0,0 +1 @@
+complete

checkpoints/sparkbet9m/checkpoint-000000222000/metadata.json

@@ -0,0 +1,10 @@
+{
+  "format": "cortex-sparkbet9m-full-v1",
+  "step": 222000,
+  "lineage": "cc0c9b5484e046e3b892214e660d2b7d",
+  "architecture": "c85415fc50d03a23f89ccd870ebf53a3a00c5a61017c13a28f6444fdd71b72b9",
+  "pipeline": "0a5a3d695613ed57222d0a20ae6c0c70f27ed7c33f86c9e7d0ba4651bc87f062",
+  "sha256": "661e2c3475097be19d6d7f780a10eabfad4323206f16920f3153337f8deb18dd",
+  "bytes": 115659315,
+  "saved_at": 1789958762.9045448
+}

checkpoints/sparkbet9m/checkpoint-000000222000/training.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:661e2c3475097be19d6d7f780a10eabfad4323206f16920f3153337f8deb18dd
+size 115659315

config.json

@@ -0,0 +1,40 @@
+{
+  "model_type": "bet",
+  "architectures": [
+    "BETForCausalLM"
+  ],
+  "step": 214215,
+  "vocab_size": 259,
+  "hidden_size": 324,
+  "intermediate_size": 864,
+  "prelude_layers": 1,
+  "body_blocks": 6,
+  "coda_layers": 1,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "head_dim": 54,
+  "lora_rank": 16,
+  "hyper_lanes": 2,
+  "max_position_embeddings": 1024,
+  "max_loops": 8,
+  "rope_theta": 10000.0,
+  "rms_norm_eps": 1e-06,
+  "ddl_beta_init": 1.0,
+  "ddl_k_eps": 0.01,
+  "ddl_v_sigmoid_scale": 4.0,
+  "refinement_cycles": 8,
+  "use_cache": false,
+  "tie_word_embeddings": true,
+  "pad_token_id": 256,
+  "bos_token_id": 257,
+  "eos_token_id": 258,
+  "precision": "fp16 autocast / fp32 master",
+  "auto_map": {
+    "AutoConfig": "configuration_bet.BETConfig",
+    "AutoModelForCausalLM": "modeling_bet.BETForCausalLM",
+    "AutoTokenizer": [
+      "tokenization_bet.BETByteTokenizer",
+      null
+    ]
+  }
+}

configuration_bet.py

@@ -0,0 +1,64 @@
+from transformers import PretrainedConfig
+
+
+class BETConfig(PretrainedConfig):
+    model_type = "bet"
+
+    def __init__(
+        self,
+        vocab_size=259,
+        hidden_size=324,
+        intermediate_size=864,
+        prelude_layers=1,
+        body_blocks=6,
+        coda_layers=1,
+        num_attention_heads=6,
+        num_key_value_heads=2,
+        head_dim=54,
+        lora_rank=16,
+        hyper_lanes=2,
+        max_position_embeddings=1024,
+        max_loops=8,
+        rope_theta=10_000.0,
+        rms_norm_eps=1e-6,
+        ddl_beta_init=1.0,
+        ddl_k_eps=1e-2,
+        ddl_v_sigmoid_scale=4.0,
+        refinement_cycles=8,
+        use_cache=False,
+        tie_word_embeddings=True,
+        pad_token_id=256,
+        bos_token_id=257,
+        eos_token_id=258,
+        **kwargs,
+    ):
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            is_encoder_decoder=False,
+            **kwargs,
+        )
+        self.vocab_size=int(vocab_size)
+        self.hidden_size=int(hidden_size)
+        self.intermediate_size=int(intermediate_size)
+        self.prelude_layers=int(prelude_layers)
+        self.body_blocks=int(body_blocks)
+        self.coda_layers=int(coda_layers)
+        # Common HF tooling expects num_hidden_layers even though only the body loops.
+        self.num_hidden_layers=int(prelude_layers+body_blocks+coda_layers)
+        self.num_attention_heads=int(num_attention_heads)
+        self.num_key_value_heads=int(num_key_value_heads)
+        self.head_dim=int(head_dim)
+        self.lora_rank=int(lora_rank)
+        self.hyper_lanes=int(hyper_lanes)
+        self.max_position_embeddings=int(max_position_embeddings)
+        self.max_loops=int(max_loops)
+        self.rope_theta=float(rope_theta)
+        self.rms_norm_eps=float(rms_norm_eps)
+        self.ddl_beta_init=float(ddl_beta_init)
+        self.ddl_k_eps=float(ddl_k_eps)
+        self.ddl_v_sigmoid_scale=float(ddl_v_sigmoid_scale)
+        self.refinement_cycles=int(refinement_cycles)
+        self.use_cache=bool(use_cache)

inference.py

@@ -0,0 +1,43 @@
+"""Small text-generation helper for an exported SparkBET repository."""
+from pathlib import Path
+import torch
+from safetensors.torch import load_file
+from bet_model import SparkBET,BETConfig,uniform_steps
+
+
+class Cortex:
+    def __init__(self,model,device=None):
+        self.model=model
+        self.device=torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
+        self.model.to(self.device).eval()
+
+    @classmethod
+    def from_export(cls,folder,device=None):
+        folder=Path(folder);model=SparkBET(BETConfig())
+        state=load_file(str(folder/"model.safetensors"),device="cpu")
+        if state and all(k.startswith("core.") for k in state):state={k[5:]:v for k,v in state.items()}
+        model.load_state_dict(state,strict=True)
+        return cls(model,device)
+
+    def generate_ids(self,ids,max_new_tokens=128,loops=8,temperature=0.0,top_k=None):
+        out=list(map(int,ids))
+        for _ in range(int(max_new_tokens)):
+            current=out[-self.model.c.max_seq_len:]
+            x=torch.tensor([current],device=self.device,dtype=torch.long)
+            with torch.inference_mode(),torch.autocast(self.device.type,dtype=torch.float16,enabled=self.device.type=="cuda"):
+                logits=self.model(x,uniform_steps(loops))[0,-1].float()
+            if temperature and temperature>0:
+                logits=logits/float(temperature)
+                if top_k:
+                    values,_=torch.topk(logits,min(int(top_k),logits.numel()));logits[logits<values[-1]]=-float("inf")
+                nxt=int(torch.multinomial(torch.softmax(logits,-1),1))
+            else:nxt=int(logits.argmax())
+            out.append(nxt)
+            if nxt==258:break
+        return out
+
+    def generate(self,text,max_new_tokens=128,loops=8,temperature=0.0,top_k=None):
+        ids=[257]+list(text.encode("utf-8"))
+        out=self.generate_ids(ids,max_new_tokens,loops,temperature,top_k)
+        body=bytes(i for i in out[1:] if 0<=i<=255)
+        return body.decode("utf-8",errors="replace")

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8835a993d051d3fdfad3bc0eed07849e5445851e9654c9debc4f56f97b959bb
+size 37429752

modeling_bet.py

@@ -0,0 +1,76 @@
+import os
+import torch
+import torch.nn.functional as F
+from transformers import PreTrainedModel, GenerationMixin
+from transformers.modeling_outputs import CausalLMOutput
+
+from .configuration_bet import BETConfig
+from .bet_model import BETConfig as CoreConfig, SparkBET, uniform_steps
+
+
+class BETPreTrainedModel(PreTrainedModel):
+    config_class=BETConfig
+    base_model_prefix="core"
+    supports_gradient_checkpointing=False
+    _no_split_modules=["PlainBlock","LoopedBlock"]
+
+
+class BETForCausalLM(BETPreTrainedModel,GenerationMixin):
+    def __init__(self,config):
+        super().__init__(config)
+        core_cfg=CoreConfig(
+            vocab_size=config.vocab_size,
+            hidden_size=config.hidden_size,
+            intermediate_size=config.intermediate_size,
+            prelude_layers=config.prelude_layers,
+            body_blocks=config.body_blocks,
+            coda_layers=config.coda_layers,
+            num_heads=config.num_attention_heads,
+            num_kv_heads=config.num_key_value_heads,
+            head_dim=config.head_dim,
+            lora_rank=config.lora_rank,
+            hyper_lanes=config.hyper_lanes,
+            max_seq_len=config.max_position_embeddings,
+            max_loops=config.max_loops,
+            rope_theta=config.rope_theta,
+            rms_eps=config.rms_norm_eps,
+            ddl_beta_init=config.ddl_beta_init,
+            ddl_k_eps=config.ddl_k_eps,
+            ddl_v_sigmoid_scale=config.ddl_v_sigmoid_scale,
+        )
+        self.core=SparkBET(core_cfg)
+
+    def get_input_embeddings(self):return self.core.embed
+    def set_input_embeddings(self,value):self.core.embed=value
+    def get_output_embeddings(self):return None
+    def set_output_embeddings(self,value):
+        if value is not None:raise ValueError("SparkBET uses tied input/output embeddings")
+
+    def _cycles(self,cycles=None):
+        if cycles is None:
+            cycles=int(os.environ.get("BET_EVAL_CYCLES",self.config.refinement_cycles))
+        cycles=int(cycles)
+        if not 1<=cycles<=self.config.max_loops:
+            raise ValueError(f"refinement cycles must be in [1,{self.config.max_loops}]")
+        return cycles
+
+    def forward(
+        self,input_ids=None,attention_mask=None,labels=None,cycles=None,
+        past_key_values=None,use_cache=None,return_dict=True,**kwargs,
+    ):
+        if input_ids is None:raise ValueError("input_ids is required")
+        if past_key_values is not None:raise ValueError("SparkBET does not implement a KV cache")
+        logits=self.core(input_ids,uniform_steps(self._cycles(cycles)),attention_mask=attention_mask)
+        loss=None
+        if labels is not None:
+            shift_logits=logits[:,:-1].contiguous().float();shift_labels=labels[:,1:].contiguous()
+            loss=F.cross_entropy(shift_logits.view(-1,shift_logits.size(-1)),shift_labels.view(-1),ignore_index=-100)
+        if not return_dict:return tuple(v for v in (loss,logits) if v is not None)
+        return CausalLMOutput(loss=loss,logits=logits)
+
+    def prepare_inputs_for_generation(self,input_ids,attention_mask=None,**kwargs):
+        max_len=self.config.max_position_embeddings
+        if input_ids.shape[1]>max_len:
+            input_ids=input_ids[:,-max_len:]
+            if attention_mask is not None:attention_mask=attention_mask[:,-max_len:]
+        return {"input_ids":input_ids,"attention_mask":attention_mask,"use_cache":False}

records.py

@@ -0,0 +1,37 @@
+"""Direct UTF-8 bytes. Special IDs match BET, not the old Cortex tokenizer."""
+from dataclasses import dataclass
+PAD,BOS,EOS=256,257,258
+
+class Oversize(ValueError):pass
+class InvalidRecord(ValueError):pass
+
+def ids(text):return list(text.encode('utf-8'))
+def decode(tokens):return bytes(t for t in tokens if 0<=t<256).decode('utf-8',errors='replace')
+
+def record(prefix,answer,source,limit=1024,supervise_all=False,meta=None):
+    p,a=ids(prefix),ids(answer)
+    if not a:raise InvalidRecord('Empty target: '+source)
+    tokens=[BOS]+p+a+[EOS]
+    if len(tokens)>limit+1:raise Oversize(f'{source}: {len(tokens)} IDs exceeds {limit+1}; no truncation')
+    weights=[0]+([1]*len(p) if supervise_all else [0]*len(p))+[1]*(len(a)+1)
+    return dict(ids=tokens,weights=weights,source=source,prompt_len=1+len(p),meta=meta or {})
+
+def plain_chunks(text,source,limit=1024):
+    # Lossless bytes, including split UTF-8 sequences: decoder assembles the byte stream.
+    # No false EOS at chunk boundaries. One-token overlap predicts each byte once.
+    if not isinstance(text,str) or not text.strip():raise InvalidRecord('Empty/non-string text: '+source)
+    raw_bytes=text.encode('utf-8')
+    if len(raw_bytes)>1024*1024:raise Oversize('Document exceeds the 1 MiB bounded-buffer limit; rejected intact')
+    raw=[BOS]+list(raw_bytes)+[EOS];out=[]
+    for offset in range(0,len(raw)-1,limit):
+        chunk=raw[offset:offset+limit+1]
+        out.append(dict(ids=chunk,weights=[0]+[1]*(len(chunk)-1),source=source,prompt_len=1,meta={}))
+    return out
+
+def validate(r,limit=1024):
+    assert 2<=len(r['ids'])<=limit+1
+    assert len(r['ids'])==len(r['weights'])
+    assert all(type(x)==int and 0<=x<259 for x in r['ids'])
+    assert all(x in (0,1) for x in r['weights']) and sum(r['weights'][1:])>0
+    assert r['weights'][0]==0
+    return r

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789334934.modal.206.0.step-000000000000

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48d4b0c9e0a10d9de70d1b7796adbc34b57938b5dddd436b1f38d7bb714bfc66
+size 135822

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789335534.modal.206.1.step-000000000171

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9d692d8c29667f2c0042b6ca1099273f75a339e9cd700936f711dfabca78422
+size 158142

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789336136.modal.206.2.step-000000000366

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40a73a39478912711ff653371947802ed165c30f04897f30afaae8c96848fab8
+size 161639

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789336741.modal.206.3.step-000000000564

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:545b636e5a97e86f0804bb5ab4a4b098178e1c2e79b00433b0c34ee75faa1ca7
+size 162830

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789337343.modal.206.4.step-000000000765

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:446a263b5774bb1337fb5366902e62a2f7f12c39166e31075da4b7798030ccba
+size 157658

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789337947.modal.206.5.step-000000000960

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7df742ad0a6981d0195ffae0b93cd31dfa0d22df9f4f5b5962b812b6c2f8cd1
+size 157724

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789338549.modal.206.6.step-000000001156

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d93ce4f48de9ee600d1f2440fcd115d836ad7883cae14439aa2b7dcae4ed4244
+size 162460

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789339151.modal.206.7.step-000000001355

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac026f039167ffa5fe49635ee70ab285d3d0d16537a5d5ac3bbbfb628372829f
+size 161161

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789339756.modal.206.8.step-000000001551

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:814cfe498ab8491f860ab93ecbcb89f273fac38d7c20a89de303ac2083f496e5
+size 158415

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789340360.modal.206.9.step-000000001749

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e44cbf718e07a2d2d3aa70556e06c37eef1db4c554336ffe7142b5711ebc050
+size 158044

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789340961.modal.206.10.step-000000001942

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5a6614a645cb649c05423ad75dd894f265f84d5cf1a149c1df97ef64a766770
+size 175345

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789341565.modal.206.11.step-000000002134

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00e89ac910db88c663dce569346c16a6ca5ccf88055eb99dc5b148b7c0db9559
+size 160122

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789342170.modal.206.12.step-000000002331

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0bcd7c62e5ab2fe57c4f479af442a2a770f3bbedf0155137dde5c5207a97643e
+size 159525

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789342775.modal.206.13.step-000000002529

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38da17f69430ebde38d5a22548da847f5916436f362fc6e3de5bd1376d461fd2
+size 162994

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789343379.modal.206.14.step-000000002730

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65a859e91485b662b3a7626cc9a27feffc264bf99425e9150088ccf215dd4a7c
+size 156370

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789343984.modal.206.15.step-000000002924

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8cb2ce2b3fcd2d8e8f5f2582f7e36d5437eafe6d6c5a9cb981cae9ba8d16e178
+size 146854

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789344588.modal.206.16.step-000000003101

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ac816ae4c45fade0d88ad7e285f8df9bb57a0a5b1ca1d948f4990b7b6e311ed
+size 151434

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789345191.modal.206.17.step-000000003288

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e02af65a1383a3f137f01e91d04532598e98a3068f64a8f8a7e4c5d841838e33
+size 157658

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789345794.modal.206.18.step-000000003483

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:104c93d2e63b2d6cdfed56e60f115d3a0cb63e942c05c68d255dea97578e008b
+size 160770

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789346396.modal.206.19.step-000000003680

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56279fc90af709fe03536202f965fa210ecd6f45a6b1e24b9f942268afd5a68d
+size 156980

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789346999.modal.206.20.step-000000003873

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db8225703f8e526951ce7c20ed08290d8c7935c50827d55a08f21cde267515d8
+size 176699

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789347602.modal.206.21.step-000000004067

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8ee22577825c6ee932a1af6c50a6fe4455c088d9e2b60869fb538dc3f3cbb6b
+size 155016

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789348204.modal.206.22.step-000000004259

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ea0f994944e23e8038acef7aff409f21a1f6bb9ec0525ea81d35c317468324a
+size 158335

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789348809.modal.206.23.step-000000004455

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d746e70cc250725413495a476bdd36b77f71a17a4805c14b00587c488d6338e
+size 159752

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789349415.modal.206.24.step-000000004650

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c38c43041f4a6c089a7057d4b06681960039b9144409fabec6d5271562eebca8
+size 159887

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789350017.modal.206.25.step-000000004846

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d72babddad46bb93801e058dd1a45209adaf4b5af3649056393bc89775281bd
+size 164137

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789350620.modal.206.26.step-000000005048

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:796c5dd39c9b54ed8a0c7a440aaef0fde0b771544f6ca47577a2bf2d4adccf89
+size 160366

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789351224.modal.206.27.step-000000005247

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d24ef9cfc1a68c7de9110d1c87aae75a336799ffc5be483456fcea04f2a2547
+size 161495

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789351828.modal.206.28.step-000000005445

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2958473bdccd11aa27b6e455465a1ab9d692657c3dc6f6e6d2d7429a4fae243
+size 156817

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789352433.modal.206.29.step-000000005639

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd0a3757a843afaf08f1defb579db0d44a61513e6a39fb5348e8f6432987925f
+size 158768

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789353038.modal.206.30.step-000000005834

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebe50b499995af4d5f04cdd7b455bf65765355de8a1a6a7d26d850dcf179a5e7
+size 177376

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789353643.modal.206.31.step-000000006029

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d76a57d11f49cd0d026762e4761df29ab99bbdd6bbc4dc63bbdb45ece567990
+size 159689

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789354246.modal.206.32.step-000000006227

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78de2b1f8822d966cdc42942d6d9a8a456b50b40c45505543d82aaba5930c723
+size 161588

runs/cc0c9b5484e046e3b892214e660d2b7d/events.out.tfevents.1789354851.modal.206.33.step-000000006422

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:507fff9afe7fefed5bebb50ffa25934b35d7935684391b22620e50c385a062d7
+size 158801