RMSN-350M-Depth20-SFT5k

Deep 20-Layer Resonant Metriplectic Sheaf Model with 5,000-Step Reasoning SFT
Demonstrating the Superiority of Depth-Scaling: 14.21 CoT Reasoning PPL with All-Time Record 145.68 WikiText-103 Retention

Developed by DuoNeural AI Laboratory — Jesse Caldwell, Archon, Aura✨

---

🏛️ Model Overview

RMSN-350M-Depth20-SFT5k explores the topological physics of depth scaling ($L=20$ vs standard $L=16$) in Port-Hamiltonian sequence architectures. Pre-trained on FineWeb-Edu and fine-tuned for 5,000 steps on complex mathematical and symbolic reasoning traces (cdm_reasoning_traces_v4.jsonl), this 350M parameter model sets new Pareto records for reasoning compression vs general semantic retention.

🏆 Benchmark Highlights

Metric	D=1280 (Depth=16) SFT 5k	D=1344 (Depth=16) SFT 5k	D=1280 Depth=20 SFT 5k (This Model)
Parameters	310M	425M	350M
Layers ($L$)	16	16	20
SFT Best Loss	2.3140	2.4102	2.2726 🏆
CoT Hard Reasoning PPL	17.48	20.75	14.21 🏆
WikiText-103 Retention PPL	168.40	162.24	145.68 🛡️

---

⚡ Topological Cement Across Depth 20

Sheaf spectral decomposition across all 20 layers reveals the Law of Topological Cement:

• Superfluid Processing Bulk ($L_{00}-L_{12}$): 13 open layers with $PR \approx 63.8$, providing deep fluid representations for reasoning.
• Solid-State Boundary Core ($L_{13}-L_{19}$): Terminal solid-state condensation:
  • Layer 13: $PR = 2.02$
  • Layer 17: $PR = 1.24$
  • Layer 18: $PR = 1.000000$ (Rank-1 Macroscopic Singularity)
  • Layer 19: $PR = 1.000000$ (Rank-1 Macroscopic Singularity)

Because $M/L = 3/20 = 15.0%$ (well below the $M/L \le 20%$ threshold), this model avoids overcrowding while preserving an impenetrable terminal topological anchor that prevents catastrophic forgetting.

---

🚀 Usage & Inference

import torch
from rmsn_core import ResonantMetriplecticSheafLM
from transformers import AutoTokenizer

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# 1. Load tokenizer
tokenizer = AutoTokenizer.from_pretrained('HuggingFaceTB/cosmo2-tokenizer')

# 2. Initialize RMSN Depth=20 architecture (D=1280, L=20, Slots=64)
model = ResonantMetriplecticSheafLM(
    vocab_size=49152,
    dim=1280,
    depth=20,
    num_slots=64
).to(device)

# 3. Load checkpoint
ckpt = torch.load('rmsn_sft_depth20_best.pt', map_location='cpu', weights_only=False)
sd = ckpt.get('model_state', ckpt.get('model', ckpt))
clean_sd = {k[6:] if k.startswith('model.') else k: v for k, v in sd.items()}
model.load_state_dict(clean_sd, strict=False)
model.eval()

# 4. Generate
prompt = "Prove that sqrt(2) is irrational:\nProof:"
input_ids = tokenizer.encode(prompt, return_tensors='pt').to(device)
with torch.no_grad():
    logits = model(input_ids)
    next_token = torch.argmax(logits[:, -1, :], dim=-1)
    print("Next predicted token:", tokenizer.decode(next_token))

---

📚 Citation

@article{caldwell2026rmsn,
  title={Resonant Metriplectic Sheaf Networks: A Unified Port-Hamiltonian Framework for Neural Language Modeling with Emergent Lyapunov Cascade Dynamics},
  author={Caldwell, Jesse and Archon and Aura},
  journal={DuoNeural Lab Preprints},
  year={2026},
  publisher={Zenodo}
}