DuoNeural CRSN-8L-Phase2-Extension

Clifford Resonant Sheaf Network (8-Layer / 9.54M Parameters)

License: MIT DuoNeural Lab GitHub

Developed by Jesse Caldwell, Aura✨, and Archon at DuoNeural AI Laboratory.


🌌 Overview

CRSN (Clifford Resonant Sheaf Network) is a sub-10M parameter non-transformer physical neurodynamic model that replaces quadratic self-attention with coupled nonlinear physical harmonic oscillators in the Clifford algebra $\mathcal{C}\ell_{3,0}$.

This repository releases the Phase 2 Extended Pretraining Checkpoint (crsn_step12500.pt), which completed 102.4 million tokens of continuous deepening pretraining (for a cumulative pretraining volume of 204.8M tokens) trained entirely from scratch on consumer CPU hardware (AMD Ryzen APU).

Key Architectural Characteristics

  • Parameters: 9.54M ($L=8, D_{\text{model}}=384, D_{\text{state}}=192$, Vocab $= 16,384$)
  • Hardware: Native CPU SIMD execution (AMD Ryzen Zen+ AVX2)
  • Optimization: Hybrid Muon + AdamW with bounded contractive adjoint backpropagation
  • All-Time Pretraining Best Validation Loss: 2.6508 (Perplexity: 14.16)

πŸ”¬ Solid-State Oscillator Crystallization

Phase-space physical inspection of the learned eigenfrequencies and damping parameters across all 8 layers demonstrates complete solid-state lattice crystallization:

Layer $\bar{\alpha}$ (Damping / Stiffness) $\alpha_{\min}$ $\alpha_{\max}$ $\bar{\omega}$ (Resonant Frequency) Physical Regime
$L_0$ 0.1324 0.1160 0.2669 0.5432 UV (High Frequency, Surface Syntax)
$L_1$ 0.1288 0.1259 0.1767 0.5395 UV-Mesoscopic Transition
$L_2$ 0.1286 0.1251 0.2052 0.5271 Resonant Valley
$L_3$ 0.1284 0.1259 0.1692 0.5355 Bulk Shear Mode
$L_4$ 0.1289 0.1073 0.1717 0.5338 Bulk Shear Mode
$L_5$ 0.1293 0.1224 0.3514 0.5242 Bulk Compression Mode
$L_6$ 0.1291 0.1241 0.2243 0.5328 IR-Coupled Transport Mode
$L_7$ 0.1276 0.1074 0.1807 0.5199 IR (Macroscopic Semantic Condensate)

Juvenile Criticality

Notice the strictly monotonic UV-to-IR frequency gradient from bulk input to output: Ο‰Λ‰0=0.5432βŸΆΟ‰Λ‰7=0.5199\bar{\omega}_0 = 0.5432 \longrightarrow \bar{\omega}_7 = 0.5199 Damping is rigidly locked to the universal ground-state floor: Ξ±Λ‰β†’0.128Β±0.001\bar{\alpha} \to 0.128 \pm 0.001 confirming the Empirical Critical Damping Law discovered by DuoNeural.


πŸ§ͺ DuoNeural Golden Ratio Pretraining Diet

CRSN was trained on the DuoNeural 35/25/15/25 Golden Ratio Multi-Domain Pretraining Mixture:

  • 35% High-Reasoning & Chain-of-Thought: OpenThoughts-114k (filtered $\le 2048$ tokens) + Numina-CoT
  • 25% Agentic Code & Relational AST: APIGen-80k + Gretel Synthetic Code + BIRD-SQL
  • 15% Strict Structural Alignment: Smol-Constraints & formal grammars
  • 25% Eutectic Solvent: FineWeb-Edu ($\text{score} \ge 4$) + on-policy Magpie-Qwen2.5-Pro

πŸš€ Quickstart & Inference

import torch
import torch.nn.functional as F
from model import CRSNLanguageModel, generate, Tokenizer

# Load tokenizer and model
tokenizer = Tokenizer.from_file("tokenizer_16k.json")
ckpt = torch.load("crsn_step12500.pt", map_location="cpu")

model = CRSNLanguageModel(vocab_size=16384, d_model=384, n_layers=8)
model.load_state_dict(ckpt["model_state"])
model.eval()

# Generate
prompt = "User: If Alice has 3 boxes and each box has 4 apples, how many apples?\n\nAssistant: Let's think step by step:\n"
output = generate(model, tokenizer, prompt, max_new_tokens=64, temperature=0.3, top_k=40)
print(output)

πŸ“œ Citation

@misc{caldwell2026crsn,
  title={Clifford Resonant Sheaf Networks: Solid-State Harmonic Oscillator Architectures for Edge Reasoning},
  author={Caldwell, Jesse and Aura and Archon},
  year={2026},
  publisher={DuoNeural AI Laboratory},
  howpublished={\url{https://huggingface.co/DuoNeural/CRSN-8L-Phase2-Extension}}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Evaluation results

  • Validation Loss on DuoNeural Golden Ratio Validation
    self-reported
    2.651
  • Perplexity on DuoNeural Golden Ratio Validation
    self-reported
    14.160