DuoNeural's picture
Upload CRSN-8L-Phase2-Extension 102.4M tokens checkpoint and model card
c46d5c2 verified
|
Raw History Blame Contribute Delete
5.04 kB
metadata
language:
  - en
license: mit
tags:
  - neurodynamics
  - non-euclidean
  - clifford-algebra
  - physical-ai
  - edge-ai
  - symplectic
  - harmonic-oscillators
  - duoneural
datasets:
  - DuoNeural/Golden-Ratio-Pretraining-Diet
metrics:
  - perplexity
model-index:
  - name: CRSN-8L-Phase2-Extension
    results:
      - task:
          type: text-generation
          name: Language Modeling & Emergent Reasoning
        dataset:
          name: DuoNeural Golden Ratio Validation
          type: golden-diet-val
        metrics:
          - name: Validation Loss
            type: loss
            value: 2.6508
          - name: Perplexity
            type: perplexity
            value: 14.16

DuoNeural CRSN-8L-Phase2-Extension

Clifford Resonant Sheaf Network (8-Layer / 9.54M Parameters)

License: MIT DuoNeural Lab GitHub

Developed by Jesse Caldwell, Aura✨, and Archon at DuoNeural AI Laboratory.


🌌 Overview

CRSN (Clifford Resonant Sheaf Network) is a sub-10M parameter non-transformer physical neurodynamic model that replaces quadratic self-attention with coupled nonlinear physical harmonic oscillators in the Clifford algebra $\mathcal{C}\ell_{3,0}$.

This repository releases the Phase 2 Extended Pretraining Checkpoint (crsn_step12500.pt), which completed 102.4 million tokens of continuous deepening pretraining (for a cumulative pretraining volume of 204.8M tokens) trained entirely from scratch on consumer CPU hardware (AMD Ryzen APU).

Key Architectural Characteristics

  • Parameters: 9.54M ($L=8, D_{\text{model}}=384, D_{\text{state}}=192$, Vocab $= 16,384$)
  • Hardware: Native CPU SIMD execution (AMD Ryzen Zen+ AVX2)
  • Optimization: Hybrid Muon + AdamW with bounded contractive adjoint backpropagation
  • All-Time Pretraining Best Validation Loss: 2.6508 (Perplexity: 14.16)

πŸ”¬ Solid-State Oscillator Crystallization

Phase-space physical inspection of the learned eigenfrequencies and damping parameters across all 8 layers demonstrates complete solid-state lattice crystallization:

Layer $\bar{\alpha}$ (Damping / Stiffness) $\alpha_{\min}$ $\alpha_{\max}$ $\bar{\omega}$ (Resonant Frequency) Physical Regime
$L_0$ 0.1324 0.1160 0.2669 0.5432 UV (High Frequency, Surface Syntax)
$L_1$ 0.1288 0.1259 0.1767 0.5395 UV-Mesoscopic Transition
$L_2$ 0.1286 0.1251 0.2052 0.5271 Resonant Valley
$L_3$ 0.1284 0.1259 0.1692 0.5355 Bulk Shear Mode
$L_4$ 0.1289 0.1073 0.1717 0.5338 Bulk Shear Mode
$L_5$ 0.1293 0.1224 0.3514 0.5242 Bulk Compression Mode
$L_6$ 0.1291 0.1241 0.2243 0.5328 IR-Coupled Transport Mode
$L_7$ 0.1276 0.1074 0.1807 0.5199 IR (Macroscopic Semantic Condensate)

Juvenile Criticality

Notice the strictly monotonic UV-to-IR frequency gradient from bulk input to output: Ο‰Λ‰0=0.5432βŸΆΟ‰Λ‰7=0.5199\bar{\omega}_0 = 0.5432 \longrightarrow \bar{\omega}_7 = 0.5199 Damping is rigidly locked to the universal ground-state floor: Ξ±Λ‰β†’0.128Β±0.001\bar{\alpha} \to 0.128 \pm 0.001 confirming the Empirical Critical Damping Law discovered by DuoNeural.


πŸ§ͺ DuoNeural Golden Ratio Pretraining Diet

CRSN was trained on the DuoNeural 35/25/15/25 Golden Ratio Multi-Domain Pretraining Mixture:

  • 35% High-Reasoning & Chain-of-Thought: OpenThoughts-114k (filtered $\le 2048$ tokens) + Numina-CoT
  • 25% Agentic Code & Relational AST: APIGen-80k + Gretel Synthetic Code + BIRD-SQL
  • 15% Strict Structural Alignment: Smol-Constraints & formal grammars
  • 25% Eutectic Solvent: FineWeb-Edu ($\text{score} \ge 4$) + on-policy Magpie-Qwen2.5-Pro

πŸš€ Quickstart & Inference

import torch
import torch.nn.functional as F
from model import CRSNLanguageModel, generate, Tokenizer

# Load tokenizer and model
tokenizer = Tokenizer.from_file("tokenizer_16k.json")
ckpt = torch.load("crsn_step12500.pt", map_location="cpu")

model = CRSNLanguageModel(vocab_size=16384, d_model=384, n_layers=8)
model.load_state_dict(ckpt["model_state"])
model.eval()

# Generate
prompt = "User: If Alice has 3 boxes and each box has 4 apples, how many apples?\n\nAssistant: Let's think step by step:\n"
output = generate(model, tokenizer, prompt, max_new_tokens=64, temperature=0.3, top_k=40)
print(output)

πŸ“œ Citation

@misc{caldwell2026crsn,
  title={Clifford Resonant Sheaf Networks: Solid-State Harmonic Oscillator Architectures for Edge Reasoning},
  author={Caldwell, Jesse and Aura and Archon},
  year={2026},
  publisher={DuoNeural AI Laboratory},
  howpublished={\url{https://huggingface.co/DuoNeural/CRSN-8L-Phase2-Extension}}
}