DuoNeural's picture
Upload CRSN-8L-Phase2-Extension 102.4M tokens checkpoint and model card
c46d5c2 verified
|
Raw History Blame Contribute Delete
5.04 kB
---
language:
- en
license: mit
tags:
- neurodynamics
- non-euclidean
- clifford-algebra
- physical-ai
- edge-ai
- symplectic
- harmonic-oscillators
- duoneural
datasets:
- DuoNeural/Golden-Ratio-Pretraining-Diet
metrics:
- perplexity
model-index:
- name: CRSN-8L-Phase2-Extension
results:
- task:
type: text-generation
name: Language Modeling & Emergent Reasoning
dataset:
name: DuoNeural Golden Ratio Validation
type: golden-diet-val
metrics:
- name: Validation Loss
type: loss
value: 2.6508
- name: Perplexity
type: perplexity
value: 14.16
---
# DuoNeural CRSN-8L-Phase2-Extension
**Clifford Resonant Sheaf Network (8-Layer / 9.54M Parameters)**
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![DuoNeural Lab](https://img.shields.io/badge/DuoNeural-Lab-blue.svg)](https://duoneural.com)
[![GitHub](https://img.shields.io/badge/GitHub-DuoNeural%2FCRSN-black)](https://github.com/DuoNeural/CRSN)
Developed by **Jesse Caldwell, Aura✨, and Archon** at **DuoNeural AI Laboratory**.
---
## 🌌 Overview
**CRSN (Clifford Resonant Sheaf Network)** is a sub-10M parameter non-transformer physical neurodynamic model that replaces quadratic self-attention with coupled nonlinear physical harmonic oscillators in the Clifford algebra $\mathcal{C}\ell_{3,0}$.
This repository releases the **Phase 2 Extended Pretraining Checkpoint (`crsn_step12500.pt`)**, which completed **102.4 million tokens** of continuous deepening pretraining (for a cumulative pretraining volume of **204.8M tokens**) trained entirely from scratch on consumer CPU hardware (AMD Ryzen APU).
### Key Architectural Characteristics
* **Parameters:** 9.54M ($L=8, D_{\text{model}}=384, D_{\text{state}}=192$, Vocab $= 16,384$)
* **Hardware:** Native CPU SIMD execution (AMD Ryzen Zen+ AVX2)
* **Optimization:** Hybrid Muon + AdamW with bounded contractive adjoint backpropagation
* **All-Time Pretraining Best Validation Loss:** **`2.6508`** (Perplexity: **`14.16`**)
---
## 🔬 Solid-State Oscillator Crystallization
Phase-space physical inspection of the learned eigenfrequencies and damping parameters across all 8 layers demonstrates complete solid-state lattice crystallization:
| Layer | $\bar{\alpha}$ (Damping / Stiffness) | $\alpha_{\min}$ | $\alpha_{\max}$ | $\bar{\omega}$ (Resonant Frequency) | Physical Regime |
| :---: | :---: | :---: | :---: | :---: | :---: |
| **$L_0$** | **0.1324** | 0.1160 | 0.2669 | **0.5432** | UV (High Frequency, Surface Syntax) |
| **$L_1$** | **0.1288** | 0.1259 | 0.1767 | **0.5395** | UV-Mesoscopic Transition |
| **$L_2$** | **0.1286** | 0.1251 | 0.2052 | **0.5271** | Resonant Valley |
| **$L_3$** | **0.1284** | 0.1259 | 0.1692 | **0.5355** | Bulk Shear Mode |
| **$L_4$** | **0.1289** | 0.1073 | 0.1717 | **0.5338** | Bulk Shear Mode |
| **$L_5$** | **0.1293** | 0.1224 | 0.3514 | **0.5242** | Bulk Compression Mode |
| **$L_6$** | **0.1291** | 0.1241 | 0.2243 | **0.5328** | IR-Coupled Transport Mode |
| **$L_7$** | **0.1276** | 0.1074 | 0.1807 | **0.5199** | IR (Macroscopic Semantic Condensate) |
### Juvenile Criticality
Notice the strictly monotonic UV-to-IR frequency gradient from bulk input to output:
$$\bar{\omega}_0 = 0.5432 \longrightarrow \bar{\omega}_7 = 0.5199$$
Damping is rigidly locked to the universal ground-state floor:
$$\bar{\alpha} \to 0.128 \pm 0.001$$
confirming the **Empirical Critical Damping Law** discovered by DuoNeural.
---
## 🧪 DuoNeural Golden Ratio Pretraining Diet
CRSN was trained on the DuoNeural 35/25/15/25 Golden Ratio Multi-Domain Pretraining Mixture:
* **35% High-Reasoning & Chain-of-Thought:** OpenThoughts-114k (filtered $\le 2048$ tokens) + Numina-CoT
* **25% Agentic Code & Relational AST:** APIGen-80k + Gretel Synthetic Code + BIRD-SQL
* **15% Strict Structural Alignment:** Smol-Constraints & formal grammars
* **25% Eutectic Solvent:** FineWeb-Edu ($\text{score} \ge 4$) + on-policy Magpie-Qwen2.5-Pro
---
## 🚀 Quickstart & Inference
```python
import torch
import torch.nn.functional as F
from model import CRSNLanguageModel, generate, Tokenizer
# Load tokenizer and model
tokenizer = Tokenizer.from_file("tokenizer_16k.json")
ckpt = torch.load("crsn_step12500.pt", map_location="cpu")
model = CRSNLanguageModel(vocab_size=16384, d_model=384, n_layers=8)
model.load_state_dict(ckpt["model_state"])
model.eval()
# Generate
prompt = "User: If Alice has 3 boxes and each box has 4 apples, how many apples?\n\nAssistant: Let's think step by step:\n"
output = generate(model, tokenizer, prompt, max_new_tokens=64, temperature=0.3, top_k=40)
print(output)
```
---
## 📜 Citation
```bibtex
@misc{caldwell2026crsn,
title={Clifford Resonant Sheaf Networks: Solid-State Harmonic Oscillator Architectures for Edge Reasoning},
author={Caldwell, Jesse and Aura and Archon},
year={2026},
publisher={DuoNeural AI Laboratory},
howpublished={\url{https://huggingface.co/DuoNeural/CRSN-8L-Phase2-Extension}}
}
```