File size: 565 Bytes
fe7c66d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | ---
license: apache-2.0
---
# Foundry-4
Kernel-specialized hybrid LM: 5.02B total / ~0.58B active params.
18 GDN (gated DeltaNet, fla) + 6 RoPE attention layers, 384-expert
sigmoid-routed dropless MoE (top-8, packed Triton segmm), KernelGram
n-gram tables, 4-iteration shared-weight latent ponder loop, JEV-style
judge head.
Trained 2000 steps on a curated kernel-optimization corpus
(CUDA/Triton episodes with measured RESULT/BEST/HISTORY feedback) plus
GPU-verified synthetic optimization ladders (synth4.py).
Code: model4.py / gdn.py / segmm.py in this repo.
|