okolxc/openmythos-2b-edge

OpenMythos-2B — Edge (SFT)

Qwen3.5-2B weights running inside the OpenMythos architecture (hybrid linear attention + GQA + MoE-ready blocks), converted 1:1 and then instruction-tuned on UltraChat-200k for 500 steps on a single Tesla T4.

The SFT pass closed and reversed the conversion gap: the fine-tuned Mythos model is now less perplex than the original Qwen on the same held-out data.

Base model	Qwen/Qwen3.5-2B
Params	2.393 B (bf16)
Layers	24 (18 linear-attention + 6 full-attention), 11 + 1 recurrent + 12
Attention	hybrid: Gated DeltaNet (linear) + GQA full-attention
Context	248 320 vocab, RoPE θ=1e7, 256k native positions
Weights	5.80 GB, 4 safetensors shards
Training	500 steps, UltraChat-200k, lr 1e-5 cosine, seq 192, Adafactor

📊 Verified results (Tesla T4, bf16)

Identical held-out set: 69 UltraChat-200k test samples + 5 general-text samples = 12 313 tokens, scored with the same tokenizer and the same code path for every model.

Qwen3.5-2B   (original)         held-out ppl =  7.9467   (1.00x)
OpenMythos-2B (converted)        held-out ppl = 11.8564   (1.49x)   <- before SFT
OpenMythos-2B (SFT, this repo)  held-out ppl =  6.2327   (0.78x)   <- after SFT

The conversion alone cost 1.49x perplexity; 500 steps of SFT on UltraChat recovered it and overshot, leaving the Mythos model 22% better than the Qwen it was derived from on this data.

📦 Files

file	what
model-00001..00004-of-00004.safetensors	the model (5.80 GB, bf16, 335 tensors)
model.safetensors.index.json	shard index
config.json	architecture metadata + SFT hyperparameters
sft_gqa_ultrachat_step500.safetensors	the same weights as a single file
scratch_2b/	earlier from-scratch checkpoints (different, smaller architecture) — see its README

Load with the OpenMythos architecture (see config.json for the layer map); the weights are a 1:1 copy of the converted Qwen3.5-2B tensors.

🏗️ Architecture

Embed → [11 × TransformerBlock] → [RecurrentBlock] → [12 × TransformerBlock] → RMSNorm → tied Head
                  │                    │
                  │                    └─ ACT / LoRA recurrent reasoning block
                  └─ hybrid attention: Gated DeltaNet + GQA full-attention

Each TransformerBlock = pre-norm attention + SwiGLU feed-forward, with optional Mixture-of-Experts, Mixture-of-Depths routing, Multi-Token-Prediction head, sparse-attention (DSA) and long-term KV memory available as fine-tuning targets.

✅ Conversion fidelity

Component	Transfer
Gated DeltaNet attention (18 layers)	direct copy — module parity 0.43 %
GQA full attention (6 layers)	direct copy — layer parity cos 0.996
SwiGLU feed-forward (6144)	direct copy
Norms (zero-centred RMSNorm)	direct copy
Embedding / output head	direct copy (tied)

load_state_dict on the promoted checkpoint reports 0 missing / 0 unexpected tensors.

Post-hoc compression of K and V into a shared MLA latent was measured too (SVD energy retention 86–98 %) and found lossy, so the converted checkpoint keeps per-head K/V; the compressed-latent attention path is kept in the project for training from scratch.

📜 Attribution

Weights are derived from Qwen3.5-2B (Apache-2.0). Please respect the upstream license and keep the attribution when redistributing.