Shoonya v0.4.0 — Hybrid Code Language Model with Rust Mechanistic Telemetry & Execution-Guided Evaluation

Shoonya v0.4.0 is a research-driven autoregressive language model combining a Pre-RMSNorm Transformer + Grouped-Query Attention (GQA) + SwiGLU + RoPE/NTK/YaRN + SparseMoE/Mamba hybrid architecture with a 5-crate Rust systems engine (shoonya-core, shoonya-telemetry, shoonya-eval, shoonya-ffi, shoonya-cli).

What's New in v0.4.0

1. First-Principles Causal & Positional Integrity:
   • Strictly causal Multi-Head / Grouped-Query Attention (is_causal=True SDPA fast-path) and causal left-only Sliding Window Attention.
   • Per-head [1, 1, seq_len, head_dim] Rotary Position Embeddings (RoPE), NTK-aware scaling, YaRN, and ALiBi support.
   • Pre-RMSNorm residual stream with SwiGLU feed-forward blocks and per-token symmetric INT8 / asymmetric INT4 KV-cache quantization.
2. Code-Specialized Tokenization (CodeTokenizer, vocab_size = 50,264):
   • Extends GPT-2 BPE with disjoint control tokens (<|pad|>=50257, <|fim_prefix|>=50258, <|fim_suffix|>=50259, <|fim_middle|>=50260) and atomic Python indentation tokens (<|indent_4|>=50261, <|indent_8|>=50262, <|indent_12|>=50263).
   • Compresses Python function token sequences by 37.9% (169 -> 105 tokens on binary_search) and eliminates IndentationErrors (0 / 32 across evaluation runs).
3. Rust Core (shoonya-core) & Mechanistic Telemetry (shoonya-telemetry):
   • Rust FFD Sequence Packer & FIM: Best-Fit Decreasing sequence packing and Fill-in-the-Middle (PSM/SPM) transforms.
   • Scope-Aware CTRL Repetition Penalty: Vectorized Top-K/Top-P/Min-P sampling in Rust that exempts prompt signature tokens from repetition penalty so parameter identifiers (arr, target) can be freely reused in function bodies.
   • Closed-Loop Spectral Regularization: Live Singular Value Decomposition (SVD) monitoring of Heavy-Tailed Self-Regularization (HTSR) power-law exponent $\alpha$ and Roy-Vetterli Effective Rank ($e\text{Rank}$), dynamically adjusting AdamW weight decay to maintain $\alpha \in [1.85, 2.5]$.
4. Execution-Guided Code Evaluation (shoonya-eval):
   • Parallel Rust sandbox executing generated Python and Rust (rustc) functions against unit-test assertions with static AST Cyclomatic Complexity, Halstead Volume, and AST token-role surprisal decomposition.

Benchmark Results (v0.4.0 small Checkpoint — 16,805,376 Parameters)

Metric	Value
Mean pass@1 (Execution-Guided Sandbox)	65.6%
Mean pass@2 (Execution-Guided Sandbox)	79.2%
Mean pass^2 (Multi-Trial Reliability)	55.5%
binary_search (pass@1 / pass@2)	75.0% / 100.0%
Unseen AST-Renamed Identifier (locate_in_sorted pass@1)	100.0%
IndentationError Rate	0.0% (0 / 32)
HTSR Spectral $\alpha$ / Effective Rank Ratio	1.87 (Ideal_HeavyTailed_SOTA) / 75.1%
Compute-Normalized Pareto Score	230.9

Model Configuration (config.json)

• vocab_size: 50264
• hidden_size: 256
• num_hidden_layers: 4
• num_attention_heads: 4
• num_key_value_heads: 2
• intermediate_size: 1024
• use_rms_norm: True
• pre_norm: True
• use_swiglu: True
• position_embedding_type: "rotary"

Quick Usage

import torch
from transformers import GPT2Tokenizer
from model.transformer import TransformerLM, ModelConfig
from data.code_dataset import CodeTokenizer

ckpt = torch.load("shoonya_v0_4_code_small.pt", map_location="cpu", weights_only=False)
config = ckpt["config"]
model = TransformerLM(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

tokenizer = CodeTokenizer(GPT2Tokenizer.from_pretrained("gpt2"))
prompt = "def binary_search(arr, target):\n"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(
        input_ids,
        max_length=96,
        temperature=0.2,
        top_k=10,
        repetition_penalty=1.05,
        eos_token_id=tokenizer.eos_token_id,
        exempt_prompt_tokens=True,
    )
print(tokenizer.decode(output_ids[0].tolist()))