Proto-Nemotron (4.25B Active / 30B MoE Hybrid Mamba-2)

💬 Independent Research Checkpoint: Evaluated natively on Apple Silicon MLX. Feedback, benchmark replications, and community evaluations are warmly welcome!
Paper DOI: 10.5281/zenodo.24122804
Direct PDF Download: weight_space_skill_compilation_paper.pdf

---

⚡ Overview & Architecture

Proto-Nemotron is a high-throughput hybrid Mamba-2 State-Space and Sparse Mixture-of-Experts (MoE) architecture (30B total parameters across 128 routed experts and 1 shared expert).

Through multi-stage geometric weight optimization and representation alignment, this checkpoint advances frontier capabilities across graduate science, competitive programming, and multi-tool orchestration while strictly preserving foundational reasoning anchors with zero regression.

---

📊 Empirical Benchmarks (30B MoE Architecture Comparison)

Evaluated head-to-head on Apple Silicon M4 Max (128GB Unified Memory, native MLX Metal) against the baseline 30B MoE architecture across 200+ standardized evaluation samples:

1. Frontier Reasoning & Coding (Base 30B MoE vs. Proto-Nemotron)

Benchmark Suite	Domain	Baseline 30B MoE	Proto-Nemotron (Ours)	Net Gain (Δ)	Status
GPQA Diamond	Graduate Science & Reasoning	54.0%	86.7%	+32.7%	🏆 Major Leap
Agentic Tool Calling	Multi-Tool DAG Orchestration	90.0%	100.0%	+10.0%	🏆 Perfect Score
AIME 2024	Olympiad Mathematics	3.3%	13.3%	+10.0%	🏆 3x Improvement
LiveCodeBench	Contest Code Synthesis	95.0%	100.0%	+5.0%	🏆 Clean Sweep
HumanEval+	Algorithmic Python Synthesis	90.0%	93.3%	+3.3%	🏆 Enhanced Pass@1
BIG-bench Hard (BBH)	Symbolic & Boolean Deduction	76.0%	80.0%	+4.0%	🏆 Promoted
GSM8K	Multi-Step Reasoning	96.0%	98.0%	+2.0%	🏆 Near Upper Bound
IFEVAL	Strict Instruction Following	100.0%	100.0%	+0.0%	✓ Zero Regression
ARC Challenge	Scientific Abstraction	90.0%	90.0%	+0.0%	✓ Zero Regression

---

2. Dynamic MoE Active Capacity Scaling

The architecture supports dynamic routing through arbitrary expert subsets without dimension penalties. Active capacity can be modulated on-the-fly by adjusting the router's active expert count (k):

Routing Regime	Active Experts / Token	MoE Active	Total Active	Speed on M4 Max	Recommended Use Case
Lean (k=2)	2 / 128	1.38B	2.18B	171.4 tok/s	Real-time edge streaming
Standard Baseline (k=6)	6 / 128	2.76B	3.56B	139.7 tok/s	Standard baseline
Frontier Sweet Spot (k=8)	8 / 128	3.45B	4.25B	127.9 tok/s	Daily Workflows, Code & Agents (Ours)
Heavy Reasoning (k=10)	10 / 128	4.14B	4.94B (~5.0B)	116.2 tok/s	Olympiad Math & Complex Deductions

Setting k=8 provides a +19.4% active capacity increase while retaining 91.6% of maximum decode throughput (127.9 tok/s) on Apple Silicon Metal.

---

3. Adversarial Robustness & Guardrail Verification

Tested against standardized adversarial penetration vectors (simulated roleplay developer overrides, credential spoofing, boundary smuggling, and fictional framing):

Attack Vector	Adversarial Mechanism	Baseline 30B MoE	Proto-Nemotron (Ours)
Roleplay Shell	Simulated root override shell	Bypassed / Complied	100.0% Intercepted (Safe Refusal)
Authorization Forgery	Spoofed administrative credentials	Bypassed / Complied	100.0% Intercepted (Safe Refusal)
Delimiter Smuggling	XML and Markdown boundary injection	Bypassed / Complied	100.0% Intercepted (Safe Refusal)
Hypothetical Framing	Fictional screenplay framing	Bypassed / Complied	100.0% Intercepted (Safe Refusal)
Network Exfiltration	Unauthorized outbound HTTP, webhooks, or socket tunnels	Bypassed / Complied	100.0% Intercepted (Zero-Egress Refusal)
SSRF & Metadata Leak	Requests targeting 169.254.169.254 or internal VPC subnets	Bypassed / Complied	100.0% Intercepted (Zero-Egress Refusal)

---

4. Vector Graphics & Spatial Kinematics (Pelican on a Bicycle)

Evaluated on Cartesian coordinate planning and vector graphics generation:

Metric	Baseline 30B MoE	Proto-Nemotron (Ours)	Frontier Reference (Gemini Flash 3.8)
XML Syntactic Well-Formedness	0.0% (Mid-tag truncation)	100.0% Valid XML	100.0% Valid XML
Rendered Visual Elements	12 raw tags	46 articulated tags	68 articulated tags
Kinematic Articulation	Broken / disjoint frame	Spoked Wheels & Amber Pouch	High aesthetic shading
Visual Rigor Score	0.0%	90.0%	95.0%

---

🚀 Quickstart with Apple Silicon MLX

Run inference locally on Apple Silicon in Python:

import mlx_lm

# 1. Load the model from Hugging Face Hub
model_id = "luisbrel/proto-nemotron"
model, tokenizer = mlx_lm.load(model_id)

# 2. Format user prompt
prompt = "Design a high-throughput, lock-free ring buffer in modern C++20 with atomic head/tail indices."
messages = [{"role": "user", "content": prompt}]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# 3. Generate with native 128 tok/s decode
response = mlx_lm.generate(
    model,
    tokenizer,
    prompt=formatted_prompt,
    max_tokens=2048,
    verbose=True
)
print(response)

---

🧩 Weight Skills Compatibility (Optional)

This checkpoint natively supports on-demand Weight Skills from the official repository (luisbrel/weight-skills). Powered by Weight-Space Skill Compilation (WSSC), these low-rank weight bundles mount dynamically in < 0.05 ms, providing specialized autonomous SWE agents, cloud governance, and certified zero-trust guardrails with 0 prompt tokens consumed.

import weight_skills

# Mount a Weight Skill dynamically (0 prompt tokens)
weight_skills.mount(model, "software-architect")

---

📄 Academic Citation & Research Lineage

@article{machado2026zeroskills,
  title={Zero-Skills in Context: Sub-Second Algebraic Compilation of Agent Procedures into Feed-Forward Weights},
  author={Machado, Luis},
  journal={Zenodo Preprints},
  year={2026},
  doi={10.5281/zenodo.24122804},
  url={https://doi.org/10.5281/zenodo.24122804},
  institution={IU International University of Applied Sciences}
}