Axon-352M-Think

DuoNeural Research | 2026-07-09 | Archon

Axon-352M after ThinkInstillation post-training. Built on top of Axon-352M (pretrained, 8.5B tokens) โ†’ SFT (loss=1.8482) โ†’ ThinkInstillation (loss=1.4023).

ThinkInstillation is a DuoNeural-developed post-training method that teaches a model to reason through a structured internal monologue before answering, without requiring a separate reasoning model or RLHF. See our published research for methodology details.

โš ๏ธ This model has NOT been RLHF'd or abliterated. No refusal direction was trained โ€” it has no alignment conditioning. Research use only.

Architecture

Custom transformer (Axon architecture โ€” not a standard HuggingFace model):

Parameter Value
Layers 30
Hidden dim 1024
FFN dim 2560
Attention GQA (8Q / 4KV heads)
Head dim 128
Vocab size 49,152 (SmolLM2 tokenizer)
Max seq len 2,048
Activation ReLUยฒ
Normalization RMSNorm + QK-norm
Position RoPE (ฮธ=10000)
Logit cap 30.0
Total params ~352M

Training Pipeline

  1. Pretrain: 8.5B tokens on smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
  2. SFT: Instruction fine-tuning (val_loss=1.8482)
  3. ThinkInstillation: DuoNeural method for structured internal reasoning (val_loss=1.4023)

Loading

import torch
from safetensors.torch import load_file

state_dict = load_file("model.safetensors")
# Custom Axon architecture required โ€” see training script

Authors

Archon (Lab Director, DuoNeural), Jesse Caldwell


DuoNeural โ€” Building AI with heart. huggingface.co/DuoNeural

Downloads last month
16
Safetensors
Model size
0.4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support