Zenyx-V3-SFT (1.58B mixture-of-experts, chat / instruction tuned)

The supervised fine-tune of Zenyx-V3-Base: a 1.58B-parameter mixture-of-experts language model (about 0.38B parameters active per token) written from scratch in JAX / Flax and trained on TPU v5e-8. The model was not benchmarked (the full evaluation suite is too slow on the hardware available to the author); for the numbers of the underlying base model see its card. Expect a small research model: it follows simple instructions, but it makes up facts and has no safety tuning.

At a glance

Parameters	1.58B total, about 0.38B active per token
Architecture	16 blocks of width 1,536: the first 2 with a dense feed-forward layer, the other 14 with a mixture of experts (12 routed experts, 2 active per token, plus 1 shared expert, expert width 1,408); multi-query attention (12 query heads of size 128 sharing one key/value head, low-rank queries of rank 384 and a grouped low-rank output projection); 3-stream hyper-connections; tied input/output embedding; trained at context lengths up to 4,096 tokens
Vocabulary	129,280 (the DeepSeek-V4 tokenizer; tokenizer/)
Weights	model.safetensors, bfloat16 (router gates in float32 at load time), from SFT step 10,000
Code	modeling_zenyx_v3.py (inference: KV cache, jit-compiled decoding), config.json
Base model	Zenyx-V3-Base, pre-training checkpoint at step 98,000 (that checkpoint is no longer in the base repository; the nearest kept one is step 100,000)
Fine-tuning	10,000 steps, 5.24B tokens processed (1,280,000 sequences), loss only on assistant tokens (content, tool calls and the closing end-of-sentence token); Muon for the matrices, AdamW for embeddings and gates

Chat format

A plain-text template (no new special tokens): turns are SYSTEM:, USER: and ASSISTANT: blocks separated by blank lines, and every assistant reply ends with the end-of-sentence token. If no system message is given, the default is "You are Zenyx, a helpful, honest and precise AI assistant. ...". The tokenizer carries the template:

SYSTEM: You are Zenyx, a helpful, honest and precise AI assistant. ...

USER: What is the capital of France?

ASSISTANT:

Reasoning answers may open with a <think> ... </think> block, as in the training data.

Usage

pip install jax flax safetensors transformers huggingface_hub

from huggingface_hub import hf_hub_download
hf_hub_download("Arko007/Zenyx-V3-SFT", "modeling_zenyx_v3.py", local_dir=".")      # the model code

from modeling_zenyx_v3 import ZenyxGenerator
gen = ZenyxGenerator("Arko007/Zenyx-V3-SFT")                  # downloads model.safetensors (3.2 GB) and the tokenizer; the first call compiles for a few minutes

reply = gen.chat([{"role": "user", "content": "Give me three tips for learning Python."}],
                 max_new_tokens=150, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.1)
print(reply)

# multi-turn, with a system message
messages = [{"role": "system", "content": "You answer in one short sentence."},
            {"role": "user", "content": "Why is the sky blue?"}]
print(gen.chat(messages, max_new_tokens=60))

The code runs on CPU, GPU or TPU (a GPU or TPU is strongly recommended; it needs about 8 GB of memory for the weights and caches). The decoding functions are compiled for fixed shapes, so the first request is slow and later ones are fast.

Replies produced by exactly this code (temperature 0.7, one sample each, on a CPU; a base-sized research model, so expect errors):

Give me three tips for learning Python.

Okay, let’s tackle this question about teaching programming in python using examples from various sources.

Here is an example of how to teach your computer through code:

Example:
In Python, you can create loops by defining functions that iterate over data points (e.g

Why is the sky blue?

Hmm, this query has a series of unrelated questions about optics and color. The user seems to be testing comprehension or attention-grabbing phrasing - perhaps they're looking for factual information on how light behaves differently from other colors?

I notice several potential pitfalls here:

• Confusing

Limitations

• Not benchmarked and not safety-tuned. Part of the fine-tuning data deliberately includes unfiltered assistant responses (the "rejected" side of Anthropic's HH-RLHF as re-packaged in Arko007/hh-rlhf-sft-rejected), so the model has no reliable refusal behaviour and can produce harmful or biased text. Do not deploy it to end users without your own guardrails.
• It hallucinates facts, is weak at arithmetic and long reasoning, and was trained on English.
• Small model, short fine-tune: do not rely on it for factual, medical, legal or financial answers.

Training data

The training script reads these sources (check each one's own terms; they are not redistributed here): allenai/tulu-3-sft-mixture (the backbone), lmsys/lmsys-chat-1m, Arko007/hh-rlhf-sft-rejected, the author's synthetic set Arko007/zenyx-v3-synthetic-sft-v4, MoreThought/Fable-5.1-Max-Reasoning-Filtered-5000x, saidutta69/fable-5-premium and Crownelius/High-Coder-Reasoning-Multi-Turn.

License

The weights and code are released by the author under the Apache License 2.0, but the fine-tuning data comes from sources with their own licences and terms (listed above, among them the LMSYS-Chat-1M agreement and the Tulu-3 mixture's mixed licences). Check those terms before any commercial use; the license: other label of this card reflects that.

Repository contents

Path	Contents
model.safetensors	the final SFT weights (step 10,000)
modeling_zenyx_v3.py, config.json	model code and hyper-parameters
tokenizer/	tokenizer files and the chat template

The intermediate checkpoints (every 200 steps, with optimiser states) and the XLA compile caches were removed from this repository to keep it small.