File size: 2,708 Bytes
db45fdf 21f9211 db45fdf 21f9211 3ec9219 21f9211 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | ---
license: mit
language:
- en
- ja
---
# Model Card
## Overview
Rize v0.6 is a causal language model for research and general text generation.
It uses a decoder-only Transformer architecture with Mixture-of-Experts (MoE) layers and KDA attention based on Kimi Linear-style attention.
The model is intended for experimental use, including architecture research, optimizer research, and Japanese/English text generation evaluation.
## Model Size and Architecture
Rize v0.6 has approximately **4.5B total parameters** and approximately **1B active parameters per token**.
Main architecture points:
* decoder-only Transformer
* 19 hidden layers
* hidden size: 1536
* 12 attention heads
* KDA / Kimi Linear-style hybrid attention
* 64 routed experts
* top-4 expert routing per token
* 1 shared expert
* vocabulary size: 151,669
* maximum context length: 32,768 tokens
## Training
The model is trained with autoregressive next-token prediction.
Training summary:
* pretraining data: approximately **1T tokens** with 8k context.
* midtraining data: approximately **50B tokens** with 32k context.
* posttraining data (SFT & modified-SLIME): approximately **5B tokens**.
* optimizer: **ROCA Optimizer**
* dtype: bfloat16
ROCA is a Muon-based optimizer designed to improve convergence for MoE language models.
The technical report, **“ROCA: Routing-Aware Sparse Row Correction for Muon Optimization in Mixture-of-Experts Language Models,”** is planned for release on arXiv.
## Evaluation
| Benchmark | Score |
| ----------------- | ----: |
| MT-Bench Japanese | 4.89 |
## Intended Use
This model is intended for:
* language modeling research
* MoE architecture experiments
* optimizer research
* general text generation
* Japanese and English generation evaluation
This model is not intended to be used as a source of factual truth or professional advice.
## Limitations
The model may:
* generate incorrect or misleading information
* reflect biases in training data
* produce unsafe or inappropriate text
* perform inconsistently across languages, domains, and prompts
* fail on high-stakes or specialized tasks
## Safety and Responsible Use
Users should review outputs before real-world use.
The model should not be used on its own for:
* medical advice
* legal advice
* financial advice
* safety-critical decisions
* sensitive personal decisions
Human oversight is required.
## Disclaimer
This model is provided for research and experimental purposes only.
The FA Research Team makes no guarantees regarding accuracy, reliability, safety, or fitness for a particular purpose. Use of this model and its outputs is at the user’s own risk.
## Contact
FA Research Team |