---
license: mit
language:
- en
- ja
---

# Model Card

## Overview

Rize v0.6 is a causal language model for research and general text generation.

It uses a decoder-only Transformer architecture with Mixture-of-Experts (MoE) layers and KDA attention based on Kimi Linear-style attention. 
The model is intended for experimental use, including architecture research, optimizer research, and Japanese/English text generation evaluation.

## Model Size and Architecture

Rize v0.6 has approximately **4.5B total parameters** and approximately **1B active parameters per token**.

Main architecture points:

* decoder-only Transformer
* 19 hidden layers
* hidden size: 1536
* 12 attention heads
* KDA / Kimi Linear-style hybrid attention
* 64 routed experts
* top-4 expert routing per token
* 1 shared expert
* vocabulary size: 151,669
* maximum context length: 32,768 tokens

## Training

The model is trained with autoregressive next-token prediction.

Training summary:

* pretraining data: approximately **1T tokens** with 8k context.
* midtraining data: approximately **50B tokens** with 32k context.
* posttraining data (SFT & modified-SLIME): approximately **5B tokens**.
* optimizer: **ROCA Optimizer**
* dtype: bfloat16

ROCA is a Muon-based optimizer designed to improve convergence for MoE language models. 
The technical report, **“ROCA: Routing-Aware Sparse Row Correction for Muon Optimization in Mixture-of-Experts Language Models,”** is planned for release on arXiv.

## Evaluation

| Benchmark         | Score |
| ----------------- | ----: |
| MT-Bench Japanese |  4.89 |

## Intended Use

This model is intended for:

* language modeling research
* MoE architecture experiments
* optimizer research
* general text generation
* Japanese and English generation evaluation

This model is not intended to be used as a source of factual truth or professional advice.

## Limitations

The model may:

* generate incorrect or misleading information
* reflect biases in training data
* produce unsafe or inappropriate text
* perform inconsistently across languages, domains, and prompts
* fail on high-stakes or specialized tasks

## Safety and Responsible Use

Users should review outputs before real-world use.

The model should not be used on its own for:

* medical advice
* legal advice
* financial advice
* safety-critical decisions
* sensitive personal decisions

Human oversight is required.

## Disclaimer

This model is provided for research and experimental purposes only.

The FA Research Team makes no guarantees regarding accuracy, reliability, safety, or fitness for a particular purpose. Use of this model and its outputs is at the user’s own risk.

## Contact

FA Research Team