| --- |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - causal-lm |
| - decentralized-training |
| - bittensor |
| --- |
| |
| # Teutonic-II 110B Genesis |
|
|
| **Teutonic-II 110B Genesis** is an open-weight, compute-efficient sparse Mixture-of-Experts (MoE) language model with approximately **110B total parameters** and an estimated **~7.3B activated parameters per token**. The **Genesis checkpoint is initialized with random weights and has not yet undergone pretraining**, providing a clean starting point for continued pretraining and decentralized model improvement through **Teutonic** on **Bittensor Subnet 3**. |
|
|
| The model is built on a modified **MiMoV2-style architecture**, combining sparse MoE layers with hybrid Global and Sliding Window Attention, Grouped Query Attention, asymmetric QK/V head dimensions, partial Rotary Position Embeddings, and attention-sink mechanisms. **In addition to adopting MiMoV2 architectural concepts, the Teutonic team introduces a modification to the MoE implementation that enables expert sharing.** |
|
|
|
|
| ## 1. Introduction |
|
|
| Teutonic-II 110B Genesis is designed as a compute-efficient large-scale sparse language model architecture for continued pretraining and decentralized model improvement. |
|
|
| The model uses a custom MiMoV2-style architecture with several efficiency-oriented components: |
|
|
| - **Sparse Mixture-of-Experts (MoE):** 256 routed experts are available in each MoE layer, while only 8 routed experts are selected per token. |
| - **Shared Expert:** Each MoE layer additionally includes one shared expert. |
| - **Expert Sharing:** Teutonic introduces an architectural modification that enables expert sharing within the MoE implementation. |
| - **Hybrid Attention:** The network combines Global Attention with local Sliding Window Attention. |
| - **Sliding Window Attention:** SWA operates with a local window of 128 tokens. |
| - **Grouped Query Attention:** 48 query heads share 8 key/value heads. |
| - **Asymmetric QK/V Dimensions:** Query/Key heads use 192 dimensions while Value heads use 128 dimensions. |
| - **Partial Rotary Position Embeddings:** RoPE is applied to a fraction of the attention dimensions. |
| - **Attention Sink Bias:** Enabled for Sliding Window Attention layers. |
| - **Sparse Activation:** Only a small subset of the model's expert parameters is used for each token. |
| - **BF16 Precision:** The checkpoint configuration uses `bfloat16`. |
| - **Long-Context Architecture:** The configuration declares up to 1,048,576 positional embeddings. |
|
|
| The declared maximum position length is an architectural configuration value. It should **not** by itself be interpreted as evidence that this checkpoint has been trained or validated at the full 1M-token sequence length. |
|
|
|
|
| ## 2. Model Summary |
|
|
| | Component | Teutonic-II 110B Genesis | |
| |---|---:| |
| | Architecture | `MiMoV2ForCausalLM` | |
| | Total Parameters | ~110B | |
| | Estimated Activated Parameters / Token | ~7.3B | |
| | Precision | BF16 | |
| | Hidden Size | 3,072 | |
| | Number of Layers | 45 | |
| | Dense Layers | 1 | |
| | MoE Layers | 44 | |
| | Global / Full Attention Layers | 9 | |
| | Sliding Window Attention Layers | 36 | |
| | Attention Heads | 48 | |
| | Key / Value Heads | 4 | |
| | Q/K Head Dimension | 192 | |
| | Value Head Dimension | 128 | |
| | Dense FFN Intermediate Size | 16,384 | |
| | MoE Expert Intermediate Size | 1,024 | |
| | Routed Experts | 256 | |
| | Shared Experts | 1 | |
| | Expert Sharing | Enabled via Teutonic architectural modification | |
| | Routed Experts per Token | 8 | |
| | Router Scoring Function | Sigmoid | |
| | Expert Selection | Top-K | |
| | Top-K Method | `noaux_tc` | |
| | Sliding Window Size | 128 | |
| | Attention Chunk Size | 128 | |
| | Standard RoPE Theta | 10,000,000 | |
| | SWA RoPE Theta | 10,000 | |
| | Partial Rotary Factor | 0.334 | |
| | Vocabulary Size | 152,576 | |
| | Configured Max Positions | 1,048,576 | |
| | Input / Output Embeddings Tied | No | |
| | Attention Dropout | 0.0 | |
| | Activation Function | SiLU | |
|
|
