| --- |
| language: |
| - en |
| - ru |
| license: apache-2.0 |
| tags: |
| - moe |
| - mixture-of-experts |
| - causal-lm |
| - pytorch |
| - safetensors |
| - text-generation |
| - lightweight |
| pipeline_tag: text-generation |
| --- |
| |
| # Sonexa-Beta |
|
|
| **Sonexa-Beta** is a lightweight, bilingual (English / Russian) language model powered by a custom **Mixture of Experts (MoE)** architecture. Built from scratch for ultra-low latency inference, edge devices, and consumer hardware. |
|
|
| * **Developer / Organization:** Sonexa Artificial Intelligence |
| * **Model Architecture:** Causal Language Model with Mixture of Experts (MoE) |
| * **License:** Apache-2.0 |
|
|
| --- |
|
|
| ## Architectural Specifications |
|
|
| The following metrics are dynamically extracted directly from the model weights: |
|
|
| | Metric | Measured Value | |
| | :--- | :--- | |
| | **Total Parameters** | **140,933,632** (140.93M) | |
| | **Active Parameters (All-Experts / Soft MoE)** | **140,933,632** (140.93M) | |
| | **Active Parameters (Top-1 MoE Mode)** | **70,154,752** (70.15M) | |
| | **Tied Embedding Weights** | 8,192,000 (8.19M) | |
| | **Attention Layers (GQA)** | 6,558,720 (6.56M) | |
| | **Shared Expert Capacity** | 23,592,960 (23.59M) | |
| | **Routed Experts Capacity (4x)** | 94,371,840 (94.37M) | |
| | **Hidden Size** | 512 | |
| | **Intermediate Size (SwiGLU)** | 1536 | |
| | **Attention Mechanism** | Grouped Query Attention (8 Query Heads / 2 KV Heads) | |
| | **Positional Encoding** | Rotary Position Embeddings (Real Cos/Sin RoPE) | |
| | **Vocabulary Size** | 16,000 (Byte-Level BPE) | |
| | **Context Window** | 256 tokens | |
| | **Memory Footprint (FP16)** | ~268.81 MB VRAM | |
| | **Release Artifact Size** | 537.64 MB | |
|
|
| --- |
|
|
| ## Key Characteristics |
|
|
| 1. **Bilingual Dialogue:** Trained on a balanced bilingual corpus combining human-curated English dialogues (`HuggingFaceH4/no_robots`) and filtered Russian conversational datasets. |
| 2. **Anti-Refusal Calibration:** Calibrated via output projection damping to eliminate generic AI assistant disclaimers (*'As an AI language model...'*, *'I do not have personal opinions...'*). |
| 3. **Soft MoE Routing:** Concurrent all-expert evaluation dynamic weighting for consistent knowledge routing and response quality. |
| 4. **Extreme Portability:** Fits comfortably in under 300 MB of memory in FP16, allowing fast CPU and consumer GPU execution. |
|
|
| --- |
|
|
| ## Prompt Template |
|
|
| Sonexa-Beta expects inputs formatted as follows: |
|
|
| ```text |
| <|user|> |
| Your message here |
| <|bot|> |
| Model response here<|endoftext|> |
| ``` |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{sonexa2026, |
| author = {Sonexa Artificial Intelligence}, |
| title = {Sonexa-Beta: Lightweight Bilingual Mixture-of-Experts Language Model}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| journal = {Hugging Face Model Hub}, |
| howpublished = {\url{[https://huggingface.co/](https://huggingface.co/)Cartik/Sonexa-Beta}} |
| } |
| ``` |
|
|
