---
language:
- en
- ru
license: apache-2.0
tags:
- moe
- mixture-of-experts
- causal-lm
- pytorch
- safetensors
- text-generation
- lightweight
pipeline_tag: text-generation
---

# Sonexa-Beta

**Sonexa-Beta** is a lightweight, bilingual (English / Russian) language model powered by a custom **Mixture of Experts (MoE)** architecture. Built from scratch for ultra-low latency inference, edge devices, and consumer hardware.

* **Developer / Organization:** Sonexa Artificial Intelligence
* **Model Architecture:** Causal Language Model with Mixture of Experts (MoE)
* **License:** Apache-2.0

---

## Architectural Specifications

The following metrics are dynamically extracted directly from the model weights:

| Metric | Measured Value |
| :--- | :--- |
| **Total Parameters** | **140,933,632** (140.93M) |
| **Active Parameters (All-Experts / Soft MoE)** | **140,933,632** (140.93M) |
| **Active Parameters (Top-1 MoE Mode)** | **70,154,752** (70.15M) |
| **Tied Embedding Weights** | 8,192,000 (8.19M) |
| **Attention Layers (GQA)** | 6,558,720 (6.56M) |
| **Shared Expert Capacity** | 23,592,960 (23.59M) |
| **Routed Experts Capacity (4x)** | 94,371,840 (94.37M) |
| **Hidden Size** | 512 |
| **Intermediate Size (SwiGLU)** | 1536 |
| **Attention Mechanism** | Grouped Query Attention (8 Query Heads / 2 KV Heads) |
| **Positional Encoding** | Rotary Position Embeddings (Real Cos/Sin RoPE) |
| **Vocabulary Size** | 16,000 (Byte-Level BPE) |
| **Context Window** | 256 tokens |
| **Memory Footprint (FP16)** | ~268.81 MB VRAM |
| **Release Artifact Size** | 537.64 MB |

---

## Key Characteristics

1. **Bilingual Dialogue:** Trained on a balanced bilingual corpus combining human-curated English dialogues (`HuggingFaceH4/no_robots`) and filtered Russian conversational datasets.
2. **Anti-Refusal Calibration:** Calibrated via output projection damping to eliminate generic AI assistant disclaimers (*'As an AI language model...'*, *'I do not have personal opinions...'*).
3. **Soft MoE Routing:** Concurrent all-expert evaluation dynamic weighting for consistent knowledge routing and response quality.
4. **Extreme Portability:** Fits comfortably in under 300 MB of memory in FP16, allowing fast CPU and consumer GPU execution.

---

## Prompt Template

Sonexa-Beta expects inputs formatted as follows:

```text
<|user|>
Your message here
<|bot|>
Model response here<|endoftext|>
```

---

## Citation

```bibtex
@misc{sonexa2026,
  author = {Sonexa Artificial Intelligence},
  title = {Sonexa-Beta: Lightweight Bilingual Mixture-of-Experts Language Model},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{[https://huggingface.co/](https://huggingface.co/)Cartik/Sonexa-Beta}}
}
```