| --- |
| language: en |
| library_name: transformers |
| license: apache-2.0 |
| model_name: project-828-gpt-base |
| pipeline_tag: text-generation |
| tags: |
| - code |
| - moe |
| - mixture-of-experts |
| - flash-attention |
| - custom-architecture |
| --- |
| |
| # project-828-gpt-base |
|
|
| A **398.7M parameter Mixture-of-Experts (MoE)** causal language model optimized for code generation and technical reasoning, pretrained on ~60B tokens. |
|
|
| > **398.7M total parameters, ~286M active per token** — 4 routed experts + 1 shared expert with top-2 routing per token. |
|
|
| ## Model Architecture |
|
|
| | Property | Value | |
| |:---|:---| |
| | **Architecture** | Decoder-only Transformer with MoE FFN layers | |
| | **Total Parameters** | **398.7M** | |
| | **Active Parameters / Token** | **~286M** | |
| | **Hidden Dimension ($d_{\text{model}}$)** | 768 | |
| | **Intermediate Size ($d_{\text{ff}}$)** | 760 | |
| | **Hidden Layers** | 24 | |
| | **Attention Heads / KV Heads** | 12 / 6 (GQA 2:1) | |
| | **Head Dimension** | 64 | |
| | **Routed Experts** | 4 | |
| | **Active Experts / Token** | 2 (Top-2) | |
| | **Shared Experts** | 1 | |
| | **Context Length** | 2048 (extensible to 8192 via YaRN) | |
| | **Vocabulary Size** | 49152 | |
| | **Precision** | BFloat16 Mixed Precision | |
| | **Positional Encoding** | RoPE with YaRN scaling support | |
|
|
| ### Parameter Breakdown |
|
|
| | Component | Parameters | |
| |:---|---:| |
| | Embeddings | ~37.7M | |
| | Unembedding | ~37.8M | |
| | Attention (×24 layers) | ~1.8M each | |
| | MoE FFN (×24 layers) | ~11.7M each | |
| | Layer Norms + Misc | ~0.07M | |
| | **Total** | **398.7M** | |
|
|
| ### Key Architectural Features |
|
|
| - **Grouped Query Attention (GQA)**: 12 query heads with 6 key-value heads (2:1 ratio) for memory-efficient attention |
| - **QK-Norm**: RMSNorm applied to query and key projections before RoPE for attention stability |
| - **Auxiliary-Loss-Free MoE Routing**: Sigmoid gating with dynamic bias adjustment ([DeepSeek-V3 paper](https://arxiv.org/abs/2408.15664)) — achieves near-perfect ~25% utilization per expert without auxiliary losses |
| - **SwiGLU Activation with Soft-Clamping**: `limit=7.0` prevents activation explosions during long training runs |
| - **Batched Expert Dispatch**: Sort-and-slice dispatch with `searchsorted` boundaries for contiguous memory access |
| - **RoPE with YaRN Extension**: Base context of 2048 tokens, extensible to 8192 via YaRN scaling |
|
|
| ## Training Details |
|
|
| ### Phase 1 — Pretraining (~60B tokens) |
|
|
| | Training Config | Value | |
| |:---|:---| |
| | **Hardware** | H200 GPU | |
| | **Peak Learning Rate** | 3e-4 | |
| | **Min Learning Rate** | 3e-5 | |
| | **Scheduler** | WSD (Warmup-Stable-Decay) | |
| | **Warmup Steps** | 500 | |
| | **Total Steps** | 101,726 | |
| | **Effective Batch Size** | 37 × 8 = 296 sequences | |
| | **Tokens per Step** | ~0.61M | |
| | **Gradient Clipping** | 1.0 | |
|
|
| ### Training Data Mix |
|
|
| | Dataset | Weight | Category | |
| |:---|:---:|:---| |
| | `starcoderdata` — Python | 14 | Source Code | |
| | `starcoderdata` — JavaScript | 8 | Source Code | |
| | `starcoderdata` — Java | 6 | Source Code | |
| | `starcoderdata` — TypeScript | 4 | Source Code | |
| | `starcoderdata` — C++ | 6 | Source Code | |
| | `starcoderdata` — C | 4 | Source Code | |
| | `starcoderdata` — C# | 3 | Source Code | |
| | `starcoderdata` — Go | 4 | Source Code | |
| | `starcoderdata` — Rust | 3 | Source Code | |
| | `starcoderdata` — PHP | 3 | Source Code | |
| | `fineweb-edu-dedup` | 20 | General Knowledge | |
| | `cosmopedia-v2` | 7 | General Knowledge | |
| | `wikipedia-en` | 3 | General Knowledge | |
| | `finemath-4plus` | 8 | Math / Reasoning | |
| | `stackexchange` (programming/CS) | 7 | CS / Engineering | |
|
|
| **Category breakdown**: Source Code 55% · General Knowledge 30% · Math/Reasoning 8% · CS/Engineering 7% |
|
|
| ## Quickstart |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo_id = "AkshithAI/project-828-gpt-base" |
| |
| tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| repo_id, |
| torch_dtype="bfloat16", |
| device_map="auto", |
| trust_remote_code=True |
| ) |
| |
| prompt = "def binary_search(arr, target):" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=200, |
| temperature=0.7, |
| top_p=0.9, |
| do_sample=True, |
| use_cache=False, |
| eos_token_id=tokenizer.eos_token_id, |
| pad_token_id=tokenizer.pad_token_id |
| ) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{project828, |
| author = {AkshithAI}, |
| title = {Project 828: MoE Transformer with Training Pipeline}, |
| year = {2025}, |
| publisher = {GitHub}, |
| url = {https://github.com/AkshithAI/project_828} |
| } |
| ``` |
|
|
