README.md

---
license: apache-2.0
language:
- code
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- code
- causal-lm
- anveshai
- coderlm
- anvarc
- mingru
- differential-attention
- multi-language
- fim
- 2k-context
- edge
- on-device
- mobile
model-index:
- name: AnveshAI-CoderLM-0.1B
  results: []
---
<div align="center">
  <picture>
    <img src="logo.png" width="50%" alt="AnveshAI CoderLM 0.1B">
  </picture>
</div>
<hr>
<div align="center" style="line-height: 1;">
  <a href="https://huggingface.co/AnveshAI" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-AnveshAI-ffc107?color=ffc107&logoColor=white"/></a>
  <a href="https://www.apache.org/licenses/LICENSE-2.0" target="_blank"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue?logoColor=white"/></a>
</div>
<div align="center" style="line-height: 1;">
  <a href="#"><img src="https://img.shields.io/badge/Parameters-100.2M-blue" alt="Parameters"></a>
  <img src="https://img.shields.io/badge/Architecture-AnvArc-green" alt="Architecture">
  <img src="https://img.shields.io/badge/Languages-13_Languages-orange" alt="Languages">
  <img src="https://img.shields.io/badge/Context-2048_tokens-purple" alt="Context Length">
  <img src="https://img.shields.io/badge/Precision-FP16-red" alt="Precision">
  <img src="https://img.shields.io/badge/Research%20Paper-Coming%20Soon-white">
</div>

**CoderLM** is a compact 100.2M-parameter language model purpose-built for multi-language code generation across **13 programming and markup languages** — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on the custom **AnvArc** (AnveshAI Architecture) — a 15-layer hybrid architecture combining **10× ParallelMinGRU** linear recurrence and **5× Differential Grouped-Query Attention (DifferentialGQA)** — CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling with an expanded **2,048-token context window**.

**Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware.** At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.

---

## Highlights

- **100.2M parameters** (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
- **~191MB model size** (FP16 `model.safetensors`) — fits easily in RAM on edge devices and consumer hardware
- **AnvArc Hybrid Architecture** — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
- **4× Extended Context (2,048 tokens)** — trained on full 2K sequences with stable perplexity scaling across all lengths
- **13 languages supported** — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
- **18.7% Loss Reduction over 10K Baseline** — weighted multi-language loss improved from 1.1637 down to **0.9463** (Perplexity: **2.58**)
- **Isolated-newline 16K BPE tokenizer** — strict indentation and newline preservation (`Split(Regex(r"\n"), behavior="isolated")`)
- **Task & language-oriented special tokens** — routing with `<|lang:xyz|>` tokens and Fill-in-the-Middle (`<|fim_prefix|>`, `<|fim_middle|>`, `<|fim_suffix|>`)

---

## Model Details

| Property | Value |
|----------|-------|
| **Model Name** | AnveshAI CoderLM 0.1B (2K Context) |
| **Parameters** | 100,176,325 (100.2M) |
| **Architecture** | **AnvArc** (15-Layer Hybrid: 10× ParallelMinGRU + 5× DifferentialGQA) |
| **Hidden Size ($d_{model}$)** | 768 |
| **Intermediate Size ($d_{ff}$)** | 1808 (SwiGLU) |
| **Layers** | 15 (10 MinGRU + 5 DifferentialGQA) |
| **Attention Heads** | 8 Query Heads ($d_{head}=64$) |
| **KV Heads** | 2 (Grouped Query Attention, 4:1 ratio) |
| **Vocab Size** | 16,384 (Isolated-Newline BPE) |
| **Max Context** | **2,048 tokens** |
| **Precision** | FP16 (`model.safetensors`) & FP32 |
| **Activation** | SwiGLU (SiLU) |
| **Tied Embeddings** | Yes (`lm_head.weight == tok_emb.weight`) |
| **Training Steps** | 15,000 (WSD Schedule) |
| **2K Val Loss** | **1.1459** (Perplexity: **3.15**) |
| **Top-1 / Top-5 Accuracy** | **76.84%** / **88.35%** |
| **Weighted 13-Lang Loss** | **0.9463** (Perplexity: **2.58**) |
| **Model Size on Disk** | ~191 MB (`model.safetensors`) |

---

## Comparison: New (2K Context, Step 15,000) vs Old (512 Context, Step 10,000)

The table below compares the new **Step 15,000 (2K Context)** release against the previous **Step 10,000 (512 Context)** model across all canonical benchmarks:

### 1. Overall Model Comparison

| Dimension | Old Model (Step 10,000) | New Model (Step 15,000) ★ | Improvement / Delta |
| :--- | :---: | :---: | :---: |
| **Context Window** | 512 tokens | **2,048 tokens** | **4× Context Expansion** |
| **Training Steps** | 10,000 steps | **15,000 steps** | +5,000 steps (WSD Decay) |
| **Architecture** | Hybrid MinGRU+DiffGQA | **AnvArc** (Optimized Boundary Reset) | Architecture refinement |
| **Weighted 13-Lang Loss** | 1.1637 | **0.9463** | **-0.2174 (-18.7% Loss)** ⚡ |
| **Weighted 13-Lang PPL** | 3.20 | **2.58** | **-0.62 (-19.4% PPL)** ⚡ |
| **2K Context Loss** | *N/A (OOM / truncated)* | **1.1459** | Full 2K sequence support |
| **2K Context Perplexity** | *N/A* | **3.15** | Stable long-range modeling |
| **Top-1 Next-Token Accuracy** | 75.54% | **76.84%** | **+1.30% Accuracy** |
| **Logit Scale Factor** | 1.1080 | **1.2861** | Sharper output distribution |
| **Model Size on Disk** | 191.09 MB | **191.09 MB** | Zero parameter overhead |

---

### 2. Per-Language Breakdown Comparison (13 Languages)

| Language | Dataset Weight | Old Step 10K Loss | New Step 15K Loss ★ | Delta (New vs Old) | New Step 15K Top-1 Acc |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **HTML** | 4.0% | 0.9901 | **0.0669** | **-0.9232 (-93.2%)** ⚡ | **98.14%** |
| **C** | 8.0% | 0.7320 | **0.1386** | **-0.5934 (-81.1%)** ⚡ | **97.07%** |
| **JavaScript** | 10.0% | 1.3318 | **0.2738** | **-1.0580 (-79.4%)** ⚡ | **92.97%** |
| **C++** | 10.0% | 0.9534 | **0.6324** | **-0.3210 (-33.7%)** ⚡ | **84.47%** |
| **Java** | 12.0% | 1.0126 | **0.9413** | **-0.0713 (-7.0%)** | **80.57%** |
| **Python** | 28.0% | 1.3031 | **1.0950** | **-0.2081 (-16.0%)** ⚡ | **73.24%** |
| **SQL** | 2.0% | 1.5765 | **1.1801** | **-0.3964 (-25.1%)** ⚡ | **78.03%** |
| **Rust** | 8.0% | 0.9482 | **1.1610** | +0.2128 | **75.10%** |
| **CSS** | 3.0% | 1.6704 | **1.4037** | **-0.2667 (-16.0%)** | **62.11%** |
| **Go** | 5.0% | 1.3273 | **1.5968** | +0.2695 | **65.43%** |
| **TypeScript** | 6.0% | 0.8214 | **1.6186** | +0.7972 | **66.70%** |
| **Markdown** | 2.0% | 2.5326 | **1.8514** | **-0.6812 (-26.9%)** ⚡ | **63.57%** |
| **Shell** | 2.0% | 1.3395 | **2.4868** | +1.1473 | **53.71%** |
| **Weighted Overall** | **100.0%** | **1.1637** | **0.9463** | **-0.2174 (-18.7%)** ⚡ | **76.84%** |

---

### 3. Context Length Scaling Curve (New 2K Model)

The new **AnvArc** architecture demonstrates stable perplexity across sequence lengths up to the full 2,048-token context window:

| Sequence Length | Validation Loss | Validation Perplexity | Relative Stability |
| :---: | :---: | :---: | :--- |
| **256 tokens** | **0.9724** | **2.64** | Baseline prompt evaluation |
| **512 tokens** | **1.1139** | **3.05** | Standard function body completion |
| **1,024 tokens** | **1.0211** | **2.78** | Multi-function module completion |
| **2,048 tokens** | **1.0746** | **2.93** | Full long-range context preservation |

---

## Quick Start

### Installation

```bash
pip install transformers torch safetensors
```

### Basic Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "AnveshAI/AnveshAI-CoderLM-0.1B"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float16,
    device_map="auto",
)
model.eval()

# Prompt with language prefix
prompt = "<|lang:python|>def is_even(n: int) -> bool:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=False))
```

### Fill-in-the-Middle (FIM) Usage

```python
# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = (
    "<|fim_prefix|>def compute_mean(values):\n"
    "    if not values:\n"
    "        return 0.0\n"
    "<|fim_suffix|>\n"
    "    return total / len(values)\n"
    "<|fim_middle|>"
)

inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(output[0]))
```

---

## Special Tokens

CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:

| Token | Purpose | Description |
|-------|---------|-------------|
| `<\|endoftext\|>` | End / Start of sequence | Delimiter and AnvArc recurrent state reset |
| `<\|pad\|>` | Padding | Padding token |
| `<\|fim_prefix\|>` | FIM Prefix | Precedes the context prefix in infilling tasks |
| `<\|fim_middle\|>` | FIM Middle | Signals the model to generate the middle fill |
| `<\|fim_suffix\|>` | FIM Suffix | Precedes the context suffix in infilling tasks |
| `<\|lang:python\|>` | Language routing | Directs generation to Python syntax |
| `<\|lang:java\|>` | Language routing | Directs generation to Java syntax |
| `<\|lang:javascript\|>` | Language routing | Directs generation to JavaScript syntax |
| `<\|lang:cpp\|>` | Language routing | Directs generation to C++ syntax |
| `<\|lang:c\|>` | Language routing | Directs generation to C syntax |
| `<\|lang:rust\|>` | Language routing | Directs generation to Rust syntax |
| `<\|lang:typescript\|>` | Language routing | Directs generation to TypeScript syntax |
| `<\|lang:go\|>` | Language routing | Directs generation to Go syntax |
| `<\|lang:html\|>` | Language routing | Directs generation to HTML markup |
| `<\|lang:css\|>` | Language routing | Directs generation to CSS stylesheets |
| `<\|lang:shell\|>` | Language routing | Directs generation to Shell / Bash scripts |
| `<\|lang:sql\|>` | Language routing | Directs generation to SQL queries |
| `<\|lang:markdown\|>` | Language routing | Directs generation to Markdown documents |

### Prompt Formats

**Standard Generation:**
```
<|lang:python|>def fibonacci(n):
```

**Fill-in-the-Middle (FIM):**
```
<|fim_prefix|>prefix_code<|fim_suffix|>suffix_code<|fim_middle|>
```

---

## Tips for Best Results

1. **Prefix with `<|lang:xyz|>`** — always supply the appropriate language token to activate language-specific representations in AnvArc
2. **Utilize Long Prompts up to 2,048 Tokens** — unlike the 512 baseline, multi-function context and long file headers are fully supported
3. **Use greedy decoding (`do_sample=False`) for algorithms** — produces clean, syntax-valid completions
4. **Isolated-newline whitespace** — indentation is tokenized into isolated newlines followed by 4-space indent tokens
5. **Direct function signature prompts work best** — e.g., `<|lang:c|>void reverse_string(char *s) {\n`

---

## Limitations

- **100M scale** — this is a lightweight edge model; complex multi-file architectural reasoning is beyond its capacity
- **2,048 token context window** — designed for single files, functions, and FIM completions rather than massive codebases
- **Base code model** — trained for code completion and FIM, not multi-turn conversational chat
- **Syntax sensitivity** — relies on language routing tokens for optimal formatting

---

## Architecture & Training Details

- **Base Architecture**: **AnvArc** (15-Layer Hybrid: 10× ParallelMinGRU + 5× DifferentialGQA)
- **Attention Mechanism**: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
- **Recurrent Blocks**: ParallelMinGRU with 1-token shift mixing and EOS/language document boundary reset
- **Feed-Forward**: SwiGLU with $d_{ff} = 1808$
- **Tokenizer**: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
- **Optimizer**: AdamW + Muon with Warmup-Stable-Decay (WSD) learning rate schedule
- **Precision**: FP16 mixed precision
- **Best Checkpoint**: Step 15,000 (selected by lowest 2K validation loss: 1.1459)
- **Hardware**: Kaggle GPU environment

---

## Citation

```bibtex
@misc{anveshai-coderlm-2026,
  title={AnveshAI CoderLM 0.1B},
  author={AnveshAI},
  year={2026},
  url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
}
```

---

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).

---

<p align="center">
  <b>Built by <a href="https://huggingface.co/AnveshAI">AnveshAI</a></b><br>
  <i>AnvArc: 15-Layer Hybrid MinGRU + Differential Attention Code Engine</i>
</p>