---
license: apache-2.0
language:
- code
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- code
- causal-lm
- anveshai
- coderlm
- mingru
- differential-attention
- multi-language
- fim
- edge
- on-device
- mobile
model-index:
- name: AnveshAI-CoderLM-0.1B
results: []
---
**CoderLM** is a compact 100.2M-parameter language model purpose-built for multi-language code generation across **13 programming and markup languages** — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on a novel 15-layer hybrid architecture combining **10× ParallelMinGRU** linear recurrence and **5× Differential Grouped-Query Attention (DifferentialGQA)**, CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling.
**Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware.** At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.
---
## Highlights
- **100.2M parameters** (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
- **~191MB model size** (FP16 `model.safetensors`) — fits easily in RAM on edge devices and consumer hardware
- **Runs on CPU & Edge Devices** — no discrete GPU required, works on laptops, phones, edge devices, and embedded systems
- **15-layer hybrid architecture** — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
- **13 languages supported** — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
- **Isolated-newline 16K BPE tokenizer** — strict indentation and newline preservation (`Split(Regex(r"\n"), behavior="isolated")`)
- **Task & language-oriented special tokens** — routing with `<|lang:xyz|>` tokens and Fill-in-the-Middle (`<|fim_prefix|>`, `<|fim_middle|>`, `<|fim_suffix|>`)
- **Verified 1.0397 Validation Loss** — 75.54% Top-1 Accuracy and 91.13% Top-5 Accuracy on canonical validation data
---
## Model Details
| Property | Value |
|----------|-------|
| **Model Name** | AnveshAI CoderLM 0.1B |
| **Parameters** | 100,176,325 (100.2M) |
| **Architecture** | AnveshAICoderLMForCausalLM (10× ParallelMinGRU + 5× DifferentialGQA) |
| **Hidden Size** | 768 |
| **Intermediate Size** | 1808 (SwiGLU) |
| **Layers** | 15 (10 MinGRU + 5 DifferentialGQA) |
| **Attention Heads** | 8 Query Heads ($d_{head}=64$) |
| **KV Heads** | 2 (Grouped Query Attention) |
| **Vocab Size** | 16,384 (Isolated-Newline BPE) |
| **Max Context** | 512 tokens |
| **Activation** | SwiGLU (SiLU) |
| **Tied Embeddings** | Yes (`lm_head.weight == tok_emb.weight`) |
| **Training Step** | 10,000 (Best Val Checkpoint) |
| **Eval Loss** | 1.0397 (Perplexity: 2.83) |
| **Top-1 / Top-5 Accuracy** | 75.54% / 91.13% |
| **Model Size on Disk** | ~191 MB (`model.safetensors`) |
---
## Benchmark Results
### General Multi-Language Validation (Step 10,000)
| Metric | FP32 Checkpoint | FP16 Checkpoint | Parity |
| :--- | :---: | :---: | :---: |
| **Validation Loss** | **1.0397** | **1.0398** | 99.99% |
| **Validation Perplexity** | **2.83** | **2.83** | 100.0% |
| **Top-1 Accuracy** | **75.54%** | **75.53%** | 99.99% |
| **Top-5 Accuracy** | **91.13%** | **91.13%** | 100.0% |
| **Weighted 13-Lang Loss** | **1.1637** | **1.1637** | 100.0% |
| **Weighted 13-Lang PPL** | **3.20** | **3.20** | 100.0% |
### Per-Language Breakdown (Ranked by Loss)
| Language | Dataset Weight | Validation Loss | Perplexity | Top-1 Accuracy | Top-5 Accuracy |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **C** | 8.0% | **0.7320** | 2.08 | **81.30%** | 94.48% |
| **TypeScript** | 6.0% | **0.8214** | 2.27 | **80.91%** | 93.80% |
| **Rust** | 8.0% | **0.9482** | 2.58 | **75.93%** | 92.97% |
| **C++** | 10.0% | **0.9534** | 2.59 | **78.71%** | 90.97% |
| **HTML** | 4.0% | **0.9901** | 2.69 | **74.95%** | 91.02% |
| **Java** | 12.0% | **1.0126** | 2.75 | **78.03%** | 91.70% |
| **Python** | 28.0% | **1.3031** | 3.68 | **70.70%** | 87.55% |
| **Go** | 5.0% | **1.3273** | 3.77 | **70.02%** | 87.45% |
| **JavaScript** | 10.0% | **1.3318** | 3.79 | **70.95%** | 88.13% |
| **Shell** | 2.0% | **1.3395** | 3.82 | **70.41%** | 87.74% |
| **SQL** | 2.0% | **1.5765** | 4.84 | **63.43%** | 85.74% |
| **CSS** | 3.0% | **1.6704** | 5.31 | **63.38%** | 83.01% |
| **Markdown** | 2.0% | **2.5326** | 12.59 | **55.22%** | 70.41% |
---
## Quick Start
### Installation
```bash
pip install transformers torch safetensors
```
### Basic Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "AnveshAI/AnveshAI-CoderLM-0.1B-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
dtype=torch.float16,
device_map="auto",
)
model.eval()
# Prompt with language prefix
prompt = "<|lang:python|>def is_even(n: int) -> bool:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=64,
do_sample=False,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
generated = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=False))
```
### Fill-in-the-Middle (FIM) Usage
```python
# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = (
"<|fim_prefix|>def compute_mean(values):\n"
" if not values:\n"
" return 0.0\n"
"<|fim_suffix|>\n"
" return total / len(values)\n"
"<|fim_middle|>"
)
inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0]))
```
---
## Special Tokens
CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:
| Token | Purpose | Description |
|-------|---------|-------------|
| `<\|endoftext\|>` | End / Start of sequence | Delimiter and MinGRU recurrent state reset |
| `<\|pad\|>` | Padding | Padding token |
| `<\|fim_prefix\|>` | FIM Prefix | Precedes the context prefix in infilling tasks |
| `<\|fim_middle\|>` | FIM Middle | Signals the model to generate the middle fill |
| `<\|fim_suffix\|>` | FIM Suffix | Precedes the context suffix in infilling tasks |
| `<\|lang:python\|>` | Language routing | Directs generation to Python syntax |
| `<\|lang:java\|>` | Language routing | Directs generation to Java syntax |
| `<\|lang:javascript\|>` | Language routing | Directs generation to JavaScript syntax |
| `<\|lang:cpp\|>` | Language routing | Directs generation to C++ syntax |
| `<\|lang:c\|>` | Language routing | Directs generation to C syntax |
| `<\|lang:rust\|>` | Language routing | Directs generation to Rust syntax |
| `<\|lang:typescript\|>` | Language routing | Directs generation to TypeScript syntax |
| `<\|lang:go\|>` | Language routing | Directs generation to Go syntax |
| `<\|lang:html\|>` | Language routing | Directs generation to HTML markup |
| `<\|lang:css\|>` | Language routing | Directs generation to CSS stylesheets |
| `<\|lang:shell\|>` | Language routing | Directs generation to Shell / Bash scripts |
| `<\|lang:sql\|>` | Language routing | Directs generation to SQL queries |
| `<\|lang:markdown\|>` | Language routing | Directs generation to Markdown documents |
### Prompt Formats
**Standard Generation:**
```
<|lang:python|>def fibonacci(n):
```
**Fill-in-the-Middle (FIM):**
```
<|fim_prefix|>prefix_code<|fim_suffix|>suffix_code<|fim_middle|>
```
---
## Tips for Best Results
1. **Prefix with `<|lang:xyz|>`** — always supply the appropriate language token to activate language-specific representations
2. **Keep `max_new_tokens` under 256** — the model performs best with short-to-medium completions
3. **Use greedy decoding (`do_sample=False`) for algorithms** — produces clean, syntax-valid completions
4. **Isolated-newline whitespace** — indentation is tokenized into isolated newlines followed by 4-space indent tokens
5. **Direct function signature prompts work best** — e.g., `<|lang:c|>void reverse_string(char *s) {\n`
---
## Limitations
- **100M scale** — this is a lightweight edge model; complex multi-file logic is beyond its capacity
- **512 token context window** — designed for snippets, functions, and FIM completions rather than long repositories
- **Base code model** — trained for code completion and FIM, not multi-turn conversational chat
- **Syntax sensitivity** — relies on language routing tokens for optimal formatting
---
## Architecture & Training Details
- **Base Architecture**: 15-Layer Hybrid (10× ParallelMinGRU + 5× DifferentialGQA)
- **Attention Mechanism**: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
- **Recurrent Blocks**: ParallelMinGRU with 1-token shift mixing and EOS document boundary reset
- **Feed-Forward**: SwiGLU with $d_{ff} = 1808$
- **Tokenizer**: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
- **Optimizer**: AdamW with cosine learning rate schedule
- **Precision**: FP16 mixed precision
- **Best Checkpoint**: Step 10,000 (selected by lowest validation loss: 1.0397)
- **Hardware**: Kaggle GPU environment
---
## Citation
```bibtex
@misc{anveshai-coderlm-2026,
title={AnveshAI CoderLM 0.1B},
author={AnveshAI},
year={2026},
url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
}
```
---
## License
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
---
Built by AnveshAI
15-Layer Hybrid MinGRU + Differential Attention Code Engine
