Instructions to use AnveshAI/AnveshAI-CoderLM-0.1B-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AnveshAI/AnveshAI-CoderLM-0.1B-Preview with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AnveshAI/AnveshAI-CoderLM-0.1B-Preview", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AnveshAI/AnveshAI-CoderLM-0.1B-Preview", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AnveshAI/AnveshAI-CoderLM-0.1B-Preview with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AnveshAI/AnveshAI-CoderLM-0.1B-Preview"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AnveshAI/AnveshAI-CoderLM-0.1B-Preview",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AnveshAI/AnveshAI-CoderLM-0.1B-Preview

SGLang

How to use AnveshAI/AnveshAI-CoderLM-0.1B-Preview with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AnveshAI/AnveshAI-CoderLM-0.1B-Preview" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AnveshAI/AnveshAI-CoderLM-0.1B-Preview",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AnveshAI/AnveshAI-CoderLM-0.1B-Preview" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AnveshAI/AnveshAI-CoderLM-0.1B-Preview",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AnveshAI/AnveshAI-CoderLM-0.1B-Preview with Docker Model Runner:
```
docker model run hf.co/AnveshAI/AnveshAI-CoderLM-0.1B-Preview
```

AnveshAI-CoderLM-0.1B-Preview / README.md

developeranveshraman

Update README.md

4eb9938 verified 7 days ago

preview code

Raw History Blame Contribute Delete

11.1 kB

---
license: apache-2.0
language:
- code
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- code
- causal-lm
- anveshai
- coderlm
- mingru
- differential-attention
- multi-language
- fim
- edge
- on-device
- mobile
model-index:
- name: AnveshAI-CoderLM-0.1B
  results: []
---
<div align="center">
  <picture>
    <img src="logo.png" width="50%" alt="AnveshAI CoderLM 0.1B">
  </picture>
</div>
<hr>
<div align="center" style="line-height: 1;">
  <a href="https://huggingface.co/AnveshAI" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-AnveshAI-ffc107?color=ffc107&logoColor=white"/></a>
  <a href="https://www.apache.org/licenses/LICENSE-2.0" target="_blank"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue?logoColor=white"/></a>
</div>
<div align="center" style="line-height: 1;">
  <a href="#"><img src="https://img.shields.io/badge/Parameters-100.2M-blue" alt="Parameters"></a>
  <img src="https://img.shields.io/badge/Architecture-Hybrid%20MinGRU%2BDiffGQA-green" alt="Architecture">
  <img src="https://img.shields.io/badge/Languages-13_Languages-orange" alt="Languages">
  <img src="https://img.shields.io/badge/Context-512_tokens-purple" alt="Context Length">
  <img src="https://img.shields.io/badge/Precision-FP16-red" alt="Precision">
</div>
**CoderLM** is a compact 100.2M-parameter language model purpose-built for multi-language code generation across **13 programming and markup languages** — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on a novel 15-layer hybrid architecture combining **10× ParallelMinGRU** linear recurrence and **5× Differential Grouped-Query Attention (DifferentialGQA)**, CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling.

**Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware.** At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.

---

## Highlights

- **100.2M parameters** (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
- **~191MB model size** (FP16 `model.safetensors`) — fits easily in RAM on edge devices and consumer hardware
- **Runs on CPU & Edge Devices** — no discrete GPU required, works on laptops, phones, edge devices, and embedded systems
- **15-layer hybrid architecture** — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
- **13 languages supported** — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
- **Isolated-newline 16K BPE tokenizer** — strict indentation and newline preservation (`Split(Regex(r"\n"), behavior="isolated")`)
- **Task & language-oriented special tokens** — routing with `<|lang:xyz|>` tokens and Fill-in-the-Middle (`<|fim_prefix|>`, `<|fim_middle|>`, `<|fim_suffix|>`)
- **Verified 1.0397 Validation Loss** — 75.54% Top-1 Accuracy and 91.13% Top-5 Accuracy on canonical validation data

---

## Model Details

| Property | Value |
|----------|-------|
| **Model Name** | AnveshAI CoderLM 0.1B |
| **Parameters** | 100,176,325 (100.2M) |
| **Architecture** | AnveshAICoderLMForCausalLM (10× ParallelMinGRU + 5× DifferentialGQA) |
| **Hidden Size** | 768 |
| **Intermediate Size** | 1808 (SwiGLU) |
| **Layers** | 15 (10 MinGRU + 5 DifferentialGQA) |
| **Attention Heads** | 8 Query Heads ($d_{head}=64$) |
| **KV Heads** | 2 (Grouped Query Attention) |
| **Vocab Size** | 16,384 (Isolated-Newline BPE) |
| **Max Context** | 512 tokens |
| **Activation** | SwiGLU (SiLU) |
| **Tied Embeddings** | Yes (`lm_head.weight == tok_emb.weight`) |
| **Training Step** | 10,000 (Best Val Checkpoint) |
| **Eval Loss** | 1.0397 (Perplexity: 2.83) |
| **Top-1 / Top-5 Accuracy** | 75.54% / 91.13% |
| **Model Size on Disk** | ~191 MB (`model.safetensors`) |
---
## Benchmark Results
### General Multi-Language Validation (Step 10,000)
| Metric | FP32 Checkpoint | FP16 Checkpoint | Parity |
| :--- | :---: | :---: | :---: |
| **Validation Loss** | **1.0397** | **1.0398** | 99.99% |
| **Validation Perplexity** | **2.83** | **2.83** | 100.0% |
| **Top-1 Accuracy** | **75.54%** | **75.53%** | 99.99% |
| **Top-5 Accuracy** | **91.13%** | **91.13%** | 100.0% |
| **Weighted 13-Lang Loss** | **1.1637** | **1.1637** | 100.0% |
| **Weighted 13-Lang PPL** | **3.20** | **3.20** | 100.0% |
### Per-Language Breakdown (Ranked by Loss)
| Language | Dataset Weight | Validation Loss | Perplexity | Top-1 Accuracy | Top-5 Accuracy |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **C** | 8.0% | **0.7320** | 2.08 | **81.30%** | 94.48% |
| **TypeScript** | 6.0% | **0.8214** | 2.27 | **80.91%** | 93.80% |
| **Rust** | 8.0% | **0.9482** | 2.58 | **75.93%** | 92.97% |
| **C++** | 10.0% | **0.9534** | 2.59 | **78.71%** | 90.97% |
| **HTML** | 4.0% | **0.9901** | 2.69 | **74.95%** | 91.02% |
| **Java** | 12.0% | **1.0126** | 2.75 | **78.03%** | 91.70% |
| **Python** | 28.0% | **1.3031** | 3.68 | **70.70%** | 87.55% |
| **Go** | 5.0% | **1.3273** | 3.77 | **70.02%** | 87.45% |
| **JavaScript** | 10.0% | **1.3318** | 3.79 | **70.95%** | 88.13% |
| **Shell** | 2.0% | **1.3395** | 3.82 | **70.41%** | 87.74% |
| **SQL** | 2.0% | **1.5765** | 4.84 | **63.43%** | 85.74% |
| **CSS** | 3.0% | **1.6704** | 5.31 | **63.38%** | 83.01% |
| **Markdown** | 2.0% | **2.5326** | 12.59 | **55.22%** | 70.41% |
---
## Quick Start
### Installation
```bash
pip install transformers torch safetensors
```
### Basic Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "AnveshAI/AnveshAI-CoderLM-0.1B-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float16,
    device_map="auto",
)
model.eval()
# Prompt with language prefix
prompt = "<|lang:python|>def is_even(n: int) -> bool:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
generated = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=False))
```
### Fill-in-the-Middle (FIM) Usage
```python
# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = (
    "<|fim_prefix|>def compute_mean(values):\n"
    "    if not values:\n"
    "        return 0.0\n"
    "<|fim_suffix|>\n"
    "    return total / len(values)\n"
    "<|fim_middle|>"
)
inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(output[0]))
```
---
## Special Tokens
CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:
| Token | Purpose | Description |
|-------|---------|-------------|
| `<\|endoftext\|>` | End / Start of sequence | Delimiter and MinGRU recurrent state reset |
| `<\|pad\|>` | Padding | Padding token |
| `<\|fim_prefix\|>` | FIM Prefix | Precedes the context prefix in infilling tasks |
| `<\|fim_middle\|>` | FIM Middle | Signals the model to generate the middle fill |
| `<\|fim_suffix\|>` | FIM Suffix | Precedes the context suffix in infilling tasks |
| `<\|lang:python\|>` | Language routing | Directs generation to Python syntax |
| `<\|lang:java\|>` | Language routing | Directs generation to Java syntax |
| `<\|lang:javascript\|>` | Language routing | Directs generation to JavaScript syntax |
| `<\|lang:cpp\|>` | Language routing | Directs generation to C++ syntax |
| `<\|lang:c\|>` | Language routing | Directs generation to C syntax |
| `<\|lang:rust\|>` | Language routing | Directs generation to Rust syntax |
| `<\|lang:typescript\|>` | Language routing | Directs generation to TypeScript syntax |
| `<\|lang:go\|>` | Language routing | Directs generation to Go syntax |
| `<\|lang:html\|>` | Language routing | Directs generation to HTML markup |
| `<\|lang:css\|>` | Language routing | Directs generation to CSS stylesheets |
| `<\|lang:shell\|>` | Language routing | Directs generation to Shell / Bash scripts |
| `<\|lang:sql\|>` | Language routing | Directs generation to SQL queries |
| `<\|lang:markdown\|>` | Language routing | Directs generation to Markdown documents |
### Prompt Formats
**Standard Generation:**
```
<|lang:python|>def fibonacci(n):
```
**Fill-in-the-Middle (FIM):**
```
<|fim_prefix|>prefix_code<|fim_suffix|>suffix_code<|fim_middle|>
```
---
## Tips for Best Results
1. **Prefix with `<|lang:xyz|>`** — always supply the appropriate language token to activate language-specific representations
2. **Keep `max_new_tokens` under 256** — the model performs best with short-to-medium completions
3. **Use greedy decoding (`do_sample=False`) for algorithms** — produces clean, syntax-valid completions
4. **Isolated-newline whitespace** — indentation is tokenized into isolated newlines followed by 4-space indent tokens
5. **Direct function signature prompts work best** — e.g., `<|lang:c|>void reverse_string(char *s) {\n`
---
## Limitations
- **100M scale** — this is a lightweight edge model; complex multi-file logic is beyond its capacity
- **512 token context window** — designed for snippets, functions, and FIM completions rather than long repositories
- **Base code model** — trained for code completion and FIM, not multi-turn conversational chat
- **Syntax sensitivity** — relies on language routing tokens for optimal formatting
---
## Architecture & Training Details
- **Base Architecture**: 15-Layer Hybrid (10× ParallelMinGRU + 5× DifferentialGQA)
- **Attention Mechanism**: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
- **Recurrent Blocks**: ParallelMinGRU with 1-token shift mixing and EOS document boundary reset
- **Feed-Forward**: SwiGLU with $d_{ff} = 1808$
- **Tokenizer**: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
- **Optimizer**: AdamW with cosine learning rate schedule
- **Precision**: FP16 mixed precision
- **Best Checkpoint**: Step 10,000 (selected by lowest validation loss: 1.0397)
- **Hardware**: Kaggle GPU environment
---
## Citation
```bibtex
@misc{anveshai-coderlm-2026,
  title={AnveshAI CoderLM 0.1B},
  author={AnveshAI},
  year={2026},
  url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
}
```
---
## License
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
---
<p align="center">
  <b>Built by <a href="https://huggingface.co/AnveshAI">AnveshAI</a></b><br>
  <i>15-Layer Hybrid MinGRU + Differential Attention Code Engine</i>
</p>

	---
	license: apache-2.0
	language:
	- code
	- en
	library_name: transformers
	pipeline_tag: text-generation
	tags:
	- code
	- causal-lm
	- anveshai
	- coderlm
	- mingru
	- differential-attention
	- multi-language
	- fim
	- edge
	- on-device
	- mobile
	model-index:
	- name: AnveshAI-CoderLM-0.1B
	results: []
	---
	<div align="center">
	<picture>
	<img src="logo.png" width="50%" alt="AnveshAI CoderLM 0.1B">
	</picture>
	</div>
	<hr>
	<div align="center" style="line-height: 1;">
	<a href="https://huggingface.co/AnveshAI" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-AnveshAI-ffc107?color=ffc107&logoColor=white"/></a>
	<a href="https://www.apache.org/licenses/LICENSE-2.0" target="_blank"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue?logoColor=white"/></a>
	</div>
	<div align="center" style="line-height: 1;">
	<a href="#"><img src="https://img.shields.io/badge/Parameters-100.2M-blue" alt="Parameters"></a>
	<img src="https://img.shields.io/badge/Architecture-Hybrid%20MinGRU%2BDiffGQA-green" alt="Architecture">
	<img src="https://img.shields.io/badge/Languages-13_Languages-orange" alt="Languages">
	<img src="https://img.shields.io/badge/Context-512_tokens-purple" alt="Context Length">
	<img src="https://img.shields.io/badge/Precision-FP16-red" alt="Precision">
	</div>

	CoderLM is a compact 100.2M-parameter language model purpose-built for multi-language code generation across 13 programming and markup languages — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on a novel 15-layer hybrid architecture combining 10× ParallelMinGRU linear recurrence and 5× Differential Grouped-Query Attention (DifferentialGQA), CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling.

	Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware. At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.

	---

	## Highlights

	- 100.2M parameters (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
	- ~191MB model size (FP16 `model.safetensors`) — fits easily in RAM on edge devices and consumer hardware
	- Runs on CPU & Edge Devices — no discrete GPU required, works on laptops, phones, edge devices, and embedded systems
	- 15-layer hybrid architecture — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
	- 13 languages supported — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
	- Isolated-newline 16K BPE tokenizer — strict indentation and newline preservation (`Split(Regex(r"\n"), behavior="isolated")`)
	- Task & language-oriented special tokens — routing with `<\|lang:xyz\|>` tokens and Fill-in-the-Middle (`<\|fim_prefix\|>`, `<\|fim_middle\|>`, `<\|fim_suffix\|>`)
	- Verified 1.0397 Validation Loss — 75.54% Top-1 Accuracy and 91.13% Top-5 Accuracy on canonical validation data

	---

	## Model Details

	\| Property \| Value \|
	\|----------\|-------\|
	\| Model Name \| AnveshAI CoderLM 0.1B \|
	\| Parameters \| 100,176,325 (100.2M) \|
	\| Architecture \| AnveshAICoderLMForCausalLM (10× ParallelMinGRU + 5× DifferentialGQA) \|
	\| Hidden Size \| 768 \|
	\| Intermediate Size \| 1808 (SwiGLU) \|
	\| Layers \| 15 (10 MinGRU + 5 DifferentialGQA) \|
	\| Attention Heads \| 8 Query Heads ($d_{head}=64$) \|
	\| KV Heads \| 2 (Grouped Query Attention) \|
	\| Vocab Size \| 16,384 (Isolated-Newline BPE) \|
	\| Max Context \| 512 tokens \|
	\| Activation \| SwiGLU (SiLU) \|
	\| Tied Embeddings \| Yes (`lm_head.weight == tok_emb.weight`) \|
	\| Training Step \| 10,000 (Best Val Checkpoint) \|
	\| Eval Loss \| 1.0397 (Perplexity: 2.83) \|
	\| Top-1 / Top-5 Accuracy \| 75.54% / 91.13% \|
	\| Model Size on Disk \| ~191 MB (`model.safetensors`) \|

	---

	## Benchmark Results

	### General Multi-Language Validation (Step 10,000)

	\| Metric \| FP32 Checkpoint \| FP16 Checkpoint \| Parity \|
	\| :--- \| :---: \| :---: \| :---: \|
	\| Validation Loss \| 1.0397 \| 1.0398 \| 99.99% \|
	\| Validation Perplexity \| 2.83 \| 2.83 \| 100.0% \|
	\| Top-1 Accuracy \| 75.54% \| 75.53% \| 99.99% \|
	\| Top-5 Accuracy \| 91.13% \| 91.13% \| 100.0% \|
	\| Weighted 13-Lang Loss \| 1.1637 \| 1.1637 \| 100.0% \|
	\| Weighted 13-Lang PPL \| 3.20 \| 3.20 \| 100.0% \|

	### Per-Language Breakdown (Ranked by Loss)

	\| Language \| Dataset Weight \| Validation Loss \| Perplexity \| Top-1 Accuracy \| Top-5 Accuracy \|
	\| :--- \| :---: \| :---: \| :---: \| :---: \| :---: \|
	\| C \| 8.0% \| 0.7320 \| 2.08 \| 81.30% \| 94.48% \|
	\| TypeScript \| 6.0% \| 0.8214 \| 2.27 \| 80.91% \| 93.80% \|
	\| Rust \| 8.0% \| 0.9482 \| 2.58 \| 75.93% \| 92.97% \|
	\| C++ \| 10.0% \| 0.9534 \| 2.59 \| 78.71% \| 90.97% \|
	\| HTML \| 4.0% \| 0.9901 \| 2.69 \| 74.95% \| 91.02% \|
	\| Java \| 12.0% \| 1.0126 \| 2.75 \| 78.03% \| 91.70% \|
	\| Python \| 28.0% \| 1.3031 \| 3.68 \| 70.70% \| 87.55% \|
	\| Go \| 5.0% \| 1.3273 \| 3.77 \| 70.02% \| 87.45% \|
	\| JavaScript \| 10.0% \| 1.3318 \| 3.79 \| 70.95% \| 88.13% \|
	\| Shell \| 2.0% \| 1.3395 \| 3.82 \| 70.41% \| 87.74% \|
	\| SQL \| 2.0% \| 1.5765 \| 4.84 \| 63.43% \| 85.74% \|
	\| CSS \| 3.0% \| 1.6704 \| 5.31 \| 63.38% \| 83.01% \|
	\| Markdown \| 2.0% \| 2.5326 \| 12.59 \| 55.22% \| 70.41% \|

	---

	## Quick Start

	### Installation

	```bash
	pip install transformers torch safetensors
	```

	### Basic Usage

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model_name = "AnveshAI/AnveshAI-CoderLM-0.1B-Preview"

	tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
	model = AutoModelForCausalLM.from_pretrained(
	model_name,
	trust_remote_code=True,
	dtype=torch.float16,
	device_map="auto",
	)
	model.eval()

	# Prompt with language prefix
	prompt = "<\|lang:python\|>def is_even(n: int) -> bool:\n"

	inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
	with torch.no_grad():
	output = model.generate(
	**inputs,
	max_new_tokens=64,
	do_sample=False,
	pad_token_id=tokenizer.pad_token_id,
	eos_token_id=tokenizer.eos_token_id,
	)

	generated = output[0][inputs["input_ids"].shape[1]:]
	print(tokenizer.decode(generated, skip_special_tokens=False))
	```

	### Fill-in-the-Middle (FIM) Usage

	```python
	# Format: <\|fim_prefix\|> prefix <\|fim_suffix\|> suffix <\|fim_middle\|>
	fim_prompt = (
	"<\|fim_prefix\|>def compute_mean(values):\n"
	" if not values:\n"
	" return 0.0\n"
	"<\|fim_suffix\|>\n"
	" return total / len(values)\n"
	"<\|fim_middle\|>"
	)

	inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
	with torch.no_grad():
	output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

	print(tokenizer.decode(output[0]))
	```

	---

	## Special Tokens

	CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:

	\| Token \| Purpose \| Description \|
	\|-------\|---------\|-------------\|
	\| `<\\|endoftext\\|>` \| End / Start of sequence \| Delimiter and MinGRU recurrent state reset \|
	\| `<\\|pad\\|>` \| Padding \| Padding token \|
	\| `<\\|fim_prefix\\|>` \| FIM Prefix \| Precedes the context prefix in infilling tasks \|
	\| `<\\|fim_middle\\|>` \| FIM Middle \| Signals the model to generate the middle fill \|
	\| `<\\|fim_suffix\\|>` \| FIM Suffix \| Precedes the context suffix in infilling tasks \|
	\| `<\\|lang:python\\|>` \| Language routing \| Directs generation to Python syntax \|
	\| `<\\|lang:java\\|>` \| Language routing \| Directs generation to Java syntax \|
	\| `<\\|lang:javascript\\|>` \| Language routing \| Directs generation to JavaScript syntax \|
	\| `<\\|lang:cpp\\|>` \| Language routing \| Directs generation to C++ syntax \|
	\| `<\\|lang:c\\|>` \| Language routing \| Directs generation to C syntax \|
	\| `<\\|lang:rust\\|>` \| Language routing \| Directs generation to Rust syntax \|
	\| `<\\|lang:typescript\\|>` \| Language routing \| Directs generation to TypeScript syntax \|
	\| `<\\|lang:go\\|>` \| Language routing \| Directs generation to Go syntax \|
	\| `<\\|lang:html\\|>` \| Language routing \| Directs generation to HTML markup \|
	\| `<\\|lang:css\\|>` \| Language routing \| Directs generation to CSS stylesheets \|
	\| `<\\|lang:shell\\|>` \| Language routing \| Directs generation to Shell / Bash scripts \|
	\| `<\\|lang:sql\\|>` \| Language routing \| Directs generation to SQL queries \|
	\| `<\\|lang:markdown\\|>` \| Language routing \| Directs generation to Markdown documents \|

	### Prompt Formats

	Standard Generation:
	```
	<\|lang:python\|>def fibonacci(n):
	```

	Fill-in-the-Middle (FIM):
	```
	<\|fim_prefix\|>prefix_code<\|fim_suffix\|>suffix_code<\|fim_middle\|>
	```

	---

	## Tips for Best Results

	1. Prefix with `<\|lang:xyz\|>` — always supply the appropriate language token to activate language-specific representations
	2. Keep `max_new_tokens` under 256 — the model performs best with short-to-medium completions
	3. Use greedy decoding (`do_sample=False`) for algorithms — produces clean, syntax-valid completions
	4. Isolated-newline whitespace — indentation is tokenized into isolated newlines followed by 4-space indent tokens
	5. Direct function signature prompts work best — e.g., `<\|lang:c\|>void reverse_string(char *s) {\n`

	---

	## Limitations

	- 100M scale — this is a lightweight edge model; complex multi-file logic is beyond its capacity
	- 512 token context window — designed for snippets, functions, and FIM completions rather than long repositories
	- Base code model — trained for code completion and FIM, not multi-turn conversational chat
	- Syntax sensitivity — relies on language routing tokens for optimal formatting

	---

	## Architecture & Training Details

	- Base Architecture: 15-Layer Hybrid (10× ParallelMinGRU + 5× DifferentialGQA)
	- Attention Mechanism: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
	- Recurrent Blocks: ParallelMinGRU with 1-token shift mixing and EOS document boundary reset
	- Feed-Forward: SwiGLU with $d_{ff} = 1808$
	- Tokenizer: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
	- Optimizer: AdamW with cosine learning rate schedule
	- Precision: FP16 mixed precision
	- Best Checkpoint: Step 10,000 (selected by lowest validation loss: 1.0397)
	- Hardware: Kaggle GPU environment

	---

	## Citation

	```bibtex
	@misc{anveshai-coderlm-2026,
	title={AnveshAI CoderLM 0.1B},
	author={AnveshAI},
	year={2026},
	url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
	}
	```

	---

	## License

	This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).

	---

	<p align="center">
	<b>Built by <a href="https://huggingface.co/AnveshAI">AnveshAI</a></b><br>
	<i>15-Layer Hybrid MinGRU + Differential Attention Code Engine</i>
	</p>