AnveshAI/AnveshAI-CoderLM-0.1B-Preview

---

CoderLM is a compact 100.2M-parameter language model purpose-built for multi-language code generation across 13 programming and markup languages — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, and Markdown. Built on a novel 15-layer hybrid architecture combining 10× ParallelMinGRU linear recurrence and 5× Differential Grouped-Query Attention (DifferentialGQA), CoderLM delivers high-efficiency code generation and fill-in-the-middle infilling.

Small enough to run on any device — including edge devices like smartphones, tablets, Raspberry Pi, and other embedded hardware. At just ~191MB (FP16), CoderLM brings multi-language code generation to devices where larger models simply can't run. No high-end GPU required — it runs comfortably on CPU or mobile accelerators.

---

Highlights

• 100.2M parameters (87.6M non-embedding) — ultra-compact model with tied input/output embeddings
• ~191MB model size (FP16 model.safetensors) — fits easily in RAM on edge devices and consumer hardware
• Runs on CPU & Edge Devices — no discrete GPU required, works on laptops, phones, edge devices, and embedded systems
• 15-layer hybrid architecture — 10× ParallelMinGRU linear recurrent blocks + 5× DifferentialGQA with RoPE
• 13 languages supported — Python, Java, JavaScript, C++, C, Rust, TypeScript, Go, HTML, CSS, Shell, SQL, Markdown
• Isolated-newline 16K BPE tokenizer — strict indentation and newline preservation (Split(Regex(r"\n"), behavior="isolated"))
• Task & language-oriented special tokens — routing with <|lang:xyz|> tokens and Fill-in-the-Middle (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>)
• Verified 1.0397 Validation Loss — 75.54% Top-1 Accuracy and 91.13% Top-5 Accuracy on canonical validation data

---

Model Details

Property	Value
Model Name	AnveshAI CoderLM 0.1B
Parameters	100,176,325 (100.2M)
Architecture	AnveshAICoderLMForCausalLM (10× ParallelMinGRU + 5× DifferentialGQA)
Hidden Size	768
Intermediate Size	1808 (SwiGLU)
Layers	15 (10 MinGRU + 5 DifferentialGQA)
Attention Heads	8 Query Heads ($d_{head}=64$)
KV Heads	2 (Grouped Query Attention)
Vocab Size	16,384 (Isolated-Newline BPE)
Max Context	512 tokens
Activation	SwiGLU (SiLU)
Tied Embeddings	Yes (lm_head.weight == tok_emb.weight)
Training Step	10,000 (Best Val Checkpoint)
Eval Loss	1.0397 (Perplexity: 2.83)
Top-1 / Top-5 Accuracy	75.54% / 91.13%
Model Size on Disk	~191 MB (model.safetensors)

---

Benchmark Results

General Multi-Language Validation (Step 10,000)

Metric	FP32 Checkpoint	FP16 Checkpoint	Parity
Validation Loss	1.0397	1.0398	99.99%
Validation Perplexity	2.83	2.83	100.0%
Top-1 Accuracy	75.54%	75.53%	99.99%
Top-5 Accuracy	91.13%	91.13%	100.0%
Weighted 13-Lang Loss	1.1637	1.1637	100.0%
Weighted 13-Lang PPL	3.20	3.20	100.0%

Per-Language Breakdown (Ranked by Loss)

Language	Dataset Weight	Validation Loss	Perplexity	Top-1 Accuracy	Top-5 Accuracy
C	8.0%	0.7320	2.08	81.30%	94.48%
TypeScript	6.0%	0.8214	2.27	80.91%	93.80%
Rust	8.0%	0.9482	2.58	75.93%	92.97%
C++	10.0%	0.9534	2.59	78.71%	90.97%
HTML	4.0%	0.9901	2.69	74.95%	91.02%
Java	12.0%	1.0126	2.75	78.03%	91.70%
Python	28.0%	1.3031	3.68	70.70%	87.55%
Go	5.0%	1.3273	3.77	70.02%	87.45%
JavaScript	10.0%	1.3318	3.79	70.95%	88.13%
Shell	2.0%	1.3395	3.82	70.41%	87.74%
SQL	2.0%	1.5765	4.84	63.43%	85.74%
CSS	3.0%	1.6704	5.31	63.38%	83.01%
Markdown	2.0%	2.5326	12.59	55.22%	70.41%

---

Quick Start

Installation

pip install transformers torch safetensors

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "AnveshAI/AnveshAI-CoderLM-0.1B-Preview"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    dtype=torch.float16,
    device_map="auto",
)
model.eval()

# Prompt with language prefix
prompt = "<|lang:python|>def is_even(n: int) -> bool:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated, skip_special_tokens=False))

Fill-in-the-Middle (FIM) Usage

# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = (
    "<|fim_prefix|>def compute_mean(values):\n"
    "    if not values:\n"
    "        return 0.0\n"
    "<|fim_suffix|>\n"
    "    return total / len(values)\n"
    "<|fim_middle|>"
)

inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(output[0]))

---

Special Tokens

CoderLM uses structured special tokens for sequence control, language conditioning, and code infilling:

Token	Purpose	Description
<|endoftext|>	End / Start of sequence	Delimiter and MinGRU recurrent state reset
<|pad|>	Padding	Padding token
<|fim_prefix|>	FIM Prefix	Precedes the context prefix in infilling tasks
<|fim_middle|>	FIM Middle	Signals the model to generate the middle fill
<|fim_suffix|>	FIM Suffix	Precedes the context suffix in infilling tasks
<|lang:python|>	Language routing	Directs generation to Python syntax
<|lang:java|>	Language routing	Directs generation to Java syntax
<|lang:javascript|>	Language routing	Directs generation to JavaScript syntax
<|lang:cpp|>	Language routing	Directs generation to C++ syntax
<|lang:c|>	Language routing	Directs generation to C syntax
<|lang:rust|>	Language routing	Directs generation to Rust syntax
<|lang:typescript|>	Language routing	Directs generation to TypeScript syntax
<|lang:go|>	Language routing	Directs generation to Go syntax
<|lang:html|>	Language routing	Directs generation to HTML markup
<|lang:css|>	Language routing	Directs generation to CSS stylesheets
<|lang:shell|>	Language routing	Directs generation to Shell / Bash scripts
<|lang:sql|>	Language routing	Directs generation to SQL queries
<|lang:markdown|>	Language routing	Directs generation to Markdown documents

Prompt Formats

Standard Generation:

<|lang:python|>def fibonacci(n):

Fill-in-the-Middle (FIM):

<|fim_prefix|>prefix_code<|fim_suffix|>suffix_code<|fim_middle|>

---

Tips for Best Results

1. Prefix with <|lang:xyz|> — always supply the appropriate language token to activate language-specific representations
2. Keep max_new_tokens under 256 — the model performs best with short-to-medium completions
3. Use greedy decoding (do_sample=False) for algorithms — produces clean, syntax-valid completions
4. Isolated-newline whitespace — indentation is tokenized into isolated newlines followed by 4-space indent tokens
5. Direct function signature prompts work best — e.g., <|lang:c|>void reverse_string(char *s) {\n

---

Limitations

• 100M scale — this is a lightweight edge model; complex multi-file logic is beyond its capacity
• 512 token context window — designed for snippets, functions, and FIM completions rather than long repositories
• Base code model — trained for code completion and FIM, not multi-turn conversational chat
• Syntax sensitivity — relies on language routing tokens for optimal formatting

---

Architecture & Training Details

• Base Architecture: 15-Layer Hybrid (10× ParallelMinGRU + 5× DifferentialGQA)
• Attention Mechanism: Differential GQA with RoPE ($\theta=10000$) and learnable Value-residual ($V_{res}$)
• Recurrent Blocks: ParallelMinGRU with 1-token shift mixing and EOS document boundary reset
• Feed-Forward: SwiGLU with $d_{ff} = 1808$
• Tokenizer: 16,384 vocabulary Byte-Pair Encoding with isolated newline regex
• Optimizer: AdamW with cosine learning rate schedule
• Precision: FP16 mixed precision
• Best Checkpoint: Step 10,000 (selected by lowest validation loss: 1.0397)
• Hardware: Kaggle GPU environment

---

Citation

@misc{anveshai-coderlm-2026,
  title={AnveshAI CoderLM 0.1B},
  author={AnveshAI},
  year={2026},
  url={https://huggingface.co/AnveshAI/AnveshAI-CoderLM-0.1B}
}

---

License

This model is released under the Apache 2.0 License.

---

Built by AnveshAI
15-Layer Hybrid MinGRU + Differential Attention Code Engine