File size: 6,537 Bytes
e194537
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
935a074
e194537
935a074
e194537
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
935a074
 
 
 
e194537
 
 
 
935a074
e194537
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
935a074
e194537
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
935a074
e194537
 
 
 
 
 
 
 
f7ca116
e194537
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
---
language:
- code
- en
license: apache-2.0
datasets:
- Avelina/python-edu-cleaned
- bigcode/starcoderdata
library_name: transformers
pipeline_tag: text-generation
tags:
- code
- python
- code-completion
- fill-in-the-middle
- fim
- llama
---

# GrimmCoder_v2

**GrimmCoder_v2** is a ~107M-parameter LLaMA-style decoder-only transformer trained from scratch exclusively on Python source code for both **left-to-right completion** and **Fill-in-the-Middle (FIM)** code completion.

This release represents **Phase 2 finetuning** on `Avelina/python-edu-cleaned` (774.1M tokens, 6,000 steps with cosine decay `1e-4 → 1e-5`), initialized from the 8B-token `bigcode/starcoderdata` pretrain base checkpoint.

## Model Summary

| Property | Value |
|---|---|
| Architecture | LLaMA (`LlamaForCausalLM`, pre-norm, RoPE, SwiGLU, GQA, tied embeddings) |
| Parameters | ~106.98M total (~100.69M non-embedding) |
| Context Length | 2048 tokens |
| Hidden Size (`d_model`) | 768 |
| Layers | 16 |
| Attention Heads | 12 query heads / 4 KV heads (GQA) |
| FFN Intermediate Size | 2048 (SwiGLU) |
| Vocabulary | 8192 (byte-level BPE trained on Python) |
| Training Data | `Avelina/python-edu-cleaned` (finetuned from `bigcode/starcoderdata` pretrain) |
| Tokens / Checkpoint | ~8.70B tokens total (7.93B pretrain + 774.1M finetune, step 6,000, val loss 1.2260) |

## Evaluation (HumanEval)

Evaluated on [`openai/openai_humaneval`](https://huggingface.co/datasets/openai/openai_humaneval) (164 Python problems) using `transformers` (`bfloat16`) on CUDA:

| Model Version | Mode & Decoding | `pass@1` | Solved | Syntax Valid |
|---|---|---|---|---|
| **GrimmCoder_v2** | Left-to-Right: greedy (`do_sample=False`) | **9.15%** | **15 / 164** | **137 / 164 (83.5%)** |
| **GrimmCoder_v2** | Left-to-Right: `t=0.2, rep=1.1` | **9.15%** | **15 / 164** | **158 / 164 (96.3%)** |
| **GrimmCoder_v2** | **FIM (PSM): greedy (`do_sample=False`)** | **9.15%** | **15 / 164** | **151 / 164 (92.1%)** |
| **GrimmCoder_v2** | **FIM (PSM): `t=0.2, rep=1.1`** | **8.54%** | **14 / 164** | **153 / 164 (93.3%)** |
| GrimmCoder (Base Pretrain) | Left-to-Right: greedy (`do_sample=False`) | 5.49% | 9 / 164 | 126 / 164 (76.8%) |
| GrimmCoder (Base Pretrain) | Left-to-Right: `t=0.2, rep=1.1` | 4.88% | 8 / 164 | 155 / 164 (94.5%) |
| GrimmCoder (Base Pretrain) | FIM (PSM): greedy (`do_sample=False`) | 4.27% | 7 / 164 | 122 / 164 (74.4%) |

### Solved Problems
- **L2R Greedy (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `34`, `45`, `48`, `50`, `52`, `53`, `55`, `60`
- **L2R t0.2_rep1.1 (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `34`, `45`, `48`, `50`, `53`, `58`, `60`, `61`
- **FIM Greedy (15)**: `HumanEval/16`, `22`, `23`, `28`, `29`, `30`, `31`, `42`, `45`, `48`, `50`, `53`, `55`, `58`, `60`
- **FIM t0.2_rep1.1 (14)**: `HumanEval/12`, `16`, `22`, `23`, `28`, `31`, `34`, `35`, `42`, `45`, `48`, `53`, `58`, `60`
- **Overall Distinct Problems Solved Across All Evaluated Modes**: **20 / 164** (`HumanEval/12, 16, 22, 23, 28, 29, 30, 31, 34, 35, 42, 45, 48, 50, 52, 53, 55, 58, 60, 61`)

## Special Tokens

| Token | ID | Purpose |
|---|---|---|
| `<\|endoftext\|>` | `0` | BOS / EOS / document delimiter |
| `<\|fim_prefix\|>` | `1` | FIM prefix marker |
| `<\|fim_middle\|>` | `2` | FIM middle marker (model generates from here) |
| `<\|fim_suffix\|>` | `3` | FIM suffix marker |
| `<\|pad\|>` | `4` | Padding token |

## Usage (`transformers`)

### 1. Left-to-Right Code Completion

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Bayhakl1/GrimmCoder_v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = """def binary_search(arr, target):
    lo, hi = 0, len(arr) - 1
"""
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.2,
    top_p=0.95,
    eos_token_id=[0, 1, 2, 3, 4],
    pad_token_id=4,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### 2. Fill-in-the-Middle (FIM)

GrimmCoder_v2 natively supports Fill-in-the-Middle (FIM) to insert code between a **prefix** (code before the cursor) and a **suffix** (code after the cursor). The model was trained 50/50 on **PSM** and **SPM** formats.

> [!NOTE]
> Special tokens require vertical pipe characters: `<|fim_prefix|>`, `<|fim_suffix|>`, and `<|fim_middle|>`.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Bayhakl1/GrimmCoder_v2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prefix = "def add(a: int, b: int) -> int:\n    "
suffix = "\n\nprint(add(2, 3))\n"

# --- Method A: String Template (PSM format) ---
# Format: <|fim_prefix|> prefix <|fim_suffix|> suffix <|fim_middle|>
fim_prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
inputs = tokenizer(fim_prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

# --- Method B (Alternative): Direct Token IDs ---
# pre_id, mid_id, suf_id = 1, 2, 3
# p_ids = tokenizer.encode(prefix, add_special_tokens=False)
# s_ids = tokenizer.encode(suffix, add_special_tokens=False)
# input_ids = torch.tensor([[pre_id] + p_ids + [suf_id] + s_ids + [mid_id]], device=model.device)
# inputs = {"input_ids": input_ids, "attention_mask": torch.ones_like(input_ids)}

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
    temperature=0.2,
    top_p=0.95,
    eos_token_id=[0, 1, 2, 3, 4],
    pad_token_id=4,
)

# Extract and decode only the generated middle tokens
middle_ids = outputs[0][inputs["input_ids"].shape[1]:]
middle = tokenizer.decode(middle_ids, skip_special_tokens=True)

# Reconstruct complete code
print("--- Generated Middle ---")
print(repr(middle))
print("\n--- Complete Code ---")
print(prefix + middle + suffix)
```

#### SPM Mode (Suffix-Prefix-Middle)
If you prefer SPM format, format the prompt as:
```python
fim_prompt = f"<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix}"
```

## Limitations

- Trained exclusively on Python source code; not intended as a general natural-language chat or instruction-following model.
- Base / completion model: responds best to code prompts and FIM markers rather than conversational Q&A.