sgc-522m

sgc (Single-GPU Coder) is a 522M-parameter decoder-only transformer for Python and C++ code, pretrained from scratch on 103B tokens in 76.3 hours on one NVIDIA GH200 GPU, then post-trained with supervised fine-tuning (SFT) and reinforcement learning (RL). This repository holds the final model, after RL. It reasons in a <think> block, then answers with code.

• Paper: One GPU, 0.1 Trillion Tokens: Pretraining a Sub-Billion-Parameter Code Model
• Code, training and evaluation records: github.com/indrajeetadityaroy9/single-gpu-code-llm
• Models: sgc-522m-base (pretrained), sgc-522m-sft (SFT), sgc-522m (RL, this model)

Results

Model	HumanEval	HumanEval+
sgc-522m-base	26.8	22.0
sgc-522m-sft	32.9	31.7
sgc-522m	40.9	38.4

pass@1 with greedy decoding, scored by EvalPlus. The pretrained model is prompted with code to complete; the SFT and RL models with the chat template below. On 512 held-out KodCode problems (4 samples each at temperature 0.6, top-p 0.95, up to 16,384 tokens), RL raised pass@1 from 0.334 to 0.426 and cut unfinished responses from 918 to 266 of 2,048.

Prompt format

The chat template renders <|bos|>, the user message, then \n\n<think>\n. The model writes its reasoning, closes it with </think>, then gives the answer with the code in a Markdown block. The end of the response is <|bos|>. generation_config.json samples at temperature 0.6 and top-p 0.95; the evaluation above is greedy. The context is 32,768 tokens.

Usage

vLLM, with the reasoning split from the answer (the repository's server script registers the sgc reasoning parser; each response message has a reasoning and a content field):

git clone https://github.com/indrajeetadityaroy9/single-gpu-code-llm && cd single-gpu-code-llm
scripts/setup.sh
scripts/serve.sh iaroy/sgc-522m

vLLM alone: vllm serve iaroy/sgc-522m --trust-remote-code --dtype bfloat16; the answer is the text after </think>.

Transformers (5.x, tested with 5.21.0):

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("iaroy/sgc-522m")
model = AutoModelForCausalLM.from_pretrained("iaroy/sgc-522m", trust_remote_code=True, dtype="bfloat16")
messages = [{"role": "user", "content": "Write a Python function that returns the n-th Fibonacci number."}]
ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)["input_ids"]
out = model.generate(ids, max_new_tokens=4096, do_sample=True, temperature=0.6, top_p=0.95)
text = tokenizer.decode(out[0][ids.shape[1]:], skip_special_tokens=True)
reasoning, _, answer = text.partition("</think>")
print(answer)

Training

Stage	Data	Steps	Time on one GH200
Pretraining	103B tokens of Python and C++ code, web text and math (iaroy/sgc-data); context 8,192, then 32,768 at the repository level	69,874	76.3 h
SFT	reasoning traces from Nemotron-Cascade SFT (code), Nemotron SFT Competitive Programming v2 and KodCode-V1-SFT-R1; 2 epochs, 11.32B tokens	7,676	11.2 h
RL	KodCode problems, rewarded by their unit tests	39	2 h

Limitations

A small model: it fails most of the problems these benchmarks pose, and 13% of its held-out responses still loop or reach the token limit without closing </think>. It was trained for Python and C++.