Configuration Parsing Warning:In tokenizer_config.json: "tokenizer_config.chat_template" does not match any of the allowed types
sgc-522m-base
sgc (Single-GPU Coder) is a 522M-parameter decoder-only transformer for Python and C++ code. This repository holds the pretrained model: trained from scratch on 103B tokens of code, web text and math in 76.3 hours on one NVIDIA GH200 GPU, first with a context of 8,192 tokens at the file level, then 32,768 at the repository level. It completes code and fills in the middle; it is not a chat model. The post-trained models are sgc-522m-sft and sgc-522m.
- Paper: One GPU, 0.1 Trillion Tokens: Pretraining a Sub-Billion-Parameter Code Model
- Code, training and evaluation records: github.com/indrajeetadityaroy9/single-gpu-code-llm
- Pretraining data: iaroy/sgc-data
Results
pass@1 with greedy decoding, scored by each benchmark's official tool.
| Benchmark | pass@1 |
|---|---|
| HumanEval / HumanEval+ (EvalPlus) | 26.8 / 22.0 |
| MBPP / MBPP+ (EvalPlus) | 44.4 / 36.5 |
| MBPP, 3-shot (lm-evaluation-harness) | 32.2 |
| HumanEval-C++ / MBPP-C++ (MultiPL-E) | 21.7 / 36.8 |
| HumanEval infilling: single-line / multi-line / random-span | 55.3 / 26.1 / 11.3 |
Usage
Code completion: give the start of a file or function; the model continues it and ends with <|bos|>, its
end-of-text token. The tokenizer adds <|bos|> at the start of every encoding, so do not add it yourself.
Fill-in-the-middle, in prefix-suffix-middle order (the tokenizer adds the leading <|bos|>):
<｜fim▁begin｜>{prefix}<｜fim▁hole｜>{suffix}<｜fim▁end｜>
The model writes the middle and ends it with <|bos|>. Load it with transformers (trust_remote_code=True) or serve
it with vLLM (vllm serve iaroy/sgc-522m-base --trust-remote-code --dtype bfloat16).
- Downloads last month
- 30