| --- |
| license: mit |
| language: |
| - code |
| tags: |
| - code-generation |
| - python |
| - instruction-tuned |
| - sft |
| - from-scratch |
| - wsm |
| library_name: pytorch |
| --- |
| |
| # code-llm-1b-sft — the instruction-tuned 1.1B Python model (mix42 arm, val-best) |
|
|
| Supervised fine-tune of the 1.1B base model in [`../base`](../base). Same architecture, same |
| tokenizer, same 1024-token context; the difference is that this checkpoint answers a task stated in |
| prose or as a signature (`### Task ... ### Code`) instead of merely continuing code. |
|
|
| > **This folder is the SFT release**, and it is the best of the 1B fine-tuning arms on the combined |
| > benchmark: **97/110 on the instruction column and 91/110 on the signature column** (human 0/1/2 |
| > grading, 11 tasks x 5 seeds). The base scores 83/110 and 61/110 on the same ruler. |
|
|
| ## Architecture |
|
|
| | | | |
| |---|---| |
| | Parameters | **1,095,845,888** (bf16), 143 tensors | |
| | Layers | 20 | |
| | d_model / heads | 2048 / 16 (head_dim 128) | |
| | FFN | SwiGLU, d_ff 5120 (gate / up / down) | |
| | Position | RoPE, theta 500000 | |
| | Norm | RMSNorm, learned `scale`, eps 1e-6 (pre-norm) | |
| | Attention | causal SDPA, fused QKV projection, no biases anywhere | |
| | Embeddings | untied (`embed` + `lm_head`) | |
| | Vocabulary | 32,000 byte-level BPE (inherited from pretraining, not retrained) | |
| | Context | 1024 tokens | |
| | Precision | bf16 (released file) | |
|
|
| Parameter count is the sum of the 143 released tensors and equals the count of the instantiated |
| `CodeLLM` class, not the (slightly wrong) closed form in `ModelConfig.num_params`. |
|
|
| ## Fine-tuning recipe |
|
|
| - **Init**: `checkpoints_wsm_1b/merged` (the pretraining merged weights in `../base`). |
| - **Data**: `data_mix_gold` — the gold instruction set mixed with signature-style examples at 5:1, |
| **91.78M tokens** in total; the loss-mask covers only the answer span. |
| - **Optimisation**: 2,801 steps at lr 2e-5, batch 4 x grad-accum 8 (32 sequences of 1024 tokens per |
| step), seed 42, weight decay and warmup as in `sft_train.py`. |
| - **Selection**: this file is the **validation-loss-best** checkpoint (SFT val loss 0.6519 at step |
| 2,000 of 2,801) — not the final step. A pre-training val batch was scored alongside as an |
| anti-forgetting gate; it did cross its +0.05 red line mid-run, which is the other reason the |
| released file is a mid-run checkpoint rather than the last one. |
| - **Hardware**: a single RTX 5090, 32 GB. |
|
|
| ## Results (human-graded, 0/1/2 per generation, 11 tasks x 5 seeds, maximum 110 per column) |
|
|
| | model | instruction column | signature column | |
| |---|---|---| |
| | base (`../base`) | 83/110 | 61/110 | |
| | **this checkpoint (mix42 best_val)** | **97/110** | **91/110** | |
| | mix42, final step (not released) | 98/110 | 85/110 | |
| | gold-only arm, best_val | 97/110 | 67/110 | |
| |
| The two columns use the same tasks, seeds and generation settings; only the prompt and the judging |
| entry point differ. The mix42 arm does not significantly beat the gold-only arm on the instruction |
| column (+0/+1 over five seeds, inside seed noise ~4 points), but it is clearly better on signatures |
| (+24) — and its training set is 1.47x larger, so the difference is a mix-plus-volume effect, not a |
| clean single-variable result. |
| |
| ## Known limits (read before trusting a number) |
| |
| - **`mergesort` never passes** (0/5 on both columns); `binary_search` is 0/5 when prompted by |
| signature and usually returns `None`/`True`/`False` instead of `-1` on failure — graded 1 for |
| recognisable structure, not correctness. `two_sum` is strong by signature (5/5) and weaker in prose. |
| - **Missing imports**: 18 of 110 generations in the signature column reference `math`/`re`/`functools` |
| without importing them; they are graded 1 (structure right, would raise `NameError`). Scoring them 2 |
| would move the two mix42 variants from 176 to 194 out of 220 — no ordering against the other arms |
| changes. |
| - **Single seed for this release decision** (SFT seed 42), five generation seeds for grading; per-arm |
| spread across seeds is about 4 points, so treat gaps under ~6 points as noise. |
| - **Template collapse**: generation is noticeably homogeneous — `fibonacci` produces one identical |
| implementation across all five seeds — consistent with the 5:1 signature-data ratio. |
| - **Context is 1024 tokens**, and the model truncates rather than refusing: prompts longer than that |
| are cut, not summarised. |
| - Trained only on code plus synthetic instructions; no personal data. Outputs may reproduce patterns |
| from the training corpus despite copyright filtering. |
|
|
| ## Files |
|
|
| | File | What | |
| |---|---| |
| | `model.safetensors` | bf16 weights, 143 tensors, 2,191,706,272 bytes — verified bit-identical to `fine-tuning/ckpt_1b_mix42_merged/best_val/checkpoint.pt` after the fp32→bf16 cast | |
| | `config.json` | architecture config taken from the checkpoint itself, plus `torch_dtype: bf16` | |
| | `tokenizer.json` | byte-level BPE, 32,000 tokens (same file as `../base`) | |
| | `SHA256SUMS.txt` | `9ecc6e2d99e81ff0df1655fd714c26e76a556f8b18aa87572adecf9cd0c4f00f model.safetensors` | |
| | `arch_facts.json` | provenance: source checkpoint, step, token count, source config | |
|
|
| Verification artifacts (the hash, key/shape, bitwise-equality, strict-load and CPU-forward logs) stay in the training workspace and are **not** part of this release; `SHA256SUMS.txt` is what lets anyone re-check the weights independently. |
|
|
| Loading requires the model class from the training repository (`sft_train.py`, class `CodeLLM`, |
| `ModelConfig(**config.json)`). This is not a `transformers` architecture and has no `AutoModel` |
| mapping. `RMSNorm` outputs fp32 while the weights are bf16, so run inference under |
| `torch.autocast("cpu", dtype=torch.bfloat16)` or in fp32 — a bare bf16 CPU forward raises a dtype |
| error inside `nn.Linear` (training always ran under autocast and never hit it). |
|
|
| Prompt template used in evaluation: |
|
|
| ``` |
| ### Task |
| <instruction> |
| |
| ### Code |
| ``` |
|
|
| ## Links |
|
|
| Design record, fine-tuning log with corrections, every judged generation and the ablation report: |
| https://github.com/TightXu/Simple_Code_LLM. MIT licensed. |
|
|
