--- license: mit language: - code tags: - code-generation - python - instruction-tuned - sft - from-scratch - wsm library_name: pytorch --- # code-llm-1b-sft — the instruction-tuned 1.1B Python model (mix42 arm, val-best) Supervised fine-tune of the 1.1B base model in [`../base`](../base). Same architecture, same tokenizer, same 1024-token context; the difference is that this checkpoint answers a task stated in prose or as a signature (`### Task ... ### Code`) instead of merely continuing code. > **This folder is the SFT release**, and it is the best of the 1B fine-tuning arms on the combined > benchmark: **97/110 on the instruction column and 91/110 on the signature column** (human 0/1/2 > grading, 11 tasks x 5 seeds). The base scores 83/110 and 61/110 on the same ruler. ## Architecture | | | |---|---| | Parameters | **1,095,845,888** (bf16), 143 tensors | | Layers | 20 | | d_model / heads | 2048 / 16 (head_dim 128) | | FFN | SwiGLU, d_ff 5120 (gate / up / down) | | Position | RoPE, theta 500000 | | Norm | RMSNorm, learned `scale`, eps 1e-6 (pre-norm) | | Attention | causal SDPA, fused QKV projection, no biases anywhere | | Embeddings | untied (`embed` + `lm_head`) | | Vocabulary | 32,000 byte-level BPE (inherited from pretraining, not retrained) | | Context | 1024 tokens | | Precision | bf16 (released file) | Parameter count is the sum of the 143 released tensors and equals the count of the instantiated `CodeLLM` class, not the (slightly wrong) closed form in `ModelConfig.num_params`. ## Fine-tuning recipe - **Init**: `checkpoints_wsm_1b/merged` (the pretraining merged weights in `../base`). - **Data**: `data_mix_gold` — the gold instruction set mixed with signature-style examples at 5:1, **91.78M tokens** in total; the loss-mask covers only the answer span. - **Optimisation**: 2,801 steps at lr 2e-5, batch 4 x grad-accum 8 (32 sequences of 1024 tokens per step), seed 42, weight decay and warmup as in `sft_train.py`. - **Selection**: this file is the **validation-loss-best** checkpoint (SFT val loss 0.6519 at step 2,000 of 2,801) — not the final step. A pre-training val batch was scored alongside as an anti-forgetting gate; it did cross its +0.05 red line mid-run, which is the other reason the released file is a mid-run checkpoint rather than the last one. - **Hardware**: a single RTX 5090, 32 GB. ## Results (human-graded, 0/1/2 per generation, 11 tasks x 5 seeds, maximum 110 per column) | model | instruction column | signature column | |---|---|---| | base (`../base`) | 83/110 | 61/110 | | **this checkpoint (mix42 best_val)** | **97/110** | **91/110** | | mix42, final step (not released) | 98/110 | 85/110 | | gold-only arm, best_val | 97/110 | 67/110 | The two columns use the same tasks, seeds and generation settings; only the prompt and the judging entry point differ. The mix42 arm does not significantly beat the gold-only arm on the instruction column (+0/+1 over five seeds, inside seed noise ~4 points), but it is clearly better on signatures (+24) — and its training set is 1.47x larger, so the difference is a mix-plus-volume effect, not a clean single-variable result. ## Known limits (read before trusting a number) - **`mergesort` never passes** (0/5 on both columns); `binary_search` is 0/5 when prompted by signature and usually returns `None`/`True`/`False` instead of `-1` on failure — graded 1 for recognisable structure, not correctness. `two_sum` is strong by signature (5/5) and weaker in prose. - **Missing imports**: 18 of 110 generations in the signature column reference `math`/`re`/`functools` without importing them; they are graded 1 (structure right, would raise `NameError`). Scoring them 2 would move the two mix42 variants from 176 to 194 out of 220 — no ordering against the other arms changes. - **Single seed for this release decision** (SFT seed 42), five generation seeds for grading; per-arm spread across seeds is about 4 points, so treat gaps under ~6 points as noise. - **Template collapse**: generation is noticeably homogeneous — `fibonacci` produces one identical implementation across all five seeds — consistent with the 5:1 signature-data ratio. - **Context is 1024 tokens**, and the model truncates rather than refusing: prompts longer than that are cut, not summarised. - Trained only on code plus synthetic instructions; no personal data. Outputs may reproduce patterns from the training corpus despite copyright filtering. ## Files | File | What | |---|---| | `model.safetensors` | bf16 weights, 143 tensors, 2,191,706,272 bytes — verified bit-identical to `fine-tuning/ckpt_1b_mix42_merged/best_val/checkpoint.pt` after the fp32→bf16 cast | | `config.json` | architecture config taken from the checkpoint itself, plus `torch_dtype: bf16` | | `tokenizer.json` | byte-level BPE, 32,000 tokens (same file as `../base`) | | `SHA256SUMS.txt` | `9ecc6e2d99e81ff0df1655fd714c26e76a556f8b18aa87572adecf9cd0c4f00f model.safetensors` | | `arch_facts.json` | provenance: source checkpoint, step, token count, source config | Verification artifacts (the hash, key/shape, bitwise-equality, strict-load and CPU-forward logs) stay in the training workspace and are **not** part of this release; `SHA256SUMS.txt` is what lets anyone re-check the weights independently. Loading requires the model class from the training repository (`sft_train.py`, class `CodeLLM`, `ModelConfig(**config.json)`). This is not a `transformers` architecture and has no `AutoModel` mapping. `RMSNorm` outputs fp32 while the weights are bf16, so run inference under `torch.autocast("cpu", dtype=torch.bfloat16)` or in fp32 — a bare bf16 CPU forward raises a dtype error inside `nn.Linear` (training always ran under autocast and never hit it). Prompt template used in evaluation: ``` ### Task ### Code ``` ## Links Design record, fine-tuning log with corrections, every judged generation and the ablation report: https://github.com/TightXu/Simple_Code_LLM. MIT licensed.