add README.md

README.md

@@ -0,0 +1,120 @@
+---
+license: mit
+language:
+  - code
+tags:
+  - code-generation
+  - python
+  - instruction-tuned
+  - sft
+  - from-scratch
+  - wsm
+library_name: pytorch
+---
+
+# code-llm-1b-sft — the instruction-tuned 1.1B Python model (mix42 arm, val-best)
+
+Supervised fine-tune of the 1.1B base model in [`../base`](../base). Same architecture, same
+tokenizer, same 1024-token context; the difference is that this checkpoint answers a task stated in
+prose or as a signature (`### Task ... ### Code`) instead of merely continuing code.
+
+> **This folder is the SFT release**, and it is the best of the 1B fine-tuning arms on the combined
+> benchmark: **97/110 on the instruction column and 91/110 on the signature column** (human 0/1/2
+> grading, 11 tasks x 5 seeds). The base scores 83/110 and 61/110 on the same ruler.
+
+## Architecture
+
+| | |
+|---|---|
+| Parameters | **1,095,845,888** (bf16), 143 tensors |
+| Layers | 20 |
+| d_model / heads | 2048 / 16 (head_dim 128) |
+| FFN | SwiGLU, d_ff 5120 (gate / up / down) |
+| Position | RoPE, theta 500000 |
+| Norm | RMSNorm, learned `scale`, eps 1e-6 (pre-norm) |
+| Attention | causal SDPA, fused QKV projection, no biases anywhere |
+| Embeddings | untied (`embed` + `lm_head`) |
+| Vocabulary | 32,000 byte-level BPE (inherited from pretraining, not retrained) |
+| Context | 1024 tokens |
+| Precision | bf16 (released file) |
+
+Parameter count is the sum of the 143 released tensors and equals the count of the instantiated
+`CodeLLM` class, not the (slightly wrong) closed form in `ModelConfig.num_params`.
+
+## Fine-tuning recipe
+
+- **Init**: `checkpoints_wsm_1b/merged` (the pretraining merged weights in `../base`).
+- **Data**: `data_mix_gold` — the gold instruction set mixed with signature-style examples at 5:1,
+  **91.78M tokens** in total; the loss-mask covers only the answer span.
+- **Optimisation**: 2,801 steps at lr 2e-5, batch 4 x grad-accum 8 (32 sequences of 1024 tokens per
+  step), seed 42, weight decay and warmup as in `sft_train.py`.
+- **Selection**: this file is the **validation-loss-best** checkpoint (SFT val loss 0.6519 at step
+  2,000 of 2,801) — not the final step. A pre-training val batch was scored alongside as an
+  anti-forgetting gate; it did cross its +0.05 red line mid-run, which is the other reason the
+  released file is a mid-run checkpoint rather than the last one.
+- **Hardware**: a single RTX 5090, 32 GB.
+
+## Results (human-graded, 0/1/2 per generation, 11 tasks x 5 seeds, maximum 110 per column)
+
+| model | instruction column | signature column |
+|---|---|---|
+| base (`../base`) | 83/110 | 61/110 |
+| **this checkpoint (mix42 best_val)** | **97/110** | **91/110** |
+| mix42, final step (not released) | 98/110 | 85/110 |
+| gold-only arm, best_val | 97/110 | 67/110 |
+
+The two columns use the same tasks, seeds and generation settings; only the prompt and the judging
+entry point differ. The mix42 arm does not significantly beat the gold-only arm on the instruction
+column (+0/+1 over five seeds, inside seed noise ~4 points), but it is clearly better on signatures
+(+24) — and its training set is 1.47x larger, so the difference is a mix-plus-volume effect, not a
+clean single-variable result.
+
+## Known limits (read before trusting a number)
+
+- **`mergesort` never passes** (0/5 on both columns); `binary_search` is 0/5 when prompted by
+  signature and usually returns `None`/`True`/`False` instead of `-1` on failure — graded 1 for
+  recognisable structure, not correctness. `two_sum` is strong by signature (5/5) and weaker in prose.
+- **Missing imports**: 18 of 110 generations in the signature column reference `math`/`re`/`functools`
+  without importing them; they are graded 1 (structure right, would raise `NameError`). Scoring them 2
+  would move the two mix42 variants from 176 to 194 out of 220 — no ordering against the other arms
+  changes.
+- **Single seed for this release decision** (SFT seed 42), five generation seeds for grading; per-arm
+  spread across seeds is about 4 points, so treat gaps under ~6 points as noise.
+- **Template collapse**: generation is noticeably homogeneous — `fibonacci` produces one identical
+  implementation across all five seeds — consistent with the 5:1 signature-data ratio.
+- **Context is 1024 tokens**, and the model truncates rather than refusing: prompts longer than that
+  are cut, not summarised.
+- Trained only on code plus synthetic instructions; no personal data. Outputs may reproduce patterns
+  from the training corpus despite copyright filtering.
+
+## Files
+
+| File | What |
+|---|---|
+| `model.safetensors` | bf16 weights, 143 tensors, 2,191,706,272 bytes — verified bit-identical to `fine-tuning/ckpt_1b_mix42_merged/best_val/checkpoint.pt` after the fp32→bf16 cast |
+| `config.json` | architecture config taken from the checkpoint itself, plus `torch_dtype: bf16` |
+| `tokenizer.json` | byte-level BPE, 32,000 tokens (same file as `../base`) |
+| `SHA256SUMS.txt` | `9ecc6e2d99e81ff0df1655fd714c26e76a556f8b18aa87572adecf9cd0c4f00f  model.safetensors` |
+| `arch_facts.json` | provenance: source checkpoint, step, token count, source config |
+
+Verification artifacts (the hash, key/shape, bitwise-equality, strict-load and CPU-forward logs) stay in the training workspace and are **not** part of this release; `SHA256SUMS.txt` is what lets anyone re-check the weights independently.
+
+Loading requires the model class from the training repository (`sft_train.py`, class `CodeLLM`,
+`ModelConfig(**config.json)`). This is not a `transformers` architecture and has no `AutoModel`
+mapping. `RMSNorm` outputs fp32 while the weights are bf16, so run inference under
+`torch.autocast("cpu", dtype=torch.bfloat16)` or in fp32 — a bare bf16 CPU forward raises a dtype
+error inside `nn.Linear` (training always ran under autocast and never hit it).
+
+Prompt template used in evaluation:
+
+```
+### Task
+<instruction>
+
+### Code
+```
+
+## Links
+
+Design record, fine-tuning log with corrections, every judged generation and the ablation report:
+https://github.com/TightXu/Simple_Code_LLM. MIT licensed.