README.md

---
license: mit
language:
  - code
tags:
  - code-generation
  - python
  - from-scratch
  - pretrained
library_name: pytorch
---

# code-llm-435m — a Python code-completion model trained from scratch on one RTX 5090

A 435M-parameter decoder-only Transformer for Python code completion. Data pipeline, tokenizer,
architecture, training loop, checkpoint merging and evaluation were all written and run by one person
on one consumer GPU — this repository holds the weights; the code, design record, ablation report and
failure log live in the GitHub repository linked below.

> **This folder is the pretrained base.** Its weights are the *merged* (weight-averaged) product of the
> pretraining run: under this training scheme the merged checkpoint, not the last step, is the finished
> pretrained model — hence the folder name. The instruction-tuned variant is in [`../sft`](../sft).

## Architecture

| | |
|---|---|
| Parameters | **434,680,832** (bf16) |
| Layers | 22 |
| d_model / heads | 1024 / 16 (head_dim 64) |
| FFN | SwiGLU, d_ff 4096 |
| Position | RoPE, theta 500000 |
| Norm | RMSNorm (pre-norm) |
| Attention | causal SDPA (no biases anywhere) |
| Embeddings | untied (input embed + output head) |
| Vocabulary | 32,000 byte-level BPE, trained on the filtered corpus (≈3.5 chars/token) |
| Context | 1024 tokens |
| Precision | bf16 (released file) |

## Training

- **Data**: three Python sources (codeparrot / the_stack / star_coder), six-layer quality filtering,
  copyright filtering, metadata stripping; concatenated into **one continuous 31.0 B-token stream**
  (no dataset boundaries, no optimizer resets — an earlier version was silently retrained on the same
  8 B tokens twice by a resume bug, and the design change makes that class of bug impossible).
- **Schedule**: WSM — constant learning rate, then a 30,000-step cooldown, then **weighted averaging
  of the last 10,000 checkpoints** (this file is that merged model, not the last step).
- **Hardware**: a single RTX 5090, 32 GB, Blackwell sm_120.

## What it can and cannot do

**Can**: complete short Python functions when given a signature and the opening indentation.
On a human-graded benchmark (10 docstring-free tasks × 5 seeds, scored 0/1/2, max 100) this model
scores **66/100**; the earlier 353M version — same architecture, same GPU — scores **6/100**. The
difference is the data pipeline, not the architecture. Sample, verbatim (temperature 0.2, seed 0;
this is a partial-credit example, not a showcase):

```python
def quicksort(arr):
    if len(arr) <= 1:
        return arr
    else:
        pivot = arr[0]
        left = [x for x in arr if x < pivot]
        right = [x for x in arr if x == pivot]      # <- wrong: should be > pivot
        return quicksort(left) + [pivot] + quicksort(right)
```

The recursion, the base case and the partition are there; one comparison operator is wrong, so the
function drops elements. That is what "66/100" looks like at this scale — the structure is learned
before the detail is, which is exactly why the benchmark is human-graded rather than pass/fail.

**Cannot**: follow instructions — this is a completion model, not a chat model (supervised
fine-tuning is documented separately in the GitHub repo). It is blind to docstrings: at this scale a
from-scratch model reads a docstring as the end of the function, so standard HumanEval
docstring prompts score ≈0 and docstring-free prompts are used instead. At 18 B tokens the same
pipeline still scored 0/50 on a five-algorithm suite — implementation ability appears between
18 B and 31 B tokens, which the ablation report documents rather than hides.

## Files

| File | What |
|---|---|
| `model.safetensors` | bf16 weights, 157 tensors — verified bit-identical to the training checkpoint after the fp32→bf16 cast |
| `config.json` | architecture config exactly as stored in the training checkpoint |
| `tokenizer.json` | byte-level BPE, 32,000 tokens |
| `SHA256SUMS.txt` | artifact hash |

Loading it requires the model class from the training repository (`src/train.py`, class `CodeLLM`
with `ModelConfig(**config.json)`); `torch.load` of a state dict built by hand will not reproduce the
forward pass described above.

## Intended use and limits

Research and education: understanding what a few-hundred-million-parameter model actually learns when
trained end to end on real data. Not for production code generation, not for instruction following,
not a substitute for a competent developer. Trained only on code; no personal data. Outputs may
reproduce patterns (and licensing quirks) from the training corpus despite copyright filtering.

## Links

Everything else — `DESIGN.md` (decisions and why), `CHALLENGES.md` (every bug and contaminant),
the ablation report, the fine-tuning log with corrections, and every evaluated generation — is in the
GitHub repository. MIT licensed.