fix links: GitHub URL + base/sft cross-links
Browse files
README.md
CHANGED
|
@@ -14,7 +14,7 @@ library_name: pytorch
|
|
| 14 |
# code-llm-435m · sft — the instruction-tuned variant
|
| 15 |
|
| 16 |
The instruction-tuned version of the 435M from-scratch code model whose pretrained weights are in
|
| 17 |
-
[`
|
| 18 |
context — only the weights differ. It reads a plain-English request (*"write a function that checks
|
| 19 |
whether a number is prime"*) or continues a bare `def` signature, and writes Python.
|
| 20 |
|
|
@@ -24,7 +24,7 @@ that did.
|
|
| 24 |
|
| 25 |
## Architecture
|
| 26 |
|
| 27 |
-
Identical to [`
|
| 28 |
parameters, 22 layers, d_model 1,024 / 16 heads, SwiGLU (d_ff 4,096), RoPE (theta 500,000), RMSNorm,
|
| 29 |
causal SDPA, untied embeddings, context 1,024, released file in bf16.
|
| 30 |
|
|
@@ -44,7 +44,7 @@ causal SDPA, untied embeddings, context 1,024, released file in bf16.
|
|
| 44 |
Roughly 30 post-training runs were evaluated to reach this one. The approaches that did **not** work —
|
| 45 |
instruction-free training, teaching the two objectives in two stages, self-distillation, weight
|
| 46 |
averaging ("soups"), ratio sweeps — are documented with their numbers, including the mistakes, in the
|
| 47 |
-
GitHub repository.
|
| 48 |
|
| 49 |
## Results
|
| 50 |
|
|
@@ -98,5 +98,5 @@ from the training corpus despite filtering and deduplication.
|
|
| 98 |
## Links
|
| 99 |
|
| 100 |
`DESIGN.md` (decisions and why), `CHALLENGES.md` (every bug and contaminant), the ablation report, the
|
| 101 |
-
post-training log with its corrections, and the evaluation harness are in the GitHub repository
|
| 102 |
-
licensed.
|
|
|
|
| 14 |
# code-llm-435m · sft — the instruction-tuned variant
|
| 15 |
|
| 16 |
The instruction-tuned version of the 435M from-scratch code model whose pretrained weights are in
|
| 17 |
+
[`TightX/code-llm-435m-base`](https://huggingface.co/TightX/code-llm-435m-base). Same architecture, same 32,000-token byte-level BPE vocabulary, same 1,024-token
|
| 18 |
context — only the weights differ. It reads a plain-English request (*"write a function that checks
|
| 19 |
whether a number is prime"*) or continues a bare `def` signature, and writes Python.
|
| 20 |
|
|
|
|
| 24 |
|
| 25 |
## Architecture
|
| 26 |
|
| 27 |
+
Identical to [`TightX/code-llm-435m-base`](https://huggingface.co/TightX/code-llm-435m-base) — see that card for the full table. In short: 434,680,832
|
| 28 |
parameters, 22 layers, d_model 1,024 / 16 heads, SwiGLU (d_ff 4,096), RoPE (theta 500,000), RMSNorm,
|
| 29 |
causal SDPA, untied embeddings, context 1,024, released file in bf16.
|
| 30 |
|
|
|
|
| 44 |
Roughly 30 post-training runs were evaluated to reach this one. The approaches that did **not** work —
|
| 45 |
instruction-free training, teaching the two objectives in two stages, self-distillation, weight
|
| 46 |
averaging ("soups"), ratio sweeps — are documented with their numbers, including the mistakes, in the
|
| 47 |
+
GitHub repository at https://github.com/TightXu/Simple_Code_LLM.
|
| 48 |
|
| 49 |
## Results
|
| 50 |
|
|
|
|
| 98 |
## Links
|
| 99 |
|
| 100 |
`DESIGN.md` (decisions and why), `CHALLENGES.md` (every bug and contaminant), the ablation report, the
|
| 101 |
+
post-training log with its corrections, and the evaluation harness are in the GitHub repository at
|
| 102 |
+
https://github.com/TightXu/Simple_Code_LLM. MIT licensed.
|