Upload README.md with huggingface_hub

README.md

@@ -7,239 +7,64 @@ tags:
 - educational
 ---
 
-# CUDA2LLM trained checkpoint
+# CUDA2LLM Trained Checkpoints
 
-This repository contains the trained checkpoint produced by the pure CUDA C implementation in `cuda2llm`.
+This repository contains checkpoints produced by the pure CUDA C implementation from `cuda2llm`. The code path uses hand-written CUDA kernels for the transformer forward pass, backward pass, AdamW update, checkpoint loading, and text generation.
 
-Artifacts:
-- `model.bin`: trained model checkpoint with parameters and AdamW state.
-- `vocab.bin`: character vocabulary.
-- `full_run.log`: complete RunPod verification log, including chapter CUDA checks, gradient check, 5000-step training, and generation samples.
-- `train.bin` and `val.bin`: Tiny Shakespeare character token streams used for this run.
+## What This Run Proves
 
-The run used an NVIDIA RTX A4000 pod with CUDA 12.4. The implementation is intentionally explicit CUDA C, not PyTorch training.
+- The chapter CUDA programs compile and pass their checks.
+- The full GPT program compiles with CUDA 12.4.
+- `prepare_data`, `grad_check`, `train_gpt`, and `generate_gpt` all run on an NVIDIA RTX A4000.
+- A 5000-step training run completed and generated text from the final checkpoint.
 
-## Verification Tail
+## Important Quality Note
+
+The final sample is not coherent Shakespeare. It is a small character-level model trained on a small corpus with an intentionally explicit educational implementation. It learns character labels, line breaks, archaic cadence, and local word shapes, but it does not reliably learn plot, speaker intent, or sentence-level meaning.
+
+The validation curve shows overfitting after about step 2000. The final checkpoint is useful as proof that the scheduled run completed; the step-2000 region is the better demo checkpoint by validation loss.
+
+## Loss Curve
+
+```text
+step  1000 | train loss 1.6028 | val loss 1.7558
+step  2000 | train loss 1.2770 | val loss 1.5911
+step  3000 | train loss 1.1062 | val loss 1.6381
+step  4000 | train loss 0.7658 | val loss 1.8825
+step  5000 | train loss 0.7931 | val loss 2.0483
+```
+
+## Final Sample Excerpt
 
 ```text
---- code/ch10 ---
-nvcc -O3 -arch=native -lm softmax.cu -o softmax
-nvcc -O3 -arch=native -lm reduce_row.cu -o reduce_row
-code/ch10/reduce_row
-OK rowmax
-OK rowsum
-code/ch10/softmax
-one row of large scores through both kernels (a demonstration):
-  logits:    101.3     99.1    102.7     12.0    100.0    103.9     97.2    100.8
-  naive:       nan      nan      nan 0.000000      nan      nan      nan      nan
-  safe:   0.051216 0.005675 0.207688 0.000000 0.013958 0.689551 0.000849 0.031064
-OK softmax_v65
-softmax_v65: N=8192 V=65  0.046 ms  ~229 GB/s effective
-OK softmax_v1000
-softmax_v1000: N=8192 V=1000  0.173 ms  ~948 GB/s effective
-OK softmax_rowsum
---- code/ch11 ---
-nvcc -O3 -arch=native -lm layernorm.cu -o layernorm
-nvcc -O3 -arch=native -lm gelu.cu -o gelu
-code/ch11/gelu
-     x    gelu(x)
-    -2    -0.0454
-    -1    -0.1588
-     0     0.0000
-     1     0.8412
-     2     1.9546
-OK gelu
-code/ch11/layernorm
-N=8192 rows, C=384 channels
-OK layernorm_out
-OK layernorm_mean
-OK layernorm_rstd
-N=1000 rows, C=100 channels
-OK layernorm_ragged
---- code/ch12 ---
-nvcc -O3 -arch=native -lm encoder.cu -o encoder
-nvcc -O3 -arch=native -lm crossentropy.cu -o crossentropy
-code/ch12/crossentropy
-OK softmax
-OK crossentropy
-uniform logits: mean loss 4.174387, expected ln(65) = 4.174387
-OK uniform_loss_4.17
-code/ch12/encoder
-OK encoder
---- code/ch13 ---
-nvcc -O3 -arch=native -lm attention.cu -o attention
-code/ch13/attention
-OK attention_out
-OK attention_att
-OK attention_causal
---- code/ch16 ---
-cc -O2 -lm -o toy_backprop toy_backprop.c
-code/ch16/toy_backprop
-forward: z = 7.0, y = 49.0
-backward: dx = 42.0, dw = 28.0, db = 14.0
-finite difference dx ~= 42.002
-OK toy_backprop
-=== GPT BUILD ===
-rm -f prepare_data train_gpt generate_gpt grad_check
-cc -O2 -lm -o prepare_data prepare_data.c
-nvcc -O3 -arch=native -lm -o train_gpt train_gpt.cu
-nvcc -O3 -arch=native -lm -o generate_gpt generate_gpt.cu
-nvcc -O3 -arch=native -lm -o grad_check grad_check.cu
-=== DATA ===
-curl -L -o input.txt https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
-  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                 Dload  Upload   Total   Spent    Left  Speed
-
-  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
-100 1089k  100 1089k    0     0  4798k      0 --:--:-- --:--:-- --:--:-- 4819k
-tokens: 1115394 | vocab: 65 | train: 1003854 | val: 111540
-=== GRAD CHECK ===
-analytic  -0.074804
-numerical -0.074863
-diff      0.000060
-OK grad_check_wte0
-=== TRAIN 5000 ===
-parameters: 10770816
-step     1 | loss 4.2383 | lr 0.000010 | 7300 tok/s
-step   100 | loss 2.4751 | lr 0.001000 | 7039 tok/s
-step   200 | loss 2.3741 | lr 0.000999 | 6967 tok/s
-val    250 | loss 2.2469
-step   300 | loss 2.1620 | lr 0.000996 | 7033 tok/s
-step   400 | loss 1.8983 | lr 0.000992 | 7017 tok/s
-step   500 | loss 1.7598 | lr 0.000985 | 7035 tok/s
-val    500 | loss 1.9397
-step   600 | loss 1.7097 | lr 0.000977 | 7024 tok/s
-step   700 | loss 1.7504 | lr 0.000967 | 7036 tok/s
-val    750 | loss 1.8425
-step   800 | loss 1.6235 | lr 0.000955 | 7022 tok/s
-step   900 | loss 1.5730 | lr 0.000942 | 7052 tok/s
-step  1000 | loss 1.6028 | lr 0.000927 | 7029 tok/s
-val   1000 | loss 1.7558
-saved model.bin at step 1000
-step  1100 | loss 1.6541 | lr 0.000911 | 7029 tok/s
-step  1200 | loss 1.6146 | lr 0.000893 | 7051 tok/s
-val   1250 | loss 1.6679
-step  1300 | loss 1.4642 | lr 0.000873 | 7035 tok/s
-step  1400 | loss 1.3288 | lr 0.000853 | 7034 tok/s
-step  1500 | loss 1.3054 | lr 0.000831 | 7043 tok/s
-val   1500 | loss 1.6317
-step  1600 | loss 1.3982 | lr 0.000807 | 7063 tok/s
-step  1700 | loss 1.3385 | lr 0.000783 | 7052 tok/s
-val   1750 | loss 1.6163
-step  1800 | loss 1.2542 | lr 0.000758 | 7047 tok/s
-step  1900 | loss 1.3436 | lr 0.000732 | 7050 tok/s
-step  2000 | loss 1.2770 | lr 0.000705 | 7062 tok/s
-val   2000 | loss 1.5911
-saved model.bin at step 2000
-step  2100 | loss 1.2688 | lr 0.000678 | 7052 tok/s
-step  2200 | loss 1.1882 | lr 0.000650 | 7042 tok/s
-val   2250 | loss 1.6044
-step  2300 | loss 1.3317 | lr 0.000622 | 7046 tok/s
-step  2400 | loss 1.1211 | lr 0.000593 | 7036 tok/s
-step  2500 | loss 1.2006 | lr 0.000564 | 7059 tok/s
-val   2500 | loss 1.6112
-step  2600 | loss 1.1568 | lr 0.000536 | 7065 tok/s
-step  2700 | loss 1.2186 | lr 0.000507 | 7067 tok/s
-val   2750 | loss 1.6321
-step  2800 | loss 1.1401 | lr 0.000478 | 7069 tok/s
-step  2900 | loss 1.0421 | lr 0.000450 | 7064 tok/s
-step  3000 | loss 1.1062 | lr 0.000422 | 7057 tok/s
-val   3000 | loss 1.6381
-saved model.bin at step 3000
-step  3100 | loss 0.9624 | lr 0.000395 | 7069 tok/s
-step  3200 | loss 1.0200 | lr 0.000368 | 7073 tok/s
-val   3250 | loss 1.6753
-step  3300 | loss 1.0242 | lr 0.000342 | 7089 tok/s
-step  3400 | loss 1.0527 | lr 0.000317 | 7055 tok/s
-step  3500 | loss 0.9729 | lr 0.000293 | 7063 tok/s
-val   3500 | loss 1.7316
-step  3600 | loss 0.9506 | lr 0.000269 | 7077 tok/s
-step  3700 | loss 0.9488 | lr 0.000247 | 7067 tok/s
-val   3750 | loss 1.7931
-step  3800 | loss 0.9607 | lr 0.000227 | 7055 tok/s
-step  3900 | loss 0.9824 | lr 0.000207 | 7056 tok/s
-step  4000 | loss 0.7658 | lr 0.000189 | 7031 tok/s
-val   4000 | loss 1.8825
-saved model.bin at step 4000
-step  4100 | loss 0.8902 | lr 0.000173 | 7044 tok/s
-step  4200 | loss 0.7832 | lr 0.000158 | 7047 tok/s
-val   4250 | loss 1.9381
-step  4300 | loss 0.8469 | lr 0.000145 | 7036 tok/s
-step  4400 | loss 0.8653 | lr 0.000133 | 7053 tok/s
-step  4500 | loss 0.8695 | lr 0.000123 | 7035 tok/s
-val   4500 | loss 1.9576
-step  4600 | loss 0.7353 | lr 0.000115 | 7048 tok/s
-step  4700 | loss 0.7704 | lr 0.000108 | 7046 tok/s
-val   4750 | loss 1.9924
-step  4800 | loss 0.7446 | lr 0.000104 | 7032 tok/s
-step  4900 | loss 0.8131 | lr 0.000101 | 7042 tok/s
-step  5000 | loss 0.7931 | lr 0.000100 | 7054 tok/s
-val   5000 | loss 2.0483
-saved model.bin at step 5000
-=== GENERATE ROMEO ===
 ROMEO:
 Ay not sustom's master and by earor-woo's love;
 She damned life, lest have had some removed,
 Lest in neither slander man, and might have
 Because the moon his commission.
+```
 
-MARCIUS:
-No, I'll consent the choler is prince:
-I am gone to great the last of my master;
-That's smules not put to my counsel.
-
-LARTIUS:
-He'll down away; alas, my liege, may stand
-And in the truth of one sons with that for night
-But upon'sy. Let thou go;
-How sometimes thee on the ground the vault?
-Though thy wife will then to thee to thine,
-And doth me, thy force should thee from thy brother
-As thou didst uncontent? I think thee, that would ens
-Of gracious that here were most under his pleasing
-pay a wordfather, a mother; a boote ardone
-And this one in so to be a child, and so meek
-To lie in; and so, as the which I do ha
-=== GENERATE JULIET ===
-JULIET:
-And fearn gather nothing by thy lance.
-Is my daughter's most reason can strengthen's stand
-My separance up and hereport of their children,
-My mildness and constant seize in purge
-With informal enrage, that the habit ours.
-Say thou never have me once what I have:
-Not hear then beat his; 'twere thy toother such
-And see the gates that which he can say.
-
-KING EDWARD IV:
-The gods be thereof there! O thee, climbs:
-This is there: 'for thou re's more there's service:
-'Tis thou there?
-Why, villain, if thou know'st this noble armed,
-It must are the alive. But what is this?
-
-GLOUCESTER:
-It should be, there's no consort or charge.
-
-YORK:
-At what command?
-
-CLARENCE:
-It is. For that docts but loss this thought of thee,
-As doubt the truth of the whole head.
-
-GLOUCESTER:
-Your brother is and Polixenes,
-Ho
-=== ARTIFACTS ===
--rw-r--r-- 1 root root 124M Jul  6 02:12 model.bin
--rw-r--r-- 1 root root 3.9M Jul  6 00:34 train.bin
--rw-r--r-- 1 root root 436K Jul  6 00:34 val.bin
--rw-r--r-- 1 root root   73 Jul  6 00:34 vocab.bin
+This is best described as Shakespeare-like texture with weak semantics.
+
+## Files
+
+```text
+model.bin
+vocab.bin
+train.bin
+val.bin
+full_run.log
+checkpoints/model-interim-20260706-011839.bin
+checkpoints/model-interim-20260706-020538.bin
+samples/interim_sample_20260706-011839.txt
+samples/interim_sample_20260706-020538.txt
+```
+
+## Final Artifact Hashes
+
+```text
 75eb8311e2bdec22b0c0e40084fb39fb1ed190662d3cd51f26e9f4b37ee06691  model.bin
 464eb51d66b5b643889064b82c75607e2d756df11ec960e5e6aaaf03c71734be  vocab.bin
 a3f591a5c0dfa7f54a2ec0ab39c4561d6475f1af28264f38076932ca425d5037  train.bin
 836220b36a012c19cd285816001700117edef4f8e1098130b234d4b8fa766d86  val.bin
-=== DONE ===
-2026-07-06T02:12:17+00:00
 ```