VityaVitalich commited on
Commit
2301c97
Β·
verified Β·
1 Parent(s): 818f9d5

Model card: name revisions by global training step

Browse files
Files changed (1) hide show
  1. README.md +21 -11
README.md CHANGED
@@ -37,23 +37,33 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
37
  import torch
38
 
39
  repo = "dlab-spp/mt-3b-base"
40
- tok = AutoTokenizer.from_pretrained(repo) # identical at every revision
41
  model = AutoModelForCausalLM.from_pretrained(
42
- repo, revision="step-0", dtype=torch.bfloat16, device_map="auto"
43
  )
44
  ```
45
 
46
- | Revision | Midtraining step | LR phase |
47
- |---|---|---|
48
- | `step-0` | 0 / 72,895 | β€” (init from `vanilla-3b-base` step 225,000) |
49
- | `step-25000` | 25,000 / 72,895 | linear decay |
50
- | `step-50000` | 50,000 / 72,895 | linear decay |
51
- | `step-72895` | 72,895 / 72,895 | linear decay β€” same weights as `main` |
52
-
53
- `main` always holds the finished model (step 72,895).
 
 
 
 
 
 
 
 
54
  Only model weights are published β€” optimizer and RNG state are not included, so these revisions support evaluation, probing, and fine-tuning, but not exact resumption of the original run.
55
 
56
- Steps are counted from the start of **midtraining**. Midtraining resumed from pretraining step 225,000, so the earlier part of this model's history is the pretraining trajectory in [`dlab-spp/vanilla-3b-base`](https://huggingface.co/dlab-spp/vanilla-3b-base) (revisions `step-25000` … `step-225000`). Those checkpoints are shared and are not duplicated here; `step-0` is the exact fork point.
 
 
57
 
58
  ## Intended use
59
  Research on alignment and safety. As a base model it is meant for continuation, probing, or further fine-tuning; it is not instruction-tuned and can produce incorrect or unsafe content.
 
37
  import torch
38
 
39
  repo = "dlab-spp/mt-3b-base"
40
+ tok = AutoTokenizer.from_pretrained(repo, revision="step-225000")
41
  model = AutoModelForCausalLM.from_pretrained(
42
+ repo, revision="step-225000", dtype=torch.bfloat16, device_map="auto"
43
  )
44
  ```
45
 
46
+ | Revision | Stage | Tokens seen | Midtraining step | LR phase |
47
+ |---|---|---|---|---|
48
+ | `step-25000` | pretraining | ~49.2B | β€” | stable |
49
+ | `step-50000` | pretraining | ~98.3B | β€” | stable |
50
+ | `step-75000` | pretraining | ~147B | β€” | stable |
51
+ | `step-100000` | pretraining | ~197B | β€” | stable |
52
+ | `step-125000` | pretraining | ~246B | β€” | stable |
53
+ | `step-150000` | pretraining | ~295B | β€” | stable |
54
+ | `step-175000` | pretraining | ~344B | β€” | stable |
55
+ | `step-200000` | pretraining | ~393B | β€” | stable |
56
+ | `step-225000` | midtraining start | ~442B | 0 / 72,895 | β€” |
57
+ | `step-250000` | midtraining | β€” | 25,000 / 72,895 | linear decay |
58
+ | `step-275000` | midtraining | β€” | 50,000 / 72,895 | linear decay |
59
+ | `step-297895` | midtraining | β€” | 72,895 / 72,895 | linear decay β€” same weights as `main` |
60
+
61
+ `main` always holds the finished model (step 297,895).
62
  Only model weights are published β€” optimizer and RNG state are not included, so these revisions support evaluation, probing, and fine-tuning, but not exact resumption of the original run.
63
 
64
+ Revisions are named by global training step. Up to `step-225000` this is the pretraining run of [`dlab-spp/vanilla-3b-base`](https://huggingface.co/dlab-spp/vanilla-3b-base), and those revisions have the same weights as the revisions of the same name there. Midtraining started from [`dlab-spp/vanilla-3b-base`](https://huggingface.co/dlab-spp/vanilla-3b-base) `step-225000` with the vocabulary extended to 49,280 rows and ran 72,895 more steps, replacing the final learning-rate cooldown. The cooldown revisions of [`dlab-spp/vanilla-3b-base`](https://huggingface.co/dlab-spp/vanilla-3b-base) (`step-240000`, `step-254313`) are not part of this model's training history.
65
+
66
+ Revisions up to `step-200000` are vanilla pretraining checkpoints and ship the vanilla tokenizer (vocabulary 49,152; same BPE vocabulary and merges, without `<assistant>` and the constitution tokens), so each revision runs with its own tokenizer. Load the tokenizer with the same `revision=` as the weights.
67
 
68
  ## Intended use
69
  Research on alignment and safety. As a base model it is meant for continuation, probing, or further fine-tuning; it is not instruction-tuned and can produce incorrect or unsafe content.